Download data warehouse and architecture and more Lecture notes Data Mining in PDF only on Docsity!
,QWURGXFWLRQWR'DWD:DUHKRXVHV
DQG2/$37HFKQRORJLHV
Data Mining Lecture 2 2
2YHUYLHZ
á :KDWLVDGDWDZDUHKRXVH"
á $PXOWLGLPHQVLRQDOGDWDPRGHO
á 'DWDZDUHKRXVHDUFKLWHFWXUH
á 'DWDZDUHKRXVHLPSOHPHQWDWLRQ
á )XUWKHUGHYHORSPHQWRIGDWDFXEHWHFKQRORJ\
á )URPGDWDZDUHKRXVLQJWRGDWDPLQLQJ
Data Mining Lecture 2 3
:KDWLV'DWD:DUHKRXVH"
á 'HILQHGLQPDQ\GLIIHUHQWZD\VEXWQRWULJRURXVO\
≤ $GHFLVLRQVXSSRUWGDWDEDVHWKDWLVPDLQWDLQHGVHSDUDWHO\
IURPWKHRUJDQL]DWLRQ∑VRSHUDWLRQDOGDWDEDVH
≤ 6XSSRUWLQIRUPDWLRQSURFHVVLQJE\SURYLGLQJDVROLGSODWIRUP
RIFRQVROLGDWHGKLVWRULFDOGDWDIRUDQDO\VLV
á ¥$GDWDZDUHKRXVHLVDVXEMHFWRULHQWHGLQWHJUDWHG
WLPHYDULDQWDQGQRQYRODWLOHFROOHFWLRQRIGDWDLQ
VXSSRUWRIPDQDJHPHQW∑VGHFLVLRQPDNLQJSURFHVVμ
≥:+,QPRQ
á 'DWDZDUHKRXVLQJ
≤ 7KHSURFHVVRIFRQVWUXFWLQJDQGXVLQJGDWDZDUHKRXVHV
Data Mining Lecture 2 4
'DWD:DUHKRXVH≥6XEMHFW2ULHQWHG
á 2UJDQL]HGDURXQGPDMRUVXEMHFWVVXFKDVFXVWRPHU
SURGXFWVDOHV
á )RFXVLQJRQWKHPRGHOLQJDQGDQDO\VLVRIGDWDIRU
GHFLVLRQPDNHUVQRWRQGDLO\RSHUDWLRQVRUWUDQVDFWLRQ
SURFHVVLQJ
á 3URYLGHDVLPSOHDQGFRQFLVHYLHZDURXQGSDUWLFXODU
VXEMHFWLVVXHVE\H[FOXGLQJGDWDWKDWDUHQRWXVHIXOLQ
WKHGHFLVLRQVXSSRUWSURFHVV
Data Mining Lecture 2 5
'DWD:DUHKRXVH≥,QWHJUDWHG
á &RQVWUXFWHGE\LQWHJUDWLQJPXOWLSOHKHWHURJHQHRXV
GDWDVRXUFHV
≤ UHODWLRQDOGDWDEDVHVIODWILOHVRQOLQHWUDQVDFWLRQUHFRUGV
á 'DWDFOHDQLQJDQGGDWDLQWHJUDWLRQWHFKQLTXHVDUH
DSSOLHG
≤ (QVXUHFRQVLVWHQF\LQQDPLQJFRQYHQWLRQVHQFRGLQJ
VWUXFWXUHVDWWULEXWHPHDVXUHVHWFDPRQJGLIIHUHQWGDWD
VRXUFHV
á (J+RWHOSULFH FXUUHQF\WD[EUHDNIDVWFRYHUHGHWF ≤ :KHQGDWDLVPRYHGWRWKHZDUHKRXVHLWLVFRQYHUWHG
Data Mining Lecture 2 6
'DWD:DUHKRXVH≥7LPH9DULDQW
á 7KHWLPHKRUL]RQIRUWKHGDWDZDUHKRXVHLV
VLJQLILFDQWO\ORQJHUWKDQWKDWRIRSHUDWLRQDOV\VWHPV
≤ 2SHUDWLRQDOGDWDEDVH FXUUHQWYDOXHGDWD
≤ 'DWDZDUHKRXVHGDWD SURYLGHLQIRUPDWLRQIURPDKLVWRULFDO
SHUVSHFWLYH HJSDVW\HDUV
á (YHU\NH\VWUXFWXUHLQWKHGDWDZDUHKRXVH
≤ &RQWDLQVDQHOHPHQWRIWLPHH[SOLFLWO\RULPSOLFLWO\
≤ %XWWKHNH\RIRSHUDWLRQDOGDWDPD\RUPD\QRWFRQWDLQ¥WLPH
HOHPHQWμ
Data Mining Lecture 2 7
'DWD:DUHKRXVH≥1RQ9RODWLOH
á $SK\VLFDOO\VHSDUDWHVWRUHRIGDWDWUDQVIRUPHG
IURPWKHRSHUDWLRQDOHQYLURQPHQW
á 2SHUDWLRQDOXSGDWHRIGDWDGRHVQRWRFFXULQWKH
GDWDZDUHKRXVHHQYLURQPHQW
≤ 'RHVQRWUHTXLUHWUDQVDFWLRQSURFHVVLQJUHFRYHU\DQG
FRQFXUUHQF\FRQWUROPHFKDQLVPV
≤ 5HTXLUHVRQO\WZRRSHUDWLRQVLQGDWDDFFHVVLQJ
á LQLWLDOORDGLQJRIGDWDDQGDFFHVVRIGDWD
Data Mining Lecture 2 8
'DWD:DUHKRXVHYV+HWHURJHQHRXV'%
á 7UDGLWLRQDOKHWHURJHQHRXV'%LQWHJUDWLRQ
≤ %XLOGZUDSSHUVPHGLDWRUVRQWRSRIKHWHURJHQHRXVGDWDEDVHV
≤ 4XHU\GULYHQDSSURDFK
á :KHQDTXHU\LVSRVHGWRDFOLHQWVLWHDPHWDGLFWLRQDU\LVXVHG WRWUDQVODWHWKHTXHU\LQWRTXHULHVDSSURSULDWHIRULQGLYLGXDO KHWHURJHQHRXVVLWHVLQYROYHGDQGWKHUHVXOWVDUHLQWHJUDWHGLQWR DJOREDODQVZHUVHW á &RPSOH[LQIRUPDWLRQILOWHULQJFRPSHWHIRUUHVRXUFHV
á 'DWDZDUHKRXVH XSGDWHGULYHQKLJKSHUIRUPDQFH
≤ ,QIRUPDWLRQIURPKHWHURJHQHRXVVRXUFHVLVLQWHJUDWHGLQDGYDQFHDQG
VWRUHGLQZDUHKRXVHVIRUGLUHFWTXHU\DQGDQDO\VLV
Data Mining Lecture 2 9
'DWD:DUHKRXVHYV2SHUDWLRQDO'%
á 2/73 RQOLQHWUDQVDFWLRQSURFHVVLQJ ≤ 0DMRUWDVNRIWUDGLWLRQDOUHODWLRQDO'% ≤ 'D\WRGD\RSHUDWLRQV SXUFKDVLQJLQYHQWRU\EDQNLQJPDQXIDFWXULQJ SD\UROOUHJLVWUDWLRQDFFRXQWLQJHWF á 2/$3 RQOLQHDQDO\WLFDOSURFHVVLQJ ≤ 0DMRUWDVNRIGDWDZDUHKRXVHV\VWHP ≤ 'DWDDQDO\VLVDQGGHFLVLRQPDNLQJ á 'LVWLQFWIHDWXUHV 2/73YV2/$3 ≤ 8VHUDQGV\VWHPRULHQWDWLRQ FXVWRPHUYVPDUNHW ≤ 'DWDFRQWHQWV FXUUHQWGHWDLOHGYVKLVWRULFDOFRQVROLGDWHG ≤ 'DWDEDVHGHVLJQ (5DSSOLFDWLRQYVVWDUVXEMHFW ≤ 9LHZ FXUUHQWORFDOYVHYROXWLRQDU\LQWHJUDWHG ≤ $FFHVVSDWWHUQV XSGDWHYVUHDGRQO\EXWFRPSOH[TXHULHV Data Mining Lecture 2 10
2/73YV2/$
OLTP OLAP users clerk, IT professional knowledge worker function day to day operations decision support DB design application-oriented subject-oriented data current, up-to-date detailed, flat relational isolated
historical, summarized, multidimensional integrated, consolidated usage repetitive ad-hoc access read/write index/hash on prim. key
lots of scans unit of work short, simple transaction complex query # records accessed tens millions #users thousands hundreds DB size 100MB-GB 100GB-TB metric transaction throughput query throughput, response
Data Mining Lecture 2 11
:K\6HSDUDWH'DWD:DUHKRXVH"
á +LJKSHUIRUPDQFHIRUERWKV\VWHPV
≤ '%06≥WXQHGIRU2/73 DFFHVVPHWKRGVLQGH[LQJ
FRQFXUUHQF\FRQWUROUHFRYHU\
≤ :DUHKRXVH≥WXQHGIRU2/$3 FRPSOH[2/$3TXHULHV
PXOWLGLPHQVLRQDOYLHZFRQVROLGDWLRQ
á 'LIIHUHQWIXQFWLRQVDQGGLIIHUHQWGDWD
≤ PLVVLQJGDWD 'HFLVLRQVXSSRUWUHTXLUHVKLVWRULFDOGDWDZKLFK
RSHUDWLRQDO'%VGRQRWW\SLFDOO\PDLQWDLQ
≤ GDWDFRQVROLGDWLRQ '6UHTXLUHVFRQVROLGDWLRQ DJJUHJDWLRQ
VXPPDUL]DWLRQ RIGDWDIURPKHWHURJHQHRXVVRXUFHV
≤ GDWDTXDOLW\ GLIIHUHQWVRXUFHVW\SLFDOO\XVHLQFRQVLVWHQWGDWD
UHSUHVHQWDWLRQVFRGHVDQGIRUPDWVZKLFKKDYHWREH
UHFRQFLOHG
Data Mining Lecture 2 12
2YHUYLHZ
á :KDWLVDGDWDZDUHKRXVH"
á $PXOWLGLPHQVLRQDOGDWDPRGHO
á 'DWDZDUHKRXVHDUFKLWHFWXUH
á 'DWDZDUHKRXVHLPSOHPHQWDWLRQ
á )XUWKHUGHYHORSPHQWRIGDWDFXEHWHFKQRORJ\
á )URPGDWDZDUHKRXVLQJWRGDWDPLQLQJ
Data Mining Lecture 2 19
$'DWD0LQLQJ4XHU\/DQJXDJH'04/ /DQJXDJH3ULPLWLYHV
á &XEH'HILQLWLRQ )DFW7DEOH
GHILQHFXEH FXEHBQDPH!> GLPHQVLRQBOLVW!@
PHDVXUHBOLVW!
á 'LPHQVLRQ'HILQLWLRQ 'LPHQVLRQ7DEOH
GHILQHGLPHQVLRQ GLPHQVLRQBQDPH!DV
DWWULEXWHBRUBVXEGLPHQVLRQBOLVW!
á 6SHFLDO&DVH 6KDUHG'LPHQVLRQ7DEOHV
≤ )LUVWWLPHDV¥FXEHGHILQLWLRQμ ≤ GHILQHGLPHQVLRQ GLPHQVLRQBQDPH!DV GLPHQVLRQBQDPHBILUVWBWLPH!LQFXEH FXEHBQDPHBILUVWBWLPH!
Data Mining Lecture 2 20
'HILQLQJD6WDU6FKHPDLQ'04/
GHILQHFXEHVDOHVBVWDU>WLPHLWHPEUDQFKORFDWLRQ@
GROODUVBVROG VXP VDOHVBLQBGROODUV DYJBVDOHV
DYJ VDOHVBLQBGROODUV XQLWVBVROG FRXQW
GHILQHGLPHQVLRQWLPHDV WLPHBNH\GD\GD\BRIBZHHN
PRQWKTXDUWHU\HDU
GHILQHGLPHQVLRQLWHPDV LWHPBNH\LWHPBQDPHEUDQG
W\SHVXSSOLHUBW\SH
GHILQHGLPHQVLRQEUDQFKDV EUDQFKBNH\EUDQFKBQDPH
EUDQFKBW\SH
GHILQHGLPHQVLRQORFDWLRQDV ORFDWLRQBNH\VWUHHWFLW\
SURYLQFHBRUBVWDWHFRXQWU\
Data Mining Lecture 2 21
'HILQLQJD6QRZIODNH6FKHPDLQ'04/
GHILQHFXEHVDOHVBVQRZIODNH>WLPHLWHPEUDQFKORFDWLRQ@
GROODUVBVROG VXP VDOHVBLQBGROODUV DYJBVDOHV
DYJ VDOHVBLQBGROODUV XQLWVBVROG FRXQW
GHILQHGLPHQVLRQWLPHDV WLPHBNH\GD\GD\BRIBZHHN
PRQWKTXDUWHU\HDU
GHILQHGLPHQVLRQLWHPDV LWHPBNH\LWHPBQDPHEUDQG
W\SHVXSSOLHU VXSSOLHUBNH\VXSSOLHUBW\SH
GHILQHGLPHQVLRQEUDQFKDV EUDQFKBNH\EUDQFKBQDPH
EUDQFKBW\SH
GHILQHGLPHQVLRQORFDWLRQDV ORFDWLRQBNH\VWUHHW
FLW\ FLW\BNH\SURYLQFHBRUBVWDWHFRXQWU\
Data Mining Lecture 2 22
'HILQLQJD)DFW&RQVWHOODWLRQLQ'04/
GHILQHFXEHVDOHV>WLPHLWHPEUDQFKORFDWLRQ@
GROODUVBVROG VXP VDOHVBLQBGROODUV DYJBVDOHV DYJ VDOHVBLQBGROODUV XQLWVBVROG FRXQW GHILQHGLPHQVLRQWLPHDV WLPHBNH\GD\GD\BRIBZHHNPRQWKTXDUWHU \HDU GHILQHGLPHQVLRQLWHPDV LWHPBNH\LWHPBQDPHEUDQGW\SH VXSSOLHUBW\SH GHILQHGLPHQVLRQEUDQFKDV EUDQFKBNH\EUDQFKBQDPHEUDQFKBW\SH GHILQHGLPHQVLRQORFDWLRQDV ORFDWLRQBNH\VWUHHWFLW\SURYLQFHBRUBVWDWH FRXQWU
GHILQHFXEHVKLSSLQJ>WLPHLWHPVKLSSHUIURPBORFDWLRQWRBORFDWLRQ@ GROODUBFRVW VXP FRVWBLQBGROODUV XQLWBVKLSSHG FRXQW GHILQHGLPHQVLRQWLPHDVWLPHLQFXEHVDOHV GHILQHGLPHQVLRQLWHPDVLWHPLQFXEHVDOHV GHILQHGLPHQVLRQVKLSSHUDV VKLSSHUBNH\VKLSSHUBQDPHORFDWLRQDV ORFDWLRQLQFXEHVDOHVVKLSSHUBW\SH GHILQHGLPHQVLRQIURPBORFDWLRQDVORFDWLRQLQFXEHVDOHV GHILQHGLPHQVLRQWRBORFDWLRQDVORFDWLRQLQFXEHVDOHV
Data Mining Lecture 2 23
0HDVXUHV 7KUHH&DWHJRULHV
á GLVWULEXWLYH LIWKHUHVXOWGHULYHGE\DSSO\LQJWKH
IXQFWLRQWRQDJJUHJDWHYDOXHVLVWKHVDPHDVWKDW
GHULYHGE\DSSO\LQJWKHIXQFWLRQRQDOOWKHGDWDZLWKRXW
SDUWLWLRQLQJ
á (JFRXQW VXP PLQ PD[
á DOJHEUDLF LILWFDQEHFRPSXWHGE\DQDOJHEUDLFIXQFWLRQ
ZLWK 0 DUJXPHQWV ZKHUH0LVDERXQGHGLQWHJHU HDFK
RIZKLFKLVREWDLQHGE\DSSO\LQJDGLVWULEXWLYHDJJUHJDWH
IXQFWLRQ
á (JDYJ PLQB1 VWDQGDUGBGHYLDWLRQ
á KROLVWLF LIWKHUHLVQRFRQVWDQWERXQGRQWKHVWRUDJHVL]H
QHHGHGWRGHVFULEHDVXEDJJUHJDWH
á (JPHGLDQ PRGH UDQN Data Mining Lecture 2^24
$&RQFHSW+LHUDUFK\ 'LPHQVLRQ ORFDWLRQ
all
Europe North_America
Germany Spain Canada Mexico
Vancouver
L. Chan M. Wind
all
region
office
country
city Frankfurt Toronto
Data Mining Lecture 2 25
0XOWLGLPHQVLRQDO'DWD
á 6DOHVYROXPHDVDIXQFWLRQRISURGXFW PRQWKDQGUHJLRQ
Product
Region
Month
Dimensions: Product, Location, Time Hierarchical summarization paths
Industry Region Year
Category Country Quarter
Product City Month Week
Office Day
Data Mining Lecture 2 26
$6DPSOH'DWD&XEH
Total annual sales
Date of TV in U.S.A.
Product
Country
sum
sum
TV
VCR
PC
1Qtr 2Qtr (^) 3Qtr 4Qtr U.S.A
Canada
Mexico
sum
Data Mining Lecture 2 27
&XERLGV&RUUHVSRQGLQJWRWKH&XEH
all
product (^) date country
product,date product,country date, country
product, date, country
0-D(apex) cuboid
1-D cuboids
2-D cuboids
3-D(base) cuboid
Data Mining Lecture 2 28
%URZVLQJD'DWD&XEH
á 9LVXDOL]DWLRQ
á 2/$3FDSDELOLWLHV
á ,QWHUDFWLYHPDQLSXODWLRQ
Data Mining Lecture 2 29
7\SLFDO2/$32SHUDWLRQV
á 5ROOXS GULOOXS VXPPDUL]HGDWD ≤ E\FOLPELQJXSKLHUDUFK\RUE\GLPHQVLRQUHGXFWLRQ á 'ULOOGRZQ UROOGRZQ UHYHUVHRIUROOXS ≤ IURPKLJKHUOHYHOVXPPDU\WRORZHUOHYHOVXPPDU\RUGHWDLOHG GDWDRULQWURGXFLQJQHZGLPHQVLRQV á 6OLFHDQGGLFH ≤ SURMHFWDQGVHOHFW á 3LYRW URWDWH ≤ UHRULHQWWKHFXEHYLVXDOL]DWLRQ'WRVHULHVRI'SODQHV á 2WKHURSHUDWLRQV ≤ GULOODFURVV LQYROYLQJ DFURVV PRUHWKDQRQHIDFWWDEOH ≤ GULOOWKURXJK WKURXJKWKHERWWRPOHYHORIWKHFXEHWRLWVEDFNHQG UHODWLRQDOWDEOHV XVLQJ64/
Data Mining Lecture 2 30
2YHUYLHZ
á :KDWLVDGDWDZDUHKRXVH"
á $PXOWLGLPHQVLRQDOGDWDPRGHO
á 'DWDZDUHKRXVHDUFKLWHFWXUH
á 'DWDZDUHKRXVHLPSOHPHQWDWLRQ
á )XUWKHUGHYHORSPHQWRIGDWDFXEHWHFKQRORJ\
á )URPGDWDZDUHKRXVLQJWRGDWDPLQLQJ
Data Mining Lecture 2 37
(IILFLHQW'DWD&XEH&RPSXWDWLRQ
á 'DWDFXEHFDQEHYLHZHGDVDODWWLFHRIFXERLGV
≤ 7KHERWWRPPRVWFXERLGLVWKHEDVHFXERLG
≤ 7KHWRSPRVWFXERLG DSH[ FRQWDLQVRQO\RQHFHOO
≤ +RZPDQ\FXERLGVLQDQQGLPHQVLRQDOFXEHZLWK/OHYHOV"
á 0DWHULDOL]DWLRQRIGDWDFXEH
≤ 0DWHULDOL]HHYHU\ FXERLG IXOOPDWHULDOL]DWLRQ QRQH QR
PDWHULDOL]DWLRQ RUVRPH SDUWLDOPDWHULDOL]DWLRQ
≤ 6HOHFWLRQRIZKLFKFXERLGVWRPDWHULDOL]H
á %DVHGRQVL]HVKDULQJDFFHVVIUHTXHQF\HWF
∏(^ +
n i i
T L
Data Mining Lecture 2 38
&XEH2SHUDWLRQ
á &XEHGHILQLWLRQDQGFRPSXWDWLRQLQ'04/ GHILQHFXEHVDOHV>LWHPFLW\\HDU@ VXP VDOHVBLQBGROODUV FRPSXWHFXEHVDOHV á 7UDQVIRUP LW LQWR D 64/OLNH ODQJXDJH ZLWK D QHZ RSHUDWRU FXEHE\LQWURGXFHGE\UD\HWDO∑ 6(/(&7LWHPFLW\\HDU680 DPRXQW )5206$/( &8%(%<LWHPFLW\\HDU á 1HHGFRPSXWHWKHIROORZLQJURXS%\V GDWHSURGXFWFXVWRPHU GDWHSURGXFW GDWHFXVWRPHU SURGXFWFXVWRPHU GDWH SURGXFW FXVWRPHU
(city) (item)
(year)
(city, item) (city, year) (item, year)
(city, item, year)
Data Mining Lecture 2 39
&XEH&RPSXWDWLRQ 52/$3%DVHG0HWKRG
á (IILFLHQWFXEHFRPSXWDWLRQPHWKRGV
≤ 52/$3EDVHGFXELQJDOJRULWKPV $JDUZDOHWDO∑
≤ $UUD\EDVHGFXELQJDOJRULWKP =KDRHWDO∑
≤ %RWWRPXSFRPSXWDWLRQPHWKRG %H\HU 5DPDUNULVKQDQ∑
á 52/$3EDVHGFXELQJDOJRULWKPV
≤ 6RUWLQJKDVKLQJDQGJURXSLQJRSHUDWLRQVDUHDSSOLHGWRWKH
GLPHQVLRQDWWULEXWHVLQRUGHUWRUHRUGHUDQGFOXVWHUUHODWHG
WXSOHV
≤ *URXSLQJLVSHUIRUPHGRQVRPHVXEDJJUHJDWHVDVD¥SDUWLDO
JURXSLQJVWHSμ ≤ $JJUHJDWHVPD\EHFRPSXWHGIURPSUHYLRXVO\FRPSXWHG DJJUHJDWHVUDWKHUWKDQIURPWKHEDVHIDFWWDEOH
Data Mining Lecture 2 40
,QGH[LQJ2/$3'DWD %LWPDS,QGH[
á ,QGH[RQDSDUWLFXODUFROXPQ á (DFKYDOXHLQWKHFROXPQKDVDELWYHFWRU ELWRSLVIDVW á 7KHOHQJWKRIWKHELWYHFWRU RIUHFRUGVLQWKHEDVHWDEOH á 7KHLWKELWLVVHWLIWKHLWKURZRIWKHEDVHWDEOHKDVWKH YDOXHIRUWKHLQGH[HGFROXPQ á QRWVXLWDEOHIRUKLJKFDUGLQDOLW\GRPDLQV
C u st R eg io n T yp e C 1 A s ia R e ta il C 2 E u ro p e D e a le r C 3 A s ia D e a le r C 4 A m e ric a R e ta il C 5 E u ro p e D e a le r
R ecID R etail D ealer 1 1 0 2 0 1 3 0 1 4 1 0 5 0 1
R ecID Asia E u ro p e Am erica 1 1 0 0 2 0 1 0 3 1 0 0 4 0 1 0 5 0 0 1
Base table Index on Region Index on Type
Data Mining Lecture 2 41
,QGH[LQJ2/$3'DWD -RLQ,QGLFHV
á -RLQLQGH[ -, 5LG6LG ZKHUH 5 5LG´ 6 6LG´ á 7UDGLWLRQDOLQGLFHVPDSWKHYDOXHVWRDOLVW RIUHFRUGLGV ≤ ,WPDWHULDOL]HVUHODWLRQDOMRLQLQ-,ILOHDQG VSHHGVXSUHODWLRQDOMRLQ≥DUDWKHUFRVWO
RSHUDWLRQ á ,QGDWDZDUHKRXVHVMRLQLQGH[UHODWHVWKH YDOXHVRIWKHGLPHQVLRQVRIDVWDUWVFKHPD WRURZVLQWKHIDFWWDEOH ≤ (JIDFWWDEOH 6DOHVDQGWZRGLPHQVLRQV FLW\DQGSURGXFW á $MRLQLQGH[RQFLW\PDLQWDLQVIRUHDFK GLVWLQFWFLW\DOLVWRI5,'VRIWKH WXSOHVUHFRUGLQJWKH6DOHVLQWKHFLW
≤ -RLQLQGLFHVFDQVSDQPXOWLSOHGLPHQVLRQV
Data Mining Lecture 2 42
(IILFLHQW3URFHVVLQJ2/$34XHULHV
á 'HWHUPLQHZKLFKRSHUDWLRQVVKRXOGEHSHUIRUPHGRQWKH
DYDLODEOHFXERLGV
≤ WUDQVIRUPGULOOUROOHWFLQWRFRUUHVSRQGLQJ64/DQGRU2/$
RSHUDWLRQVHJGLFH VHOHFWLRQSURMHFWLRQ
á 'HWHUPLQHWRZKLFKPDWHULDOL]HGFXERLG V WKHUHOHYDQW
RSHUDWLRQVVKRXOGEHDSSOLHG
á ([SORULQJLQGH[LQJVWUXFWXUHVDQGFRPSUHVVHGYVGHQVH
DUUD\VWUXFWXUHVLQ02/$
Data Mining Lecture 2 43
0HWDGDWD5HSRVLWRU\
á 0HWDGDWDLVWKHGDWDGHILQLQJZDUHKRXVHREMHFWV,WKDVWKH IROORZLQJNLQGV ≤ 'HVFULSWLRQRIWKHVWUXFWXUHRIWKHZDUHKRXVH á VFKHPDYLHZGLPHQVLRQVKLHUDUFKLHVGHULYHGGDWD GHIQGDWDPDUW ORFDWLRQVDQGFRQWHQWV ≤ 2SHUDWLRQDOPHWDGDWD á GDWDOLQHDJH KLVWRU\RIPLJUDWHGGDWDDQGWUDQVIRUPDWLRQSDWK FXUUHQF
RIGDWD DFWLYHDUFKLYHGRUSXUJHG PRQLWRULQJLQIRUPDWLRQ ZDUHKRXVH XVDJHVWDWLVWLFVHUURUUHSRUWVDXGLWWUDLOV ≤ 7KHDOJRULWKPVXVHGIRUVXPPDUL]DWLRQ ≤ 7KHPDSSLQJIURPRSHUDWLRQDOHQYLURQPHQWWRWKHGDWDZDUHKRXVH ≤ 'DWDUHODWHGWRV\VWHPSHUIRUPDQFH á ZDUHKRXVHVFKHPDYLHZDQGGHULYHGGDWDGHILQLWLRQV ≤ %XVLQHVVGDWD á EXVLQHVVWHUPVDQGGHILQLWLRQVRZQHUVKLSRIGDWDFKDUJLQJSROLFLHV
Data Mining Lecture 2 44
'DWD:DUHKRXVH%DFN(QG7RROVDQG8WLOLWLHV
á 'DWDH[WUDFWLRQ ≤ JHWGDWDIURPPXOWLSOHKHWHURJHQHRXVDQGH[WHUQDOVRXUFHV á 'DWDFOHDQLQJ ≤ GHWHFWHUURUVLQWKHGDWDDQGUHFWLI\WKHPZKHQSRVVLEOH á 'DWDWUDQVIRUPDWLRQ ≤ FRQYHUWGDWDIURPOHJDF\RUKRVWIRUPDWWRZDUHKRXVHIRUPDW á /RDG ≤ VRUWVXPPDUL]HFRQVROLGDWHFRPSXWHYLHZVFKHFNLQWHJULW\ DQGEXLOGLQGLFLHVDQGSDUWLWLRQV á 5HIUHVK ≤ SURSDJDWHWKHXSGDWHVIURPWKHGDWDVRXUFHVWRWKH ZDUHKRXVH
Data Mining Lecture 2 45
2YHUYLHZ
á :KDWLVDGDWDZDUHKRXVH"
á $PXOWLGLPHQVLRQDOGDWDPRGHO
á 'DWDZDUHKRXVHDUFKLWHFWXUH
á 'DWDZDUHKRXVHLPSOHPHQWDWLRQ
á )XUWKHUGHYHORSPHQWRIGDWDFXEHWHFKQRORJ\
á )URPGDWDZDUHKRXVLQJWRGDWDPLQLQJ
Data Mining Lecture 2 46
'LVFRYHU\'ULYHQ([SORUDWLRQRI'DWD&XEHV
á +\SRWKHVLVGULYHQ H[SORUDWLRQE\XVHUKXJHVHDUFKVSDFH á 'LVFRYHU\GULYHQ 6DUDZDJLHWDO∑ ≤ SUHFRPSXWHPHDVXUHVLQGLFDWLQJH[FHSWLRQVJXLGHXVHULQWKHGDWD DQDO\VLVDWDOOOHYHOVRIDJJUHJDWLRQ ≤ ([FHSWLRQ VLJQLILFDQWO\GLIIHUHQWIURPWKHYDOXHDQWLFLSDWHGEDVHG RQDVWDWLVWLFDOPRGHO ≤ 9LVXDOFXHVVXFKDVEDFNJURXQGFRORUDUHXVHGWRUHIOHFWWKHGHJUHH RIH[FHSWLRQRIHDFKFHOO ≤ &RPSXWDWLRQRIH[FHSWLRQLQGLFDWRU PRGHOLQJILWWLQJDQGFRPSXWLQJ 6HOI([S,Q([SDQG3DWK([SYDOXHV FDQEHRYHUODSSHGZLWKFXEH FRQVWUXFWLRQ
Data Mining Lecture 2 47
([DPSOHV 'LVFRYHU\'ULYHQ'DWD&XEHV
Data Mining Lecture 2 48
&RPSOH[$JJUHJDWLRQDW0XOWLSOH*UDQXODULWLHV
0XOWL)HDWXUH&XEHV
á 0XOWLIHDWXUHFXEHV 5RVVHWDO &RPSXWHFRPSOH[TXHULHV LQYROYLQJPXOWLSOHGHSHQGHQWDJJUHJDWHVDWPXOWLSOHJUDQXODULWLHV á ([*URXSLQJE\DOOVXEVHWVRI^LWHPUHJLRQPRQWK`ILQGWKH PD[LPXPSULFHLQ IRUHDFKJURXSDQGWKHWRWDOVDOHVDPRQJ DOOPD[LPXPSULFHWXSOHV VHOHFWLWHPUHJLRQPRQWKPD[ SULFH VXP 5VDOHV IURPSXUFKDVHV ZKHUH\HDU FXEHE\LWHPUHJLRQPRQWK VXFKWKDW5SULFH PD[ SULFH á &RQWLQXLQJWKHODVWH[DPSOHDPRQJWKHPD[SULFHWXSOHVILQGWKH PLQDQGPD[VKHOIOLYHDQGILQGWKHIUDFWLRQRIWKHWRWDOVDOHV GXHWRWXSOHWKDWKDYHPLQVKHOIOLIHZLWKLQWKHVHWRIDOOPD[ SULFHWXSOHV
