Sentiment-based Classification of Radical Text on the Web

Abstract

The total number of webpages has grown substantially since the birth of the Internet. So too have the number of webpages dedicated to radical yet subtle content. As these new circumstances have necessitated a guided data collection method, one that can sidestep the laborious manual methods that have been classically utilized, simple keyword analysis has not been sufficient to identify radical sites on Web 1.0 – pro-extremist, anti-extremist, and news sites, for example, may use the same keywords to discuss the same event but have a very different motivation. In an effort to explore this problem, we completed an exercise involving the use of a web-crawler to collect 20,000 webpages from five sentiment-based classes to assess their differences: (1) radical Right sites; (2) radical Islamic sites; (3) anti-extremist sites; (4) news source sites discussing extremism; and (5) sites that did not discuss extremism. Parts-of-Speech (POS) tagging was used to identify 198 of the most frequent keywords within the data, and the sentiment value for each of these keywords was calculated for each webpage using sentiment analysis. With these values, a decision tree was applied to three classification models. Results suggest that radical Islamic text can be classified at a much higher rate of success than radical Right text.

Full text

6HQWLPHQWEDVHG&ODVVLILFDWLRQRI5DGLFDO
7H[WRQWKH:HE
5\DQ6FULYHQVDQG5LFKDUG)UDQN
,QWHUQDWLRQDO&\EHU&ULPH5HVHDUFK&HQWUH 
6FKRRORI&ULPLQRORJ\6LPRQ)UDVHU8QLYHUVLW\
%XUQDE\&DQDGD
^UVFULYHQUIUDQN`#VIXFD

Abstract ² 7KH WRWDO QXPEHU RI ZHESDJHV KDV JURZQ
VXEVWDQWLDOO\ VLQFH WKH ELUWK RI WKH ,QWHUQHW 6R WRR KDYH WKH
QXPEHU RI ZHESDJHV GHGLFDWHG WR UDGLFDO \HW VXEWOH FRQWHQW $V
WKHVHQHZFLUFXPVWDQFHVKDYHQHFHVVLWDWHGDJXLGHGGDWDFROOHFWLRQ
PHWKRGRQHWKDWFDQVLGHVWHSWKHODERULRXVPDQXDOPHWKRGVWKDW
KDYH EHHQ FODVVLFDOO\ XWLOL]HG VLPSOH NH\ZRUG DQDO\VLV KDV QRW
EHHQVXIILFLHQWWRLGHQWLI\UDGLFDOVLWHVRQ:HE±SURH[WUHPLVW
DQWLH[WUHPLVW DQG QHZV VLWHV IRU H[DPSOH PD\ XVH WKH VDPH
NH\ZRUGV WR GLVFXVV WKH VDPH HYHQW EXW KDYH D YHU\ GLIIHUHQW
PRWLYDWLRQ,QDQHIIRUWWRH[SORUHWKLVSUREOHPZHFRPSOHWHGDQ
H[HUFLVH LQYROYLQJ WKH XVH RI D ZHEFUDZOHU WR FROOHFW 
ZHESDJHV IURP ILYH VHQWLPHQWEDVHG FODVVHV WR DVVHVV WKHLU
GLIIHUHQFHV  UDGLFDO 5LJKW VLWHV  UDGLFDO ,VODPLF VLWHV 
DQWLH[WUHPLVW VLWHV  QHZV VRXUFH VLWHV GLVFXVVLQJ H[WUHPLVP
DQGVLWHVWKDWGLGQRWGLVFXVVH[WUHPLVP3DUWVRI6SHHFK326
WDJJLQJ ZDV XVHG WR LGHQWLI\  RI WKH PRVW IUHTXHQW NH\ZRUGV
ZLWKLQ WKH GDWD DQG WKH VHQWLPHQW YDOXH IRU HDFK RI WKHVH
NH\ZRUGV ZDV FDOFXODWHG IRU HDFK ZHESDJH XVLQJ VHQWLPHQW
DQDO\VLV :LWK WKHVH YDOXHV D GHFLVLRQ WUHH ZDV DSSOLHG WR WKUHH
FODVVLILFDWLRQPRGHOV5HVXOWVVXJJHVWWKDWUDGLFDO,VODPLFWH[WFDQ
EHFODVVLILHG DW DPXFKKLJKHU UDWH RIVXFFHVV WKDQ UDGLFDO 5LJKW
WH[W
Keywords - Sentiment Analysis; Decision Trees; Extremism
,,1752'8&7,21
 6LQFH WKH PLGV ZH KDYH VHHQ D UDSLG JURZWK LQ WKH
QXPEHURIUDGLFDOZHEVLWHVDFWLQJDVDKXERIWKHPRYHPHQWWKH\
UHSUHVHQW,QPDQ\UHVSHFWVWKH\VHUYHDVDFRQYHUJHQFHVHWWLQJ
IRUQHZFRPHUVWRIDPLOLDUL]HWKHPVHOYHVZLWKUDGLFDOWHDFKLQJV
>@FRQQHFWLQJXVHUVWRRWKHUVLPLODUZHEVLWHVZHEIRUXPV
DQGSDJHVRQWKH'DUN:HEWRQDPHEXWDIHZ>@&RXQWHU
H[WUHPLVPRUJDQL]DWLRQV FRQWLQXH WR ORRN IRU ZD\V WR LGHQWLI\
WKHVH VLWHV >@ \HW WKH ,QWHUQHW LV FRQVWDQWO\ JURZLQJ DQG
LQIRUPDWLRQLVEHLQJJHQHUDWHGDWDUDSLGSDFH>@7KLVKDVOHG
WR D JURZLQJ IORRG RI GDWD UHVXOWLQJ LQ PDQXDO GDWD DQDO\VLV
EHFRPLQJ OHVV HIILFLHQW RU HQWLUHO\ LQIHDVLEOH >@ ,Q UHVSRQVH
UHVHDUFKHUV KDYH GHYHORSHG GDWD FROOHFWLRQ WRROV WKDW DOORZ
GHFLVLRQVWREHPDGHDERXWWKHWKRXVDQGVRIZHESDJHVWKDWDUH
H[WUDFWHGDQGE\H[WHQVLRQFODVVLI\UDGLFDOWH[WRQWKH:HE
 5HVHDUFKHUVKDYHH[SORUHGWKLVFULWLFDOSRLQWRIGHSDUWXUHYLD
DXWRPDWHG FRPSXWDWLRQDO DQG VHPLDXWRPDWHG WRROV ERWK WR
LPSURYH WKH PHWKRGV RI FROOHFWLRQ DQG LGHQWLILFDWLRQ RI
H[WUHPLVW FRQWHQW &KHQ¶V Dark Web Project IRU H[DPSOH
SURGXFHGDQXPEHURIFRPSXWDWLRQDODQGGDWDFHQWULFVWXGLHVRQ
WKHFRQWHQW DQG VWUXFWXUH RI VXUIDFH OHYHO DQG 'DUN :HE VLWHV
FRQWDLQLQJH[WUHPLVWPDWHULDO>@,QOLJKWRIWKLVFRPSUHKHQVLYH
ZRUNWKHHQWLUHSURFHVVZDVDXWRPDWHGDQGWKH\GLGQRWDVVHVV
WKHVXEWOH ODQJXDJH ± LQ WKH (QJOLVK WH[W ± IRXQG RQ ZHEVLWHV
IHDWXULQJH[WUHPLVWFRQWHQW$UJXDEO\DIXOO\DXWRPDWHGV\VWHP
VKRXOGEHDYRLGHGZKHQKXPDQLQWHOOLJHQFHLVUHTXLUHG>@LH
LGHQWLI\LQJVXEWOHIRUPVRIUDGLFDOWH[WLQ(QJOLVK
 ,QWKHZRUNRI0HLDQG)UDQNWKHDXWKRUVGHYHORSHGDPRGHO
WKDW FRPELQHG VHQWLPHQW DQDO\VLV DQG D ZHEFUDZOHU ZLWK D
GHFLVLRQWUHHWRGLIIHUHQWLDWHSURH[WUHPLVWZHESDJHVIURPDQWL
H[WUHPLVW SDJHV QHZV SDJHV DQG SDJHV WKDW GLG QRW UHODWH WR
H[WUHPLVP +HUH WKH RYHUDOO JRDO RI WKLV VHPLDXWRPDWHG
DSSURDFKZDVWRFUHDWHDWRROWKDW PDGHSUHGLFWLRQV DERXW WKH
FRQWHQW IRXQG RQ WKH VLWHV LW GRZQORDGHG >@ :KLOH WKLV
WHFKQLTXH PDQDJHG WR FODVVLI\ H[WUHPLVW WH[W DW D KLJK UDWH RI
DFFXUDF\ LH  ERWK H[WUHPLVWEDVHG ZHEIRUXPV DQG
ZHEVLWHV RI H[WUHPLVW RUJDQL]DWLRQV ZHUH DQDO\]HG ZLWKLQ RQH
PRGHO$UJXDEO\WKHVHQWLPHQWIRXQGRQERWKW\SHVRISODWIRUPV
DUH GLIIHUHQW LQ WHUPV RI WRQH DQG VXEMHFW PDWWHU ,QGHHG WKLV
SUREOHPUHTXLUHVIXUWKHUH[SORUDWLRQ
,,0(7+2'6
7KHSXUSRVHRIWKLVVWXG\ZDVWRDGGWRWKHOLWHUDWXUHE\XVLQJ
DVHQWLPHQWDQDO\VLVWRRODQGDGHFLVLRQWUHHWRGLIIHUHQWLDWHSUR
H[WUHPLVWZHESDJHVIURPDQWLH[WUHPLVWSDJHVQHZVSDJHVDQG
SDJHVWKDWGLGQRWUHODWHWRH[WUHPLVP7KLVZDVGRQHE\EXLOGLQJ
WKUHHWH[WEDVHG FODVVLILFDWLRQ PRGHOV&KDSWHU,,$ )LUVWZH
XVHGWKH7HUURULVPDQG([WUHPLVP1HWZRUN([WUDFWRU7(1(
WR FROOHFW GDWD &KDSWHU  DQG 2SHQ1/3¶V 3DUWVRI6SHHFK
326DQDO\VLVWRGHYHORSDOLVWRINH\ZRUGV&KDSWHU,,&7KH
VHQWLPHQW H[SUHVVHG ZLWKLQ WKH ZHESDJHVZDV WKHQ FDOFXODWHG 
EDVHG RQ WKH 326 NH\ZRUGV GHWDLOLQJ WKH UHODWLYH VHQWLPHQW
VFRUHVIRUHDFKSDJH IRU HDFK RI WKH NH\ZRUGV &KDSWHU,,'
)LQDOO\DGHFLVLRQWUHHZDVEXLOWEDVHGRQWKHVHQWLPHQWVFRUHV
&KDSWHU,,(ZKLFKSURGXFHGWKHFODVVLILFDWLRQUHVXOWVIRUWKUHH
FODVVLILFDWLRQPRGHOV&KDSWHU,,)VHH)LJXUH
A. Webpage Data
5HVHDUFKHUVKDYHVXJJHVWHGWKDWWKHVHQWLPHQWIRXQGRQERWK
UDGLFDO 5LJKW > @ DQG UDGLFDO ,VODPLF > @ ZHEVLWHV DUH
HOXVLYH$VDPHDQVRIH[SORULQJWKLVFODLPRQWKHSURH[WUHPLVW
FRQWHQWDQGHYDOXDWLQJZKHWKHUWKHGHFLVLRQWUHHFRXOGFODVVLI\
FRQWHQW EDVHG RQ WKRVH VXEWOH LGHRORJLFDO PRWLYDWLRQV ILYH
GLVWLQFWDQGQRQRYHUODSSLQJFODVVHVRISDJHVZHUHDWWDLQHG
1) 7KH radical Right FODVV ZDV VHOHFWHG WKURXJK WZR
PHWKRGV  D *RRJOH VHDUFK RI WKH NH\ZRUGV
µH[WUHPLVWZHEVLWHV¶µZKLWHVXSUHPDF\ZHEVLWHV¶DQG
RWKHUVLPLODUZRUGVDQGXVLQJDQLQGH[RIH[WUHPLVW
VLWHV JHQHUDWHG E\ UHVHDUFKHUV DQG VRXUFHV IRU
2016 European Intelligence and Security Informatics Conference
978-1-5090-2857-3/16 $31.00 © 2016 IEEE
DOI 10.1109/EISIC.2016.41
104

H[DPSOHV VHH > @ 7KLV FODVV FRQWDLQHG 
ZHESDJHVVSUHDGRYHUZHEVLWHV
2) 7KH radical Islamic FODVV ZDV VHOHFWHG WKURXJK WKH
NH\ZRUGV µH[WUHPLVW ZHEVLWHV¶ µMLKDG RUJDQL]DWLRQV¶
DQG SUHYLRXVO\ FRPSLOHG LQGH[HV >@ 7KLV FODVV
FRQWDLQHGZHESDJHVIURPZHEVLWHV
3) $Qanti-extremist FODVV LQFOXGHG LQWHOOLJHQFHDJHQFLHV
DQG RUJDQL]DWLRQV GHGLFDWHG WR FRXQWHULQJ H[WUHPLVP
DQG ZHUH LGHQWLILHG YLD DQ RQOLQH VHDUFK RI VXFK
NH\ZRUGVDVµFRXQWHUH[WUHPLVP¶DQGµDQWLH[WUHPLVP¶
JURXSV IRU H[DPSOH 7KH VDPSOH FRQWDLQHG 
ZHESDJHVIURPZHEVLWHV
4) $ news FODVV LQFOXGHG ZHEVLWHV LI WKH\ ZHUH VRXUFHV
WKDWUHSRUWHGRQH[WUHPLVWHYHQWV+HUHLWZDVDVVXPHG
WKDW WKH\ SUHVHQWHG D VWRU\ LQ D PRUH ³LPSDUWLDO´
PDQQHUWKDQSURH[WUHPLVWRUDQWLH[WUHPLVWVLWHV7KLV
VDPSOHFRQWDLQHGZHESDJHVIURPZHEVLWHV
5) $ILQDO FODVVotherVHUYHGDVWKHFRQWUROJURXSLQWKH
VDPSOH ZKHUH RQH ZRXOG QRW H[SHFW WR XQFRYHU
PDWHULDO UHODWLQJ WR H[WUHPLVP DQG LQFOXGHG WRSLFV
UHODWLQJ WR DUW DQG FXOWXUH HGXFDWLRQ HQWHUWDLQPHQW
IRRG WUDYHO DQG OLIHVW\OH VRFLDO PHGLD RQOLQH EORJV
DXWRPRELOHVVSRUWVDQGEXVLQHVV)RUWKLVFODVV
ZHESDJHVIURPZHEVLWHVZHUHVDPSOHG

7KUHH PRGHOV ZHUH WKHQ FRQVWUXFWHG IRU WKH SXUSRVHV RI
FRPSDULQJDQGFRQWUDVWLQJWKHLUFODVVLILFDWLRQUHVXOWV
Two-class model:radical RightDQGIslamicYHUVXV
anti-extremistnewsDQGotherZHESDJHV
Four-class model:radical RightDQGradicalIslamic
YHUVXV  anti-extremist YHUVXV  news YHUVXV 
otherZHESDJHV
Five-class model:radical RightYHUVXV  radical
IslamicYHUVXVanti-extremistYHUVXVnewsYHUVXV
other ZHESDJHV
B. Terrorism and Extremism Network Extractor (TENE)
7KH7HUURULVP DQG([WUHPLVP1HWZRUN([WUDFWRU 7(1(
DFXVWRPZULWWHQFRPSXWHUSURJUDPWKDWZDVGHVLJQHGWRFROOHFW
YDVWDPRXQWVRIGDWDRQOLQHDXWRPDWLFDOO\EURZVHGDQGFDSWXUHG
DOORI WKHFRQWHQWIURPWKH ZHEVLWHVXVHGIRUFODVVLILFDWLRQ
EHWZHHQ-XO\  DQG -XO\IRU D PRUHGHWDLOHG
GHVFULSWLRQRIWKHFUDZOHUVHH>@
C. Parts-of-Speech (POS) Tagging
7KH LQLWLDO VWHS LQ DQDO\]LQJ WKH FRQWHQW RQ ZHESDJHV
DXWRPDWLFDOO\ZDVWRGHWHUPLQHWKHWRSLFVRIGLVFXVVLRQ7RGR
WKLVZHZDQWHG WR WDNH D GDWDGULYHQ DSSURDFK DQG LVRODWH WKH
SDUWLFXODUQRXQV ZLWK WKHKLJKHVWUDWH RI RFFXUUHQFHZLWKLQWKH
GDWDFROOHFWHG :HDVVXPHGWKDW WKHPRVWIUHTXHQWO\ GLVFXVVHG
WRSLFVZRXOGPRVWOLNHO\EHWKHRQHVLQZKLFKH[WUHPLVWFRQWHQW
ZDVOLNHO\WREH GHWHFWHG EDVHG RQ WKH ZRUN RI >@ )XUWKHU
QRXQVZHUHFKRVHQEHFDXVHWKH\DUHWKHZRUGVPRVWOLNHO\WREH
VXUURXQGHGE\ UHOHYDQW VHQWLPHQW WHUPV >@ QDPHVDQGSODFHV
DUHRIWHQGHVFULEHGRUGHQRWHGE\WKHDGMHFWLYHVOLQNHGWRWKHP
DQG DGMHFWLYHV DUH RIWHQ WKH ZRUGV WKDW KDYH VHQWLPHQW YDOXHV
DWWDFKHGWRWKHP>@%\VSHFLI\LQJDGMHFWLYHVDVNH\ZRUGVWKH
VHQWLPHQWRIWKHZRUGLWVHOIZRXOGEHORVW
3DUWVRI6SHHFK 326 WDJJLQJ ZDV XVHG WR SHUIRUP WKLV
DQDO\VLV326LVDGDWDDQDO\VLVPHWKRGWKDWFROOHFWVDQGGLYLGHV
VXSSOLHG WH[W LQWR ZRUG JURXSLQJV VXFK DV QRXQV DQG YHUEV
EDVHGRQWKHLUXVDJHDQGSRVLWLRQZLWKLQWKHVHQWHQFHIRUPRUH
LQIRUPDWLRQVHH>@
)URPHDFKFODVVZHVHOHFWHGRIWKHPRVWIUHTXHQWQRXQV
DQG DJJUHJDWHG WKHP LQWR RQH OLVW ZKLFK DIWHU UHPRYDO RI
GXSOLFDWHVDQGPLVFDWHJRUL]HGZRUGVV\PEROVDQGQRQZRUGV
IRU H[DPSOH ZH HQGHG ZLWK  NH\ZRUGV 7KLV ILQDO OLVW
IRUPHG WKH NH\ZRUG OLVW IRU WKH VHQWLPHQW DQDO\VLV 'RPDLQ
H[SHUWVPD\UHSODFHRUH[WHQGWKHOLVWRINH\ZRUGVZLWKUHOHYDQW
GRPDLQVSHFLILF ZRUGV +RZHYHU WKH H[SORUDWRU\ DQG GDWD
GULYHQQDWXUHRIWKHVWXG\VXJJHVWHGWKDWZHUHO\RQWKHGDWDWR
LGHQWLI\WKHUHOHYDQWNH\ZRUGV
D. Sentiment Analysis
$IWHU WKH NH\ZRUG OLVW ZDV GHYHORSHG LW ZDV QHFHVVDU\ WR
LGHQWLI\DQGHYDOXDWHWKHFRQWH[WVXUURXQGLQJWKHNH\ZRUGV7R
DOORZDSURSHUDQDO\VLVRIWKHZHESDJHVVHQWLPHQWDQDO\VLVZDV
XVHGWRKLJKOLJKWUHOHYDQWWH[W
6HQWLPHQWDQDO\VLVLVDGDWD FROOHFWLRQDQGDQDO\VLVPHWKRG
WKDW DOORZV IRU WKH DSSOLFDWLRQ RI VXEMHFWLYH ODEHOV DQG
FODVVLILFDWLRQV>@,WFDQHYDOXDWHWKHRSLQLRQVRILQGLYLGXDOVE\
RUJDQL]LQJGDWDLQWRGLVWLQFWFODVVHVDQGVHFWLRQVDQGDVVLJQLQJ
DQ LQGLYLGXDO¶V VHQWLPHQW ZLWK D QHJDWLYH RU SRVLWLYH SRODULW\
YDOXH>@,WDOVRSURYLGHV DPRUHWDUJHWHGYLHZRIDGDWDVHW E\

)LJXUH
±
'DWD&ROOHFWLRQDQG7UHH*HQHUDWLRQ3URFHVV
105

DOORZLQJIRUWKHGHPDUFDWLRQEHWZHHQFDVHVWKDWDUHVRXJKWDIWHU
DQGWKRVH ZLWKRXW DQ\ QRWDEOHUHOHYDQFH6LQFH WKH SXUSRVHRI
WKHFXUUHQWVWXG\ZDV QRW WR SXVK WKH ERXQGDULHV RIVHQWLPHQW
DQDO\VLVDOJRULWKP6HQWL6WUHQJWK>@ZDVXWLOL]HG
6HQWL6WUHQJWK LV -DYDEDVHG VRIWZDUH WKDW XVHV D VSHFLILF
DOJRULWKP WR UXQ WKURXJK ODUJH YROXPHV RI WH[W DQG FUHDWH
VHQWLPHQW VFRUHV IRU WKH VXSSOLHG GRFXPHQWV :KLOH WKHUH DUH
VHYHUDOFRQILJXUDWLRQVRI WKH VRIWZDUH ZHXWLOL]HGDNH\ZRUG
IRFXVHGPHWKRGDVDFHQWUDOIHDWXUHRI6HQWL6WUHQJWKLVLWVDELOLW\
WR HYDOXDWH VHQWLPHQW DURXQG DQ\ JLYHQ NH\ZRUG >@ 7KLV
SURFHVV LQYROYHV WKH XWLOL]DWLRQ RI D GLFWLRQDU\ RI FDWDORJXHG
WHUPV DQG +DUYDUG¶V JHQHUDO LQTXLUHU GDWDEDVH WR GHWHUPLQH
VHQWLPHQWYDOXHV>@,WORFDWHVZRUGVWKDWFRUUHVSRQGZLWKLWV
GLFWLRQDU\DQGGDWDEDVHDQGWKHQLWHPSOR\VDVWHPPLQJPHWKRG
WRHYDOXDWHDWH[WE\DVVLJQLQJSRODULW\YDOXHVRIHLWKHUSRVLWLYH
RUQHJDWLYHWRWKH ZRUGV 9DOXHV DUH DXJPHQWHG E\ FKDUDFWHUV
WKDWFDQLQIOXHQFHWKHYDOXHVDVVLJQHGWRWKHWH[WVXFKDVERRVWHU
ZRUGVQHJDWLYHZRUGVUHSHDWHGOHWWHUVUHSHDWHGQHJDWLYHWHUPV
DQWDJRQLVWLFZRUGVSXQFWXDWLRQDQGRWKHUGLVWLQFWLYHFKDUDFWHUV
VXLWHGIRUVWXG\LQJDQRQOLQHFRQWH[W>@IRUPRUHLQIRUPDWLRQ
RQ6HQWL6WUHQJWKVHH>@
E. Decision Tree
$WRROLQPDFKLQHOHDUQLQJ DQG GDWD PLQLQJ WKDW LV ZLGHO\
DFFHSWHG DPRQJVW DFDGHPLFV :DLNDWR (QYLURQPHQW IRU
.QRZOHGJH$QDO\VLV:(.$ZDVUXQ RQ WKH VHQWLPHQW GDWD
IRUHDFKRIWKHWKUHHFODVVLILFDWLRQPRGHOV,WVVWDQGDUG-WUHH
FODVVLILFDWLRQPHWKRGZDVXVHGEHFDXVHLWFRQWDLQVDQDOJRULWKP
IRUWH[WFODVVLILFDWLRQWKDWDOORZVIRUDUXOHEXLOGLQJSURFHVV>@
:(.$ZDVDSSOLHGXVLQJIROGFODVVLILFDWLRQRQWKHRXWSXWWHG
VHQWLPHQWWDEOHV VHDUFKLQJ IRU GLIIHUHQFHV LQVHQWLPHQWYDOXHV
EHWZHHQWKHFODVVHVRIVLWHV$VDUHVXOWWKLVDOJRULWKPSURGXFHG
DGHFLVLRQWUHHZLWKUHFXUVLYHOHDYHVDQGEUDQFKHVZLWKHDFKOHDI
UHSUHVHQWLQJDVSHFLILFVHWRIVHQWLPHQWWKUHVKROGV
,WLV ZLWKWKHVHWKUHVKROGVWKDWWKH GHFLVLRQWUHHHVWDEOLVKHG
ZKHWKHUDFHUWDLQZHESDJHZDVRUJDQL]HGLQWRWKHradicalright
radical Islamic pro-extremist anti-extremist news RU other
FODVV)RUH[DPSOHLIWKHNH\ZRUGµPDUULDJH¶ZDVLQFOXGHGLQ
WKHWH[WZLWKDVHQWLPHQWYDOXHRIOHVVWKDQRUHTXDOWRDQGWKH
WHUPµSULFH¶ZDVDOVRLQWKHWH[WZLWKDVHQWLPHQWYDOXHRIOHVV
WKDQRUHTXDOWRDYDOXHRIWKHQWKH-SUXQHGWUHHLQGLFDWHG
WKDWRIWKHSDJHVIHOOZLWKLQWKHnewsFODVVLILFDWLRQ
F. Classification Results
 $IWHU WKH - DOJRULWKPZDV UXQ RQ WKH VHQWLPHQWGDWDIRU
HDFK RI WKH WKUHH PRGHOV :(.$ FUHDWHG D PHDVXUH WKDW
VSHFLILHGWKHQXPEHURI DFFXUDWHSDJHV WKDWILWLQWRHDFKRIWKH
FRUUHFWFODVVHV,QWXUQDFRQIXVLRQPDWUL[GLVSOD\HGWKHQXPEHU
RISDJHVWKDWZHUHFRUUHFWO\DQGLQFRUUHFWO\LGHQWLILHGLHIDOVH
SRVLWLYHV DQG IDOVH QHJDWLYHV LQ HDFK PRGHO PHDVXUHG ZLWK
SUHFLVLRQDQGUHFDOO3UHFLVLRQLVDPHDVXUHZKLFKUHSUHVHQWVWKH
H[DFWQHVVRIWKHUXOHV,WFDQEHWKRXJKWRIDVWKHQXPEHURIWUXH
SRVLWLYHHQWLWLHVLGHQWLILHG E\WKHFODVVLILHU RXW RIWKHVHW RI DOO
HQWLWLHV LGHQWLILHG DV SRVLWLYH 7KH ODUJHU WKH YDULDELOLW\ LQ WKH
UHVXOWVWKHVPDOOHUWKHSUHFLVLRQZLOOEH5HFDOOLVDPHDVXUHRI
FRPSOHWHQHVVDQGLVWKHQXPEHURISRVLWLYHHQWLWLHVidentifiedRXW
RIall WUXHSRVLWLYHHQWLWLHV$ UHFDOOVFRUHRI SHUIHFWPHDQV
WKDWDOOWUXHSRVLWLYHVDUHLGHQWLILHGZLWKRXWDQ\IDOVHQHJDWLYHV
,,,5(68/76
7KHUHVXOWV ZHUH LQWHUSUHWHGLQWZR ZD\V )LUVWWKHRYHUDOO
FODVVLILFDWLRQ UHVXOWV ZHUH HYDOXDWHG IRU HDFK FODVVLILFDWLRQ
PRGHOIROORZHGE\ WKH TXDOLW\ RI WKH HQWLUH UXOH VHW IRU HDFK
+HUHWKHDQDO\VLVZDVODUJHO\IRFXVHGRQWKHSURH[WUHPLVWFODVV
A. Two-Class Model: Results and Quality of Rule-Set
7KH WZRFODVV WUHH DQDO\VLV LQGLFDWHG WKDW  RI
ZHESDJHV ZHUH VXFFHVVIXOO\ FODVVLILHG DFURVV FODVVHV 7KH
FRPELQHG radical Right DQG radical Islamic FODVV ZDV
VXFFHVVIXOO\ FODVVLILHG DW D FRQVLGHUDEO\ ORZHU UDWH WKDQ WKH
FRPELQHGanti-extremistnewsDQGotherFODVVLHRI
SDJHVZHUHDFFXUDWHO\LGHQWLILHG)XUWKHUPRUHWKHUHFDOORIWKH
HQWLUH UXOHVHW LQGLFDWHG WKDW DFWXDO WZR UDGLFDO FODVVHV ZHUH
FODVVLILHG DW D UDWH RI  LQ FRPSDULVRQ WR WKH QRQUDGLFDO
FODVVHV¶UHFDOOPHDVXUHRIVHHSUHFLVLRQDQGUHFDOOLQ7DEOH

B. Four-Class Model: Results and Quality of Rule-Set
7KH IRXUFODVV WUHH DQDO\VLV UHYHDOHG WKDW  RI
ZHESDJHV ZHUH DFFXUDWHO\ FODVVLILHG DFURVV FODVVHV
,QWHUHVWLQJO\news SDJHVKDGWKH KLJKHVWQXPEHURI FRUUHFWO\
FODVVLILHGSDJHV LH IROORZHGE\WKHFRPELQHG WZR
UDGLFDOFODVVHVLHZKLFKDJDLQVKRZHGPRGHUDWHO\KLJK
SUHFLVLRQ UDWHV LH  DQG PRGHUDWHO\ ORZ UHFDOO UDWHV
LHNewsFODVVVLWHVZHUHDOVRPLVFODVVLILHGDFURVV WKH
UHPDLQLQJWKUHHFODVVHVDWWKHKLJKHVWIUHTXHQF\VHH7DEOH
C. Five-Class Model: Results and Quality of Rule-Set
7KH ILYHFODVV WUHH DQDO\VLV LQGLFDWHG WKDW  RI WKH
SDJHVZHUHDFFXUDWHO\FODVVLILHG2ISDUWLFXODULQWHUHVWKHUHZHUH
WKHradical Right-DQGradical Islamic SDJHV)LUVWWKHFRQIXVLRQ
PDWUL[LQGLFDWHGWKDWDPHUHRIradical RightSDJHVZHUH
VXFFHVVIXOO\FODVVLILHG:LWKLQWKLVSDUWLFXODUFODVVnewsSDJHV
ZHUHPLVLGHQWLILHG DWWKHKLJKHVWUDWH LQWKHPRGHO RI
LWV SDJHV ZHUH PLVFODVVLILHG DV UDGLFDO ULJKWZLQJ 6HFRQG
 RI SDJHV WKDW IHDWXUHG UDGLFDO ,VODPLF FRQWHQW ZHUH
DFFXUDWHO\FODVVLILHGNewsSDJHVZHUHDOVRPLVFODVVLILHGDWWKH
KLJKHVWUDWHLHZLWKLQWKHradical Islamic FODVVNews
SDJHVDJDLQKDGWKHKLJKHVWQXPEHURIFRUUHFWO\FODVVLILHGSDJHV
DFURVVWKH HQWLUH VDPSOH VLPLODUWR WKH FODVVLILFDWLRQ
UHVXOWVLQWKHIRXUFODVVPRGHO0HDVXULQJWKHTXDOLW\RIWKHUXOH
IRU WKH ILYHFODVV PRGHO WKH DFWXDO radical Right SDJHV ZHUH
FODVVLILHGDW D ORZ UHFDOO UDWH RI  $WWKHRWKHU HQG RI WKH
VSHFWUXPERWKWKH SUHFLVLRQDQGUHFDOOPHDVXUHVLQGLFDWHGWKDW
radical IslamicSDJHVZHUHFODVVLILHGDWDKLJKHUUDWHRIVXFFHVV
WKDQWKH radical Right FODVVRIWKH radical IslamicSDJHV
ZHUHVXFFHVVIXOO\ FODVVLILHG DQGDFWXDOSDJHVZLWKLQ WKLVFODVV
ZHUHVXFFHVVIXOO\FODVVLILHGDWDUDWHRI6HH7DEOH
,9&21&/86,216
 :HVRXJKWWREXLOGRQDVHQWLPHQWJXLGHGZHEFUDZOHUWKDW
FRXOG DFFXUDWHO\ FODVVLI\ WKH VXEWOH \HW UDGLFDOEDVHG WH[W RQ
UDGLFDO 5LJKW DQG UDGLFDO ,VODPLF ZHESDJHV $ IHZ QRWDEOH
ILQGLQJV ZHUH SURGXFHG )LUVW FODVVLILFDWLRQ UHVXOWV IURP WKH
WZRDQGIRXUFODVVPRGHOVVXJJHVWHGWKDWWKHVHQWLPHQWJXLGHG
ZHEFUDZOHUODFNHGWKH RYHUDOO DELOLW\WRGLIIHUHQWLDWHEHWZHHQ
WKHVHQWLPHQWIRXQGRQWKHVHOHFWHGZHEVLWHVWKDWGLGDQGGLGQRW
SURPRWHUDGLFDO LGHRORJLHV 7KLVILQGLQJFRQWUDVWV WKDW RI >@
ZKHUH SURH[WUHPLVW ZHEIRUXP DQG ZHESDJH GDWD ZHUH
106

FODVVLILHGDWDKLJKUDWHRIDFFXUDF\6HFRQGWKHILYHFODVVPRGHO
VKRZHG WKDW UDGLFDO ,VODPLF SDJHV ZHUH FODVVLILHG DW D PXFK
KLJKHU UDWH RI VXFFHVV WKDQ UDGLFDO ULJKWZLQJ SDJHV 3UHYLRXV
UHVHDUFKVXJJHVWVWKDWWKHVHQWLPHQWRQ UDGLFDO 5LJKW ZHEVLWHV
HJ>@DQGUDGLFDO,VODPLFZHEVLWHVLH>@LVSUHVHQWHG
LQDVXEWOHPDQQHUERWKWRDSSHDOWRDZLGHUDXGLHQFHDQGUHFUXLW
QHZPHPEHUVIRUH[DPSOH +RZHYHURXU UHVXOWVOHQGVXSSRUW
IRU >@¶V DVVHUWLRQ WKDW ZKLOH UDGLFDO ,VODPLF VLWHV DWWHPSW WR
OHJLWLPL]HWKHLUHIIRUWVE\SUHVHQWLQJWKHPVHOYHVDVQHZVVRXUFH
ZHEVLWHVWKHVHQWLPHQWLVDOPRVWDOZD\VUHODWHGWRUDGLFDOWRSLFV
LH GLVFXVVLRQV RI YLROHQFH 7KDW VDLG D PRUH LQGHSWK
DQDO\VLVLV QHHGHGWR DVVHVVZKHWKHUWKHUHLVDFOHDUGLVWLQFWLRQ
EHWZHHQDQH[WUHPLVWZHEVLWHIHDWXULQJHOXVLYHPHVVDJHVDQGD
QHZVVLWH7KLVSUREOHPFRXOGEHH[SORUHGXVLQJRWKHUWRROVRI
FODVVLILFDWLRQVXFK DV UDQGRPIRUHVWVRU%D\HVLDQ PHWKRGV WR
VXSSRUW YHFWRU PDFKLQHV DQG QHXUDO QHWZRUNV )XWXUH VWXGLHV
VKRXOG DOVR LQWHJUDWH D TXDOLWDWLYH XQGHUVWDQGLQJ RI KRZ
PDFKLQHOHDUQLQJWRROVPDNHGHFLVLRQVDERXWWKHZHESDJHVWKDW
DUHYLVLWHG 'RLQJ VRPD\LQFUHDVH WKHUHOLDELOLW\RIWKH UHVXOWV
DQGLQFUHDVHWKHOLNHOLKRRGRILGHQWLI\LQJUDGLFDOWH[WRQOLQH
95()(5(1&(6
 / %DFN ³$U\DQV 5HDGLQJ $GRUQR &\EHU&XOWXUH DQG 7ZHQW\)LUVW
&HQWXU\ 5DFLVP´LQ Ethnic and Racial Studies, 25  SS

 /%RZPDQ*ULHYH³$QWLDERUWLRQ([WUHPLVP2QOLQH´LQFirst Monday,
14
 0&DLDQL 'GHOOD 3RUWD & :DJHPDQQMobilizing on the Extreme
Right: Germany, Italy, and the United States2[IRUG2[IRUG8QLYHUVLW\
3UHVV
 + &KHQ Dark Web: Exploring and Data Mining the Dark Side of the
Web1HZ<RUN6SULQJH
 .&RKHQ) -RKDQVVRQ/.DDWL-& 0RUN³'HWHFWLQJ/LQJXLVWLF
0DUNHUV IRU 5DGLFDO 9LROHQFH LQ 6RFLDO 0HGLD´ LQ Terrorism and
Political Violence, 26SS
 5)HOGPDQ³7HFKQLTXHV DQG$SSOLFDWLRQVIRU 6HQWLPHQW$QDO\VLV´LQ
Communications of the ACMSS
 0 +DOO (  )UDQN + *HRIIUH\ % 3IDKULQJHU 3  5HXWHPDQQ  ,
:LWWHQ ³7KH :(.$ 'DWD 0LQLQJ 6RIWZDUH$Q 8SGDWH´ LQ SIGKDD
Explorations, 11SS
 0 +X  / %LQJ ³0LQLQJ DQG 6XPPDUL]LQJ &XVWRPHU 5HYLHZV´ LQ
Proceedings of ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining
 ,QWHUQHW:RUOG 6WDWVInternet Growth Statistics 5HWULHYHG IURP
KWWSZZZLQWHUQHWZRUOGVWDWVFRPHPDUNHWLQJKWP
 -0HL5)UDQN ³6HQWLPHQW&UDZOLQJ([WUHPLVW &RQWHQW&ROOHFWLRQ
WKURXJKD6HQWLPHQW$QDO\VLV*XLGHG :HE&UDZOHU´LQProceedings of
the International Conference on Advances in Social Networks Analysis
and Mining
 06DJHPDQ³7KH6WDJQDWLRQLQ7HUURULVP5HVHDUFK´LQTerrorism and
Political ViolenceSS
 5 6FULYHQV *  'DYLHV 5  )UDQN  - 0HL ³6HQWLPHQWEDVHG
,GHQWLILFDWLRQRI5DGLFDO $XWKRUV 6,5$´ LQ Proceedings of the 2015
IEEE ICDM Workshop on Intelligence and Security Informatics
 36HLE'0-DQEHNGlobal Terrorism and New Media: The Post-Al
Qaeda Generation/RQGRQ5RXWOHGJH
 0 7KHOZDOO  . %XFNOH\ ³7RSLFEDVHG 6HQWLPHQW $QDO\VLVIRU WKH
6RFLDO:HE7KH 5ROHRI 0RRGDQG,VVXHUHODWHG:RUGV´LQ Journal of
the American Society for Information Science and Technology 
SS
 77KHW- 1D&.KRR³$VSHFWEDVHG 6HQWLPHQW$QDO\VLVRI0RYLH
5HYLHZV RQ 'LVFXVVLRQ %RDUGV´ LQ Journal of Information Science
SS
 <7VIDWL*:HLPDQQ³ZZZWHUURULVPFRP7HUURURQWKH,QWHUQHW´
LQStudies in Conflict & Terrorism, 25SS
 3UHGLFWHG3DJHV  

Radical RightDQGRadical
Islamic
Anti-extremist, News DQG
Other
3UHFLVLRQ 5HFDOO
$FWXDO3DJHV
Radical RightDQG
Radical Islamic   
Anti-extremist,
News DQG Other   

7DEOH
±
&RQIXVLRQ0DWUL[IURPWKH-7UHH$QDO\VLVIRUWKH7ZR&ODVV0RGHO
 3UHGLFWHG3DJHV   

Radical RightDQG
Radical IslamicAnti-extremist News Other 3UHFLVLRQ 5HFDOO
$FWXDO3DJHV
Radical RightDQG
Radical Islamic     
Anti-extremist      
News      
Other      

7DEOH
±
&RQIXVLRQ0DWUL[IURPWKH-7UHH$QDO\VLVIRUWKH)RX
U
&ODVV0RGHO
 3UHGLFWHG3DJHV   

Radical
Right
Radical
Islamic Anti-extremist News Other 3UHFLVLRQ 5HFDOO
$FWXDO3DJHV
Radical Right       
Radical Islamic       
Anti-extremist       
News       
Other       

7DEOH±&RQIXVLRQ0DWUL[IURPWKH-7UHH$QDO\VLVIRUWKH)LYH&ODVV0RGHO

107