Conference PaperPDF Available

Proceedings of the First Workshop on Educational Data Visualization (EdViz-2019)

March 2019

Conference: Companion Proceedings of the 9th International Learning Analytics & Knowledge Conference (LAK19)
At: ASU, Tempe, AZ

Authors:

Nirmal Patel

Playpower Labs

Derek Lomas

Delft University of Technology

Collin Sellman

Arizona State University

The primary goal of this workshop is to produce open source data visualizations that help communicate results of learning analytics (LA) research to educators. Instructors are increasing their use of data to drive instruction, and various results of LA research are useful towards this end. However, the actionable insights discovered by the LA community are often inaccessible to educators due to their relative complexity. In this case, it is possible to use data visualization to communicate actionable insights about learners to optimize instruction effectively. Visualizations of learner data can make it easy for teachers and other education stakeholders to take evidence-based action. Organizers of the workshop intend to invite authors to describe and implement educational data visualizations that can aid decision making in online and offline classrooms. The workshop will result in a gallery of open source educational data visualizations that can be freely used by the LA community.

An example game state from RumbleBlocks and how it would be described to TRESTLE.

…

Distribution of Cronbach's alpha when an item within an assessment is dropped. The points in blue color signify Cronbach's alpha for the overall assessment and the points in black signify Cronbach's alpha when one of the assessment items is dropped.

…

Many of the 100 students did not attempt any exercise (long grey bars). The exercises Gradient and OurShapes were attempted the most and mostly correctly solved (large green area)

…

A visualized TRESTLE tree of a sample of 100 solutions to a level in RumbleBlocks. In the top visualization clusters are colored by their likelihood of succeeding on the level (yellow for passing, purple for failing). In the lower three visualizations the same tree is re--colored according to different physical properties of game solutions.

…

The assessment and item information show up when we hover over the dots of the plot.

…

Figures - uploaded by Nirmal Patel

Content may be subject to copyright.

Content uploaded by Nirmal Patel

Content may be subject to copyright.

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

Workshop(on(Educational(Data(Visualization(

Nirmal'Patel'

A5$B;)C#"(-$1,(

+&"*$5D;5$B;)C#"5$1,8.)*(

Derek'Lomas'

6#5E%(:+&'#",&%B()E(F#.G+)5)HB(

I8<85)*$,D%2<#5E%8+5(

Collin'Sellman'

0"&J)+$(K%$%#(:+&'#",&%B'

.)55&+8,#55*$+D$,28#<2'

ABSTRACTL(FG#( ;"&*$"B(H)$5()E(%G&,(C)"M,G);(&,(%)(;")<2.#();#+( ,)2".#(<$%$('&,2$5&J$%&)+,(

%G$%( G#5;( .)**2+&.$%#( "#,25%,( )E( 5#$"+&+H( $+$5B%&.,( =-0 @( "#,#$".G( %)( #<2.$%)",8( N+,%"2.%)",(

$"#(&+."#$,&+H( %G#&"( 2,#( )E(<$%$( %)( <"&'#(&+,%"2.%&)+/($+<( '$"&)2,( "#,25%,()E(-0( "#,#$".G( $"#(

2,#E25(%)C$"<,(%G&,(#+<8(O)C#'#"/(%G#($.%&)+$15#(&+,&HG%,(<&,.)'#"#<(1B(%G#(-0( .)**2+&%B(

$"#()E%#+(&+$..#,,&15#(%)(#<2.$%)",(<2#(%)(%G#&"("#5$%&'#(.)*;5#P&%B8(N+(%G&,(.$,#/(&%(&,(;),,&15#(

%)( 2,#( <$%$( '&,2$5&J$%&)+( %)( .)**2+&.$%#( $.%&)+$15#( &+,&HG%,( $1)2%( 5#$"+#",( %)( );%&*&J#(

&+,%"2.%&)+(#EE#.%&'#5B8(Q&,2$5&J$%&)+,()E(5#$"+#"(<$%$(.$+(*$M#(&%(#$,B(E)"(%#$.G#",($+<()%G#"(

#<2.$%&)+(,%$M#G)5<#",(%)(%$M#(#'&<#+.#31$,#<($.%&)+8(R"H$+&J#",()E(%G#(C)"M,G);(&+%#+<(%)(

&+'&%#( $2%G)",( %)( <#,."&1#( $+<( &*;5#*#+%( #<2.$%&)+$5( <$%$( '&,2$5&J$%&)+,( %G$%( .$+( $&<(

<#.&,&)+( *$M&+H( &+( )+5&+#( $+<( )EE5&+#( .5$,,"))*,8( FG#( C)"M,G);( C&55( "#,25%( &+( $( H$55#"B( )E(

);#+(,)2".#(#<2.$%&)+$5(<$%$('&,2$5&J$%&)+,(%G$%(.$+(1#(E"##5B(2,#<(1B(%G#(-0(.)**2+&%B8(

KeywordsL(6$%$(Q&,2$5&J$%&)+/(6$%$36"&'#+(N+,%"2.%&)+/(N+E)"*$%&)+(!)**2+&.$%&)+(

1( INTRODUCTION(

0( "#.#+%( ,2"'#B( &+( %G#( :+&%#<( K%$%#,( E)2+<( %G$%( STU( )E( %G#( V3WX( %#$.G#",( 2,#( $( .)*1&+$%&)+( )E(

$.$<#*&.(<$%$($+<(+)+3$.$<#*&.(<$%$(%)(2+<#",%$+<(%G#&"(,%2<#+%,Y(;#"E)"*$+.#8(O)C#'#"/(7ZU()E(

%G#(,2"'#B#<(%#$.G#",($5,)("#;)"%#<(%G$%(%G#"#(C$,(%))(*2.G(<$%$(E)"(%G#*(%)(5))M($%8((O)C(.$+(C#(

G#5;(#<2.$%)",(*$M#(,#+,#()E(5$"H#($*)2+%,()E(,%2<#+%(<$%$[(6$%$('&,2$5&J$%&)+(&,()+#()E(%G#(*),%(

C&<#5B( 2,#<( %#.G+&\2#,( %G$%( G#5;( ;#);5#( *$M#( ,#+,#( )E( 5$"H#( $*)2+%,( )E( +2*#"&.$5( &+E)"*$%&)+8(

]"$;G&.$5( "#;"#,#+%$%&)+,( )E( <$%$( .$+( 1#( 2,#<( '#"B( #EE#.%&'#5B( %)( .)**2+&.$%#( .)+%#P%3,;#.&E&.(

&+E)"*$%&)+8((

^#;)"%&+H( )E( 5#$"+#"( <$%$( &,( )+#( )E( %G#( .)"+#",%)+#,( )E( -0( "#,#$".G/( $+<( %G#( -0( .)**2+&%B( G$,(

<#'#5);#<(<)*$&+3,;#.&E&.( <$%$( '&,2$5&J$%&)+,( %)( ,G)C( ,%2<#+%(5#$"+&+H(&+(<&EE#"#+%(.)+%#P%,8(K)*#(

)E( %G#,#( '&,2$5&J$%&)+,( #*#"H#<( E")*( %G#( -#$"+&+H( K.&#+.#( "#,#$".G( .)**2+&%B( =#8H8/( -#$"+&+H(

!2"'#,/@( CG&5#( )%G#"( '&,2$5&J$%&)+,( G$'#( $( .5),#( $EE&+&%B( C&%G( .5$,,"))*( ;"$.%&.#( =#8H8/( !2""&.252*(

A$.&+H(A5)%,8@(05%G)2HG(%G#,#('&,2$5&J$%&)+,($"#(,5)C5B(*$M&+H(%G#&"(C$B(&+%)(%G#(G$+<,()E(#<2.$%)",/(

*$+B( )E( %G#,#( '&,2$5&J$%&)+,( $"#( +)%( "#$<&5B( $'$&5$15#( E)"( "#;")<2.%&)+( &+( %G#( );#+( ,)2".#( <$%$(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

$+$5B,&,( #+'&")+*#+%,( ,2.G( $+<( ^( $+<( AB%G)+8( FG&,( C)"M,G);( $&*,( %)( ;")<2.#( $+( );#+( ,)2".#(

H$55#"B()E(#<2.$%&)+(<$%$('&,2$5&J$%&)+,(%G$%($"#(#$,&5B("#2,$15#(1B(-0("#,#$".G#",($+<(;"$.%&%&)+#",8(

( (

Figure'1a'(left)'and'1b'(right):'The'figure'on'the'left'shows'a'Curriculum'Pacing'Plot,'showing'the'

progression'of'a'classroom'through'a'yearlong'period.'The'figure'on'the'right'shows'a'Mastery'

Matrix'for'a'classroom,'which'allows'easy'identification'of'struggling'students'and'difficult'topics.'

Figure'2:'Learning'Pathway'Visualization,'showing'frequently'taken'learning'paths'taken'by'

students'in'an'online'curriculum.'

F)(*$M#(<$%$('&,2$5&J$%&)+,($(%))5(E)"(M+)C5#<H#(<&,.)'#"B/(%G#B(G$'#(%)(1#(*$<#(<)*$&+(,;#.&E&.8(0(

,.$%%#"(;5)%/(CG#+($<$;%#<(%)($(.)+%#P%()E(.5$,,"))*/(.$+( 1#.)*#( $( !2""&.252*(A$.&+H(A5)%(=E&H2"#(

W$/@($( G#$%*$;( C&%G(,%2<#+%( $+<( ,M&55(<$%$( .$+(1#( %2"+#<( &+%)($( _$,%#"B( _$%"&P(=E&H2"#(W1/@( $+<( $(

-#$"+&+H(A$%GC$B(Q&,2$5&J$%&)+(C&%G(#<2.$%&)+$5($.%&'&%&#,($,(+)<#,(.$+(<&,;5$B(G&HGC$B,()E(,%2<#+%(

5#$"+&+H(=E&H2"#(X8@(055(%G#,#('&,2$5&J$%&)+,($"#(1$,#<(2;)+(.)**)+5B(2,#<(H"$;G&.$5("#;"#,#+%$%&)+,(

)E( <$%$/( 12%( C&%G( $( 5&%%5#(%C#$M&+H/( %G#B( %2"+( &+%)( %))5,( %G$%( 5#%( 2,( #P%"$.%( *#$+&+HE25( &+E)"*$%&)+(

E")*(#<2.$%&)+$5( <$%$8(_)"#)'#"/( 2,#E25(&+E)"*$%&)+( '&,2$5&J$%&)+,(.$+( G#5;(,2;;)"%( $+<(#+H$H#( $(

"$+H#( )E( <&EE#"#+%( ,%$M#G)5<#",8( `)"( #P$*;5#/( 5#$"+#"( <$%$( '&,2$5&J$%&)+,( .$+( G#5;( .2""&.252*(

.))"<&+$%)",( &+( ,.G))5,( 2+<#",%$+<( ,%2<#+%( 1#G$'&)"( &+( $( *$++#"( %G$%( .$+( &+E)"*( &+,%"2.%&)+8(

-#$"+&+H( A$%GC$B,( )E( $( _RR!( .$+( G#5;( &+,%"2.%&)+$5( <#,&H+#"( <&,.)'#"( CG#%G#"( ,%2<#+%,( $"#(

;")H"#,,&+H(%G")2HG(%G#(.2""&.252*($,(#P;#.%#<()"(+)%8( aG#"#$,(*$+B(<$%$(*&+&+H(%#.G+&\2#,(.$+(

;")<2.#(b15$.M(1)P#,/c('&,2$5&J$%&)+,($"#(G2*$+("#$<$15#8(FG&,("#$<$1&5&%B($55)C,(<&"#.%(,%$M#G)5<#",(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

=#8H8/(%#$.G#",($+<(&+,%"2.%&)+$5(<#,&H+#",@(%)(."&%&\2#($,,2*;%&)+,(*$<#(1B(<$%$(,.&#+%&,%,/(CG)($"#(

)E%#+("#*)'#<(E")*(%G#(.)+%#P%()E(<$%$(;")<2.%&)+8(

-0( "#,#$".G( G$,( %G#( ;)%#+%&$5( %)( &+E)"*( "#$53C)"5<( &+,%"2.%&)+/(12%( 1"&+H&+H( "#,#$".G( E&+<&+H,( &+%)(

"#$53C)"5<("#\2&"#,(#EE#.%&'#(.)**2+&.$%&)+(%)(,%$M#G)5<#",()2%,&<#(%G#(-0(.)**2+&%B8(FG&,(.$+(1#(

<&EE&.25%(1#.$2,#( #<2.$%)",(2,2$55B(E&+<("#,25%,()E( 5#$"+&+H( $+$5B%&.,( .G$55#+H&+H( %)( 2+<#",%$+<8( a#(

1#5&#'#(%G$%(H"$;G&.,(.$+(G#5;(2,(1"&<H#(%G&,(.)**2+&.$%&)+(H$;($+<(*$M#(%G#("#,25%,()E(-0("#,#$".G(

*)"#(E"2&%E258(

2( ORGANIZATION(

FG#(C)"M,G);(C&55(1#( )"H$+&J#<( $,( $(G$5E3<$B(#'#+%/($,(%G&,(&,(%G#(E&",%(<$%$('&,2$5&J$%&)+(C)"M,G);(

1#&+H( )"H$+&J#<( &+( %G#( .)**2+&%B8( FG#( )"H$+&J#",( C&55( &+'&%#( $2%G)",( %)( <#'#5);( $+<( &*;5#*#+%(

);#+(,)2".#(<$%$( '&,2$5&J$%&)+,($+<( <#,."&1#( %G#*( &+(;),%#",/( ,G)"%( ;$;#",/()"( E255(;$;#",8(A),%#",(

C&55(<#,."&1#('&,2$5&J$%&)+,(%G$%($"#("#5#'$+%(%)(%G#(.)**2+&%B(12%( $"#(,%&55( 1#&+H( <#'#5);#<8(KG)"%(

;$;#",(C&55(<#,."&1#(C)"M,(%G$%($"#(,2EE&.&#+%5B(*$%2"#(12%(G$'#+Y%(1##+(%#,%#<(&+(%G#(E&#5</($+<(E255(

;$;#",(C&55(<#,."&1#(+)'#5('&,2$5&J$%&)+,(%G$%($"#(*$%2"#($+<(G$'#(1##+(%#,%#<(C&%G(%G#(,%$M#G)5<#",(

&+(%G#(E&#5<8(FG#(;$"%&.&;$%&)+()E(%G#(C)"M,G);( C&55( 1#(*&P#</($+<( <#5#H$%#,()%G#"(%G$+(%G#( $2%G)",(

C&55(1#( &+'&%#<(%)( "#H&,%#"($+<(%$M#(1#+#E&%()E(%G#(C)"M,G);8(FG#(C)"M,G);(C&55(.)+%$&+($( ,#"&#,()E(

'&,2$5&J$%&)+(<#*),/( $+<( #$.G(;"#,#+%#"(C&55(,G)C(G)C(#'#"B)+#(&+(%G#( $2<&#+.#(.$+( 2,#( %G#();#+(

,)2".#( '&,2$5&J$%&)+8( 02%G)",( C&55( 1#( G&HG5B( #+.)2"$H#<( %)( <#'#5);( %G#&"( '&,2$5&J$%&)+,( &+( ^( $+<(

AB%G)+/($+<($55()E(%G#('&,2$5&J$%&)+(;")H"$*,(C&55(1#(2;5)$<#<(&+($(]&%O21("#;),&%)"B(%G$%(C&55("#*$&+(

E"##5B($..#,,&15#(%)(%G#(-0(.)**2+&%B8(a#(#P;#.%(WT(%)(79(;$"%&.&;$+%,(%)($%%#+<()2"(C)"M,G);8(4)(

,;#.&$5(#\2&;*#+%()%G#"(%G$+($(;")I#.%)"(C&55(1#("#\2&"#<8(

(

3( OBJECTIVES(

FG#( H)$5( )E( %G&,( C)"M,G);( &,( %)( $<$;%( #P&,%&+H( '&,2$5&J$%&)+,( )"( ;")<2.%( +)'#5( '&,2$5&J$%&)+,( )E(

#<2.$%&)+$5(<$%$(%G$%(.$+(.)**2+&.$%#($.%&)+$15#(&+E)"*$%&)+(%)(#<2.$%)",8(a#(C&55(&+'&%#($2%G)",(

%)( <#,."&1#( $+<( &*;5#*#+%( #<2.$%&)+$5( <$%$( '&,2$5&J$%&)+,( "#5#'$+%( %)( $( "$+H#( )E( #<2.$%&)+$5(

.)+%#P%,(,2.G($,('$"&)2,(%B;#,()E(<&H&%$5(5#$"+&+H(#+'&")+*#+%,(,2.G($,(_RR!,/('&"%2$5(,.G))5,/(V3WX(

$+<(2+&'#",&%B(.5$,,"))*,/( $+<( )%G#"(#P;5)"$%)"B( 5#$"+&+H( #+'&")+*#+%,( 5&M#( $,( #<2.$%&)+$5( H$*#,8(

a#(C&55(#+.)2"$H#($2%G)",(%)(;")<2.#(&+%#"$.%&'#('&,2$5&J$%&)+,(,)(%G$%(%G#(2,#",(.$+($.%&'#5B(#+H$H#(

C&%G(%G#(<$%$( $+<( 2,#(%G#(H"$;G,($,( %))5,( %)( #P;5)"#( <$%$( $+<( 2+<#",%$+< ( ,%2<#+%,( "$%G#"( %G$+( $(

,+$;,G)%()E(<$%$(%G$%(%G#B(.$+(5))M($%($+<("#E5#.%()+8(0,(%G#();#+3,)2".#(.)<#(&,($+(#*;G$,&,()E(%G#(

C)"M,G);/($2%G)",(C&55($5,)(1#($,M#<(%)(C"&%#(;")H"$*,(%G$%($"#(#$,B(%)(2,#/(#8H8/(;")'&<&+H(E2+.%&)+,(

%G$%(%$M#( C#553<#E&+#<( <$%$( ,%"2.%2"#,( =#8H8/( <$%$( %$15#,/( H"$;G,@( $,( &+;2%( $+<( ;")<2.#( <#,&"#<(

'&,2$5&J$%&)+,($,()2%;2%8(F);&.,(E)"(%G#(<$%$('&,2$5&J$%&)+,(C&55(1#L(

•!K%2<#+%(5#$"+&+H($+<(1#G$'&)"(

•!K%2<#+%(M+)C5#<H#($+<(*$,%#"B(

•!K%2<#+%(5#$"+&+H(%"$I#.%)"&#,($+<(;").#,,#,(

•!K%2<#+%(*&,.)+.#;%&)+,(

•!A")15#*3,)5'&+H(,%"$%#H&#,(

•!F#$.G&+H(,%"$%#H&#,(

•!!)55$1)"$%&'#(5#$"+&+H(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

•!!5$,,"))*(5#$"+&+H(

•!d*)%&)+$5(,%$%#,(

•!!5&.M,%"#$*(<$%$(

•!K%2<#+%(H")2;,($+<(%G#&"(<&EE#"#+.#,(

•!!)*;$"&,)+()E()1,#"'#<($+<(&<#$5(,%2<#+%(1#G$'&)"(

•!:,$H#($+<(#EE&.$.B()E(#<2.$%&)+$5(.)+%#+%(

•!6#*B,%&EB&+H(b15$.M(1)Pc(*$.G&+#(5#$"+&+H(*)<#5,(

(

0+B()%G#"(%);&.,( %G$%( $"#( "#5#'$+%(%)(#<2.$%&)+$5(<$%$($+<(#+'&")+*#+%,(C&55($5,)(1#( &+.52<#<8( a#(

G);#( %G$%( %G#( '&,2$5&J$%&)+,(;")<2.#<( <2"&+H( %G&,( C)"M,G);( .$+( $.%( $,( C)"M#<( #P$*;5#,( )E( '&,2$5(

<#,&H+(;$%%#"+,(%G$%(.$+(1#($;;5&#<(%)(#<2.$%&)+$5(<$%$(E")*($("$+H#()E(<&EE#"#+%(,)2".#,($+<(,#"'#(

$,($(\2&.M("#E#"#+.#(H2&<#(E)"(-0(.)**2+&%B8(

(

REFERENCES((

6$%$(e2$5&%B(!$*;$&H+8(=X9Wf@8(?2%1()%*,&1/(%&-(@,%+2,*/(@2'&A(7B"C1(D-C+%1'"&%4(E%1%F'^#%"&#'#<(

E")*(G%%;,Lgg<$%$\2$5&%B.$*;$&H+8)"Hg"#,)2".#gCG$%3;$"#+%,3$+<3%#$.G#",3%G&+M3$1)2%3

#<2.$%&)+3<$%$g(

V)#<&+H#"/(V8(^8/(>$M#"/(^8(K8/(!2++&+HG$*/(V8/(KM)H,G)5*/(08/(-#1#"/(>8/(h(K%$*;#"/(i8(=X9W9@8(0(<$%$(

"#;),&%)"B( E)"( %G#( d6_( .)**2+&%BL( FG#( AK-!( 6$%$KG);8(O$+<1))M( )E( #<2.$%&)+$5( <$%$(

*&+&+H/(Z7/(Z73Tj8(

A$%#5/( 48/( KG$"*$/( 08/( K#55*$+/( !8/( h( -)*$,/( 68( =X9Wf@8( !2""&.252*( A$.&+HL( 0( 4#C( 0;;")$.G( %)(

6&,.)'#"( N+,%"2.%&)+$5( A"$.%&.#,( &+( !5$,,"))*,8( N+(3&1,*&%1'"&%4( !"&5,*,&+,( "&( 3&1,44'.,&1(

@C1"*'&.(G8/1,#/(=;;8(7ZT37TW@8(K;"&+H#"/(!G$*8(

(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

Implications,of,Instructor,Analytics,Use,Patterns,for,,,,,,,,,,,,,,,,,,,,,,,,,,,

the,Design,of,Actionable,Educational,Data,Visualizations,

Alyssa&Friend&Wise&

4#A(?)"B(:+&'#",&%C(

$5C,,$8A&,#D+C28#<2((

Yeonji&Jung&

4#A(?)"B(:+&'#",&%C(

C#)+E&8E2+FD+C28#<2((

ABSTRACTG( HI&,( ;$;#"( )JJ#",( &+,&FI%,( %)( &+J)"*( #'&<#+.#31$,#<( 5#$"+&+F( $+$5C%&.,( <#,&F+(

%I")2FI( %I#( ;"#,#+%$%&)+( )J( $+( #*;&"&.$55C3<#"&'#<( *)<#5( )J( &+,%"2.%)"( $+$5C%&.,( 2,#8( HI#(

*)<#5("#;"#,#+%,(B#C(#5#*#+%,()J(%I#(A$C,(&+(AI&.I(&+,%"2.%)",(*$B#(,#+,#()J($+<("#,;)+<(

%)(%I#($+$5C%&.,( <$%$( $,( ;$"%()J(%I#&"( ;#<$F)F&.$5( <#.&,&)+3*$B&+F(;").#,,/(AI&.I(.$+($,,&,%(

#<2.$%&)+$5( '&,2$5&K$%&)+( <#,&F+#",( &+( .I)),&+F( $*)+F( %I#( *C"&$<( <$%$( "#;"#,#+%$%&)+,(

;),,&15#( %)( ;")<2.#( &+%#";"#%$15#( $+<( $.%&)+$15#( 5#$"+&+F( $+$5C%&.,( ,C,%#*,8( L+,%"2.%)"(

$+$5C%&.,(2,#(&,(,I)A+($,( $(*25%&3;I$,#(;").#,,( <&'&<#<($."),,(%I#(5$"F#"($.%&'&%&#,()J(,#+,#3

*$B&+F( $+<( ;#<$F)F&.$5( "#,;)+,#8( M#+,#3*$B&+F( ;").#,,( *)'#,( J")*( $( F#+#"$5( $"#$( )J(

.2"&),&%C(AI&.I(&+,%"2.%)",(.$+(<#'#5);(&+%)(*)"#(,;#.&J&.(N2#,%&)+,(%I")2FI(&+%#"$.%&)+(A&%I(

%I#( <$%$/( %)( "#$<&+F( %I#( <$%$( %)( &<#+%&JC( +)%#A)"%IC( ;$%%#"+ ,( $+<( $;;"$&,&+F( %I#( ;$%%#"+,O(

&*;)"%$+.#(&+( %I#( .)2",#8( P#<$F)F&.$5("#,;)+,#,( &+')5'#( %$B&+F( %I#( J)"*()J($.%&)+,(=AI)5#3

.5$,,(,.$JJ)5<&+F/(%$"F#%#<(,.$JJ)5<&+F/($+<("#'&,&+F(.)2",#(<#,&F+@()"($<);%&+F($(A$&%3$+<3,##(

I)5<&+F( ;$%%#"+/( $+<Q)"( <##;5C( "#J5#.%&+F( )+( ;#<$F)FC8( 6"$A&+F( )+( %I&,( *)<#5/( ,;#.&J&.(

"#.)**#+<$%&)+,( $"#( *$<#( J)"( I)A( 5#$"+&+F( $+$5C%&.,( <#,&F+( .$+( $5&F+( &+J)"*$%&)+(

;"#,#+%$%&)+(A&%I( .)"#( &+,%"2.%&)+$5(;"$.%&.#,( $+<( #*1#<( J#$%2"#,(%)( ,2;;)"%( ;").#,,#,( )J(

2,#(&+()"<#"(%)(1#(*),%(&*;$.%J25()+(%#$.I&+F($+<(5#$"+&+F8(

KeywordsG( -#$"+&+F( $+$5C%&.,( 2,#/( 6$%$3&+J)"*#<( <#.&,&)+3*$B&+F/( H#$.I&+F( $+$5C%&.,/(

-#$"+&+F($+$5C%&.,(<#,&F+/(R<2.$%&)+$5(<$%$('&,2$5&K$%&)+(

1, INTRODUCTION,

SI&5#( %I#( ;").#,,( )J( 2,&+F( $+$5C%&.,( <$%$( %)( &+J)"*( ;#<$F)F&.$5( <#.&,&)+,( &,( $.B+)A5#<F#<( %)( 1#(

.)*;5#T( =U#")<)%)2( #%( $58/( V9WX@/( 5&%%5#( &,( B+)A+( $1)2%( %I#( <#%$&5,( )J( I)A( &%( )..2",( &+( $2%I#+%&.(

%#$.I&+F( .)+%#T%,8( 6$%$3&+J)"*#<( ;#<$F)F&.$5( <#.&,&)+3*$B&+F( ;").#,,( &+')5'#,( *)"#( %I$+( E2,%(

&+,%"2.%)",O( 2;%$B#( )J( 5#$"+&+F( $+$5C%&.,( %))5,Y( "$%I#"/( &%( #+%$&5,( &+,%"2.%)",O( %"$+,5$%&)+( )J( %))53

;")'&<#<( &+J)"*$%&)+( &+%)( 5).$55C3*#$+&+FJ25(B+)A5#<F#( $+<( ,21,#N2#+%5C( 2,#( )J( &%(%)( F2&<#( %I#&"(

;#<$F)F&.$5($.%&)+,(=Z)5#+$$"( [( '$+(!$*;#+/( V9W\@8( RT$*&+&+F( ,2.I(&+J)"*$%&)+(2,#( &,(."&%&.$5( %)(

&*;$.%( #<2.$%&)+$5( ;"$.%&.#( $+<( &+J)"*( %I#( <#,&F+( )J( 5#$"+&+F( $+$5C%&.,( &+( &+%#";"#%$15#( $+<(

$.%&)+$15#('&,2$5&K$%&)+8( HI&,( ;$;#"( J&55,( $( F$;( &+( %I#( &+J)"*$%&)+( $'$&5$15#( %)( #<2.$%&)+$5(

'&,2$5&K$%&)+( <#,&F+#",(%)( *$B#( #'&<#+.#31$,#<( <#,&F+( <#.&,&)+,( 1C( ;"#,#+%&+F( $+( #*;&"&.$55C3

<#"&'#<( *)<#5( )J( &+,%"2.%)"( $+$5C%&.,( 2,#( %)( F2&<#( %I#( <#,&F+( $+<( &*;5#*#+%$%&)+( J)"( 5#$"+&+F(

$+$5C%&.,8((

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

2, A,BRIEF,OVERVIEW,OF,THE,MODEL,DEVELOPMENT,

HI#(*)<#5(A$,(<#'#5);#<(1$,#<()+(#*;&"&.$5(.$,#(,%2<&#,(.)+<2.%#<(A&%I(J&'#(2+&'#",&%C(&+,%"2.%)",(

AI)(2,#<($(5#$"+&+F($+$5C%&.,(<$,I1)$"<(&+(%I#&"(%#$.I&+F(<2"&+F(%I#(.)2",#()J($(,#*#,%#"8(L+3<#;%I(

&+%#"'&#A,( A#"#( F2&<#< ( 1C( %I#( =5&*&%#<@( #T&,%&+F( 5&%#"$%2"#( )+( %I#( %);&.( =Z)5#+$$"( [( '$+( !$*;#+/(

V9W\Y('$+(-##2A#+/('$+(S#"*#,B#"B#+/(R"B#+,/([(]2**#5/(V9WXY(^#"1#"%(#%($58/(V9W7@($+<(&+')5'#<(

&+,%"2.%)",O( %I&+B3$5)2<( A$5B3%I")2FI( )J( %I #&"( $+$5C%&.,( 2,#/(,I)A&+F( %I#( "#5#'$+%( '&,2$5&K$%&)+,( %)(

.)+."#%&K#(%I#&"("#,;)+,#,8(HI#(<$,I1)$"<(2,#<(1C(%I#,#(&+,%"2.%)",( A$,( <#'#5);#<($+<( ")55#<()2%(

1C(%I#(2+&'#",&%CO,(L+,%"2.%&)+$5(H#.I+)5)FC(%#$*(1$,#<()+(.)+,25%$%&)+,(A&%I(#$.I(&+,%"2.%)"($1)2%(

%I#( B&+<,( )J( ,%2<#+%( $.%&'&%C( $+<( ;#"J)"*$+.#( &+J)"*$%&)+( %I#C(A)25<( 5&B#( %)( ,##( J)"( )+#( )J( %I#&"(

.)2",#,8( HI#( ;#",)+$5&K#<( <$,I1)$"<( &+')5'#<(%I"##( %)( J)2"(<&,%&+.%( '&#A,( =#8F8( ,%2<#+%( $..#,,( )J(

.)2",#( ,&%#($+<( "#,)2".#,/( '&<#)( '&#A#",I&;/( )+5&+#( N2&K( "#,25%,( )"( ,2"'#C( "#,;)+,#,@/( #$.I( AI&.I(

$.%#<( $,( $+( &+<#;#+<#+%( )'#"'&#A( &+%)( %I#(<$%$( =,##( _&F2"#( W( $+<( V@8(0+( &+<2.%&'#( N2$5&%$%&'#(

$+$5C,&,(A$,( .)+<2.%#<( )+( %I#(&+%#"'&#A(%"$+,."&;%,($+<(,2"J$.#<(%A#+%C( #*#"F#+%( %I#*#,( "#5$%#<(

%)(&+,%"2.%)",O(;"$.%&.#,()J($+$5C%&.,(2,#8(HI#(%I#*#,(A#"#(%I#+(,C+%I#,&K#<(&+%)($(,&%2$%#<(*)<#5()J(

&+,%"2.%)"( $+$5C%&.,( 2,#( %I$%( &,( ;"#,#+%#<( &+( %I#( +#T%( ,#.%&)+8( HI#( J255( ,%2<C/( &+.52<&+F( <#%$&5,( $+<(

#'&<#+.#( J)"( #$.I( %I#*#( $+<( %I#( *)<#5( <#'#5);*#+%( ;").#,,/( &,( "#;)"%#<( &+( S&,#( $+<( `2+F( =&+(

"#'&#A@8(

(( (

Figure&1.&Example&Dashboard&View&of&Resource&Access&&&&&&Figure&2.&Example&Dashboard&View&of&Quiz&Results&

3, A,SITUATED,MODEL,OF,INSTRUCTOR,ANALYTICS,USE,

HI#( *)<#5( .)+,&,%,( )J( *25%&;5#( ;I$,#,( )J( $.%&'&%C( #*1#<<#<( &+( $( %A)3;$"%( ,%"2.%2"#( A&%I( ,#+,#3

*$B&+F($+<(;#<$F)F&.$5("#,;)+,#($,(<&,%&+.%($,;#.%,()J(;"$.%&.#,(=,##(_&F2"#(7@8(M2.I(,%"2.%2"#($5&F+,(

A&%I(%I#(*$E)"&%C()J(;"&)"(,%2<&#,(AI&.I(I$'#(<#,."&1#<(&+,%"2.%)"($+$5C%&.,(2,#($,(J&",%(<#%#"*&+&+F(

$+( 2+<#",%$+<&+F( )J( AI$%( %I#( &+J)"*$%&)+( %#55,( $1)2%( %I#( .)2",#( $+<( %I#+( .)+,&<#"&+F( ;)%#+%&$5(

$.%&)+,(&+("#,;)+,#(%)(&%(=U#")<)%)2(#%($58/(V9WXY('$+(-##2A#+(#%($58/(V9Wa@8((

-))B&+F( &+,&<#( #$.I( ;$"%( )J( %I#( ;").#,,/( ,#+,#3*$B&+F( 1#F&+,( A&%I( $+( &+,%"2.%)"O,(F#+#"$5( $"#$( )J(

.2"&),&%C(=#8F8(.5$,,35#'#5()"(&+<&'&<2$535#'#5(#+F$F#*#+%/(2,#J25+#,,()J(.)2",#(*$%#"&$5,@8(SI&5#(;"&)"(

,%2<&#,(,2FF#,%(%I$%( &+,%"2.%)",( .$+(.)*#( %)( $+$5C%&.,(A&%I(J255C3J)"*#<( N2#,%&)+,(=6C.BI)JJ(#%($58/(

V9WV@(#&%I#"( 1$,#<( )+(;"&)"($+$5C%&.,( 2,#( )"( %I#&"()A+( *#%I)<,()J( <$%$(.)55#.%&)+( $+<($+$5C,&,, )"(

%I$%( %I#C( *$C( E2,%( "#,;)+<( %)( %I#(<$%$( $,(;"#,#+%#<( =U#")<)%)2(#%( $58/( V9WX@/( %I&,(*)<#5( )JJ#",( $(

%I&"<(;),,&15#( ;$%IG( %I$%( &+,%"2.%)",( *$C( ,%$"%( %I#&"( $+$5C%&.,( 2,#( A&%I( $( F#+#"$5( $"#$( )J( .2"&),&%C8(

0"#$,()J(.2"&),&%C(.$+(<#'#5);(&+%)(*)"#(,;#.&J&.(N2#,%&)+,(%I")2FI(&+%#"$.%&)+(A&%I(%I#(<$%$/(AI&.I(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

.$+(%I#+( 1#($+,A#"#<(A&%I(*)"#( .$"#J25(#T$*&+$%&)+(=#8F8(&<#+%&JC&+F(;)%#+%&$5("#5$%&)+,I&;,(&+(%I#(

<$%$(%I$%(&+,%"2.%)",(I);#(%)(#T;5)"#(J2"%I#"@(=Z)5#+$$"([('$+(!$*;#+/(V9W\@8((

Interpret Data

Sense-Making Pedagogical Response

Get Oriented through

Focused Attention

Find Absolute & Relative

Reference Points

Read Data

Trian gul ate

Contextualize

Make Attribution

Explain Pattern

AFFECTIVE PROCESSES

Area of

Curiosity

Question

Generation Wait-and-See

Reflect on

Pedagogy

Check

Impact

Tak e A cti on

Whole-Class Scaffolding

Targeted Scaffolding

Revise Course Design

(

Figure&3.&A&Situated&Model&of&Instructor&Analytics&Use&

6$%$(&+%#";"#%$%&)+()..2",(%I")2FI(%A)(<&,%&+.%($+<(#N2$55C(&*;)"%$+%(;I$,#,()J($.%&'&%CG("#$<&+F(%I#(

<$%$( %)( &<#+%&JC( *#$+&+FJ25( ;$%%#"+,( $+<( %I#+(F#+#"$%&+F( #T;5$+$%&)+,( %I$%( $<<"#,,( %I#( ;$%%#"+,O(

&*;)"%$+.#( J)"( %I#(.)2",#8( L+( "#$<&+F( %I#( <$%$/( &+,%"2.%)",( F#%( )"&#+%#<( %)( '&,2$5&K$%&)+,( %I")2FI(

;$C&+F($%%#+%&)+(%)($(,;#.&J&.(;&#.#()J(&+J)"*$%&)+()"($(+)%&.#$15#(;$%%#"+($,(&+&%&$5($+.I)",/($+<(%I#+(

#T;$+<&+F(%I#&"('&#A()2%A$"<,(%)( #T;5)"#( %I#(<&JJ#"#+%(B&+<,( )J( &+J)"*$%&)+()JJ#"#<(1C(%I#()'#"$55(

$+$5C%&.,8( HI&,( <&JJ#",( J")*( ;"#'&)2,( ,%2<&#,(AI&. I( ,2FF#,%#<( $( ,#N2#+.#( )J( J&",%( F#%%&+F( )"&#+%#< /(

%I#+( $;;5C&+F( J).2,#<( $%%#+%&)+( %)( ,;# .&J&.( <$%$( =^$+( -##2A#+( #%( $58/( V9WX@8( M2.I( <&JJ#"#+.#( *$C(

<#;#+<( )+( I)A( %I#( $+$5C%&.,( '&#A( &,( )"F$+&K#<( $,( .#"%$&+( B&+<,( )J( 5$C)2%( <#,&F+( $+<( &+J)"*$%&)+(

$""$+F#*#+%(.$+( $55)A( &+,%"2.%)",( %)( F#%( )"&#+%#<( J&",%Y( %I2,( 1)%I( $.%&'&%C( ,#N2#+.#,( ,I)25<( 1#(

.)+,&<#"#<($,( ;),,&1&5&%&#,8( L+("#$<&+F( %I#( <$%$/(%I#"#( &,($(+##<( J)"("#J#"#+.#( ;)&+%,()J(.)*;$"&,)+(

AI&.I(&+,%"2.%)",(.$+(J##5(.)+J2,#<($1)2%(AI$%(%)(2,#($,Y(#&%I#"($1,)52%#(=#8F8(<)(,%2<#+%,( #+F$F#(

A&%I( $%( 5#$,%( Xab( )J( %I#( ;")'&<#<( *$%#"&$5,@( )"( "#5$%&'#( =#8F8( <)#,( ,%2<#+%( #+F$F#*#+%( .I$+F#(

<2"&+F(%I#(.)2",#/(I)A( <)#,( %I&,( C#$"O,( ;$.&+F(%"$E#.%)"C(.)*;$"#(%)( 5$,%( C#$"O,@8( !)+,&<#"$%&)+()J(

I)A( %)( 1#,%( ;")'&<#( #T;5&.&%( "#J#"#+.#(; )&+%,(%)( $&<( 2,#",( &+( +$'&F$%&+F( %I")2FI( <$%$( &,( $( F")A&+F(

$"#$( )J( $%%#+%&)+( &+( %I#( 5&%#"$%2"#( =J)"( #T$*;5#( ,##(P$%#5/( MI$"*$/( M#55*$+/( [( -)*$,/( V9W\@8( L+(

#T;5$&+&+F( ;$%%#"+,/( &+,%"2.%)",( #T%#+<( %I#( *#$+&+F( )J( ;$%%#"+,( &<#+%&J&#<( 1C( #T;5$&+&+F( =)"(

N2#,%&)+&+F@( %I#&"( &*;5&.$%&)+,( $,( "#5$%#<( %)( %I#&"( .)2",#8( L+,%"2.%)",( )J%#+( %"C( %)( %"&$+F25$%#( %I#(

;$%%#"+,(A&%I($<<&%&)+$5( &+J)"*$%&)+( =#8F8( .5$,,( )1,#"'$%&)+@(%)( .)+J&"*( %I#&"(&+%#";"#%$%&)+8(SI#+(

%I&,(,2;;)"%,( %I#(&+%#";"#%$%&)+,/(&+,%"2.%)",(*$C(2,#(%I#&"(.)+%#T%2$5(B+)A5#<F#()J(%I#(.)2",#($+<(

,%2<#+%,( %)( #T;5$&+( AI$%( %I#( "#,25%,( *&FI%( *#$+( $+<( *$B#( $%%"&12%&)+,( %)( ;)%#+%&$5( .$2,#,(

=Z)5#+$$"([('$+(!$*;#+/(V9W\@8(SI#+(%I#(#T%#"+$5(&+J)"*$%&)+( $+<($+$5C%&.,(<$%$(<)(+)%($5&F+/(&%(

.$+( 5#$<( &+,%"2.%)",( %)( N2#,%&)+(%I#( $+$5C%&.,( =6$K)/( M%#;$+#B/( !I$2I$+/( [( 6)"+/( V9WX@( $+<Q)"(

I#,&%$%#(%)(%$B#($.%&)+(=U#")<)%)2(#%($58/(V9WX@8(L+($<<&%&)+(%)(.)F+&%&'#(;").#,,&+F()J(;$%%#"+,/(<$%$(

&+%#";"#%$%&)+(.$+(;")')B#($JJ#.%&'#("#,;)+,#,( ,2.I($,( ,2";"&,#/( <&,$;;)&+%*#+%/()"(E)C($,("#;)"%#<(

&+(%I#(5&%#"$%2"#(=S&,#/(cI$)/([(U$2,B+#.I%/(V9Wd@8(

_)55)A&+F( ,#+,#3*$B&+F/( ;#<$F)F&.$5("#,;)+,#( &,( &+,%"2.%)",O( <#.&,&)+,( $+<Q)"( .I$+F#,(&+( %I&+B&+F(

1$,#<( )+( %I#( $+$5C%&.,8( HI#( *),%( .)**)+( "#,;)+,#( %C;#(&,(%$B&+F( $.%&)+( %)A$"<,( =W@( %I#( AI)5#3

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

.5$,,/(=V@( ;$"%&.25$"( ,%2<#+%,/()"(=7@(.)2",#( *$%#"&$5,( =#8F8( U#")<)%)2( #%($58/(V9WXY(Z)5#+$$"( [( '$+(

!$*;#+/( V9W\Y( eI$B$E/( 05#'#+/( [( Z.-$"#+/( V9WX@8( !I#.B&+F( AI#%I#"( %I#( $.%&)+,( %$B#+( I$'#(

$.I&#'#<(%I#(&+%#+<#<(&*;$.%(&,($(J&+$5(;I$,#(%)(.5),#(%I#(5));Y(I)A#'#"(%I&,(<)#,(+)%($5A$C,()..2"(

=6$K)(#%($58/(V9WX@8(0+)%I#"(.)**)+("#,;)+,#(%)($+$5C%&.,(&,(%)($<);%($(I)5<&+F(;$%%#"+()J(A$&%&+F(%)(

,##(AI$%(A&55(I$;;#+($,( *)"#( <$%$(&,(*$<#($'$&5$15#( =U#")<)%)2(#%($58/(V9WX@8( 6##;("#J5#.%&)+($+<(

,I&J%,(&+( I)A( $+( &+,%"2.%)"( .)+. #;%2$5&K#,( %I#&"( ;#<$F)FC( &,( $ ( +#A/( &+%#"#,%&+F( "#,;)+,#( %C;#( %I$%(

I$,(+)%("#.#&'#<(*2.I($%%#+%&)+(&+(%I#(5&%#"$%2"#(=.8J8(Z)5#+$$"([('$+(!$*;#+/(V9W\@(12%(*$C(I$'#(

F"#$%#"($+<(5)+F#"35$,%&+F(#JJ#.%,(%I$+(,&*;5#($<E2,%*#+%,(%)(%#$.I&+F8(

L+( $<<&%&)+( %)( %I#( 2+&<&"#.%&)+$5( ;$%I( %I")2FI( %I#( <&JJ#"#+%( ;I$,#,( <#,."&1#<( $1)'#/( &+,%"2.%)"(

$+$5C%&.,(2,#(*$C($5,)()..2"(&%#"$%&'#5C(A&%I&+($+<($."),,(#$.I()J(%I#(%A)(5$"F#"(;$"%,(=#8F8($..#,,(%)(

<$%$( 5#$<,( %)( +#A( N2#,%&)+,Y( $.%&)+,( %$B#+( %)( %#,%( $+( &+&%&$5( &+%#";"#%$%&)+( &+J52#+.#( 1$.B( %)( %I#(

&+%#";"#%$%&)+(&%,#5J@8(

4, IMPLICATIONS,FOR,LEARNING,ANALYTICS,DESIGN,

HI&,(,&%2$%#<( *)<#5( )JJ#",($( .5#$"(,%$"%&+F( ;5$.#( J)"( # JJ)"%,( %)( <#,&F+( 5#$"+&+F( $+$5C%&.,( %)( ,2;;) "%(

&+,%"2.%)",O( ;#<$F)F&.$5( <#.&,&)+3*$B&+F( ;"$.%&.#,/(AI&.I( .$+( F2&<#( <#,&F+#",( &+(%I&+B&+F( $I#$<(%)(

&+,%"2.%)",O( $+$5C%&.,(2,#(<2"&+F(%I#(<#,&F+(;").#,,(=eI$B$E(#%($58/( V9WX@8( L+(*$B&+F(#'&<#+.#31$,#<(

<#,&F+( <#.&,&)+,/( &%( &,( ."&%&.$5( %)( A)"B( <&"#.%5C( A&%I( #<2.$%&)+$5( ,%$B#I)5<#",( %I")2FI)2%( %I#(

<#'#5);*#+%( ;").#,,8( HI&,( ;").#,,/( I)A#'#"/( "#N2&"#,( *)"#( %I$+( E2,%( $,B&+F( ,%$B#I)5<#",( AI$%(

&+J)"*$%&)+( %I#C( A)25<( 5&B#( %)( 5))B( $%( $+<( 2,#/( ,&+.#( &%( &,( $( <&JJ&.25%( N2#,%&)+( %)( $+,A#"( &+( %I#(

$1,#+.#( )J( ;"&)"( #T;#"&#+.#( A)"B&+F( A&%I( ,2.I( <$%$8( Z)"#( <#%$&5,( ,I)25<( 1#( .)+,&<#"#<( &+( %I&,(

;").#,,/(&+.52<&+F(I)A(F$&+&+F($..#,,(%)(%I#(<$%$(.)+%"&12%#,(%)(,I&J%,(&+(&+,%"2.%)",O(2+<#",%$+<&+F(

)"( +#A( N2#,%&)+( F#+#"$%#<( %)( %I#( <$%$8( HI&,(I&FI5&FI%,( %I#( +##<( J)"( #'&<#+.#31$,#<( <#,&F+( %)( 1#(

$%%#+%&'#( %)( $.%&)+$1&5&%C( AI#"#( f$+$5C%&.,( .)++#.%( A&%I( #<2.$%&)+( $+<( %I#( .I$+F#,( %I$%(

$<*&+&,%"$%)",/( %#$.I#",( $+<( ,%2<#+%,( A$+%( %I#( %))5,( %)( *$B#( &+( )"<#"( %)( ,2;;)"%( %I#&"( #'#"C<$C(

5#$"+&+F/(%#$.I&+F( $+<($,,#,,*#+%( A)"Bg(=_#"F2,)+( #%( $58/(V9Wh/(;8( i@8( HI#( .)"#($"#$,( J)"( $%%#+%&)+(

$"#(;"#,#+%#<(&+(%I#(J)55)A&+F(,#%,()J(<#,&F+("#.)**#+<$%&)+,(1$,#<()+(%I#(,&%2$%#<(*)<#58((

4.1,Learning,analytics,design,should,align,information,structures,with,instructors’,

pedagogical,concerns.,

?*.%&'@,( '&5"*#%1'"&( 5*"#( 12,( $,*/$,+1'A,( "5( '&/1*B+1"*/C( &"1(-%1%( /1*B+1B*,/D( L+( $+$5C%&.,(

<#'#5);*#+%/(&%(&,(#$,C(=$+<()J%#+(+#.#,,$"C@(%)(,%$"%(%I&+B&+F($1)2%(%I#(<$%$(&+(%I#(J)"*(&+(AI&.I(&%(

&,(*$<#( $'$&5$15#( =#8F8( )"F$+&K#<( $5;I$1#%&.$55C/( 1C( %C;#( )J( &+%#"$.%&)+/( )"( 1C( ,C,%#*3%&*#@8( >2%(

&+,%"2.%)",( )J%#+( %I&+B( &+( < &JJ#"#+%( .$%#F)"&#,G( A##B,( )"( 2+&%,( )J( $( .5$,,/( ,#%,( )J( $,,).&$%#<( .)2",#(

$.%&'&%&#,8( HI&,( <&,.)++#.%&)+( ."#$%#,( $( ."&%&.$5( 1$""&#"( J)"( #JJ#.%&'#( $+$5C%&.,( 2,#/( 12%( .$+( 1#(

$<<"#,,#<(1C( #T;5&.&%5C( #5&.&%&+F(&+,%"2.%)"(.)+.#;%2$5&K$%&)+,()J( I)A( %I#C(%I&+B( $1)2%(%I#&"( .)2",#(

$+<( %I#( <&JJ#"#+%( #5#*#+%,( %I$%(.)*;),#( &%( =AI&.I( &,( $( N2&%#( <&JJ#"#+%( B&+<( )J( N2#,%&)+( %)( $,B(

&+,%"2.%)",(%I$+(AI$%(B&+<,()J(&+J)"*$%&)+(%I#C(A)25<(5&B#(%)(B+)A@8(0%%#+%&)+(%)(%I&,(&,,2#(*$C($5,)(

"$&,#(%I#(+##<(J)"(="#@.)+,&<#"&+F(5#$"+&+F(<#,&F+(1#J)"#($+$5C%&.,($"#(12&5%(,)(%I$%(%I#(%A)(.$+(1#(&+(

$5&F+*#+%(=-).BC#"(#%($58/(V9W7@8(

74'.&(12,(1'#'&.( "5( /8/1,#( %&-( '&/1*B+1"*/E( $*%+1'+,/D(M&*&5$"( %)( %I#( ;"&)"( .)+,&<#"$%&)+( )+( %I#(

)"F$+&K$%&)+( )J( &+J)"*$%&)+/(%I#( %&*&+F( )J( $..#,,( %)( &+J)"*$%&)+( +##<,( %)( 1#( .)+,&<#"#<( J")*( %I#(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

;#",;#.%&'#( )J(%I#( &+,%"2.%)"8( HI#( <#J#""#<( 2;<$%#( )J( <$%$( "#J"#,I( .$+( 5&* &%( %I#( 2,#J25+#,,( )J( %I#(

$+$5C%&.,(J)"(&+,%"2.%)",(AI)(A$+%(%)($..#,,(%I#(<$,I1)$"<(&**#<&$%#5C(;"&)"(%)($(.5$,,($,("#;)"%#<(

&+( S&,#( $+<( `2+F( =&+( "#'&#A@8( SI&5#( .)+,%$+%( <$%$( 2;<$%&+F( J)"( %I#( #+%&"#( ,C,%#*( *$C( +)%( 1#(

"#$5&,%&./($55)A&+F(&+,%"2.%)",(%)(2;<$%#(%I#&"(<$%$()+(<#*$+<(&+(,&%2$%&)+,()J(+##<(.)25<(1#()+#(A$C(

%)($<<"#,,(%I#(,&%2$%&)+8(

4.2,Learning,analytics,should,be,designed,to,support,processes,of,use.,,

F#G,-(/B$$"*1(5"*(HB,/1'"&(.,&,*%1'"&(%&-(#%'&1,&%&+,D(0(B#C(#5#*#+%()J(%I#('$52#(;");),&%&)+(J)"(

&+,%"2.%)"($+$5C%&.,( 2,#(&,(%I$%($+$5C%&.,(<$%$("#,;)+<,(%)( &*;)"%$+%( N2#,%&)+,()+(AI&.I(&+,%"2.%)",(

.$+(%$B#($.%&)+8(0,($+(&%#"$%&'#(;").#,,(A&%I&+($+<($."),,(#$.I()J(%I#(%A)(5$"F#"(;$"%,()J(%I#(*)<#5/(

N2#,%&)+,()J%#+(#*#"F#()"($"#("#J&+#<(%I")2FI(J2"%I#"(#T$*&+$%&)+()J(%I#(<$%$8(L*;)"%$+%5C/($*&<,%(

I#.%&.(%#$.I&+F( ,.I#<25#,( ,2.I( N2#,%&)+,(*$C( +)%(1#( J)"*#<( )"("#*#*1#"#<( $."),,($+$5C%&.,(2,#(

,#,,&)+,8(0+$5C%&.,(%))5,(.$+(1#(<#,&F+#<(%)(,2;;)"%(%I&,(;").#,,(1C(&+.52<&+F(J#$%2"#,(AI&.I(,2;;)"%(

%I#(F#+#"$%&)+($+<(*$&+%#+$+.#()J(N2#,%&)+,(=$+<(;#"I$;,($+,A#",@8(_)"(#T$*;5#/($(N2#,%&)+($"#$(

$,,).&$%#<(A&%I($('&,2$5&K$%&)+(.)25<()JJ#"($(,#%()J(#<&%$15#/(%$&5)"$15#(N2#,%&)+,(A&%I($<</(<#5#%#($+<(

#<&%(J2+.%&)+,(%I$%(;")'&<#(J5#T&1&5&%C()J(2,#8(j2#,%&)+,(.$+(1#(*$&+%$&+#<($."),,(,#,,&)+,()J(2,#($+<(

$++)%$%#<(A&%I($+,A#",(&+,%"2.%)",(J&+<(&+(%I#(<$%$()"(%$F,(J)"(J2%2"#(J)55)A32;8(

3&+"*$"*%1,(A'/B%4(%'-/(1"(5'&-(,&1*8($"'&1/(1"(12,(%&%481'+/D(0+)%I#"(&*;)"%$+%(.)+,&<#"$%&)+(&,(I)A(

%)(J$.&5&%$%#(&+,%"2.%)",(%)(J&+<(#+%"C(;)&+%,(A&%I(AI&.I(%I#C(.$+(F#%()"&#+%#<(%)(%I#($+$5C%&.,8(]$%I#"(

%I$+(1#F&++&+F( A&%I( $+()'#"'&#A($+<( %I#+( <&FF&+F(&+($(;$"%&.25$"( ;$"%/(&+,%"2.%)",(*$C( )J%#+( 1#F&+(

A&%I(,)*#(;$"%()J(%I#($+$5C%&.,(%I$%(%I#C(.$+(*$B#(,#+,#()J($+<(#T;$+<(&%()2%A$"<,(=S&,#([(`2+F/(&+(

"#'&#A@8(^&,2$5($&<,(.$+(1#(&+.)";)"$%#<(&+%)($+$5C%&.,(%)(,2;;)"%(%I&,(;").#,,Y(J)"(#T$*;5#/(&+($(5$"F#(

*$%"&T( )J( <$%$( %)FF5#$15#( %))5,( AI&.I( I&FI5&FI%( &+J)"*$%&)+( 1C( ")A,( )"( .)52*+,( .)25<( I#5;( 2,#",(

J).2,( %I#&"( $%%#+%&)+( )+( J&+<&+F( .#"%$&+( B&+<,( )J( ;$%%#"+,( )"( #T%#+<#<( ,#N2#+.#,( )J( '&,2$5(

&+J)"*$%&)+(AI&.I(.$+($55)A(F")2;&+F(%)(5#%(2,#",($%%#+<(%)(&+<&'&<2$5(A##B,()"(N2&KK#,(&+(%2"+8((

I,4$( '&/1*B+1"*/( 1"( 5'&-( %&-( ;"*J( ;'12( *,5,*,&+,( $"'&1/( 5"*( -%1%('&1,*$*,1%1'"&D( 0( J2"%I#"(

.)+,&<#"$%&)+( %)( ,2;;)"%( &+,%"2.%)",(&+( *$B&+F( 2,#( )J( $+$5C%&.,( &,( )JJ#"&+F( ,2;;)"%( J)"( J&+<&+F(

"#J#"#+.#(;)&+%,( A&%I( AI&.I(%)(*$B#( ,#+,#( )J(%I#(<$%$8(P")'&<&+F( $..#,,(%)(,&*&5$"(<$%$(J")*(;"&)"(

%#"*,()"( )'#"$".I&+F(%"#+<,( J")*(,&*&5$"( .)2",#,()"( %))5,(J)"( *$B&+F(.)*;$"&,)+,($."),,( %&*#(.$+(

;")'&<#( I&FI( 5#'#5( "#5$%&'#( "#J#"#+.#( ;)&+%,8( 01,)52%#( "#J#"#+.#( ;)&+%,( *$C( 1#( #T;5&.&%5C( #5&.&%#<(

%I")2FI( $( ;").#,,( )J( F2&<#<( "#J5#.%&)+ ( %I")2FI( AI&.I( &+,%"2.%)",( $"%&.25$%#( %I#&"( #T;#.%$ %&)+,( J)"(

.5$,,($.%&'&%C/(#+F$F#*#+%()"(;#"J)"*$+.#(&+(%#"*,()J(%I#(*#%"&.,($'$&5$15#8((

F#G,-(54%./(5"*(4%1,*(-,+'/'"&/(1"(1%J,(%+1'"&(%&-( +2,+J( '#$%+1D( k+#( )J( %I#( ;#<$F)F&.$5("#,;)+,#,(

%I$%(&+,%"2.%)",(.$+(.)**)+5C( *$B#(&,(A$&%3$+<3,##($;;")$.I(%I$%(<#5$C,($.%&)+(%&55(J2"%I#"(<$%$(&,(

$'$&5$15#8(RJJ#.%&'#+#,,()J( %I&,( ,%"$%#FC( "#N2&"#,(%I#(&+,%"2.%)"( %)( "#*#*1#"(%I#( ,&%2$%&)+( %I#C( $"#(

A$&%&+F(%)( B+)A(*)"#($1)2%( $+<( "#%2"+( %)( "##T$*&+#(&%( $%( ,)*#(J2%2"#(<$%#8( U)A#'#"/( &%(&,( N2&%#(

;),,&15#( %I$%( %I&,( +#'#"( )..2",8( ]$%I#"( %I$+( "#5C&+F( )+( &+,%"2.%)"( *#*)"C/( $+$5C%&.,( <#,&F+( .$+(

,2;;)"%(%I&,( ;").#,,( 1C( )JJ#"&+F(J#$%2"#,(%I$%( 5#%( &+,%"2.%)",(*$"B( $+<Q)"($++)%$%#( $( ;$%%#"+(%I#C(

)1,#"'#(&+( %I#( <$%$( J)"(J2%2"#(J)55)A32;8( M&*&5$"5C/( AI#+( $.%&)+(&,(%$B#+/( $+$5C%&.,( J#$%2"#,( .$+( 1#(

2,#<(%)(."#$%#(#T%#"+$5&K#<("#*&+<#",(%)(.I#.B(%I#(&*;$.%()J(%I#($.%&)+8(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

4.3,Learning,analytics,should,be,designed,to,support,sharing,and,conversations.,

I,4$('&/1*B+1"*/(1"(/2%*,(12,(%&%481'+/(G8("55,*'&.(-,K'-,&1'5',-(A',;/D(SI#+(&+,%"2.%)",(*$B#(,#+,#(

)J( %I#( $+$5C%&.,( )"( %$B#( $.%&)+,( 1$,#<( )+( &%/( &+,%"2.%)",( *$C( +##<( )"( A$+%( %)( ,I$"#( $+$5C%&.,( A&%I(

)%I#"(&+,%"2.%)",(%)(#+F$F#(&+($(;").#,,()J(.)55$1)"$%&'#(&+%#";"#%$%&)+()"(A&%I(,%2<#+%,($,($+()1E#.%(

J)"(<&,.2,,&)+($+<("#J5#.%&)+(&+(%I#(.5$,,8(U)A#'#"/(%I&,(;"$.%&.#(*$C("$&,#(;)%#+%&$5(;"&'$.C(.)+.#"+,(

"#F$"<&+F(%I#(2,#()J($+$5C%&.,3$,3#'&<#+.#8(k+#(A$C(%)(J$.&5&%$%#(%I#(2,#()J($+$5C%&.,($,($(*#<&$%&)+$5(

)1E#.%(=S&,#/(V9Wd@(&,(%)(*$B#(&%(#$,C(J)"(&+,%"2.%)",(%)( ,A&%.I( %)($('&#A(&+(AI&.I(,%2<#+%(&<#+%&%&#,(

I$'#(1##+("#*)'#<()"(I&<<#+8(

5, IMPLICATIONS,FOR,LEARNING,ANALYTICS,IMPLEMENTATION,

L+($<<&%&)+( %)( "#%I&+B&+F(5#$"+&+F( $+$5C%&.,(<#,&F+( &+( %I#( .)+%#T%()J(&+,%"2.%&)+$5( ;"$.%&.#,/(&%(&,($5,)(

&*;)"%$+%(%)(.)+,&<#"(A$C,(&+(AI&.I($+$5C%&.,(2,#(.$+(1#(J$.&5&%$%#<(%I")2FI(;#<$F)F&.$5(,2;;)"%(J)"(

%I#( ;").#,,( )J( 2,#( &%,#5J8( HI&,( &,( ."&%&.$5( %)( J$.&5&%$%#( &+,%"2.%)",O( %"$+,5$%&)+( )J( &+J)"*$%&)+( &+%)(

$.%&)+$15#(&+,&FI%,(AI&.I(.$+(J##<(1$.B(%)(%I#(#'&<#+.#31$,#<(<#,&F+(;").#,,#,8(k+#(;)%#+%&$5(A$C(

&,(%)(.)+,&<#"(A$C,(%)(#<2.$%#(&+,%"2.%)",(&+(I)A(%)(A)"B(A&%I(<$%$(%)(&+J)"*(%I#&"(%#$.I&+F8(HI&,(.$+(

1#( <)+#( 2;J")+%( %I")2FI( ,%"2.%2"#<( &+,%"2.%)"( <$%$( %"$&+&+F( )"( &+3,&%2( A&%I( %I#(&+%")<2.%&)+( )J( $(

;#<$F)F&.$5($+$5C%&.,(.)$.I(AI)(."#$%#,($(,#"&#,()J(,.$JJ)5<,(%)(,2;;)"%(&+,%"2.%)",(&+(%I&,(%"$+,5$%&)+(

;").#,,8( 0+$5C%&.,( .)$.I( ,2;;)"%,( .$+( %$B#( %I#( J)"*( )J( ;#"&)<&.( #*$&5,/( )+#3)+3)+#( .)$.I&+F(

,#,,&)+,/( <#;$"%*#+%31$,#<( A)"B,I);,( $+<( %I#( .25%&'$%&)+( )J( 5).$5( &+,%"2.%)"( .)**2+&%&#,8(_)"(

#T$*;5#/(#*$&5(*#,,$F#,(.$+(I&FI5&FI%(;$"%&.25$"(;#<$F)F&.$5(N2#,%&)+,(=#8F8(I)A(.$+(L(J&+<($+<(I#5;(

,%2<#+%,( AI)( ,##*( 2+#+F$F#<( &+( %I#( J&",%( J#A( A##B,l@/( #T;5$&+&+F( I)A( %) ( 2,#( %I#(<$,I1)$"<(%)(

$+,A#"( &%( =#8F8( );#+( %I#( ,%2<#+%3.)2",#( &+%#"$.%&)+( F"&<( $+<( 5))B( $."),,( %I#( ")A,( J)"( .)+,&,%#+ %5C(

5&FI%(.)5)"#<(.#55,@/($+<(%I#+(<&,.2,,&+F($.%&)+,(%I$%(.)25<(1#(%$B#+(&+("#,;)+,#(=#8F8(,;#$B(A&%I(%I#*(

&+<&'&<2$55C(%)(J&+<()2%(AI$%(&,(F)&+F()+($+<(*$B#(%I#*($A$"#(%I$%(C)2($"#(&+'#,%#<(&+(%I#&"(,2..#,,/(

I&FI5&FI%( J)"( %I#( AI)5#( .5$,,( I$1&%,()J( ,2..#,,J25( ,%2<#+%,@8(L+( %I&,( A$C/( ;#<$F)F&.$55C( *#$+&+FJ25(

N2#,%&)+,/($+,A#",/($+<($.%&)+,($"#(5&+B#<(%)F#%I#"(%)(J"$*#($+$5C%&.,(2,#/("$%I#"(%I$+(,%$"%&+F(A&%I(

$(<$%$3.#+%"&.()"&#+%$%&)+8(HI#(,$*#(&,,2#,(.$+(1#(#+F$F#<(A&%I()+($(1")$<#"(,.$5#(%I")2FI()+#3)+3

)+#(.)$.I&+F()"(A)"B,I);,(&+(AI&.I(&+,%"2.%)",($"#(,2;;)"%#<(&+(A)"B&+F(%I")2FI(%I#,#(,#N2#+.#,(

2,&+F(<$%$(J")*( %I#&"( )A+( .5$,,#,8(!)55$1)"$%&'#( &+%#";"#%$%&)+( A&%I( $( ,$*;5#( <$%$( ,#%(.$+( 1#( $5,)(

&*;5#*#+%#<( %)( <&,.2,,( .)**)+( .I$55#+F#,( &+( $+$5C%&.,( 2,#( $+<( <&,$*1&F2$%#( %I#( *#$+&+F( )J(

$+$5C%&.,(%I")2FI(<&$5)F2#(A&%I()%I#"(;$"%&.&;$+%,8(L+(%I#(5)+F(%#"*/(5).$5(&+,%"2.%)"(.)**2+&%&#,()J(

;"$.%&.#( $")2+<( $+$5C%&.,( 2,#( .$+( 1#( .25%&'$%#<( AI#"#( ,2.I( .)+%#T%2$5&K#</(#*1#<<#</( )+F)&+F(

,2;;)"%( +#%A)"B,( $"#( *)"#( #JJ#.%&'#( %I$+( ,I)"%3%#"*( &+J)"*$%&)+( <#5&'#"C( =6$"5&+F3U$**)+<( [(

]&.I$"<,)+/( V99i@8( P2%( %)F#%I#"/( %I#( ,#%,( )J( "#.)**#+<$%&)+,( I&FI5&FI%( %I#( &*;)"%$+.#( )J(

#,%$15&,I&+F( );;)"%2+&%&#,( J)"( J2%2"#( #JJ)"%,( &+( 1)%I( 5#$"+&+F( $+$5C%&.,( <#,&F+( $+<( &*;5#*#+%$%&)+(

1$,#<()+(#*;&"&.$5(J&+<&+F,(J)"(,2;;)"%&+F(&+,%"2.%)",O(,&%2$%#<(2,#()J($+$5C%&.,8((

6, CONCLUSION,

HI&,(;$;#"()JJ#",(&+,&FI%,(%)( &+J)"*(#'&<#+.#31$,#<( 5#$"+&+F( $+$5C%&.,(<#,&F+(<#.&,&)+,(%I")2FI(%I#(

;"#,#+%$%&)+( )J( $+(#*;&"&.$55C3<#"&'#<( *)<#5( )J(&+,%"2.%)"( $+$5C%&.,( 2,#8( 0+( 2+<#",%$+<&+F( )J( %I#(

;"$.%&.#,( &+,%"2.%)",( #+F$F#( &+( AI#+(2,&+F( $+$5C%&.,( &+( %I#&"( %#$.I&+F( .$+( F2&<#( #<2.$%&)+$5(

'&,2$5&K$%&)+( <#,&F+#",( &+( .I)),&+F( $*)+F( %I#( *C"&$<( <$%$( "#;"#,#+%$%&)+,( ;),,&15#( %)( ;")<2.#(

&+%#";"#%$15#( $+<($.%&)+$15#(5#$"+&+F( $+$5C%&.,( ,C,%#*,8(_2%2"#(A)"B( .$+( &+'#,%&F$%#(%I#(&*;$.%( )J(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

%I#( <#.&,&)+,( %$B#+( %)( '$5&<$%#( )"( "#J&+#( %I#( "#.)**#+<$%&)+,(*$<#( $1)'#8( L+($<<&%&)+/( 1")$<#"(

<$%$(.)55#.%&)+()+( &+,%"2.%)"( $+$5C%&.,(2,#( &+.52<&+F(5)F3J&5#("#.)"<,/( #T;#"&#+.#(,$*;5&+F( <$%$/($+<(

.5$,,"))*()1,#"'$%&)+,(.$+( J2"%I#"("#'#$5(I)A($+$5C%&.,(2,#()..2",($+<( J##<,(1$.B(&+%)(&+,%"2.%)",O(

%#$.I&+F( ;"$.%&.#,8( H)F#%I#"( %I#,#( #JJ)"%,( I#5;( ,%"#+F%I#+( %I#( 5&+#,( )J( .)**2+&.$%&)+( 1#%A##+(

,%$B#I)5<#",($+<( <#,&F+#",/( $+<(I#5;(2,(*)"#( $,( $( J&#5<( %)A$"<,(#'&<#+.#31$,#<( 5#$"+&+F( $+$5C%&.,(

<#,&F+($,($(;"$.%&.#(%)(,2;;)"%(%#$.I&+F($+<(5#$"+&+F/(J),%#"&+F(#<2.$%&)+$5(,2..#,,8(

REFERENCES,,

6$"5&+F3U$**)+</( -8/( [( ]&.I$"<,)+/( 48( =V99i@8( ]#,#$".I( "#'&#AQ%#$.I#"( 5#$"+&+FG( SI$%(

*$%%#",8(F-B+%1'"&%4(4,%-,*/2'$C(LL=a@/(dh3a78(

6$K)/( M8( -8/(M%#;$+#B/( 48( ]8/( !I$2I$+/( 08/( [( 6)"+/( >8( =V9WX/( Z$C@8( RT$*&+&+F( &+,%"2.%)"( 2,#( )J(

5#$"+&+F($+$5C%&.,8(L+(P").##<&+F,()J(12,(MN=O(!I3(!"&5,*,&+,(FP1,&-,-(7G/1*%+1/("&( IB#%&(

Q%+1"*/('&(!"#$B1'&.(R8/1,#/(=;;8(Va9d3VaW9@8(4#A(?)"B/(4?G(0!Z8((

6C.BI)JJ/( 08( -8/( c&#5B#/( 68/( >m5%*$++/( Z8/( !I$%%&/( Z8( 08/( [( M.I")#<#"/( :8( =V9WV@8( 6#,&F+( $+<(

&*;5#*#+%$%&)+( )J( $( 5#$"+&+F( $+$5C%&.,( %))5B&%( J)"( %#$.I#",8(S"B*&%4( "5( F-B+%1'"&%4(

T,+2&"4".8(9(R"+',18C(=U=7@/(a\nXh8(

_#"F2,)+/(]8/(>"$,I#"/(08/(!5)A/(68/(!));#"/(08/(U&55$&"#/(o8/(Z&%%5#*#&#"/(`8/(]&#+%&#,/(>8/(:55*$+/(H8/( [(

^2)"&B$"&/( ]8(=V9Wh@8( ]#,#$".I( #'&<#+.#( )+( %I#( 2,#( )J( 5#$"+&+F( $+$5C%&.,G( L*;5&.$%&)+,( J)"(

#<2.$%&)+( ;)5&.C8( L+( ]8( ^2)"&B$"&/( [( `8( !$,%$+)3Z2+)K( =R<,8@/( 7( FB*"$,%&( 5*%#,;"*J( 5"*(

%+1'"&("&(4,%*&'&.(%&%481'+/8(-2T#*1)2"FG(`)&+%(]#,#$".I(!#+%"#(M.&#+.#(J)"(P)5&.C(]#;)"%8((

U#")<)%)2/( !8/( ]&#+%&#,/( >8/( >)"))A$/( 08/( c<"$I$5/( c8/( U5),%$/( Z8/( [( 4$C<#+)'$/( o8( =V9WX/( Z$".I@8(

L*;5#*#+%&+F( ;"#<&.%&'#(5#$"+&+F( $+$5C%&.,( )+( $( 5$"F#( ,.$5#G( %I#( %#$.I#"p,( ;#",;#.%&'#8(

L+()*"+,,-'&./("5(12,(R,A,&12(3&1,*&%1'"&%4(6,%*&'&.(7&%481'+/(9(:&";4,-.,(!"&5,*,&+,(=;;8(

VhX3VXW@8(4#A(?)"B/(4?G(0!Z8((

-).BC#"/( -8/( U#$%I.)%#/( R8/( [( 6$A,)+/( M8( =V9W7@8( L+J)"*&+F( ;#<$F)F&.$5($.%&)+G( 05&F+&+F( 5#$"+&+F(

$+$5C%&.,(A&%I(5#$"+&+F(<#,&F+8(7#,*'+%&(V,2%A'"*%4(R+',&1'/1C(UO=W9@/(Wd7i3Wdai8(

Z)5#+$$"/(L8/([('$+(!$*;#+/(!8(=V9W\@8(U)A(%#$.I#",(*$B#(<$,I1)$"<(&+J)"*$%&)+( $.%&)+$15#8(3FFF(

T*%&/%+1'"&/("&(6,%*&'&.(T,+2&"4".',/8((

P$%#5/(48/(MI$"*$/(08/(M#55*$+/(!8/([(-)*$,/(68(=V9W\/(`2+#@8(!2""&.252*( P$.&+FG(0(4#A(0;;")$.I(%)(

6&,.)'#"(L+,%"2.%&)+$5(P"$.%&.#,(&+(!5$,,"))*,8(L+()*"+,,-'&./("5(3&1,*&%1'"&%4(!"&5,*,&+,("&(

3&1,44'.,&1(TB1"*'&.(R8/1,#/(=;;8(7da37aW@8(!I$*/(MA&%K#"5$+<G(M;"&+F#"8(

'$+(-##2A#+/( 08/( `$+,,#+/(`8/(R"B#+,/( o8/( [(>"#B#5*$+,/( Z8( =V9Wa@8(H#$.I#"("#F25$%&)+( )J( .)F+&%&'#(

$.%&'&%&#,( <2"&+F( ,%2<#+%( .)55$1)"$%&)+G( RJJ#.%,( )J( 5#$"+&+F( $+$5C%&.,8(!"#$B1,*/( 9(

F-B+%1'"&C(0N/(\93id8(

'$+(-##2A#+/(08/('$+(S#"*#,B#"B#+/(Z8/(R"B#+,/(o8/([(]2**#5/(48(=V9WX@8(Z#$,2"&+F(%#$.I#"(,#+,#(

*$B&+F(,%"$%#F&#,()J(5#$"+&+F($+$5C%&.,G($( .$,#( ,%2<C8(6,%*&'&.W(X,/,%*+2( %&-( )*%+1'+,C(Y=W@/(

dV3a\8(

^#"1#"%/( q8/( 62'$5/( R8/( q5#"BT/( `8/( o)'$#"%,/( M8/( [( M$+%),/( `8( -8( =V9W7@8( -#$"+&+F( $+$5C%&.,( <$,I1)$"<(

$;;5&.$%&)+,8(7#,*'+%&(V,2%A'"*%4(R+',&1'/1C(UO=W9@/(Wa993Wa9i8(

S&,#/(08(_8(=V9Wd/( Z$".I@8( 6#,&F+&+F ( ;#<$F)F&.$5( &+%#"'#+%&)+,( %)( ,2;;)"%( ,%2<#+%( 2,#( )J( 5#$"+&+F(

$+$5C%&.,8( L+()*"+,,-'&./( "5( 12,( 5"B*12( '&1,*&%1'"&%4( +"&5,*,&+,( "&( 4,%*&'&.( %&%481'+/( %&-(

J&";4,-.,(=;;8(V973VWW@8(4#A(?)"B/(4?G(0!Z8((

S&,#/(08(_8/([(`2+F/(?8(=&+("#'&#A@8(H#$.I&+F(A&%I($+$5C%&.,G(H)A$"<,($(,&%2$%#<(*)<#5()J(&+,%"2.%&)+$5(

<#.&,&)+3*$B&+F8(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

S&,#/(08(_8/(cI$)/(?8/([( U$2,B+#.I%/(M8(=V9Wd@8(-#$"+&+F($+$5C%&.,(J)"()+5&+#(<&,.2,,&)+,G( R*1#<<#<(

$+<(#T%"$.%#<($;;")$.I#,8(S"B*&%4("5(6,%*&'&.(7&%481'+/C(==V@/(d\3XW8(

eI$B$E/(_8/(05#'#+/( ^8/( [(Z.-$"#+/( >8( Z8( =V9WX/( M#;%#*1#"@8(RJJ#.%,( )J( $(%#$.I#"( <$,I1)$"<( J)"($+(

&+%#55&F#+%( %2%)"&+F( ,C,%#*( )+( %#$.I#"( B+)A5#<F#/( 5#,,)+( ;5$++&+F/( 5#,,)+,( $+<( ,%2<#+%(

5#$"+&+F8( L+( )*"+,,-'&./( "5( 12,( MN=O( 3&1,*&%1'"&%4( !"&5,*,&+,( "&( 7*1'5'+'%4( 3&1,44'.,&+,( '&(

F-B+%1'"&(=;;8(7Wa37Vi@8(!I$*/(MA&%K#"5$+<G(M;"&+F#"8((

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

“What&do&students&know,&how&long&does&it&take&them&to&know?”&at&

a&Glance&for&Teachers&and&Instructional&Designers&

Truong'Sinh+An,+Majd+Edriss,+Agathe+Merceron,+Thuy'Anh+Nguyen+

?%3'B!;,()%$-('C!*D!1<<6(%=!E/(%,/%-!

'$3*,F4-(,B9&,G*3'6**H9/*+0!%=$(--9+&I=GF+&(69/*+0!+%$/%$*,G2%3'B4B*/B-/B36%9=%0!

'B3C&,B9,F3C%,9=%GF+&(69/*+!

++++++++++++++++ABSTRACTJ! K,! 'B(-! <&<%$0! L%! <$%-%,'! 'B%! )(-3&6(M&'(*,-! $%&6(M%=! (,! 'B%! 6%&$,(,F! &,&6C'(/-!

-%$)(/%!D*$! 'B%! .%&$,(,F! #*+<&,(*,!1<<6(/&'(*,9!;-%$!(,'%$&/'(*,-!-'*$%=!2C! 'B%!&<<6(/&'(*,!

D*$+!'B%!%)(=%,/%!D*$!'B%-%!)(-3&6(M&'(*,-9! NB%! =(&F$&+-!D*$!'%&/B%$-! %,&26%! 'B%+! '*! F$&-<!

&'!&!F6&,/%!LB(/B!'*<(/-!'B%($!-'3=%,'-!+&-'%$!*$!,*'!-*!'B&'!'B%C!/&,!<$%<&$%!'B%($!,%O'!/6&--!

&//*$=(,F6C9!NB%! -&+%!=(&F$&+!*DD%$-!&==('(*,&6!*<'(*,-0!6(H%!'B%! '*'&6!,3+2%$! *D! &''%+<'-!

D*$!(,-'$3/'(*,&6! =%-(F,%$-!-*!'B&'!'B%C! /&,!$%D6%/'!*,!'B%! =(DD(/36'C!6%)%6!*D!'B%!%O%$/(-%-9!1,!

&==('(*,&6! )(-3&6(M&'(*,! D*$! (,-'$3/'(*,&6!= %-(F,%$-!-B*L-! 'B%! '(+%! -'3=%,'-! -<%,=! *,! %&/B!

6%&$,(,F!*2I%/'9!NB%!.1!-%$)(/%!(-!2%(,F!$%&6(M%=!&-!&,!.NK4N**69!

+++++++++++++++KeywordsJ!'$&DD(/46(FB'!=(&F$&+-0!O1PK!-'&'%+%,'-0!6%&$,(,F!6*/H%$0!%6&-'(/-%&$/B0!F$&D&,&!

1& INTRODUCTION&

!NB%! .%&$,(,F! #*+<&,(*,! 1<<6(/&'(*,! >.#1A! (-! =%)%6*<%=! (,! 'B%! -+&$'! 6%&$,(,F! <$*I%/'"!'*! D('! 'B%!

,%%=-!*D!D3664'(+%!%+<6*C%%-!LB*!'&H%!<&$'!(,!&,!Q,%$FC!#*,-36'&,'!'$&(,(,F!(,!&!#B&+2%$!*D!#$&D'-9!

.#1!/&,!2%!'B*3FB'!*D!&-!&!6%&$,(,F!+&,&F%+%,'!-C-'%+!>.REA!L('B!'L*!=(-'(,/'()%!D%&'3$%-9!S($-'0!

'B%! =(F('&6! 6%&$,(,F! $%-*3$/%-! &$%! -'*$%=! /%,'$&66C! (,! &! $%<*-('*$C! &,=! /&,! 2%! &//%--%=! L('B*3'!

$%<6(/&'(*,! LB%,! &!/*3$-%! (-!'&3FB'! (,!=(DD%$%,'! (,-'('3'(*,-9! E%/*,=0!('! (,/63=%-!&! $%/*++%,=&'(*,!

-%$)(/%! D*$! 6%&$,%$-! LB(/B! -%6%/'-! &<<$*<$(&'%! /*,'%,'-0! &-! L%66! &-! &!6%&$,(,F!&,&6C'(/-! -%$)(/%! '*!

=(DD%$%,'!-'&H%B*6=%$-0!(,!<&$'(/36&$!'*! '%&/B%$-! &,=!(,-'$3/'(*,&6!=%-(F,%$-9! .#1!(-! (,=%<%,=%,'! *D!

&,C! '*<(/! &,=! &,C! (,-'('3'(*,! &,=0! 'B%$%D*$%0! /&,! 2%! 3-%=! (,! *'B%$! /*,'%O'-!&,=! D*$! *'B%$! /*3$-%-!

>T$&3--!%'!&69!U:"VA9!

1! /*3$-%! (,! .#1! &-! (,! +&,C! .RE! /&,! 2%! =()(=%=! (,'*!-%/'(*,-0! LB(/B! /&,! 2%! =()(=%=! (,'*! 6%&$,(,F!

3,('-9!1! 6%&$,(,F! 3,('! /*,'&(,-! =(DD%$%,'!6%&$,(,F!$%-*3$/%-!&6-*!/&66%=!6%&$,(,F! *2I%/'-!>.WA!-3/B!&-!

'%O'-0!)(=%*-0!&,(+&'(*,-0!P7S!D(6%-0!*'B%$!+%=(&4'C<%-0!&,=!%O%$/(-%-9!NB%-%!6%&$,(,F!*2I%/'-!/&,!2%!

$%3-%=!(,!*'B%$!/*3$-%-9!N*!-3<<*$'!'B%!<%=&F*F(/&6!/*,/%<'!&=*<'%=!(,!.#1!&-!L%66!&-!'*!(+<6%+%,'!

'B%! $%/*++%,=&'(*,! &,=! 6%&$,(,F! &,&6C'(/-! -%$)(/%-0! +%'&=&'&! &$%! &--*/(&'%=! L('B!&,C! 6%&$,(,F!

*2I%/'9!NB%-%!+%'&=&'&!/*,'&(,!&+*,F! *'B%$-!&'!6%&-'!*,%! 6%&$,(,F!*2I%/'()%!&,=! &! 'C<(/&6!6%&$,(,F!

'(+%9!1!6%&$,(,F! 3,('!(-!$%,=%$%=!&-!&,!&//*$=(*,!L('B!&!-<%/(D(/!-%X3%,'(&6!-'$3/'3$%0!-%%! S(F3$%! "9!

NB%!'*<! ('%+! (-! 'B%!6(-'!*D! 'B%! 6%&$,(,F! *2I%/'()%-!*D! &66!6%&$,(,F!*2I%/'-!*D! 'B&'! 3,('9!1! 6%&$,%$!/&,!

$&'%!%&/B! 6%&$,(,F! *2I%/'()%!&,=!-*! $%D6%/'! *,!B*L!+3/B! -YB%!H,*L-!&6$%&=C!*,! 'B&'!'*<(/0!D$*+!"!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

"!B''<-JYY<$*I%H'92%3'B4B*/B-/B36%9=%Y-+&$'6%&$,(,FY!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

>H,*L!,*'B(,FA!'*!Z!>%O<%$'A9!NB(-!('%+!(-!D*66*L%=!2C!'B%!-%X3%,/%!*D!'B%!.W-!*D!'B&'!3,('9!K,!S(F3$%!

"0! 'B(-! 6(-'! (,/63=%-! &! -%'! *D! %O%$/(-%-! -B*L,! (,! *$&,F%9! S*66*L(,F! &66! .W-0! 'B%! ,%O'! ('%+! (,! 'B%!

&//*$=(*,4)(%L!(-! &F&(,!'B%!6(-'!*D! 6%&$,(,F!*2I%/'()%-9!?C!$&'(,F!'B%+0!&! -'3=%,'!/&,!$%D6%/'!*,! B*L!

+3/B!-YB%!H,*L-!&D'%$!6%&$,(,F!'B%!3,('9!NB%!D*66*L%$!('%+!&66*L-!-'3=%,'-!'*!<$*)(=%!D%%=2&/H!*,!

'B%! 'C<(/&6! 6%&$,(,F! '(+%! D*$! 'B&'! 3,('! >D$*+! "0! L&C! '**! 6(''6%! '(+%! '*! Z0! L&C!'**! +3/BA! &,=! F()%!

/*++%,'-9!NB%!6&-'!('%+!(,!'B%!6(-'!*<%,-!&!=(-/3--(*,!'B$%&=!*,!'B&'!3,('9!NB%-%!6&-'!'L*!('%+-!&$%!

+&$H%=!&!#*++3,(/&'(*,4'**6-!(,!S(F3$%!"9!

Figure+1:!A+learning+unit+in+LCA+

NB%!D($-'!&(+!*D!'B%!6%&$,(,F!&,&6C'(/-!>.1A!-%$)(/%!(-!'*!-3<<*$'!'%&/B%$-!&,=!(,-'$3/'(*,&6!=%-(F,%$-9!

73$(,F!'B%! <$*I%/'0! 'B$%%!+%%'(,F-!L('B!'B$%%!'%&/B%$-!>5[8A!*D!'B%!#B&+2%$! *D! #$&D'-! B&)%! '&H%,!

<6&/%!'*!-%,-%!'B%($!,%%=-!&,=!'*!=(-/3--!<$*<*-%=!-*63'(*,-9!NB%!*3'/*+%!-'$%--%=!'B%!(+<*$'&,/%!

*D!&! -(+<6%! &,=!3,&+2(F3*3-! )(-3&6(M&'(*,J! '%&/B%$-! -B*36=! /6%&$6C!3,=%$-'&,=!LB&'! 'B%C! -%%!&,=!

,*'! 2%! *)%$LB%6+%=! L('B! '**! +3/B9! NB%$%D*$%0! L%! B&)%! *<'%=! D*$! L%664H,*L,! =(&F$&+-! 'B&'!

'%&/B%$-!&$%!D&+(6(&$!L('B9!NB%!.1!-%$)(/%!-B*36=!%,&26%!'%&/B%$-!'*!2%!&L&$%!*D!B*L!+&,C!-'3=%,'-!

&$%!+&-'%$(,F0!&$%!(,!'B%!<$*/%--!*D!+&-'%$(,F!*$!=*!,*'!+&-'%$!&'!&66!'B%!'*<(/-!*D!&!6%&$,(,F!3,('0!-*!

'B&'! 'B%C! /&,! <$%<&$%! 'B%($! ,%O'! /6&--! &//*$=(,F! '*! 'B%! 6%&$,(,F! ,%%=-! *D! 'B%($! -'3=%,'-9! NB%! .1!

-%$)(/%! -B*36=! %,&26%! (,-'$3/'(*,&6! =%-(F,%$-! '*! (+<$*)%! 'B%! 6%&$,(,F! *2I%/'-! 'B%C! /$%&'%! (,!

/**<%$&'(*,!L('B!'B%!'%&/B%$-9!S*$!'B%+!'**0!('!(-!(+<*$'&,'!'*!3,=%$-'&,=!LB&'!'B%C!-%%9!\*L%)%$0!

'B%C!+&C! ,%%=!'*! %O<6*$%!+*$%! -'3=%,'-]!(,'%$&/'(*,-! '*!2%! &L&$%!*D! LB%'B%$! 'B%! $%-*3$/%-! 'B%C!

=%)%6*<!B&)%!'B%!$(FB'!6%,F'B!*$!'B%!$(FB'!6%)%6!*D!=(DD(/36'C9!!

K,!'B%!,%O'! -%/'(*,0!L%! =%-/$(2%!'B%!(,'%$&/'(*,-!=&'&!-'*$%=!2C!'B%!-C-'%+9!K,!'B%!D*66*L%$!-%/'(*,0!

'B%! =(&F$&+-! D*$! '%&/B%$-! &,=! (,-'$3/'(*,&6! =%-(F,%$-! &$%! <$%-%,'%=9! NB(-! <&<%$! %,=-! L('B! &!

/*,/63-(*,!&,=!D3'3$%!L*$H-9!!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

2& DATA&AND&TOOLS&

#*+<$%B%,-()%! 3-%$! (,'%$&/'(*,-! &$%! -'*$%=! &-! O1PKU!-'&'%+%,'-! (,! .%&$,(,F! .*/H%$89! QO&+<6%-! *D!

-'*$%=! (,'%$&/'(*,-! (,/63=%! 'B%! *<%,(,F! *D! &! 6%&$,(,F! 3,('0! *<%,(,F! &,=! /6*-(,F! *D! %)%$C! -(,F6%!

6%&$,(,F!*2I%/'0! -%6D4&--%--+%,'! *D!%&/B! 6%&$,(,F! *2I%/'()%0!&''%+<'!(,!-*6)(,F!&,!%O%$/(-%0!-'&$'(,F0!

<&3-(,F!*$!X3(''(,F!&!)(=%*!%'/9!1-!&,!%O&+<6%0!/*,-(=%$!'B%!D*66*L(,F!O1PK!-'&'%+%,'J!

!"#$%&'(#)"!"#*+',-./$0.1*#)"#02345464076%839820$:932725:%:$5;3203%:$<#"=>"(1)"

""#?9(+#)"!"#@<#)"#/&&A)BB$<CD9&EF'?B9,A$A@B?9(+.B$D.G9(9<#"=>"(2)"

""#'+H9%&#)"!"(3)"

""""#@<#)"#/&&A.)BB?8/042E+91&/I/'%/.%/1C9E<9BJK@&9*LMNO;6PQ;ROO0:STU<8T*V0W7/;RQXH$YV;9C5GZV:X[G#>"

""""#<98@D@&@'D#)"!"#&\A9#)"#/&&A)BB$<CD9&EF'?B9,A$A@B$%&@?@&@9.B]19.&@'D#>"

""""""#D$*9#)"!"#<9IM^#)"#S$1A/\.@U$C@.%/9"_(1D<C$F9D#"="="

""=>"

""#(9.1C&#)"!"(4)"

""""#.%'(9#)"!"#.%$C9<#)";E6>"#*@D#)"I0>"#*$,#)"0"=>"

""""#(9.A'D.9#)"#`a#M@9"b$..9(<$*A8.c&&@F1DF.*9DF9"@.&"<@9"Yd%/.&*9DF9"$D"b$..9(<$*A8"<@9"P18&"+9@"

9@D9("+9.&@**&9D"e9*A9($&1("$18D$/*9D"U$DDEa#f#>"

""""#<1($&@'D#)"#ge;Y0Z00h#>"

""""#9,&9D.@'D.#)"!"

""""""#/&&A.)BB.C9/G(i43j+91&/I/'%/.%/1C9i43j<9B,$A@B9,&9D.@'D.B]19.&@'De\A9#)"#%/'@%9Z1C&@AC9#>"

""""""#/&&A.)BB.C9/G(i43j+91&/I/'%/.%/1C9i43j<9B,$A@B9,&9D.@'D.B%'((9%&T9.A'D.9g$&&9(D#)"`"

""""""""#M@9"b$..9(<$*A8.c&&@F1DF.*9DF9"@.&"<@9"Yd%/.&*9DF9"$D"b$..9(<$*A8"<@9"P18&"+9@"9@D9("

+9.&@**&9D"e9*A9($&1("$18D$/*9D"U$DDE#>"

""""""""#M9("b$..9(<$*A8<(1%U"@.&"$+/cDF@F"?'D"<9("(9C$&@?9D"P18&891%/&@FU9@&"1D<"<9("P18&&9*A9($&1(E#"

""""""f"

""""="

""=>"

""#%'D&9,&#)"!"(5)"

""""#AC$&8'(*#)"#*''<C9E/GUI+9(C@DE<9#>"(6)"

""""#.&$&9*9D&#)"!"#@<#)"#<+23;k79I3k6:I4;0$I+%k+I<$$<;3kk+352#"=>"(7)"

""""#%'D&9,&l%&@?@&@9.#)"!"(8)"

""""""#A$(9D&#)"`"

""""""""!"#@<#)"#/&&A.)BB?8/042E+91&/I/'%/.%/1C9E<9BJK@&9*LMNO;6PQ;ROO0:STU<8T*V0W7/;RQXH$YV;9F#>"

""""""""""#<98@D@&@'D#)"!"#&\A9#)"#/&&A)BB$<CD9&EF'?B9,A$A@B$%&@?@&@9.B@D&9($%&@'D#"="="

""""""f>"

""""""#F('1A@DF#)"`"

""""""""!"#@<#)"#/&&A.)BB?8/042E+91&/I/'%/.%/1C9E<9B$A@BC%*.B%'1(.9.B_^S#>"

""""""""""#<98@D@&@'D#)"!"#&\A9#)"#/&&A)BB$<CD9&EF'?B9,A$A@B$%&@?@&@9.B%'1(.9#"="

""""""""=>"!"#@<#)"#/&&A.)BB?8/042E+91&/I/'%/.%/1C9E<9BJK@&9*LMNO;6PQ;ROOGNN#>"

""""""""""#<98@D@&@'D#)"!"#&\A9#)"#/&&A)BB$<CD9&EF'?B9,A$A@B$%&@?@&@9.B*'<1C9#"="

""""""""=>"!"#@<#)"#/&&A.)BB?8/042E+91&/I/'%/.%/1C9E<9BJK@&9*LMNO;6PQ;ROO0:STU<8T*V0W7/;RQXH$YV;9F#>"

""""""""""#<98@D@&@'D#)"!"#&\A9#)"#/&&A)BB$<CD9&EF'?B9,A$A@B$%&@?@&@9.B@D&9($%&@'D#"="="

""""""f"

""""=>"

""""#9,&9D.@'D.#)"!"

""""""#/&&A)BB$<CD9&i43jF'?B9,A$A@B$%&@?@&@9.B%'1(.9#)"`"#_^SI0I0km_^S#"f"(9)"

""""="

""=>"

""#&@*9.&$*A#)"#7;0kI;2I2;e02)2;)06E0676;;n;;);;#>"(10)"

""#@<#)"#;0$k569;I<kk8I4735I53;+I7+27552<3%k9#"(11)"="

NB%!O1PK!-'&'%+%,'!L('B!'B%!F()%,!'-!(11)!&2*)%!/*,'&(,-!'B%!(,D*$+&'(*,!'B&'!&!/$,+'5'+(%+1"*!(1)!=(=!

%&/;,*!(2)!&!/$,+'5'+(?@,/1'"&!(3)!L('B!'B%!-B*L,!*,/@41!(4)!*,!&!-<%/(D(/!1'#,/1%#$!(10)9!NB%!/+%4,-(

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

U!B''<-JYYF('B329/*+Y&=6,%'YO1PK4E<%/!

8!!B''<-JYYF('B329/*+Y.%&$,(,F.*/H%$Y6%&$,(,F6*/H%$!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

/+"*,!*D! :9Z! (,=(/&'%-! 'B&'! 'B%! F()%,!-*63'(*,! (-! <&$'(&66C! /*$$%/'_! 'B%! X3%-'(*,! L&-! =(-<6&C%=! D*$! &!

-@*%1'"&!*D! "! +(,3'%! &,=! ""! -%/*,=-9! S*$! D3$'B%$! &,&6C-(-0! 'B%! F()%,! *,/$"&/,!&,=! 'B%! +"**,+1(

*,/$"&/,( $%11,*&!&$%! &6-*! -'*$%=9! S*$! 'B%! <3$<*-%! *D! 2%''%$! 3,=%$-'&,=(,F! 'B%! =&'&0! D3$'B%$!

(,D*$+&'(*,!(-!23,=6%=!(,!'B%!+"&1,A1!(5)9!NB%!-'&'%+%,'!$%D%$%,/%!(7)!6(,H-!'*!'B%!<$(*$!-'*$%=!O1PK!

-'&'%+%,'!*,!&!B(FB%$!6%)%6!LB(/B!&66*L-!'*!23(6=!&!F$&<B!*D!'B%!6%&$,(,F!2%B&)(*$_!O1PK!-'&'%+%,'-!

*,! 'B%! -&+%! 6%)%60! %9F9! +36'(<6%!&''%+<'-! *D! %&/;,*'&.!'B%! -&+%! X3% -'(*,0! L(66! $%D%$! '*! 'B%!-&+%!

B(FB%$!-'&'%+%,'!(7)!L('B(,!'B(-!6%&$,(,F!-%--(*,!`!'B%($!1'#,/1%#$/!(10)!B%6<!'*!*$=%$!'B%!&''%+<'-9!

K,D*$+&'(*,! &2*3'!'B%! <&$%,'!(8),!6(H%! 'B%! %O%$/(-%!'B(-! X3%-'(*,! 2%6*,F-!'*0! &,=! F$*3<(,F!B%6<-! '*!

=(-'(,F3(-B! (D! &! 6%&$,(,F! *2I%/'0! B%$%! 'B%! F()%,!X3%-'(*,! (3)0! (-! 3-%=! (,! -%)%$&6! 6%&$,(,F! 3,('-! *$!

/*3$-%-9!1-!'B%!-&+%!/*3$-%!/&,!$3,! -%)%$&6! '(+%-0! 'B%! $4%15"*#!*D! 'B%!B*-'!(,-'('3'(*,!(6)!&,=!'B%!

(,'%$,&6!+"@*/,(/2"*1(&%#,B'-!(,!'B&'!<6&'D*$+!(9)!B%6<!'*!=(-'(,F3(-B!2%'L%%,!'B%!(,-'&,/%-9!

NB%!)(-3&6(M&'(*,-! &$%! $%&6(M%=! (,! 'B%!.1!-%$)(/%!&-!&!<63F4(,!*D!'B%!a$&D&,&^!D$&+%L*$H9! b%!3-%!'B%!

/1%1,#,&1C5"*;%*-'&.!D%&'3$%! *D! 6%&$,(,F! 6*/H%$! -(,/%! )%$-(*,! U! '*! -C,/! 'B%! -'&'%+%,'-! L('B!

%6&-'(/-%&$/BZ!D$*+!LB(/B!a$&D&,&!$%&=-!'B%!=&'&9!K,('(&6!(+<*$'!(-!$%&6(M%=!3-(,F!&,!*L,!'**69!

3& VISUALIZATION&

Figure+2:!Many+of+the+100+students+did+not+attempt+any+exercise+(long+grey+bars).+The+exercises+

Gradient+and+OurShapes+were+attempted+the+most+and+mostly+correctly+solved+(large+green+area)!

NB%!)(-3&6(M&'(*,!=%<(/'%=!(,!S(F3$%!U!%,&26%-!'%&/B%$-!'*!F$&-<!&'!&!F6&,/%!'B%!<%$D*$+&,/%!6%)%6!*D!

'B%($!-'3=%,'-! &'!'B%! 6%)%6!*D! &! 6%&$,(,F! 3,('9! K'!3-%-! 'B%!L%664H,*L,! '$&DD(/46(FB'!+%'&<B*$! 3-%=! (,!

*'B%$!L*$H-!&-!L%660!D*$!%O&+<6%! (,!>7*66c$!d!E'%(D!U:"UA9!N%&/B%$-!-%%!D*$!%&/B!%O%$/(-%!*D! 'B%!3,('!

B*L! +&,C! -'3=%,'-! &$%! (,! F$%%,! `! /*$$%/'!-*63'(*,! `0! C%66*L! `! -*63'(*,!<&$'(&66C! /*$$%/'! `0! $%=! `!

L$*,F!-*63'(*,!`0!*$!F$%C!`!,*!&''%+<'9!NB%!'(+%!-<&,!&,=! 'B%!'B$%-B*6=!)&63%-!D$*+!$%=!'*! C%66*L!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

^!B''<-JYYF('B329/*+YF$&D&,&YF$&D&,&!

Z!B''<-JYYF('B329/*+Y%6&-'(/Y%6&-'(/-%&$/B!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

&,=!D$*+!C%66*L!'*!F$%%,!/&,!2%!/B*-%,!2C!'B%!3-%$9!S(F3$%!"0!'*<!/*$,%$!$(FB'0!-B*L-!'B%!'B$%-B*6=!

)&63%-!:9V!&,=!:98J!(D!&,! %O%$/(-%!B&-! F*'!2%'L%%,!8"e!&,=!V:e!*D!'B%!+&O(+&6!-/*$%0!('!/*3,'-!&-!

<&$'(&66C!/*$$%/'!&,=!(-!/*6*$4/*=%=!(,!C%66*L9!NB%$%!&$%!-%)%$&6!*<'(*,-!'*!/&6/36&'%!'B%!<%$D*$+&,/%!

*D!&!-'3=%,'!*,!&,!%O%$/(-%9!NB%!=%D&36'!)&63%!-%'!D*$!'%&/B%$-!(-!-(+<6C!'B%!-/*$%!*D!'B%!6&-'!&''%+<'!

&-!('!$%D6%/'-!'B%!2%-'!'B%!/3$$%,'!H,*L6%=F%!*D!-'3=%,'-!-*!'B&'!'%&/B%$-!/&,!&=I3-'!'B%($!,%O'!6%--*,!

&//*$=(,F6C9!W'B%$!+%'$(/-!&$%!&)&(6&26%!&,=!/&,!2%!/B*-%,!(,!&!=$*<4=*L,!6(-'9!NB%C!&$%!<$(+&$(6C!D*$!

(,-'$3/'(*,&6! =%-(F,%$-! '*! F()%! 'B%+! &L&$%,%--! *,! B*L!=(DD(/36'! *$! %&-C!('! L&-! D*$!'B%! -'3=%,'-! '*!

-*6)%!'B&'! %O%$/(-%9!1-!&,!&/'(*,0!(,-'$3/'(*,&6!=%-(F,%$-!(,!/**<%$&'(*,!L('B!'%&/B%$-!+(FB'! $%L*$H!

'B&'! %O%$/(-%! '*! +&H%! ('! %&-(%$! *$! +*$%! =(DD(/3 6'!'* ! -*6)%0! *$0! 6%&)%! ('! &-! (-9! NB%-%! +%'$(/-! &$%! 'B%!

&)%$&F%0! +(,(+&6! &,=! +&O(+&6! -/*$%! *,! &66! &''%+<'-0! &-! L%66! &-! 'B%!'*'&6! ,3+2%$! *D! &''%+<'-9! 1!

F%,%$&6!6&$F%!,3+2%$!*D!&''%+<'-!+(FB'!(,=(/&'%!'B&'!'B%!=(DD(/36'C!6%)%6!(-!,*'!&<<$*<$(&'%9!

Figure+3:!Overview+of+three+learning+units!!

7(-/3--(*,-!L('B! '%&/B%$-!B&)%!-B*L,!'B&'!'B%C!&6-*!,%%=!&!F%,%$&6!*)%$)(%L!'*!<6&,! $%+%=(&'(*,!

/6&--%-!6(H%! &'!'B%!%,=!*D!&!/*3$-%!'*! '&/H6%!&F&(,!X3%-'(*,-!'B&'!B&)%!,*'!2%%,! L%66!3,=%$-'**=!2C!

'B%($! -'3=%,'-9! NB%C! ,%%=! &,! *)%$)(%L! &'! 'B%! /*3$-%! 6%)%6_! (D! 'B%C! =%'%/'! &,! (+<*$'&,'! <&$'! *D!

-'3=%,'-!(,!'B%!$%=!*$! C%66*L!&$%&0!'B%C!+(FB'!L&,'!'*!-<*'!'B%!<$*26%+&'(/!'*<(/-0!=$(66(,F!=*L,!&'!

'B%!-%/'(*,!6%)%60!'B%,!&'!'B%!6%&$,(,F!3,('!6%)%6!&-!-B*L,!(,!S(F3$%!80!&,=!'B%,!(,'*!'B%!3,('!('-%6D!&,=!

F%'! 'B%! )(-3&6(M&'(*,! <$%-%,'%=! (,! S(F3$%! U9! S*$! 'B(-! -('3&'(*,0! L%! B&)%! =%)%6*<%=! &! -(+(6&$!

)(-3&6(M&'(*,J! F$%%,0! C%66*L0! $%=! &,=! F$%C9! E'&$'(,F! D$*+! 'B%! )(-3&6(M&'(*,! =%<(/'%=! (,! S(F3$%! U0! 'B%!

&FF$%F&'(*,! &'! 'B%! 6%&$,(,F! 3,('! 6%)%6! 3-%-!'B%! L%664H,*L,! +%'B*=! *D! +&<<(,F! 'B%! )&63%-! *D! &,!

*$=%$%=! /&'%F*$(/&6! )&$(&26%! '*!*$=(,&6! ,3+2%$-0! &-! D*$! %O&+<6%! %O<6&(,%=! (,! >\&,0! T&+2%$! d! P%(!

U:"UA! <9! V^9! NB%! )&63%-! F$%C0! $%=0! C%66*L! &,=! F$%%,! (,! 'B(-! *$=%$! &$%! +&<<%=! '*! :0! "0! U! &,=! 8!

$%-<%/'()%6C9! N&H%! 'B%! %O&+<6%! *D! &! 3,('! L('B! 'B$%%! %O%$/(-%-9! #*,-(=%$! &! -'3=%,'! LB*! -*6)%=!

/*$$%/'6C!'L*!%O%$/(-%-!`!F$%%,!/*6*$!/*=%!`!&,=!=(=!,*'!&''%+<'!'B%!'B($=!%O%$/(-%!`!F$%C!/*6*$!/*=%9!

NB%! &FF$%F&'(*,! )&63%! &'! 'B%! 3,('! 6%)%6! D*$! 'B(-! -'3=%,'! (-! >8f8f:AY8! [! U0! LB(/B! (-!/*6*$4/*=%=! (,!

C%66*L9!KD!'B%!'B($=!%O%$/(-%!L&-!L$*,F6C!-*6)%=0!'B%!&FF$%F&'(*,!)&63%!L(66!2%!>8f8f"AY8![!U980!LB(/B!

(-!&6-*! /*6*$4/*=%=! (,! C%66*L9! \*L%)%$0! (D! 'B%! 'B($=! %O%$/(-%! L&-! <&$'(&66C! /*$$%/'0! 'B%! &FF$%F&'(* ,!

)&63%!L(66!2%!>8f8fUAY8![!U9g!&,=!/*6*$4/*=%=!(,!F$%%,9! NB%!-&+%!<$*/%=3$%!(-!3-%=!'*!<$*=3/%!'B%!

)(-3&6(M&'(*,-!&'! 'B%! -%/'(*,!*$! /*3$-%! 6%)%69!NB(-!<$*/%=3$%! /&,! 2%!&<<6(%=!LB&'%)%$! +%'$(/-! B&)%!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

2%%,! 3-%=! '*! <$*=3/%! 'B%! )(-3&6(M&'(*,! *D! &66! 'B%! %O%$/(-%-! *D! &! 6%&$,(,F! 3,('9! NB%! -&+%! H(,=! *D!

)(-3&6(M&'(*,! B&-! 2%%,! (+<6%+%,'%=! D*$! 'B%! -%6D4&--%--+%,'-! *,! 'B%! 6%&$,(,F! *2I%/'()%-9! N%&/B%$-!

/&,!-%%!&'!&! F6&,/%! B*L!'B%($!-'3=%,'-! &--%--! 'B%($! *L,! H,*L6%=F%!*,!%&/B!6%&$,(,F!*2I%/'()%!*D!&!

6%&$,(,F!3,('0!&,=0!&-!&2*)%0!*2'&(,!&,!*)%$)(%L!&'!'B%!/*3$-%0!-%/'(*,!&,=!6%&$,(,F!3,('!6%)%69!!

S*$!(,-'$3/'(*,&6!=%-(F,%$-!&-!L%66!&-!D*$!'B%!$%/*++%,=%$!-%$)(/%0!('!(-!(+<*$'&,'!'*!H,*L!LB%'B%$!

'B%!'C<(/&6!6%&$,(,F!'(+%!(,=(/&'%=!(,!'B%!+%'&=&'&!D*$! %&/B!6%&$,(,F!*2I%/'!(-!$%&6(-'(/9!N*!'B(-!%,=0!

L%!<$*<*-%!&!)(-3&6(M&'(*,!'B&'!-B*L-!,*'! *,6C!'B%!/%,'$&6!'%,=%,/C!23'!&6-*!'B%!=(-<%$-(*,!*D! 'B%!

*)%$&66!'(+%! -'3=%,'-! -<%,=! *,! &! 6%&$,(,F!*2I%/'9!NB%!)(-3&6(M&'(*,! (-! &! -%X3%,/%! *D!-(+<6(D(%=!2*O4

<6*'-_!%&/B!2*O!$%<$%-%,'-!&!-%--(*,!L('B!'B%!2*''*+!*D!'B%!2*O!2%(,F!'B%!+(,(+3+!'(+%!-<%,'!2C!&!

-'3=%,'! *,! 'B&'! *2I%/'! (,! 'B&'! -%--(*,! &,=! 'B%! '*<! *D! 'B%! 2*O! 'B%! +&O(+&6! '(+%_! &)%$&F%! '(+%! (-!

=$&L,!&-!&!6(,%!(,!'B%! 2*O_! 'B%! 'C<(/&6!6%&$,(,F! '(+%! &-!F()%,!(,! 'B%! +%'&4=&'&! (-! &6-*!$%<$%-%,'%=0!

-%%! S(F3$%! ^9! W,! B(FB%$! 6%)%6-0! -3/B! &-! &! 6%&$,(,F! 3,('0! %&/B! 2*O! $%<$%-%,'-! 'B%! '(+%! -<%,'! *,! &66!

6%&$,(,F!*2I%/'-!&,=!%&/B!-'3=%,'!(-!/*,-(=%$%=!2C!'B%!*)%$&66!'(+%!-<%,'!>-3+!*D!&66!-%--(*,-A9!

Figure+4:+In+the+first+session,+173+students+accessed+the+object.+The+minimal+time+spent+was+about+

1+second,+the+maximum+above+1250+seconds+and+the+average+about+20+seconds.+In+a+second+

session,+102+students+accessed+that+object.+The+typical+learning+time+is+given+by+the+red+line+

4& CONCLUSION&AND&FUTURE&WORKS&

NB%!)(-3&6(M&'(*,-!<$%-%,'%=!(,! 'B(-! <&<%$! &$%!D*$!'%&/B%$-!&,=! (,-'$3/'(*,&6! =%-(F,%$-9! ?C! -B*L(,F!

'B%+! B*L! +&,C!-'3=%,'-! &$%!(,! 'B%! F$%%,0!C%66*L0! $%=!&,=! F$%C! &$%&-0!'%&/B%$-! /&,!<$%<&$%! 'B%($!

,%O'!/6&--!&//*$=(,F!'*!'B%!,%%=-!*D!'B%($!-'3=%,'-9!?C!-B*L(,F!'B%+!%)(=%,/%!*D!B*L!6*,F!-'3=%,'-!

-<%,=!*,! 6%&$,(,F!$%-*3$/%-!*$!B*L!+&,C!&''%+<'-!'B%C!+&H%!*,!%O%$/(-%-0!(,-'$3/'(*,&6!=%-(F,%$-!

/&,!$%D6%/'!*,!'B%!6%,F'B! &,=!=(DD(/36'C!6%)%6! *D!'B%!$%-*3$/%-!&,=! &=&<'!'B%+9!E'3=%,'-!/&,!-%%!'B%!

-&+%!=(&F$&+-!&-!'%&/B%$-!=*0!23'!L('B!'B%($!*L,!=&'&!(,-'%&=!*D!'B%!D366!/6&--9!

1,!*2)(*3-!<$%$%X3(-('%!D*$! 'B%! =(&F$&+-! '*! 2%! 3-%D36!(-! 'B&'! 'B%! %O%$/(-%-! B&)%! &! B(FB! X3&6('C! &,=!

6%&$,(,F! *2I%/'()%-! &$%! L%66! D*$+36&'%=9! NB(-! $%X3($%-! -*+%! %DD*$'9! NB%! =(&F$&+-! -3<<*$'! &! +*$%!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>!

#$%&'()%!#*++*,-!.(/%,-%0!1''$(23'(*,!4!5*,#*++%$/(&645*7%$()-!89:!;,<*$'%=!>##!?@45#457!89:A!

&/'()%!<%=&F*FC!-'C6%! 6(H%! 'B%!(,)%$'%=! /6&--$**+9! QO<%$(%,/%! -B*L-!'B&'! ('!$%X3($%-! -*+%! '$&(,(,F!

D*$! '%&/B%$-! '*! (,'%F$&'%! 'B%! =(&F$&+-! (,! 'B%($! =&(6C! $*3'(,%9! ! S3$'B%$! L*$H-! (,/63=%! =(&F$&+-!

-B*L(,F! '*! (,-'$3/'(*,&6! =%-(F,%$-! 'B%! <&'B-! 'B&'! -'3=%,'-! D*66*L! LB(6%! ,&)(F&'(,F! 'B$*3FB! 'B%!

6%&$,(,F!*2I%/'-!&,=!'B%!6%&$,(,F!3,('-9!NB(-!6%&$,(,F!&,&6C'(/-!-%$)(/%!(-!2%(,F!$%&6(M%=!&-!&,!.NK4N**6!

-*!'B&'!('!/&,!2%!3-%=!L('B!&,C!.NK4/*+<6(&,'!6%&$,(,F!-C-'%+9!

1/H,*L6%=F%+%,'-J!NB%!&3'B*$-!'B&,H!'B%!LB*6%!E+&$'!.%&$,(,F!'%&+9!NB(-!L*$H!(-!<&$'(&66C!

-3<<*$'%=!2C!'B%!a%$+&,!S%=%$&6!R(,(-'$C!*D!Q=3/&'(*,!&,=!h%-%&$/B!F$&,'!,3+2%$!:"P7"V::U?.""

REFERENCES&!

7*66c$0! 190! E'%(D0! P9! >U:"UA9! D,ECE%/,-( /1%1'+/( +"@*/,( ;'12( 4,%*&'&.( -%/2E"%*-( 5"*( '&/1*@+1"*/0!

P$*/%%=(,F-!*D!#*+<3'%$-!&,=!1=)&,/%=!N%/B,*6*FC!(,!Q=3/&'(*,!>#1NQ!U:"UA0!5&<*6(!

\&,0!\90!T&+2%$0!R90!P%(0!i9!>U:"UAJ!F%1%(G'&'&.(!"&+,$1/(%&-(H,+2&'?@,/!>8$=9!Q=9A9!Q6-%)(%$9!

T$&3--0! #90! R%$/%$*,0! 190! 1,0! N94E90! jL(/H62&3%$0! R90! E'%F6(/B0! E90! 1$2&,*L-H(0! E9! >U:"VA9!H,%+2'&.(

7-I%&+,-(D,E(H,+2&"4".',/(;'12(%(G"E'4,(6,%*&'&.(!"#$%&'"&(7$$4'+%1'"&J(P$*/%%=(,F-!*D!

'B%! "g'B! 1#R! b*$6=!#*,D%$%,/%! *,! R*2(6%!&,=! #*,'%O'3&6! .%&$,(,F! >+6%&$,! U:"VA0! 1#R0!

W,'9!8:!4!5*)9!"0!U:"V0!.&$,&/&0!#C<$3-0!=*(k":9""^ZY8"8gl:V98"8gl8V!

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

Visualizing*the*Solution*Space*of*Educational*Games*using*

TRESTLE*

Erik%Harpstead%

!$"+#A&#(B#55)+(:+&'#",&%C(

D$";,%#$<E.*28#<2(

Christopher%J.%MacLellan%

F)$"(G#.D+)5)AC/(H+.8(

.D"&,8*$.5#55$+E,)$"%#.D8.)*(

ABSTRACTI(JD#+(<#,&A+&+A();#+3#+<#<(#<2.$%&)+$5(A$*#,($+<()%D#"(."#$%&'#(&+,%"2.%&)+$5(

#+'&")+*#+%,( &%( &,( &*;)"%$+%( K)"( < #,&A+#",(%)( 2+<#",%$+<(LD$%(5#$"+#",( .$+( <)( L&%D&+( %D#(

,;$.#(%D#&"( A$*#,( $KK)"<( $+<( LD#%D#"(1#D$'&)",( $."),,( %D$%( ,;$.#( $"#( ,2;;)"%&+A(%D#&"(

&+,%"2.%&)+$5( A)$5,8( H+( %D&,( <#*)(L#(L&55( ;"#,#+%( ,#'#"$5( ;")%)%C;#( '&,2$5&M$%&)+( .)+.#;%,(

1$,#<()+(%D#(GNOFG-O(.)+.#;%(K)"*$%&)+($5A)"&%D*(%)()"A$+&M#(<$%$(K")*(,%2<#+%( ,)52%&)+,(

&+%)($(%"##(,%"2.%2"#($*#+$15#(%)(,#'#"$5(P&+<,()K('&,2$5&M$%&)+8(

KeywordsI(Q&,2$5&M$%&)+/(.)+.#;%(K)"*$%&)+/($5&A+*#+%/(,)52%&)+(,;$.#(

1* INTRODUCTION*

OR;5)"$%)"C( <$%$( $+$5C,&,( &,( $+( &*;)"%$+%( ,%#;( L&%D&+( %D#( #<2.$%&)+$5( <$%$( *&+&+A( ;").#,,/(

;$"%&.25$"5C(,)(&+(%D#(.)+%#R%()K(#<2.$%&)+$5(A$*#,/(LD&.D(A#+#"$%#(5$"A#($*)2+%,(;5$C#"(<$%$8(>#&+A(

$15#( %)( '&,2$55C( &+,;#.%( %"#+<,( &+( <$%$( .$+( ;")'&<#( .)+%#R%( $+<( ;#",;#.%&'#( )+( .)*;5#R(,%$%&,%&.$5(

$+$5C,#,( $+<( .$+( D#5;( A2&<#( #<2.$%&)+$5( %#.D+)5)AC( <#,&A+8( :+K)"%2+$%#5C/( D$+<( .)+<2.%&+A( ,2.D(

$+$5C,#,()+(5$"A#"(<$%$(,#%,(&,()K%#+(%))(2+L&#5<C($+<(%&*#(.)+,2*&+A(%)(1#(;"$.%&.$58(G)()'#".)*#(

%D&,(1$""&#"/(L#( <#'#5);#<(%D#( GNOFG-O($5A)"&%D*( $+<( $+( $..)*;$+C&+A(,#%()K('&,2$5&M$%&)+,(%)(D#5;(

<#,&A+#",( $+<( "#,#$".D#",( D&#"$".D&.$55C( )"A$+&M#( $+<( #R;5)"#( ,%"2.%2"#<( <$%$/( ,2.D( $,( %D#( P&+<(

A#+#"$%#<(1C(#<2.$%&)+$5(A$*#,($+<()%D#"();#+3#+<#</(."#$%&'#(&+,%"2.%&)+$5(#+'&")+*#+%,8((

:+<#",%$+<&+A(%D#( 1"#$<%D()K($;;")$.D#,(%D$%(5#$"+#",(.$+( %$P#( &+( %D#,#( #+'&")+*#+%,( $+</(*)"#(

&*;)"%$+%5C/( D)L( %D#( A$*#( "#$.%,( %)( %D),#( $;;")$.D#,/( &,( #,,#+%&$5( K)"( #+,2"&+A( #KK#.%&'#(

&+,%"2.%&)+8(H+(%D&,(;$;#"(L#(1"&#K5C(<#,."&1#(%D#(GNOFG-O($;;")$.D($+<(<#,."&1#(%L)(#R$*;5#,()K(D)L(

&%(.$+(,2;;)"%(%D#()"A$+&M$%&)+/(#R;5)"$%&)+/($+<(&+%#";"#%$%&)+()K(,%"2.%2"#<(#<2.$%&)+$5(A$*#(<$%$8(

2* TRESTLE*

GNOFG-O(&,( $( .)+.#;%( K)"*$%&)+( $5A)"&%D*( %D$%( &+."#*#+%$55C( 5#$"+,( .)+.#;%2$5( D&#"$".D&#,( A&'#+(

,%"2.%2"#<( #R$*;5#,($,( %"$&+&+A( <$%$( =B$.-#55$+/( S$";,%#$</( 05#'#+/( T( U)#<&+A#"/( V9WX@8( :+5&P#(

*),%( 5#$"+&+A( ,C,%#*,( %D$%( )+5C( ,2;;)"%( $( '#.%)"( )K( K#$%2"#( '$52#,/( GNOFG-O(,2;;)"%,( D&#"$".D&.$5(

$%%"&12%#3'$52#( 5&,%,( ="#;"#,#+%#<( $,( YC%D)+( < &.%&)+$"&#,@( %D$%( .)+%$&+( 1)%D( +)*&+$5( =#8A8/( <&,."#%#(

)1Z#.%(%C;#,@($+<(+2*#"&.(=#8A8/(R($+<(C(;),&%&)+@($%%"&12%#,($,(L#55($,($%%"&12%#,(%D$%("#K#"(%)(+#,%#<(

$%%"&12%#3'$52#(5&,%,/(LD&.D(L#("#K#"(%)($,(,%"2.%2"$5($%%"&12%#,(=#8A8/([15).PW[(*&AD%(D$'#($(+#,%#<(,#%(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

)K($%%"&12%#3'$52#,(%D$%(<#,."&1#,(&%,(5).$%&)+($+<(%C;#@8(H%($5,)(,2;;)"%,("#5$%&)+$5($%%"&12%#,(%D$%(.$+(

<#,."&1#( "#5$%&)+,D&;,( 1#%L##+( )%D#"( $%%"&12%#,/( ,2.D( $,( ,;#.&KC&+A( %D$%( [15).PW[( &,( )+( %);( )K(

[15).PV[/( #8A8/( )+=15).PW/( 15).PV@W8( GD#( '$"&#%C( )K( $%%"&12%#( %C;#,( %D$%( GNOFG-O(.$+( D$+<5#( *$P#,( &%(

1")$<5C($;;5&.$15#(%)(L&<#("$+A#()K(;)%#+%&$5(<$%$(,#%,8(

(

Figure%1.%An%example%game%state%from%RumbleBlocks%and%how%it%would%be%described%to%TRESTLE.%

\&'#+(,%"2.%2"#<( #R$*;5#,( <#,."&1#<(&+( %D&,( "#;"#,#+%$%&)+/( GNOFG-O(.$+( #+A$A#(&+(1)%D( ,2;#"'&,#<(

$+<(2+,2;#"'&,#<(5#$"+&+A()"( $( .)*1&+$%&)+()K(%D#(%L)8( F;#.&K&.$55C/(%D#(,C,%#*( .$+(5#$"+($(,D$"#<(

D&#"$".D&.$5()"A$+&M$%&)+()K(1)%D(5$1#5#<($+<(2+5$1#5#<(<$%$(%D$%(#+$15#,(5#$"+&+A(K")*()+#(P&+<()K(

<$%$(%)(1#+#K&%(%D#()%D#"(P&+<8(-#$"+&+A( L&%D&+(GNOFG-O(&,(&+."#*#+%$5/(*#$+&+A( %D$%(&%(&,( ;"#,#+%#<(

L&%D($(,#]2#+.#()K(#R$*;5#,8(:;)+("#.#&'&+A(#$.D(#R$*;5#/(GNOFG-O(,)"%,(#$.D(+#L(#R$*;5#(&+%)(&%,(

D&#"$".DC/(2;<$%&+A(&%(%)("#K5#.%(%D#(+#L(%"$&+&+A(<$%$8(G)(A2&<#(%D&,(5#$"+&+A(;").#,,(GNOFG-O(2,#,($+(

)1Z#.%&'#( K2+.%&)+( .$55#<( .$%#A)"C( 2%&5&%C/( LD&.D(&,( <#"&'#<( K")*( ;,C.D)5)A&.$5( ,%2<&#,( )K( D2*$+(

.)+.#;%( K)"*$%&)+( =^&,D#"/( W_`a@8( GD&,( )1Z#.%&'# ( K2+.%&)+( &,( ,&*&5$"( %)( %D#( &+K)"*$%&)+3A$&+( *#%"&.(

2,#<(&+(<#.&,&)+(%"##(5#$"+&+A(12%(,2;;)"%,(%D#($1&5&%C(%)(;"#<&.%($"1&%"$"C($%%"&12%#,()K(#R$*;5#,8((

GD#( D&#"$".D&.$5( P+)L5#<A#( ,%"2.%2"#( 5#$"+#<( 1C( GNOFG-O(,2;;)"%,( %L)( P#C( .$;$1&5&%&#,I( ;"#<&.%&)+(

$+<(.52,%#"&+A8(Y"#<&.%&)+(L&%D&+(GNOFG-O();#"$%#,(,&*&5$"5C(%)(5#$"+&+A8(GD#(,C,%#*($..#;%,($,( &+;2%(

#R$*;5#,(L&%D(,)*#()K(%D#&"( $%%"&12%#('$52#,(*&,,&+A8(GD#( ,C,%#*( ,)"%,( %D#,#(;$"%&$5(#R$*;5#,(&+%)(

&%,(.2""#+%()"A$+&M$%&)+(2,&+A(%D#($'$&5$15#(K#$%2"#,($+<(%D#("#,25%&+A(.52,%#"(&%(&,(,)"%#<(&+%)(&,(2,#<(

%)( ;"#<&.%( %D#( '$52#,( )K( $+C( *&,,&+A( $%%"&12 %#,8( H+( %D&,( "#A$"</( GNOFG-O(&,( ,&*&5$"( %)( )%D#"( &+,%$+.#3

1$,#<(5#$"+&+A( $;;")$.D#,/(,2.D($,(P3+#$"#,%(+#&AD1)"/(12%(&%( $2%)*$%&.$55C(<#%#"*&+#,b1$,#<()+(

%D#(<$%$bD)L( *$+C(#R$*;5#,( =%D#( P@( %)(2,#( K)"( ;"#<&.%&)+8( Y"#'&)2,(L)"P( ,2AA#,%,( %D$%( GNOFG-Oc,(

;"#<&.%&)+(;#"K)"*$+.#(&,(,&*&5$"(%)(D2*$+,()+(%D#( %$,P()K(5$1#5&+A(%D#( ,%$1&5&%C()K(15).P(,%"2.%2"#,(

A#+#"$%#<(1C(,%2<#+%,(&+($+(#<2.$%&)+$5(A$*#(=B$.-#55$+(#%($58/(V9WX@8(

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

W(0<<&%&)+$5(,;#.&K&.,( )+( %D#( ,#*$+%&.,()K(%D&,(&+,%$+.#("#;"#,#+%$%&)+( $"#( $'$&5$15#( K")*( GNOFG-Od,( )+5&+#(<).2*#+%$%&)+I(

D%%;,Iee.)+.#;%3K)"*$%&)+8"#$<%D#<).,8&)e#+e5$%#,%e&+,%$+.#f"#;"#,#+%$%&)+8D%*5(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

H+($<<&%&)+(%)(;"#<&.%&)+/(GNOFG-O(.$+(.52,%#"(#R$*;5#,(1)%D(D&#"$".D&.$55C($+<(&+%)(K5$%(.52,%#",(=#8A8/(

&+%)(P(A")2;,@8(G)(A#+#"$%#(.52,%#"&+A,/(2,#",(;"#,#+%(GNOFG-O(L&%D($(,#]2#+.#()K(.)*;5#%#()"(;$"%&$5(

#R$*;5#,( =5$1#5#<( )"( 2+5$1#5#<@/( LD&.D( &%( )"A$+&M#,(&+%)( $( .)+.#;%2$5( D&#"$".DC( 2,&+A( &%,( 5#$"+&+A(

*#.D$+&,*,8( GD#( "#,25%&+A( D&#"$".DC( .$+( 1#( <&"#.%5C( "#%2"+#<( $,( $( .52,%#"&+A( )K( %D#( <$%$8(

0<<&%&)+$55C/(GNOFG-O(D$,( *25%&;5#(;),%3;").#,,&+A(")2%&+#,(%D$%(.$+(%"$+,5$%#(%D#,#( D&#"$".D&#,(&+%)(

K5$%(.52,%#"&+A,()K(%D#(#R$*;5#,8(^)"(#R$*;5#/(&%(.$+(,%$"%($%(%D#("))%()K(%D#(D&#"$".DC(=LD&.D(.)+%$&+,(

$55(#R$*;5#,@($+<(;")A"#,,&'#5C(1"#$P(%D&,(%);35#'#5(.52,%#"&+A(&+%)(;")A"#,,&'#5C(,*$55#"($+<(,*$55#"(

A")2;,/(,%);;&+A(LD#+( &%( D$,( );%&*&M#<()+#( )K( $( "$+A#()K(*#%"&.,/( ,2.D($,(!$%#A)"C( :%&5&%C/( 0H!/()"(

>H!8(g2"($+$5C,&,()K(GNOFG-Oc,($1&5&%C(%)(.52,%#"(15).P(,%"2.%2"#,(,2AA#,%,(%D$%(&%(;")<2.#,(A")2;&+A,()K(

#R$*;5#,( %D$%( D$'#( "#$,)+$15C( D&AD( $A"##*#+%( L&%D( D2*$+3A#+#"$%#<( .52,%#"&+A,( )K( %D#( ,$*#(

15).P,( ,%"2.%2"#,( =B$.-#55$+( #%( $58/( V9W`@/( LD&.D( ,2AA#,%,( %D$%( GNOFG-O(*&AD%( 1#( 2,#<( %)( )"A$+&M#(

5$"A#(')52*#,()K(#R$*;5#,(&+($(L$C(%D$%($5&A+,(L&%D(D)L(D2*$+,(L)25<()"A$+&M#(%D#(,$*#(<$%$8(

3* VISUALIZATION*USE*CASES*

J#(D$'#(<#,&A+#<(,#'#"$5('&,2$5&M$%&)+,(%)(K$.&5&%$%#(&+%#";"#%$%&)+()K(GNOFG-Oc,()2%;2%,/(%D)2AD(K#L(

D$'#( 1##+( #*;&"&.$55C( '$5&<$%#<( L&%D( %$"A#%( 2,#",8( GD#,#( '&,2$5&M$%&)+,( D$'#( 1##+( 12&5%( %)( 1#(

&+%#"$.%&'#(2,&+A(678Z,(=>),%).P/( gA&#'#%,PC/(T(S##"/(V9WW@(&+()"<#"( %)( #+$15#( $+($+$5C,%(%)(#R;5)"#(

%D#(&+K)"*$%&)+($+<(*$P#(%D#&"()L+("#K5#.%&'#(Z2<A#*#+%,($1)2%(%D#&"(<#,&A+8((

O$.D( )K( %D#( '&,2$5&M$%&)+,( ;"#,#+%#<( D#"#( L$,( <#'#5);#<( $,( ;$"%( )K( %D#( $+$5C,&,()K( ?@#A4,B4"+C/(

=!D"&,%#5(#%( $58/( V9WV@($+(#<2.$%&)+$5(A$*#( <#,&A+#<(%)(%#$.D( 1$,#<(.)+.#;%,()K(,%"2.%2"$5(,%$1&5&%C(

$+<(1$5$+.#( %)( C)2+A( .D&5<"#+8( H+( %D&,( A$*#( ;5$C#",( 12&5<( 15).P( %)L#",( %D$%( *2,%( ,2"'&'#( $+(

#$"%D]2$P#( %)( ,2..##<8( \&'#+( %D$%( %D#( A$*#( "#5&#<( )+( $( "#$5( %&*#( ;DC,&.,( #+A&+#( %)( A#+#"$%#( &%,(

K##<1$.P(&%(L$,(+)%($5L$C,(.5#$"(&K(%D#(A$*#(L$,(;")'&<&+A(.5#$"(A2&<$+.#(%)(;5$C#",(%D$%(L)25<(D#5;(

%D#*(5#$"+(&%,(%$"A#%#<(.)+.#;%,8((

3.1* TRESTLE*Tree*Visualization*

GD#(.)"#('&,2$5&M$%&)+()K(GNOFG-O(&,( $( '&,2$5( "#;"#,#+%$%&)+()K(&%,( D&#"$".D&.$5(.)+.#;%(%"##8( ^&A2"#(W(

,D)L,( %D&,( '&,2$5&M$%&)+( )K( #R$*;5#,( )K( ;5$C#"( ,)52%&)+,( %)( )+#( 5#'#5( &+(?@#A4,B4"+C/8( H+( %D&,(

'&,2$5&M$%&)+( #$.D( .&".5#( "#;"#,#+%,( $( .)55#.%&)+( )K( ,%2<#+%( ,)52%&)+,( %)( $( A$*#( 5#'#58( GD#( 5#$K(

.)+.#;%,( =K&55#<( &+( .&".5#,@( "#;"#,#+%( ,;#.&K&.( &+,%$+.#,( L&%D&+( %D#( <$%$,#%( LD&5#( %D#( %"$+,;$"#+%(

#+.5),&+A( .&".5#,( "#;"#,#+%( D&AD#"3)"<#"( .52,%#",( .)+%$&+&+A( ,21A")2;,8( GD#( ,&M#( )K( #$.D( .52,%#"(

"#;"#,#+%,( D)L( *$+C( &+,%$+.#,( $"#( A")2;#<( L&%D&+( &%8( G)( %D#( "&AD%( )K( %D#( %"##( &,( $( .)+%")5( ;$+#5(

,D)L&+A( '$"&)2,( );%&)+,( $,( L#55( $,( $+( 0%%"&12%#3Q$52#( G$15#( %D$%( ,D)L,( %D#( <&,%"&12%&)+,( )K( #$.D(

$%%"&12%#3'$52#(L&%D&+(%D#(%"##8(JD#+($(.)+.#;%()"(&+,%$+.#(&,(.5&.P#<()+/(%D#('&#L(M))*,(%)(K).2,()+(

%D$%(.)+.#;%($+<(%D#(0%%"&12%#3Q$52#(G$15#(2;<$%#,(,D)L,( %D#(<&,%"&12%&)+()K(;");#"%&#,(L&%D&+(%D#(

,#5#.%#<(.)+.#;%e&+,%$+.#8((

4)<#,(&+(%D#('&,2$5&M$%&)+(.$+( 1#( .)5)"#<($..)"<&+A(%)(%D#&"(<&KK#"#+%( $%%"&12%#,( %)( D&AD5&AD%( %"#+<,(

L&%D&+( %D#( .)+.#;%( %"##8( H+( *$&+( #R$*;5#( &+( ^&A2"#( W/( %D#( +)<#,( $"#( .)5)"#<( 1$,#<()+( LD#%D#"($(

,)52%&)+( ,2..##<#<( )"( K$&5#<( %D#( 5#'#5( &+( ]2#,%&)+($..)"<&+A(%)( %D #( A$*#d,( 5)A( <$%$8( F)52%&)+,( $"#(

.)5)"#<(C#55)L(&K( %D#C( $"#( *)"#( 5&P#5C( %)( ;$,,( %D#( 5#'#5/($+<(;2";5#(&K(%D#C($"#(*)"#(5&P#5C(%)(K$&58(H+(

%D&,( .$,#( %D#"#( &,( $( .5#$"( ,2..#,,K25( .52,%#"( =1)%%)*( )K( 5#K%( 1"$+.D@/( $+<( $( *),%5C( .5#$"( +#A$%&'#(

.52,%#"(="&AD%(1"$+.D@8(GD#()2%.)*#,()K(%D#()%D#"(%L)(*$&+(1"$+.D#,()K(%D#(%"##(=5#K%($+<("&AD%()K(%D#(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

5$"A#"(5#K%(1"$+.D@($"#( 5#,,(.5#$"($+<(L)25<(;)%#+%&$55C( L$""$+%( K2"%D#"( &+'#,%&A$%&)+8(0+($+$5C,%(.$+(

"#3.)5)"(%D#(,$*#('&,2$5&M$%&)+($..)"<&+A( %)( <&KK#"#+%( ;");#"%&#,()K(,)52%&)+,(%)(,##( &K(%D#"#($"#($+C(

.)""#5$%&)+,( &+( %"#+<,( %D$%( *&AD%( L$""$+%( &+'#,%&A$%&)+( $,( <#,&A+( &,,2#,8(GD#( ,*$55#"( #R$*;5#,( &+(

^&A2"#( W( $"#( #$.D( .)5)"#<( $..)"<&+A( %)( $( <&KK#"#+%( ;DC,&.$5( ;");#"%C( )K( %D#( ,)52%&)+( %)L#"(=#8A8/(

,C**#%"C/( 1$,#( L&<%D/( .#+%#"( )K( *$,,( D#&AD%@8H+( ;"&+.&;5#(%D#,#( ;");#"%&#,( ,D)25<( ")2AD5C(

.)""#,;)+<( L&%D( ,2..##<&+A( $%( %D#( 5#'#5( 12%( %D#"#( &,( +)%( $( .5#$"( .)""#,;)+<#+.#( %)( %D#( .)5)"&+A(

1$,#<()+(,2..#,,/(,2AA#,%&+A(%D#"#(*&AD%($+(&,,2#(L&%D(D)L(K##<1$.P(&,($,,&A+#<(&+(%D#(A$*#8(

(

Figure%2.%A%visualized%TRESTLE%tree%of%a%sample%of%100%solutions%to%a%level%in%RumbleBlocks.%In%the%

top%visualization%clusters%are%colored%by%their%likelihood%of%succeeding%on%the%level%(yellow%for%

passing,%purple%for%failing).%In%the%lower%three%visualizations%the%same%tree%is%reOcolored%according%

to%different%physical%properties%of%game%solutions.%

3.2* TRESTLE*Alignment*Visualization*

JD&5#(%D#(D&#"$".D&.$5(%"##(&,(.2""#+%5C(%D#(<#K$25%()2%;2%( K)"( '&,2$5&M$%&)+(&+( GNOFG-O/( L#(D$'#( $5,)(

<#'#5);#<( *)"#( $<'$+.#<( K)"*,( )K( '&,2$5&M$%&)+(%D$%( *$P#( 2,#( )K( %D#( GNOFG-O(<$%$( ,%"2.%2"#( &+( $(

K5$%%#+#<(K)"*8(GD#($5&A+*#+%('&,2$5&M$%&)+(,D)L+(&+(^&A2"#(7(&,(1$,#<()+($+$5C,#,(L#(D$'#(<)+#()K(

%D#( ,)52%&)+( ,;$.#( )K( ?@#A4,B4"+C/(=S$";,%#$</( B$.-#55$+/( 05#'#+/( T( BC#",/( V9Wh@8( GD&,(

'&,2$5&M$%&)+(1"#$P,(2;(%D#(GNOFG-O(%"##(&+%)("#;"#,#+%$%&'#(.52,%#",(%D$%(.$+(1#(;5)%%#<($..)"<&+A(%)(

%D#&"( ;");#"%&#,( %)( 5))P( K)"( .)""#5$%&)+,( &+( <$%$( %D $%( *&AD%( 1#( &+<&.$%&'#( )K(;")15#*,8( H+( %D#( .$,#(

,D)L+( &+( ^&A2"#( 7( $( ;"&+.&;5#( "#5#'$+%( *#%"&.(=&+( %D&,( .$,#( %D#( ,C**#%"C( )K( %D#( %)L#"@(,D )25<( 1#(

;"#<&.%&'#()K(,2..#,,K25(;#"K)"*$+.#(&+(%D#(A$*#8(L&%D&+(%D#('&,2$5&M$%&)+(%D&,(&,(<#+)%#<(1C(%D#("#<(

$+<(A"##+(,D$<#<("#A&)+,(LD#"#(&%(L)25<(1#(<#,&"$15#(K)"(,)52%&)+(.52,%#",(%)(K$55(&+(%D#(A"##+($"#$,(

LD&5#(&%(L)25<(1#($(;")15#*(&K(%D#C(*$&+5C(K#55(&+%)("#<("#A&)+,8(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

Figure%3.%An%example%of%employing%TRESTLE%to%visualize%the%alignment%of%clusters%of%player%solutions%

to%a%game.%Solutions%are%plotted%along%an%axis%for%principle%relevant%metrics%(xOaxis)%and%likelihood%

of%success%(yOaxis).%Ideally%there%would%be%a%relationship%between%principle%relevant%metrics%and%

success,%denoted%by%the%shaded%areas.%2%%

G)( ,2;;)"%( <&AA&+A( K2"%D#"( &+%)( $;;$"#+%( %"#+<,( %D#( '&,2$5&M$%&)+( ,2;;)"%,( %D#( );%&)+( )K( &+.52<&+A(

,."##+,D)%,(%D$%( .$+( 1#(5&+P#<( %)( &+,%$+.#,( L&%D&+( %D#( .52,%#"&+A8( GD#,#( ,."##+,D)%,( .$+( 1#(

.)*;),&%#<( %)A#%D#"( =5)L#"( "&AD%( )K( ^&A2"#( 7@( %)( $55)L( $+( $+$5C,%( %)( *)"#( ]2 &.P5C( #R$*&+#( LDC( $(

;$"%&.25$"(%"#+<(*&AD%( 1#( D$;;#+&+A(L&%D&+( %D#&"( &+,%"2.%&)+$5( #+'&")+*#+%($+<( LD$%($.%&)+,( %D#C(

*$C(#R;5)"#(%)(K&R(%D#(;")15#*8(

4* CONCLUSION*

g2"(A)$5(&+(<#,&A+&+A('&,2$5&M$%&)+,(K)"(GNOFG-O(&,( %)( D#5;($+$5C,%,( %)()"A$+&M#(%D#&"(<$%$(&+( L$C( %D$%(

.$+(,2;;)"%( &+%2&%&'#(#R;5)"$%&)+8(J#(D$'#( K)2+<(%D&,(%)(1#(;$"%&.25$"5C(2,#K25(L&%D&+(<$%$,#%,(K")*(

.)*;5#R(&+,%"2.%&)+$5(#+'&")+*#+%,(,2.D($,(#<2.$%&)+$5(A$*#,8(J#(D);#(%D$%()%D#",(.$+(,##(2%&5&%C(

&+(%D&,($;;")$.D($+<(.$+(K&+<($(L$C(%)($;;5C(&%(%)(%D#&"()L+(.)+%#R%,8(

REFERENCES*

>),%).P/( B8/( gA&#'#%,PC/( Q8/( T( S##"/( i8( =V9WW@8( 67(6$%$36"&'#+( 6).2*#+%,8( 3DDD( E*%&/%+1'"&/( "&(

F'/@%4'G%1'"&( %&-( !"#$@1,*( H*%$2'+//( =I=WV@/( V79WjV79_8(

D%%;,Iee<)&8)"AeW98WW9_eGQ!\8V9WW8W`k(

!D"&,%#5/(B8(\8/(F%#'#+,/(F8(B8/(B$D#"/(>8(F8/(>"&.#/(F8/(!D$*;#"/(B8/(i$C$;$5$+/(-8/(l(-)*$,/(68(=V9WV@8(

N2*15#>5).P,I( G#$.D&+A( ,.&#+.#( .)+.#;%,( %)( C)2+A( .D&5<"#+( %D")2AD( $( 2+&%C( A$*#8( H+( )*"+(

!H7JDK(LM=L(=;;8(WXVjWXX@8(HOOO8(D%%;,Iee<)&8)"AeW98WW9_e!\$*#,8V9WV8X7Whka9(

^&,D#"/( 68( S8( =W_`a@8( U+)L5#<A#( 0.]2&,&%&)+( Q&$( H+."#*#+%$5( !)+.#;%2$5( !52,%#"&+A( -#$"+&+A( m(

O5#*#+%(Y#"K)"*$+.#(O5#*#+%8(J%+2'&,(6,%*&'&./(L/(W7_jWaV8(

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

V(0(5&'#('#",&)+()K(%D&,('&,2$5&M$%&)+(&,($'$&5$15#($%(D%%;Iee#"&PD$";,%#$<8+#%e$5&A+*#+%e'&,2$5&M$%&)+8D%*5(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

S$";,%#$</(O8/(B$.-#55$+/(!8(i8/(05#'#+/(Q8/(T(BC#",/(>8(08(=V9Wh@8(:,&+A(#R%"$.%#<(K#$%2"#,(%)(&+K)"*(

$5&A+*#+%3<"&'#+(<#,&A+(&<#$,(&+($+(#<2.$%&)+$5(A$*#8(H+()*"+,,-'&./("5(12,(NL&-(%&&@%4(7!J(

+"&5,*,&+,("&(O@#%&(5%+1"*/('&(+"#$@1'&.(/8/1,#/(P(!O3(Q=R(=;;8(77V_j777`@8(4#L(?)"P/(4#L(

?)"P/(:F0I(0!B(Y"#,,8(D%%;,Iee<)&8)"AeW98WWhkeVkkXV``8Vkka7_7(

B$.-#55$+/(!8( i8/(S$";,%#$</(O8/(05#'#+/(Q8/(T(U)#<&+A#"/(U8(N8(=V9WX@8(GNOFG-OI(0( B)<#5()K(!)+.#;%(

^)"*$%&)+(&+(F%"2.%2"#<( 6)*$&+,8( 7-S%&+,/('&(!".&'1'S,(K8/1,#//(R/(W7WjWk98(N#%"&#'#<(K")*(

D%%;IeeLLL8.)A,C,8)"Ae;$;#",e0!F')5he;$;#"W98;<K(

(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

Visualizing*Cronbach’s*Alpha*for*a*Large*Number*of*Assessments*

!"#$%&'()&*+&'

A5$B;)C#"(-$1,(

$<&%B$8,D$"*$E;5$B;)C#"5$1,8.)*((

,#*+&-'.&$/-'

A5$B;)C#"(-$1,(

+&"*$5E;5$B;)C#"5$1,8.)*((

!0(12!31F(G+(%D&,(;$;#"/(C#(;"#,#+%($(+)'#5(<$%$('&,2$5&H$%&)+(%D$%(,D)C,(%D#(<&,%"&12%&)+()I(

!")+1$.DJ,( 05;D$( I)"( $( 5$"K#( +2*1#"( )I( $,,#,,*#+%,( $+<( %D#&"( &%#*,8( !")+1$.DJ,( $5;D$(

*#$,2"#(.$+(1#(2,#<(%)(&*;")'#($,,#,,*#+%,(1B("#*)'&+K(&%#*,(%D$%($"#(+)%(.)+,&,%#+%(C&%D(

)%D#"(&%#*,()I( %D#( %#,%8( LD#(#M.52,&)+( )I( &%#*,(.$+( $II#.%( %D#( $5;D$( )I(%D#( $,,#,,*#+%(&+( $(

<&II#"#+%(C$B8(LD#( ;");),#<( '&,2$5&H$%&)+(*$N#,( &%(#$,B(%)( &<#+%&IB( $,,#,,*#+%,( CD#"#(%D#(

"#*)'$5( )I( ,)*#( &%#*,( .$+( 5#$<( %)( $( ,&K+&I&.$+%( K$&+( &+( %D#( &+%#"+$5( .)+,&,%#+.B( )I( %D#(

$,,#,,*#+%(&%#*,8( LD#( '&,2$5&H$%&)+(&,(;$"%&.25$"5B( 2,#I25( CD#+( %D#( +2*1#"()I( $,,#,,*#+%,(

1#&+K($+$5BH#<(&,(5$"K#8(LD#('&,2$5&H$%&)+(;"#,#+%#<(&,($5,)();#+(,)2".#($+<("#2,$15#8(

4/%56*"7F(G+%#"$.%&'#(6$%$(O&,2$5&H$%&)+/(0,,#,,*#+%(6$%$/(!")+1$.DJ,(05;D$(

1* INTRODUCTION*

P2$5&%B( $,,#,,*#+%,( $"#( .#+%"$5( %)( %D#( <#,&K+( )I( $( K))<( .2""&.252*8( LD#B( ;")'&<#( 2,( C&%D(

*#$,2"#*#+%,( %D$%( D#5;( 2,( K$2K#( D)C( C#55( ,%2<#+%,( 2+<#",%$+<( %D#( &+,%"2.%&)+( )I( %D#( .)2",#8(

0,,#,,*#+%,(1#.)*#(#'#+(*)"#(&*;)"%$+%(&+(%D#(.)+%#M%()I()+5&+#(.2""&.25$/(1#.$2,#(%D#(&+,%"2.%)"(

&,( ;DB,&.$55B( ,#;$"$%#<( I")*( D#"( ,%2<#+%,/( $+<($,,#,,*#+%( <$%$( &,( )+#( )I(%D#( I#C( C$B,( %)( K$&+($+(

&+,&KD%( &+%)( %D#( ;")K"#,,( ) I( %D#( .)D)"%8( Q+5&+#( 5#$"+&+K( ;5$%I)"*,( $ "#( +)C( $15#( %)( #*;5)B( '$"&)2,(

N&+<,( )I( $,,#,,*#+%,/( "$+K&+K( I")*( &+3'&<#)( R2&HH#,( %)( $2%)3K"$<#<( ;")K"$**&+K($,,&K+*#+%,( %)(

N+)C(D)C(C#55(,%2<#+%,( $"#( *)'&+K( %)C$"<,(%D#( K)$5,()I( %D#(.)2",#8( >2%(%D#( *#$,2"#,()I(,%2<#+%(

N+)C5#<K#(;")'&<#<(1B(%D#,#(R2&HH#,($+<(%#,%,($"#(<&"#.%5B("#5$%#<(%)(%D#(R2$5&%B()I(%D#($,,#,,*#+%,8(

S))<( R2$5&%B( $,,#,,*#+%,( .$+( %#55( 2,( "&KD%( %D&+K,( $1)2%( ,%2<#+%,J( ;")K"#,,/( 12%( 1$<( R2$5&%B(

$,,#,,*#+%,(.$+(K&'#(2,($*1&K2)2,(&+I)"*$%&)+($1)2%(,%2<#+%(N+)C5#<K#8(

LD#"#($"#( *$+B( <&II#"#+%(*#%"&.,(%)($,,#,,( %D#(R2$5&%B( )I(%D#( %#,%,8( T)*#( )I( %D#,#( *#%"&.,( K&'#( 2,(

&+I)"*$%&)+( $1)2%( %D#( R2$5&%B( )I( %#,%(&%#*,/( CD&5#( )%D#"( *#%"&.,( K&'#( 2,( &+I)"*$%&)+( $1)2%( %D#(

R2$5&%B()I( %D#( )'#"$55(%#,%8(U)"#)'#"/( <&II#"#+%( %#,%( %D#)"&#,(,2.D( $,( .5$,,&.$5(%#,%( %D#)"B($+<( &%#*(

"#,;)+,#( %D#)"B( K&'#( 2,( <&II#"#+%( *#%"&.,( %)( *#$,2"#( D)C( #II#.%&'#( %D#( &%#*,( $+<( %#,%,( $"#( %)(

*#$,2"#(,%2<#+%(N+)C5#<K#8(LD#( %B;&.$5( *#$,2"#,(I)"( &%#*,( I")*(.5$,,&.$5( %#,%( %D#)"B($"#( ;#".#+%(

.)""#.%( $+<( ;)&+%( 1&,#"&$5(=)"( &%#*3%)%$5( .)""#5$%&)+/@(CD&5#( &%#*( "#,;)+,#( %D#)"B( D$,( &%,( )C+(

;").#<2"#,( %)( .$5.25$%#( &%#*(<&II&.25%B( $+<( <&,."&*&+$%&)+8( V)"( %#,%,/( .5$,,&.$5( %#,%( %D#)"B( ;")'&<#,(

!")+1$.DJ,($5;D$( =!")+1$.D/(WXYW@( %D$%( *#$,2"#,(%D#(&+%#"+$5(.)+,&,%#+.B()I(%D#( &%#*,/( CD&5#( &%#*(

"#,;)+,#(%D#)"B( 2,#,( $( *#$,2"#(.$55#<(%#,%( &+I)"*$%&)+(I2+.%&)+(%D$%( "#5$%#,( %#,%(&+I)"*$%&)+(C&%D(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

5$%#+%( ,%2<#+%( $1&5&%B8(055( )I( %D#,#( *#$,2"#,( K&'#( 2,( $.%&)+$15#( &+I)"*$%&)+( $1)2%( D)C( C#( .$+(

&*;")'#(%D#(%#,%(,)($,(%)(&*;")'#()2"(#,%&*$%#()I(,%2<#+%($1&5&%B8((

2* METHOD*

G+( %D&,( ;$;#"/( C#( I).2,( )+( !")+1$.DZ,( $5;D$/( CD&.D( &,( )+#( )I( %D#( *),%( C&<#5B( 2,#<( *#%"&.,( %)(

*#$,2"#(%D#( R2$5&%B( )I( %#,%,8( !")+1$.DZ,( $5;D$( %#55,( D)C( .5),#5B( &%#* ,( C&%D&+( $( %#,%( $"#( "#5$%#<( %)(

#$.D()%D#"8(G%(&,($(*#$,2"#()I(&+%#"+$5(.)+,&,%#+.B8(GI(!")+1$.DZ,($5;D$(&,('#"B(D&KD(I)"($(%#,%/(C#(.$+(

$,,2*#(%D$%($55()I(%D#( &%#*,( &+(%D#( %#,%( $"#( *#$,2"&+K( $( ,&*&5$"( N+)C5#<K#(.)+,%"2.%8(GI(!")+1$.DZ,(

$5;D$( &,( '#"B( 5)C( I)"( $( %#,%/( C#( .$+( $,,2*#( %D$%( %D#( R2#,%&)+,( )I( $+( $,,#,,*#+%( $"#( %"B&+K( %)(

*#$,2"#('#"B(<&II#"#+%()"(2+"#5$%#<(.)+,%"2.%,8(G<#$55B/(I)"($+B(.2""&.252*/(C#(C)25<(C$+%(%D#($5;D$(

%)(1#(D&KD(I)"(#'#"B($,,#,,*#+%8(>2%()I%#+(%&*#,/(C#(.$+("2+( &+%)($,,#,,*#+%,(C&%D(5)C(!")+1$.DZ,(

$5;D$8( Q+#( "#$,)+( I)"( 5)C( $5;D$( &,( %D$%( $55( )I( %D#( &%#*,( $"#( *#$,2"&+K( <&II#"#+%( N+)C5#<K#(

.)+,%"2.%,/(12%($+)%D#"("#$,)+(I)"(5)C($5;D$(&,(%D$%()+5B()+#()"(,)*#(&%#*,()I( %D#(%#,%($"#()2%5&#",/(

CD&5#()%D#"(&%#*,()I(%D#(%#,%($"#(.)""#5$%#<8(GI(C#(.$+("#*)'#(%D#,#()2%5&#"(&%#*,/(C#(.$+(*$N#(%D#(

%#,%(*)"#(.)+,&,%#+%8(

Q+#( C$B( %)( &<#+%&IB( &%#*,( %D$%( $"#( +)%( .)+,&,%#+%( C&%D( )%D#"( &%#*,( )I( %D#( %#,%( &,( %)( 5))N( $%( D)C(

!")+1$.DZ,( 05;D$( &,( $II#.%#<( CD#+($+( &%#*( &,( <");;#<()2%( )I( %D#( %#,%8( GI( %D#( $5;D$( )I( %D#( %#,%(

&+."#$,#,( $I%#"( "#*)'$5( )I( $+( &%#*/( C#( .$+( ,2"*&,#( %D$%( %D#( &%#*( %D$%( C$,( "#*)'#<( D#5;#<( %)(

&+."#$,#(%D#(&+%#"+$5(.)+,&,%#+.B()I(%D#(%#,%(&%#*,8(0(%#,%(C&%D(D&KD(&+%#"+$5(.)+,&,%#+.B(.$+(;")'&<#(

2,( C&%D( $( *)"#( "#5&$15#( *#$,2"#( )I( ,%2<#+%( N+)C5#<K#8( G*$K&+#( $( %#,%( $1)2%( I)"( 5));,( &+(

;")K"$**&+K(%D$%(.)+%$&+,(,)*#(&%#*,()+(&I[#5,#(,%$%#*#+%,(%D$%(,%2<#+%,(D$'#+Z%(5#$"+#<($1)2%8(

\#,25%,()I(%D&,(%#,%($"#(*)"#("#5&$15#(&I(C#("#*)'#($55()I(%D#(&%#*,(%#,%&+K(&I[#5,#(,%$%#*#+%,/(1#.$2,#(

%D#+/(%D#(,.)"#,(C&55(%#55(2,(*)"#($1)2%(%D#(,%2<#+%(N+)C5#<K#()I(I)"(5));,8((

L)(<#%#.%()2%5&#"(&%#*,(&+($(%#,%/(C#(.$+(2,#(&%#*("#,;)+,#(<$%$()I(,%2<#+%,(%)(.$5.25$%#(%D#($5;D$(I)"(

%D#( %#,%/( $+<( $5;D$( I)"( %D#( %#,%( $I%#"("#*)'$5( )I( #$.D( )I( %D#( %#,%( &%#*8( LD#( *#%"&.,( )1%$&+#<(

%D#"#$I%#"(.$+(D#5;( 2,( <#.&<#(CD&.D(&%#*,( I")*( %D#( %#,%(.$+(1#("#*)'#<( %)( &*;")'#(%D#(R2$5&%B()I(

%D#(%#,%8(LD&,(K&'#,(2,($(C$B(%)(2,#(<$%$(%)(&*;")'#(%D#(<#,&K+()I($,,#,,*#+%/(CD&.D/(&+(%2"+/(*&KD%(

&*;")'#(%D#(R2$5&%B()I(<$%$(.)55#.%#<(5$%#"8(

0( !")+1$.DJ,($5;D$( '$52#( )I(98]9( &,( .)+,&<#"#<( $..#;%$15#( =!)"%&+$/( WXX7@8( V)"( $( ,*$55( +2*1#"( )I(

$,,#,,*#+%,/(C#(.$+(.$5.25$%#(!")+1$.DJ,($5;D$($+<(5))N($%($(%$15#()I('$52#,(%)(I&+<()2%("#*)'$5()I(

CD&.D(&+."#$,#,(%D#( $5;D$( )I(%D#(%#,%8( >2%/(CD#+( %D#( +2*1#"()I( $,,#,,*#+%,( I")*( $( .2""&.252*(&,(

5$"K#(=#8K8(4(^(W99/@(C#(*&KD%(I&+<(&%('#"B(<&II&.25%(%)(&<#+%&IB($,,#,,*#+%,( %D$%(.$+(1#(&*;")'#<(%D#(

*),%8( _#( .$+( 5))N( $%( ,)"%#<( %$125$"( <$%$( %)( I&+<( )2%5&#"( &%#*,( %D$%( .$+( 5#$<( %)( %D#( *),%(

&*;")'#*#+%( &+( $5;D$ /( 12%( C#( C)25<( +)%( K#%( *2.D( &+,&KD%( &+ %)( %D#( <&,%"&12%&)+( )I( %D#( "#,%( )I( %D#(

<$%$8(G+(%D&,(.$,#/(&%(&,(;),,&15#(%)(2,#($(<$%$('&,2$5&H$%&)+(%D$%(D#5;,(2,(&<#+%&IB(1)%D(%D#()2%5&#"(&%#*,(

$+<( )'#"$55( ;$%%#"+( )I( %D#( $,,#,,*#+%( R2$5&%B8( LD&,( &,( %D#( *)%&'$%&)+( 1#D&+<( %D#( '&,2$5&H$%&)+(

<#,."&1#<(&+(%D&,(;$;#"8((

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

LD#('&,2$5&H$%&)+(;"#,#+%#<(&+(%D&,(;$;#"(&,($5,)();#+(,)2".#W($+<(&,(C"&%%#+(&+(\/($+<(&%(.$+(%$N#(<$%$(

&+($(,;#.&I&.(I)"*$%($+<(K#+#"$%#(%D#('&,2$5(I)"($+B(,#%()I(&+;2%(<$%$8(

3* DATA*

L)(K#+#"$%#(%D#('&,2$5&H$%&)+(2,&+K(%D#(;")'&<#<();#+(,)2".#(\(.)<#/(C#(+##<(&%#*("#,;)+,#(<$%$()I(

,%2<#+%,(I)"(<&II#"#+%($,,#,,*#+%,(&+($(,&+K5#(%$15#8(L$15#(W(<#,."&1#,(%D#(<$%$(I)"*$%($+<(,D)C,($+(

#M$*;5#(%$15#(%D$%(.$+(1#(&+K#,%#<(1B(%D#(\(;")K"$*(%)(K#+#"$%#(%D#('&,2$5&H$%&)+8(

LD#(0,,#,,*#+%(G6/(P2#,%&)+( G6/( $+<( T%2<#+%(G6(.)52*+( .$+( 1#( $+B(;),,&15#(2+&R2#(&<#+%&I&#",(I)"(

$,,#,,*#+%,/(R2#,%&)+,(C&%D&+(%D),#($,,#,,*#+%,/($+<(,%2<#+%,(CD)($%%#*;%#<(%D),#($,,#,,*#+%,8(

0,,#,,*#+%( G6,( C&55( $;;#$"( )+( %D#( `3$M&,( )I( %D#( ;5)%/( ,)( &%( &,( ,2KK#,%#<( %)( 2,#( *)"#( *#$+&+KI25(

+$*#,(&+(%D$%(.)52*+8(V)"(#'#"B(P2#,%&)+(G6(&+(%D#(<$%$/($(<)%(C&55(1#(*$<#(&+(%D#('&,2$5&H$%&)+($+<(

D)'#"&+K()'#"( %D$%( <)%( C&55("#'#$5( 1$.N( %D#( P2#,%&)+( G68( T)( &+%#";"#%$15#('$52#,(&+(%D#( P2#,%&)+( G6(

.$+($5,)(1#('#"B(2,#I258(T%2<#+%(G6(.)52*+(.$+(.)+%$&+(#&%D#"($+)+B*&H#<(G6,()"("#$5(,%2<#+%(+$*#,/(

&%(C&55(+)%(*$N#($(<&II#"#+.#(&+($+B($,;#.%()I(%D#('&,2$5&H$%&)+8(

1&8-/'9:';&$&'<6*+&$'&="'/>&+?-/'"&$&'*/@A#*/"'$6'B/=/*&$/'$)/'C#7A&-#D&$#6='

0,,#,,*#+%(G6(

=T%"&+K@(

P2#,%&)+(G6(

=T%"&+K@(

T%2<#+%(G6(

=T%"&+K@(

!)""#.%(

=W[9@(

L&*#(

=????3UU366(aaFUUFTT@(

L#,%W(

PW(

0+)+W(

b9Wc39W39W(W9F99F99(

L#,%(W(

Pb(

0+)+W(

b9Wc39W39W(W9F9WF99(

L#,%(W(

P7(

0+)+W(

b9Wc39W39W(W9F9bFb9(

L#,%(b(

PW(

0+)+b(

b9Wc39W39b(WWF79F99(

L#,%b(

Pb(

0+)+b(

b9Wc39W39b(WWF79FYY(

L#,%7(

PW(

0+)+7(

b9Wc39W39d(WYFb9FWb(

(

4* VISUALIZATION*

LD#('&,2$5&H$%&)+(&,(K#+#"$%#<(1B(2,&+K(%D#(<$%$(<#,."&1#<(&+(L$15#(W8((LD#('&,2$5&H$%&)+(;")K"$*(2,#,(

%D#( I2+.%&)+( alpha()(I")*( \( ;$.N$K#( psych(=\# '#55#/( b9W]@( %)( .$5.25$%#( %D#( !")+1$.DZ,( $5;D$8(

LD#($5;D$( '$52#( &,(.$5.25$%#<( I)"( #$.D( $,,#,,*#+%(&+.52,&'#( )I( $55(&%#*,/($+<( %D#($5;D$( '$52#( &,( "#3

.$5.25$%#<(1B( "#*)'&+K(#$.D( &%#*()I( %D#( $,,#,,*#+%,8( T)( &I($+( $,,#,,*#+%(D$,( +(%#,%( &%#*,/( +( e( W(

$5;D$('$52#,(C&55(1#(.$5.25$%#<(I)"(%D$%($,,#,,*#+%8(GI(%D#"#($"#($(%)%$5()I(*(2+&R2#($,,#,,*#+%(G6,(&+(

%D#( <$%$/( *( M( =+( e( W@( $5;D$( '$52#,( C&55( 1#( .$ 5.25$%#<8( Q+.#( %D#( .$5.25$%&)+( &,( I&+&,D#</( $55( )I( %D#,#(

'$52#,(C&55(1#('&,2$5&H#<(%)K#%D#"8((

0+(#M$*;5#()I(%D#('&,2$5&H$%&)+(&,(,D)C+(&+(V&K2"#(W8(LD#(`3$M&,()I(%D#(;5)%(,D)C,(%D#(0,,#,,*#+%(G68(

LD#(?3$M&,()I(%D#(;5)%("$+K#,(I")*(9(%)(W($+<(;)&+%,(%D#(!")+1$.DZ,($5;D$(I)"(%D#($,,#,,*#+%()+(%D#(

`3$M&,8(G+(%D#('&,2$5&H$%&)+/(#'#"B($,,#,,*#+%(&,("#;"#,#+%#<($,($('#"%&.$5(5&+#(C&%D(*25%&;5#(<)%,8(LD#(

<)%,($"#()I(%C)(.)5)",F(152#($+<(15$.N8(LD#(152#(<)%,(,D)C(%D#($5;D$()I($,,#,,*#+%(C&%D)2%(<");;&+K(

$+B( &%#*/( CD&5#( %D#( <&II#"#+%( 15$.N( <)%,( ,D)C( $5;D$( )I( %D#( $,,#,,*#+%( $I%#"( <");;&+K( )+#( )I( %D#(

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

W(D%%;F[[K&%D218.)*[+&"*$5;$%#5[#<'&H3b9WX(

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

&%#*,8(0( "#<( D)"&H)+%$5(5&+#( $%( ?(f(98]( &,( <"$C+( $,($("#I#"#+.#($5;D$('$52#( %D$%( &,( K#+#"$55B(,##+($,(

$..#;%$15#8(LD#($,,#,,*#+%,()+(%D#(`3$M&,( $"#( ,)"%#<( I")*( %D#( D&KD#,%(%)( 5)C#,%($5;D$( '$52#,(%D$%(

C#"#( .$5.25$%#<( 1B( 2,&+K( $55( )I( %D#( $,,#,,*#+%( &%#*,8( LD#( '&,2$5&H$%&)+( &,( $5,)( &+%#"$.%&'#/( $+<($,(

,D)C+(&+(V&K2"#(b/(D)'#"&+K()'#"($+B(<)%(C&55(%#55(2,(CD&.D(&%#*(%D#(<)%("#I#",(%)/($+<(D)C(%D#($5;D$(

)I(%D#($,,#,,*#+%(&,($II#.%#<(1B("#*)'&+K(%D#(,#5#.%#<(&%#*8(

E#BA*/'9:';#7$*#8A$#6='6<'3*6=8&F)G7'&-?)&'5)/='&='#$/+'5#$)#='&='&77/77+/=$'#7'"*6??/"H'1)/'

?6#=$7'#='8-A/'F6-6*'7#B=#<%'3*6=8&F)G7'&-?)&'<6*'$)/'6C/*&--'&77/77+/=$'&="'$)/'?6#=$7'#='8-&FI'

7#B=#<%'3*6=8&F)G7'&-?)&'5)/='6=/'6<'$)/'&77/77+/=$'#$/+7'#7'"*6??/"H''

E#BA*/'J:'1)/'&77/77+/=$'&="'#$/+'#=<6*+&$#6='7)65'A?'5)/='5/')6C/*'6C/*'$)/'"6$7'6<'$)/'?-6$H''

5* DISCUSSION*

LD#('&,2$5&H$%&)+()I(!")+1$.DZ,($5;D$(I)"(*$+B($,,#,,*#+%,($%()+.#(;")'&<#,(2,($('#"B(#$,B(%)(K"$,;(

)'#"'&#C( )I( %D#( $,,#,,*#+%( R2$5&%B(<$%$8( \$%D#"( %D$+( g2,%( ,##&+K( $( <&,%"&12%&)+( =)"( D&,%)K"$*@( )I(

!")+1$.DZ,($5;D$(I)"( $( ,#%()I($,,#,,*#+%,/( %D#( '&,2$5&H$%&)+(#+$15#,(2,( %)( ,##(D)C( <&II#"#+%( &%#*,(

$"#(&*;$.%&+K(%D#()'#"$55($5;D$,()I(%D#(%#,%,8((

!"#$%&'"&()*"+,,-'&./(012(3&1,*&%1'"&%4(!"&5,*,&+,("&(6,%*&'&.(7&%481'+/(9(:&";4,-.,(<67:=0>(

!"#$%&'#(!)**)+,(-&.#+,#/(0%%"&12%&)+(3(4)+!)**#".&$534)6#"&',(789(:+;)"%#<(=!!(>?34!346(789@(

(

LD#(I&",%(I#$%2"#(%)(+)%&.#(&,(CD#"#(%D#(152#(<)%()"(%D#()'#"$55($5;D$()I(%D#($,,#,,*#+%(5&#,8(GI(%D&,(&,(

1#5)C( 98]( )"( $+)%D#"( #M;#"%( <#I&+#<( %D"#,D)5</( %D#( %#,%( *$B( 1#( )I( .)+.#"+8( 0+)%D#"( +)%&.#$15#(

I#$%2"#()I(%D#( '&,2$5&H$%&)+(&,(D)C( .5),#( $55()I(%D#(15$.N( <)%,( $"#(I)"($+( $,,#,,*#+%8(GI(%D#(<)%,( $"#(

'#"B( .5),#( %)K#%D#"/( %D&,( %D$%( %D#( "#*)'$5( )I( $( ,&+K5#( &%#*( *$B( +)%( $II#.%( %D#( $5;D$( )I( %D#( %#,%(

,&K+&I&.$+%5B8(GI( %D#( 15$.N( <)%,( )I( *25%&;5#( &%#*,( .)&+.&<#/( &%( *#$+,( %D$%( %D#( .)&+.&<&+K( &%#*,(

.)+%"&12%#(,&*&5$"5B(%)(%D#(,.$5#(&+(%#"*,()I(.)+,&,%#+.B8(

LD#(+#M%(&*;)"%$+%(I#$%2"#()I(%D#(;5)%(&,($,,#,,*#+%,(C&%D(C&<#('$"&$%&)+(&+(%D#&"($5;D$('$52#,(CD#+(

&%#*,( $"#( "#*)'#<8( T)* #( 15$.N( <)%,( &+( %D#( ;5)%( K)#,( D&KD( $1)' #( %D#( 152#( <)%,/( CD&5#( ,)*#( 15$.N(

<)%,(K)(C#55(<)C+(1#5)C8(LD#(15$.N(<)%,(K)&+K($1)'#(%D#(152#(<)%(%#55(2,(%D$%("#*)'$5()I($+(&%#*(.$+(

&+."#$,#(%D#($5;D$()I(%D#($,,#,,*#+%/(CD&5#(%D#(15$.N(<)%,(K)&+K(1#5)C(%D#(152#(<)%(%#55(2,(%D$%(&I(C#(

"#*)'#(,)*#(&%#*,()I(%D#($,,#,,*#+%/(%D#($5;D$('$52#(.$+(<#."#$,#8((

G+(V&K2"#(W/( C#( .$+(,##(%D$%()+(%D#(%);(5#I%(%D#(W7%D($,,#,,*#+%(I")*(%D#(5#I%(D$,()+#(&%#*(%D$%(.$+(

1#("#*)'#<( %)(&+."#$,#(%D#( )'#"$55( $5;D$/(CD&5#("#*)'$5()I( $+)%D#"( &%#*( .$+(<#."#$,#(%D#($5;D$8(

LD#"#($"#($(I#C(*)"#($,,#,,*#+%,(1#%C##+($+($5;D$('$52#()I(98Y($+<(98]( %D$%(.$+(1#(&*;")'#<( 1B(

"#*)'&+K(.#"%$&+(&%#*,8((

6* CONCLUSION*

LD&,(<#,."&1#<('&,2$5&H$%&)+(.$+(1#(2,#<(1B(&+,%"2.%&)+$5(<#,&K+#",(%)(&<#+%&IB($,,#,,*#+%,(%D$%(+##<(

$%%#+%&)+8( P2$5&%B( %#,%,( $"#( $( '#"B( &*;)"%$+%( %))5( %)( K$2K#( ,%2<#+%( 2+<#",%$+<&+K()I( %D#( 5#$"+&+K(

*$%#"&$5/( $+<( D$'&+K( $( %#,%( C&%D( D&KD( .)+,&,%#+.B( .$+( ;")'&<#( &+,%"2.%)",( C&%D( *)"#( "#5&$15#(

#,%&*$%#,( )I( ,%2<#+%( *$,%#"B8( LD&,("#5&$1&5&%B( .$+( 5#$<( %)( $( 1#%%#"( $1&5&%B( %)( 2,#( <$%$( %)( <"&'#(

&+,%"2.%&)+8((

7* FUTURE*WORK*

LD#( '&,2$5&H$%&)+( &,( ,%&55( 5&*&%#<( &+( &%,( $1&5&%B( %)( ,D)C( )%D#"( "#5$%#<( &%#*( $+$5B,&,( *#%"&.,( ,2.D( $,(

;#".#+%(.)""#.%($+<( ;)&+%(1&,#"&$58( LD#,#( .$+( ;")'&<#( 2,#",( C&%D( *)"#(&+I)"*$%&)+($1)2%(D)C(C#55(

&%#*,(&+(%D#(;5)%($"#(;#"I)"*&+K8(0+)%D#"(I#$%2"#(%D$%(&,(*&,,&+K(&+(%D&,('&,2$5&H$%&)+(&,(,D)C&+K(D)C(

%D#("#*)'$5( )I( *25%&;5#(&%#*,( .$+( $II#.%(%D#()'#"$55( $5;D$8(05%D)2KD( %D&,( &,( *)"#(.)*;5#M( $,( %D#"#(

$"#(.)*1&+$%)"&$5(;),,&1&5&%&#,(CD#+("#*)'&+K($(.)*1&+$%&)+()I(&%#*,8(>2%($<<&+K(%D#,#(I#$%2"#,(%)(

%D#(;5)%(*&KD%(D#5;($,,#,,*#+%(<#,&K+#",(%"$.N(<)C+(;")15#*$%&.(&%#*,(&+($(*)"#(1#%%#"(C$B8((

REFERENCES**

!)"%&+$/(h8(U8(=WXX7@8(_D$%(&,( .)#II&.&#+%($5;D$i(0+( #M$*&+$%&)+()I(%D#)"B($+<($;;5&.$%&)+,8(?"@*&%4(

"5(%$$4',-($/8+2"4".8/(AB=W@/(Xc8(

!")+1$.D/( -8( h8( =WXYW@8( !)#II&.&#+%( $5;D$( $+<( %D#( &+%#"+$5( ,%"2.%2"#( )I( %#,%,8()/8+2"#,1*'C%/(=D=7@/(

bX]377d8(

\#'#55#/( _8( =b9W]@( ;,B.DF( A").#<2"#,( I)"( A#",)+$5&%B( $+<( A,B.D)5)K&.$5( \#,#$".D/( 4)"%DC#,%#"+(

:+&'#",&%B/( j'$+,%)+/( G55&+)&,/( :T0/( D%%;,F[[!\048\3;")g#.%8)"K[;$.N$K#f;,B.D(O#",&)+( f(

W8]8c8(

ResearchGate has not been able to resolve any citations for this publication.

TRESTLE: A Model of Concept Formation in Structured Domains

Article

Full-text available

Jun 2016

The literature on concept formation has demonstrated that humans are capable of learning concepts incrementally, with a variety of attribute types, and in both supervised and unsupervised settings. Many models of concept formation focus on a subset of these characteristics, but none account for all of them. In this paper, we present TRESTLE, an incremental account of probabilistic concept formation in structured domains that unifies prior concept learning models. TRESTLE works by creating a hierarchical categorization tree that can be used to predict missing attribute values and cluster sets of examples into conceptually meaningful groups. It updates its knowledge by partially matching novel structures and sorting them into its categorization tree. Finally, the system supports mixed-data representations, including nominal, numeric, relational, and component attributes. We evaluate TRESTLE's performance on a supervised learning task and an unsupervised clustering task. For both tasks, we compare it to a nonincremental model and to human participants. We find that this new categorization model is competitive with the nonincremental approach and more closely approximates human behavior on both tasks. These results serve as an initial demonstration of TRESTLE's capabilities and show that, by taking key characteristics of human learning into account, it can better model behavior than approaches that ignore them.

RumbleBlocks: Teaching science concepts to young children through a Unity game

Conference Paper

Full-text available

Jul 2012

RumbleBlocks was developed at the Entertainment Technology Center (ETC) to teach engineering principles of tower stability to children ages 4-7. The game features tower construction, tower piece removal, and tower comparison levels which were designed with feedback from early childhood educators and learning researchers, and iteratively improved with feedback from child play tests. This paper emphasizes the development process, and initial formative play tests with children. It was developed using the Unity3D game engine, allowing for export as a stand-alone application, web player, or to mobile devices. First results are promising in terms of educational effectiveness, with more studies planned for the future.

Using extracted features to inform alignment-driven design ideas in an educational game

Conference Paper

Apr 2014

As educational games have become a larger field of study, there has been a growing need for analytic methods that can be used to assess game design and inform iteration. While much previous work has focused on the measurement of student engagement or learning at a gross level, we argue that new methods are necessary for measuring the alignment of a game to its target learning goals at an appropriate level of detail to inform design decisions. We present a novel technique that we have employed to examine alignment in an open-ended educational game. The approach is based on examining how the game reacts to representative student solutions that do and do not obey target principles. We demonstrate this method using real student data and discuss how redesign might be informed by these techniques.

Knowledge Acquisition Via Incremental Conceptual Clustering

Article

Sep 1987

Doug Fisher

Conceptual clustering is an important way of summarizing and explaining data. However, the recent formulation of this paradigm has allowed little exploration of conceptual clustering as a means of improving performance. Furthermore, previous work in conceptual clustering has not explicitly dealt with constraints imposed by real world environments. This article presents COBWEB, a conceptual clustering system that organizes data so as to maximize inference ability. Additionally, COBWEB is incremental and computationally economical, and thus can be flexibly applied in a variety of domains.

What Is Coefficient Alpha? An Examination of Theory and Applications

Article

Feb 1993

Jose Cortina

Psychological research involving scale construction has been hindered considerably by a widespread lack of understanding of coefficient alpha and reliability theory in general. A discussion of the assumptions and meaning of coefficient alpha is presented. This discussion is followed by a demonstration of the effects of test length and dimensionality on alpha by calculating the statistic for hypothetical tests with varying numbers of items, numbers of orthogonal dimensions, and average item intercorrelations. Recommendations for the proper use of coefficient alpha are offered. (PsycINFO Database Record (c) 2012 APA, all rights reserved)

D-3: Data-Driven Documents

Article

Dec 2011
IEEE T VIS COMPUT GR

Data-Driven Documents (D3) is a novel representation-transparent approach to visualization for the web. Rather than hide the underlying scenegraph within a toolkit-specific abstraction, D3 enables direct inspection and manipulation of a native representation: the standard document object model (DOM). With D3, designers selectively bind input data to arbitrary document elements, applying dynamic transforms to both generate and modify content. We show how representational transparency improves expressiveness and better integrates with developer tools than prior approaches, while offering comparable notational efficiency and retaining powerful declarative components. Immediate evaluation of operators further simplifies debugging and allows iterative development. Additionally, we demonstrate how D3 transforms naturally enable animation and interaction with dramatic performance improvements over intermediate representations.

Web--based statics course with learning dashboard for instructors, Proceedings of Computers and Advanced Technology in Education

Jan 2012

A Dollár
P Steif

Dollár, A., Steif, P. (2012). Web--based statics course with learning dashboard for instructors, Proceedings of Computers and Advanced Technology in Education (CATE 2012), Napoli

Teaching Advanced Web Technologies with a Mobile Learning Companion Application

Nov 2017

C Krauss
A Merceron
T.--S An
M Zwicklbauer
S Steglich
S Arbanowski

Krauss, C., Merceron, A., An, T.--S., Zwicklbauer, M., Steglich, S., Arbanowski, S. (2017). Teaching Advanced Web Technologies with a Mobile Learning Companion Application. Proceedings of the 16th ACM World Conference on Mobile and Contextual Learning (mlearn 2017), ACM, Ont. 30 --Nov. 1, 2017, Larnaca, Cyprus, doi>10.1145/3136907.3136937

Proceedings of the First Workshop on Educational Data Visualization (EdViz-2019)

Abstract and Figures

Recommendations

Sweet Sensations: Exploring the Correlations between Classical Harmonic Ratios and Emotional Communication in Vibrotactile Feedback

Feel The Vibe: Whole body Aesthetic Vibration Experiences

The Effect of Poverty on Learning at Scale

Aesthetic Resonance

Expanding Genetic Code Expansion through Resource Facilities, Workshops, and Conferences

DSLDI 2014: second workshop on domain specific languages design and implementation

Public Participation and the Environment: Do We Know What Works?

Connect and Strive to Survive and Thrive: The Evolutionary Meaning of Communion and Agency