ArticlePDF Available

Effects of instructional conditions and experience on the adoption of a learning tool

Authors:

Abstract and Figures

This paper presents the results of a natural experiment investigating the effects of instructional conditions and experience on the adoption and sustained use of a learning tool. The experiment was conducted with undergraduate students, enrolled into four performing art courses (N=77) at a research intensive university in Canada. The students used the video annotation software CLAS for course‐based self‐assessment on their performances. Although existing research offers insights into the factors predicting students' intentions of accepting a learning tool, much less is known about factors that affect actual adoption and sustained tool use. The study explored the use of CLAS amongst undergraduate students in four courses across two consecutive semesters. Trace data of students' tool use, graph‐based measures of metacognitive monitoring, and text cohesion of video annotations were used to estimate the volume of tool use and the quality of the learning strategy and learning products created. The results confirmed that scaffolding (e.g., graded activity with instructional feedback) is required to guide students' initial tool use, although scaffolding did not have an independent significant effect on the quantity of tool use. The findings demonstrated that the use of the tool is strongly influenced by the experience an individual student gains from scaffolded conditions. That is, the students sustained their use of the learning tool in future courses even when the tool use was not graded nor was instructional feedback provided. An important implication is that students' tool use is not solely driven by motivation – rather, it is shaped by instructional conditions and experience with the tool use.
Content may be subject to copyright.
1
Effectsofinstructionalconditionsandexperienceon
theadoptionofalearningtool
DraganGasevic1,2,NeginMirriahi3,ShaneDawson4,SreckoJoksimovic1
1MorayHouseSchoolofEducation,UniversityofEdinburgh,UnitedKingdom,
HolyroodRoad,EdinburghEH88AQ,UnitedKingdom
2SchoolofInformatics,UniversityofEdinburgh,UnitedKingdom,
10CrichtonStreet,Edinburgh,MidlothianEH89LE,UnitedKingdom
3LearningandTeachingUnit,UniversityofNewSouthWales,
Level4,MathewsBuilding(F23),ViaGate11,BotanyStreet,,Sydney,NSW2052,Australia
4TeachingInnovationUnit,UniversityofSouthAustralia,
160CurrieSt,Adelaide,SA5000,Australia
Abstract:Thispaperpresentstheresultsofanaturalexperimentinvestigatingtheeffectsof
instructionalconditionsandexperienceontheadoptionandsustaineduseofalearningtool.The
experimentwasconductedwithundergraduatestudents,enrolledintofourperformingartcourses
(N=77)ataresearchintensiveuniversityinCanada.Thestudentsusedthevideoannotationsoftware
CLASforcoursebasedselfassessmentontheirperformances.Althoughexistingresearchoffersinsights
intothefactorspredictingstudents’intentionsofacceptingalearningtool,muchlessisknownabout
factorsthataffectactualadoptionandsustainedtooluse.ThestudyexploredtheuseofCLASamongst
undergraduatestudentsinfourcoursesacrosstwoconsecutivesemesters.Tracedataofstudents’tool
use,graphbasedmeasuresofmetacognitivemonitoring,andtextcohesionofvideoannotationswere
usedtoestimatethevolumeoftooluseandthequalityofthelearningstrategyandlearningproducts
created.Theresultsconfirmedthatscaffolding(e.g.,gradedactivitywithinstructionalfeedback)is
requiredtoguidestudents’initialtooluse,althoughscaffoldingdidnothaveanindependentsignificant
effectonthequantityoftooluse.Thefindingsdemonstratedthattheuseofthetoolisstrongly
influencedbytheexperienceanindividualstudentgainsfromscaffoldedconditions.Thatis,the
studentssustainedtheiruseofthelearningtoolinfuturecoursesevenwhenthetoolusewasnot
gradednorwasinstructionalfeedbackprovided.Animportantimplicationisthatstudents’tooluseis
notsolelydrivenbymotivationrather,itisshapedbyinstructionalconditionsandexperiencewiththe
tooluse.
Keywords:learningtechnologyadoption;instructionalscaffolding;selfregulatedlearning;learning
analytics
2
1 Introduction
Despitethemanyreportedbenefitsoftechnologyforfacilitatingstudentlearningandengagement
(Chen,Lambert,&Guidry,2010;LópezPérez,PérezLópez,&RodríguezAriza,2011),studieshave
identifiedthatagreatmajorityofstudents(above60%)canbeclassifiedaslimitedlearningtechnology
users(Lust,Elen,&Clarebout,2013;Lust,JuarezCollazo,Elen,&Clarebout,2012).Theobservedlimited
useisnotsimplyafunctionofpoorcoursedesign.Theauthorsalsonotedalackofstudentengagement
withtechnologiesevenwhenlearningtoolsarespecificallyembeddedintocoursedesignsthatfollow
pedagogicallysoundandempiricallyvalidatedprinciples(Lustetal.,2013,2012).Essentially,thereisan
educationalchallengetofirstmotivatestudentstoacceptthelearningtoolandsecond,tosustaintheir
useofit.
Muchresearchhasbeenundertakentounderstandtheconditionsforpromotingstudentacceptance
andlongtermadoptionoflearningtools(Cheung&Vogel,2013;Edmunds,Thorpe,&Conole,2012;
EscobarRodriguez&MongeLozano,2012).Mostprominentinthisareahasbeenthetechnology
acceptancemodel(TAM).TheTAMwasfirstproposedbyDavis(1989)andcomprisesoftwoprimary
factorsthatareperceivedtocontributetotechnologyadoption:perceivedeaseofuseandperceived
usefulness(Sánchez&Hueros,2010).Theexplanatorypowerofthismodelisfurtherextendedwhen
additionalconstructsareincorporatedsuchasselfefficacy,enjoyment,andlearninggoalorientation.
TheseadditionalconstructsprovideadditionalexplanatorypowerbeyondthatofTAMtobetter
understandstudentuseoftechnicalsystems(Yi&Hwang,2003).However,whilethereismuchtolearn
fromthesestudies,theadoptedconstructstendtoexplainfactorsinfluencingstudents’intentionsto
acceptusinglearningtools,ratherthantheiractualadoption(Clarebout,Elen,Collazo,Lust,&Jiang,
2013).Moreover,thereislimitedunderstandingoftheconditionsandpedagogicalapproachesthat
sustainstudents’useofeducationaltechnology,especiallywhentheuseofacertaintoolisoptionaland
notassessed.
1.1 LearningToolUseasSelfRegulatedLearning
Contemporaryresearchinvestigatingstudentuseofeducationtechnologiesisincreasinglysituated
withinthecontextofselfregulatedlearning(SRL)(Trevors,Duffy,&Azevedo,2014).Asoriginally
suggestedbyAzevedo(2005),SRLprovidesarobusttheoreticalframeworktoinformthestudyof
technologyenabledlearning(incomputerbasedlearningenvironments).Inthispaper,wehave
adoptedSRLasaframeworktounderstandtheconditionsthatsustainstudents’useofalearning
technology.Inparticular,weincorporatedWinneandHadwin’smodelofselfregulatedlearning(Winne,
2006;Winne&Hadwin,1998).Thismodelconsidersfiveelements:conditions,operations,products,
evaluationsandstandards(COPES)thatcollectivelyinfluenceselfregulatoryprocessesoflearning
(Winne,1996).AccordingtotheCOPESmodel,learnersusetools(cognitive,digitalorphysical)to
operateonrawinformation(e.g.,watchingvideorecordingsofalecture)inordertoconstructproducts
oftheirlearning(e.g.,recallofinformationintroducedinthevideorecordings).Toregulatetheir
learningprocess,studentsevaluatetheproductsoftheirlearning(e.g.,qualityoftheirrecall)andthe
effectivenessoftheirlearningstrategiesaccordingtointernal(e.g.,whethervideowatchingresultsin
satisfyinginformationrecallwithinthetimebudgetedforlearning)orexternalstandards(e.g.,whether
theyreceivedapassingmarkonaquizthataccompaniedthevideo).Consistentwithmodern
3
educationalpsychology,theWinneandHadwinmodel(1998)deemslearnersasactiveagentsinthe
learningprocess.Asactiveandconstructiveparticipants,learnersmonitortheirlearningandchoosethe
toolstheyaregoingtoadoptandthestandardstheywillfollowtoevaluatetheproductsoftheir
learning(Winne,1996)asapartoftheirmetacognitivecontrolandmonitoring.Thisdecisionmaking
processisbasedoninternal(e.g.,experiencewithtools,epistemicbeliefs,andpriorknowledge)and
external(e.g.,tasksmandatingtheuseofatool)conditions(Winne,2011;Winne&Hadwin,1998).
Thus,certainconditionsarerequiredforlearnerstoselectandregularlyuseaparticularlearningtool.
AspreviouslypositedbyWinne(2006)andempiricallyvalidatedinseveralstudiesconductedby
Clareboutetal.(2013),therearegenerallyfourmainconditionsthatinfluencelearners’decisions
regardingtoolselectionanduse.First,learnersneedtobeawareofthevalueofthetoolandits
availabilityintheirlearningenvironment.Second,learnersneedtorecognizethatthetoolcanbe
appliedtothespecifictaskathand.Third,evenifthelearnersarecognizantofthebenefitsofthetool
fortheassignedtask,theyneedtohavesufficientskillstoutilisetheselectedtooleffectively.Finally,
learnersneedsufficientmotivationtoinvestthetimenecessarytousethetool.Theseconditionscan
explainwhycertaintoolsarenotalwaysadoptedbylearnersdespitehavingapositivepriorexperience
(Sarfo,Elen,Clarebout,&Louw,2010).Inthiscontext,Clareboutetal.(2013)proposedthatlearners
firstneedtohavesomepriorexperiencewithatoolbeforetheirconceptionsofitcanbeusedasa
predictoroffutureuse.
1.2 InstructionalConditionsfortheSustainedUseofaLearningTool
Inthispaper,weacceptandextendClareboutetal.’s(2013)propositiontofurthersuggestthatfora
tooltohavesustaineduse,learnersmustfirstbeexposedtothelearningtool;andsecond,gainalevel
ofproficiencyinitsuse.Intheabsenceofanypreviousexperiencewithatoolorifalearnerisonly
familiarwithitinalternatecontexts(i.e.,transferacrosscontextscanbechallenging(Perkins,1985)),it
isunlikelythatlearnerswillbeabletorecognizethevalueofthetool.Thatis,twooftheconditions
suggestedbyWinne(2006),valueandawarenessofatool,arenotmet.Wepositthatinordertomeet
theseconditionsandfacilitatelearners’abilitytoeffectivelyuseatool,alevelofscaffoldingisrequired
toguidelearnersintheirinitialuseofthetoolandhowitcanbeappliedtoaparticularlearningtask
(Azevedo&Hadwin,2005;Beed,Hawkins,&Roller,1991).Theeffectsoftheinstructionalconditionson
learners’decisionmakingandtechnologyacceptanceiswelldocumentedintheliterature(Azevedo,
Moos,Greene,Winters,&Cromley,2008;Cho&Kim,2013;Garrison&ClevelandInnes,2005;McGill&
Klobas,2009;Trigwell,Prosser,&Waterhouse,1999).Basedonthisliterature,wesuggestthata
learner’sinitialexperiencewithatool,should:
haveatleastonetaskwheretheuseofthetoolisrequiredtocompleteacoursetaskandthe
taskassessed(mandatedinthecoursedesign);and
beaccompaniedwithguidanceandfeedbackonhowthestudentcanusethetooleffectivelyin
ordertocompletetheassignedlearningtasks.
Toestablishasustainedlevelofuseofaparticulartool,additionalconditionsneedtobemet.First,as
recognizedbytheresearchoneducationaltechnologyacceptanceandillustratedbyTAM,learnersneed
toperceivethetoolaseasytouseandusefulinordertopreservetheirintentiontousethetoolinthe
4
future(Sánchez&Hueros,2010).Thisisparticularlyimportantwhentheuseofatoolisoptional.In
otherwords,atoolmustbeintuitivetousewithoutanextensivelearningcurveorextraneouscognitive
loadthatcouldcreateanaddedlayerofcomplexityimpedingastudent’sabilitytocompleteanassigned
task(Devolder,vanBraak,&Tondeur,2012;Kirschner,Sweller,&Clark,2006).
Second,learnersneedtobeabletotransfertheuseofatooltonewcontexts(Salomon&Perkins,
1989).AssuggestedbyWinne(2006),learnersneedtobeabletorecognizewhenatoolcanbe
appropriatelyappliedtocompleteanewtask.Ifastudent’spreviousexperiencewithatoolissimilarto
thenewtask,theyaremorelikelytoadoptthetoolagaintocompletetherequestedtask.However,if
thecontextissignificantlydifferent,thenthestudent’sselectionofthesametechnologyislesslikely.
Winne(2006)describesthisasamediationdeficiency.Thatis,asituationwhenlearnersare“unableto
assemblebridginginformationbetweentoolsandtobelearnedinformation”(Winne,2006,p.7).The
studyreportedinthispaperfocusesonthesustainedtooluseinsimilartasksratherthanonthetransfer
acrossdifferentcontexts.
1.3 MeasurementoftheUseofEducationalTechnology
Studiesontheadoptionandeffectsofeducationaltechnologyonselfregulatedlearninghaveprimarily
beenbasedonmeasuresoflearneroperations,asdefinedintheCOPESmodel(Winne,1996,2006).
Thesemeasureshavetendedtorelyonlearners’selfreportsoftheirperceptions;use,anddegreeof
useofaparticulartoolorlearningapproach(Clareboutetal.,2013;Lustetal.,2012;Sánchez&Hueros,
2010;Yi&Hwang,2003).WhileSRLstudieshaveoftenreliedonselfreportmethodologies(e.g.,think
aloudprotocolsandsurveys),alternateoptionsarerapidlyemergingsuchastheanalysisofcaptured
tracedatafromlearners’interactionswitheducationaltechnology(Azevedo,2015).Theanalysisoftrace
datatoinformlearning,teaching,andresearchhasrecentlyamplifiedduetoagrowinginterestinthe
fieldsoflearninganalyticsandeducationaldatamining(Baker&Yacef,2009;Gašević,Dawson,&
Siemens,2015;Gasevic,Mirriahi,Long,&Dawson,2014;Siemens&Gašević,2012).Typicalmeasures
derivedfromalearner’stracedataincludethefrequencyandtimespentonthevariousoperations
performedwithlearningtechnologies.Forexample,inthecontextofselfregulatedlearning,Choand
Shen(2013)foundthatastudent’sabilitytoregulatesocialinteractionwithothers(Cho&Jonassen,
2009)wasasignificantpredictoroftheamountoftimespentonlineinalearningmanagementsystem
(LMS).Similarly,Jeske,Backhaus,andStamovRoßnagel(2014)notedthattracebasedvariables,suchas
timespentandfrequencyofnavigationthroughasequenceofresourcesinanonlinelesson,weresound
proxiesofmotivationandselfregulationstrategiesthatmediatedtheassociationbetweenlearning
experienceandtestperformanceinancontrolledexperiment.
Todate,therehasbeenlimitedresearchthathasinvestigatedtheeffectsofconditionsassociatedwith
theCOPESmodel(Winne,1996,2006)ontechnologyuseandacceptance.Whileinternalconditions
havebeenstudied,suchasselfefficacy,goalorientation,andpriorknowledge(Cho&Shen,2013;
Clareboutetal.,2013;Jeskeetal.,2014),theeffectsofexternalconditionshavereceivedlimited
researchattention.AspositedintheCOPESmodel(Winne,1996,2006),externalconditions(e.g.
gradingoflearners’selfassessmentsorsharingtheselfassessmentswithpeers)canhavesignificant
effectsonthestandardsthelearnersusetoevaluatetheproductsoftheirlearningandthelearning
strategiestheychosetoapply.While,thequantityoftheproductsoflearningandtheadoptedlearning
5
strategiesmayremainconsistent,thequalityoflearningcanbemarkedlydifferent.Forexample,inan
onlinesoftwareengineeringcourse,Gašević,Adesope,Joksimović,andKovanović(2015)demonstrated
thatthequalityofalearner’sdiscourse(operationalizedascognitivepresence)significantlyimproved
afterchangestotheinstructionaldesignandresourceshadbeenmadetoincludescaffoldinglearner
participationinadiscussionforum.However,theauthorsnotedthatthequantityofthediscussion
remainedatthesamelevelaspreviouscourseiterations.Similarly,Kuhn(1995)suggestedthatlearners
donotincreasetheirusageofanewlyacquiredlearningstrategy1,butratherapplythisstrategyina
moreeffectivemanner.Inotherwords,whenastrategyiseffectivelyapplied,thequantityremains
consistentwhilethequalityofthelearningproductincreases(Malmberg,Järvelä,&Kirschner,2014).
Hence,wepositthat:
i) theinstructionalconditionsprovidelearnerswithanopportunitytoexperiencean
educationaltechnology(oritstools).Thisexperienceinfluencesalearner’smotivationtouse
thistechnologyinfuturesimilarlearningcontexts;
ii) theinstructionalconditionsinfluencesboththequalityoflearningproductscreatedandhow
theoperations(i.e.,strategy)manifest.
1.4 ResearchQuestions
Toaddressthepropositionsoutlinedabove,thispaperreportsontheresultsofanempiricalstudythat
aimedtoaddressthefollowingresearchquestions:
RQ1. Whatistheeffectoftheinstructionalconditionsatthecourselevel(assessedvs.nonassessed)on
students’extentofuseofalearningtoolintermsofcountofannotationsproduced,qualityof
learningproductscreated,andlearningstrategyfollowed?
RQ2. Whatistheeffectofastudents’priorexperiencewithalearningtoolontheirfutureadoptionin
termsofcountsofannotationsproduced,qualityoflearningproductscreated,andlearning
strategyfollowedunderdifferentinstructionalconditions?
Thesequestionsaimtoinvestigatetheeffectsofdifferentinstructionalconditionsandexperienceon
thethreemaindimensions:i)countofannotationscreated,ii)thequalityoflearningproductscreated,
andiii)learningstrategyadoptedintheuseofvideoannotationsoftwareforselfassessmentpurposes.
Therefore,bothresearchquestionsareoperationalizedaccordingtothesethreeresearchdimensions
andtheresultssectionisorganizedaccordingly.
2 Method
2.1 StudySetting
Theresearchdesigncanbedescribedasanaturalexperiment(Dunning,2012).Thisapproachwas
drivenbysituatingthestudywithinthecontextofthecoursesavailableforstudentenrolmentata
researchintensivehighereducationinstitutioninNorthAmerica.Assuch,theresearchershadno

1AccordingtoMalmberg,Järvelä,&Kirschner(2014,p.4),alearningstrategyisdefinedas“acoordinatedsetof
studytacticsthataredirectedbyalearninggoal,andthatareaimedatacquiringanewskillorgaining
understanding(Alexander,Graham,&Harris,1998;Weinstein,1988;Winne,2001;Zimmerman,1998)”.
6
controlovertheexperimentalassignmentofthestudyparticipants.Rather,thestudywasconductedin
anecologicallyvalidsettingwherebytheassignmenttotheexperimentalconditionswasperformed
throughtheparticipants’enrolmentinthecoursesusedinthestudy.Thetracedataloggedbythevideo
annotationsoftwarecalledtheCollaborativeLectureAnnotationSystem(CLAS)(Mirriahi&Dawson,
2013;Risko,Foulsham,Dawson,&Kingstone,2013),wereincorporatedforfurtheranalysis.The
experimentalconditionsweredeterminedbytheinstructionalconditionsassociatedwitheachofthe
coursesinvolvedinthestudy.
2.2 Materials
ThestudyparticipantsusedCLAS,awebbasedapplicationforannotatingvideosofstudent
performances.ThedesignofCLASextendsotherpreviouslyestablishedvideoannotationsoftwaresuch
asMicrosoftResearchAnnotationSystem(MRAS)(Bargeron,Gupta,Grudin,&Sanocki,1999),Media
AnnotationTool(MAT)(Colasante&Fenn,2009),andDigitalVideoDigitalUniversity(DiViDu)(Hulsman,
Harmsen,&Fabriek,2009).Informedbythedesignofpriorvideotoolsandassociatedresearchinthis
domainresultedintheCLASsoftwarebeingperceivedbylearnersaseasytolearn,easytouse,and
usefulfortheirstudies(Riskoetal.,2013).ThisvalidationbyRiskoetal.ofthesoftware’sperceivedease
ofuse,easeoflearningandperceivedusefulnessaddressedanimportantpropositionvoicedinthe
theoreticalbackgroundforthisstudy.
Intermsofuserfunctionality,CLAShastwoformsofannotationfeatures:i)timestampedannotations
thatofferstudentsandinstructorswithopportunitiestocreatetimestampednotesthatareassociated
toaspecificpartofavideo;thesenotescanbeaccessedlaterforreview(Dawson,Macfadyen,Risko,
Foulsham,&Kingstone,2012;Riskoetal.,2013);ii)generalannotationsthatarenotassociatedtoany
specificpartofavideo,butallowuserstopostageneralnoteorsummaryofthevideo(ibid).Bothtypes
ofannotationscanbeeitherprivateorcollaborativeofferinganopportunitytoshareannotations
amongpeers.Furthermore,CLAShasafeatureforvisualizingthepositionoftimestampedannotations.
CLAShasadditionalfeaturesforperformingoperationsonthevideosuchaspauseplaying,resume
playing,rewind,andfastforwardasiscommonforcontemporaryvideoplayers.Alltheseoperations
arerecordedbyCLASinlogfilesandthus,eachoperationperformedinCLAShastimestampedtrace
datathatcanbeusedforresearch(Mirriahi&Dawson,2013).CLASdoesnotallowfordirect
downloadingofthevideos.Hence,studentscanonlyviewandannotatethevideoswhileusingtheCLAS
software.ThisincreasestheaccuracyofthetracedatacollectedbyCLAS,asuserscannotinteractwith
thevideosusedinthespecificcoursesoutsideoftheeducationalsoftware.
2.3 Procedures
Thestudyincludedfourundergraduatecoursesintheperformingartsdisciplineofferedinthe
2012/2013academicyear.Thefirsttwocourses(Course1andCourse2)wereofferedinthefirst
semester(Fall2012),whiletheothertwocourses(Course3andCourse4)wereofferedinthesecond
semester(Winter2013).Inallfourcourses,videorecordingsofstudents’ownperformanceswere
availableforviewingandannotatingthroughtheCLASsoftware.InCourse1,thevideoswereofgroup
performancesandtherewasnogradedrequirementforstudentstomakeanannotationorgeneral
comment.Thevideosinthethreeothercoursesinsteadfocusedonstudents’individualperformances.
7
Thatis,forCourse1,allenrolledstudentsannotatedthesamethreevideos(i.e.,threegroup
performances).Further,asthiswasagroupbasedactivity,thestudentshadanoptiontosharetheir
annotationswiththeirpeers.WhiletheuseofCLASinCourse1wasoptional,inCourse2students’use
ofCLASwasdirectlyassessed(i.e.acourserequirement).Studentshadaccesstoreviewfourvideo
recordingsoftheirindividualperformancesthroughtheCLASsoftware.
Ofthefourcourses,generalannotationsonthreeofthemweregradedandinstructorfeedback
provided.Forthefourthvideo,formativefeedbackonthegeneralannotationwasprovidedonly.
Overall,studentsmadetimestampedannotationsandonegeneralannotationpervideosummarizing
theiroverallreflectionsontheirperformance.Furthermore,Course2wasaprerequisiteforbothCourse
3andCourse4thatwereofferedinthefollowingsemester.Therequirementsandinstructional
conditionsfortheuseofCLASinCourse3wereidenticaltothoseofCourse2includinggradedgeneral
annotations.TheonlydifferencewasthatCourse3includedanadditionalvideorecordedperformance
forstudentstoview,annotate,andsubmitageneralreflectivesummaryonCLAS.Course4wassimilar
toCourses2and3intermsofthenumberofvideorecordingsoftheindividualperformances(i.e.,four
videorecordingswereaccessiblethroughCLAS).However,forCourse4,studentsdidnotreceiveagrade
norformativefeedbackontheirgeneralselfreflectiveannotationspostedintheCLASsystem.The
courserequirementsandrelationshipbetweenthecoursesandthenongraded/gradedgeneral
annotationsarespecifiedinTable1andFigure1.
Table1.Thenumbersofstudentperformancevideosandthetypeofperformancerecordedinthefour
courses
Course NumberofvideosrequestedtoannotateTypeofperformance
Course13Group
Course24Individual
Course35Individual
Course44Individual
Labelsofthegroupsofthestudentscreatedbasedonthecoursestheywereenrolledin:
Course1astudentswhotookcourse1butdidnottakeCourse2(n=23)
Course1bstudentswhotookCourses1and2(n=8)
Course2astudentswhotookCourse2,butdidnottakecourse1(n=32)
Course2bstudentswhotookCourses1and2(n=8)
Course2cstudentswhotookCourse2,butdidnottakeCourse3(n=22)
Course2dstudentswhotookCourse2and3(n=18)
Course2e
studentswhotookCourse2,butdidnottakeCourse3(n=29)
Course2fstudentswhotookCourses2and4(n=11)
Course3astudentswhotookCourse3,butdidnottakeCourse2(n=10)
Course3bstudentswhotookCourses3and2(n=18)
Course4astudentswhotookCourse4,butdidnottakeCourse2(n=9)
Course4bstudentswhotookCourses4and2(n=11)
Course1
(nongraded,social)
Course2
(graded,individual)
Course3
(graded,individual)
Course4
(nongraded,individual)
Semester1Semester2
8
Figure1.Thecoursesincludedinthestudyandthegroupsofstudentsformedbasedontheirenrolment
inindividualcourses
Throughoutthepaper,werefertothenotionofinstructionalconditionswherebywedistinguish
betweennongradedandgradedconditions.Inthegradedcondition,theinstructorslookedatthe
specificityofgoalssetbythestudentsintheirreflections.Specificityofgoalsisanindicatorofstudents’
recognitionofthepointstobeimproveduponintheirfuturework.Anexampleofthesentencesof
specificgoalthatinstructorsexpectedtoseeisgiveninthisquoteextractedfromthestudents’
annotations:Ithinkformynextlabmygoalsshallbetotrytomakeeyecontactwitheveryoneatleast
once.Anexampleofalessspecificreflectionwouldbeasimpleobservationoftheownbehavior
withoutanyspecificgoalsetforthefuturework:Istillcontinuetohaveproblemswithmakingeye
contact.Moreover,instructorsprovidedstudentswithformativefeedbackontheirperformanceand
reflectionsinthegradedinstructionalcondition.Forexample,incaseswherestudentsmissed
somethingintheirreflections,theinstructorwouldprovidefeedbackofthistype:
Youaremostsuccessfulwhenyouaretrulyassertiveinyourmusicmaking.The
beginningofthelab,youdidnotreallyhaveaclearpictureofthetempothatyou
wanted.Inconductingrecits,youhavetobesuperclearinexactlywhatyouwantand
leadtheensemble.Ifyoujustbeattime,itwon'tbesuccessful.
Partofthisleadershipcomesfromhavingaveryclearpictureofexactlywhatyouwant.”
Inthenongradedcondition,studentsdidnotreceivegradesnorformativefeedbackfromtheir
instructors.
2.4 Sample
ThestudentsenrolledinCourses14wereincludedinthestudy.Allthestudentswerealreadyenrolled
inadegreeprograminperformingartsdirectlylinkedtothecoursesincludedinthestudy.Thesample
had77uniquestudents(42female).Theaverageageatthetimeofenrolmentwas22.1withstandard
deviationof2.82.Eachcoursehadadifferentnumberofstudentsenrolled:Course1(N=31),Course2
(N=40),Course3(N=28),andCourse4(N=20).Sincethestudywasanaturalexperimentwherestudents
hadtheoptiontoenrollintoanycourse,aspertheuniversityprogramregulations,somestudentswere
enrolledinmorethanonecourseincludedinthestudy.ThenumbersofstudentsenrolledinCourses14
andtheiroverlapsareoutlinedinFigure1.
Inordertoaccountforapossibleconfoundingeffect,wereportthestudents’gradepointaverages
(GPAs)astheyarecommonlyusedasproxiesofstudents’abilityandpredictorsoffutureperformance
(Elias&MacDonald,2007;Grove,Wasserman,&Grodner,2006).TheGPAvaluescouldalsoindicatethe
differencesinskillsforselfregulatedlearning,astypicallyhigheracademicperformanceisassociated
withhigherskillsforselfregulatedlearning(Greene&Azevedo,2009).Table2reportstheresultsofthe
comparisonbetweenthegroupsofstudentsinthestudyandtheirgradepointaverages(GPAs)atthe
endoftheacademicyear.Sincethestudywasconductedasanaturalexperiment,thecontrolfor
importantconfounderswasnotpossibleintheexperimentalassignment.Theonlysignificantdifference
identifiedwasbetweenCourse2candCourse2d.StudentswithahigherGPAinCourse2weremore
9
likelytoenrollinCourse3(Table2).Thiscouldpotentiallyconfoundthecomparisonsbetweenstudents
ingroupsCourse2candCourse2d,i.e.,betweenthestudentswithinthesameexperimentalcondition
thefirstexperiencewiththegradedlearningtooluse.However,thishadnoeffectontheinvestigation
ofresearchquestionRQ2wherethesegroupsareinvestigated.Nosignificantdifferencebetweenthose
students(Course2d)andotherstudentsinCourse3(Course3b)wereobservedintheGPAvalues,and
thus,equivalencyispreservedwithrespecttoresearchquestionRQ2thatinvestigatedsustainedtool
useunderthesameinstructionalconditionsintwocourses(e.g.,gradedgeneralannotationsinthefirst
andsubsequentcourses).
Table2.ThecomparisonoftheGPAvaluesbetweentheidentifiedgroupsinthestudy.
ABCDAvs.B Avs.B
Bvs.C Cvs.D
Course1aCourse1bCourse2a Course2b
78.80
(75.00,89.05)
85.33
(81.38,88.72)
81.62
(72.35,85.74)
85.33
(81.38,88.72)
U=300.50,z=.646,
p=.519,r=0.09
U=105.50,z=1.05,
p=.294,r=.19
N/A*U=176.50,z=1.64,
p=.101,r=.26
Course2cCourse2dCourse3aCourse3b 
75.35
(66.43,84.83)
85.57
(81.96,86.88)
80.37
(75.36,88.88)
85.57
(81.96,86.88)
U=142.00,z=1.30,
p=.193,r=.23
U=310.00,z=3.05,
p=.002,r=.48
N/A*U=111.00,z=1.01,
p=.314,r=.19
Course2eCourse2fCourse4aCourse4b 
83.10
(72.60,86.13)
84.70
(80.50,87.10)
78.05
(70.60,88.00)
84.70
(80.50,87.10)
U=92.00,z=.38,
p=.704,r=.06
U=178.00,z=.58,
p=.565,r=.09
N/A*U=47.00,z=.77,
p=.441,r=.18
Legend:*Comparisonofthestudentsinrepeatedmeasurescouldnotbedone,asthosewerethesame
studentswithasingleGPAvalue(priortoenteringtotheacademicyearinwhichallthecourseswere
offered).
2.5 Variables
2.5.1 Independentvariables
Instructionalconditionsandexperience(bothbinaryvariables)wereusedasindependenteffects,i.e.,
fixedeffectsaccordingtotheterminologyofthemethodusedinouranalyses(c.f.,Section2.6).Student
enrollmentintoCourse2andCourse3representsthegradedinstructionalcondition,wherebyCourses
1and4representthenongradedinstructionalcondition.EnrollmentandcompletionofCourse2was
anindicatorofexperiencewiththetoolgainedinthegradedconditionwithformativefeedback.Thatis,
thestudentswhomovedfromCourse2toeitherCourse3orCourse4hadpriorexperiencewiththe
tool.Otherwise,allotherstudentsinallfourcourseswereconsiderednottohaveanypriortool
experience.ItshouldbenotedthatsomestudentscompletedCourse2inapreviousyearwhenthe
videoannotationsoftwarewasnotused.Theselearners(accordingtoFigure1,thosearethestudentsin
groupsCourse3aandCourse4a)wereconsideredwithoutexperienceinCourses3and4.The
connectionbetweenthedistributionofthestudentsaccordingtothefixedeffects(instructional
conditionsandexperience)andthegroupsidentifiedinFigure1isshowninTable3.
10
Table3.Theconnectionbetweenthedistributionofthestudentsaccordingtothefixedeffects
(instructionalconditionsandexperience)andthegroupsidentifiedinFigure1
InstructionalConditions
GradedNongraded
ExperienceCourse3bCourse4b
NoexperienceCourse2ae,Course3a Course1ab,Course4a
2.5.2 Dependentvariables
Thefollowingthreegroupsofdependentvariableswereusedinthestudy.
Operationsonvideo.Countsofannotationscreatedbythestudentsareusedastheprimarydependent
variabletoinvestigateourresearchquestions.Thisisduetothefactthatthetoolusedinthestudy
(CLAS)wasdesignedforvideoannotationandtheamountofitsuseisprimarilyfocusedonitsmain
functionalitycreatingvideoannotations;i.e.,weusedthevariablethatrepresentedthecountofvideo
annotationscreatedbyastudentinacourse.Giventhatthecoursesdifferedinthenumberofvideos
thestudentswererequestedtoworkwithasshowninTable1,wealsousedanothervariablethat
representedtherelativecountofannotationsastudentcreatedpervideoinacourse.Thisvariableis
computedbydividingthecountofvideoannotationscreatedbythenumberofvideosstudentswere
requiredtoworkinagivencourse.
Severalothersecondarydependentvariableswereusedtounderstandthepatternsofinteractionwith
videosunderdifferentinstructionalconditionsandwithdifferencesinexperience.Thesevariables
representedtheoccurrencesofeventsrecordedbyCLASineachcourseincludingthecountsofpause,
rewindandfastforwardevents.Similarly,thetotaltime(minutes)whentheplaybuttonwasactivated
indicatingtimelikelyspentwatchingeachvideowasused.Theeffectsofthesevariablesarereportedin
AppendixB.
Learningproducts.Toassessthequalityoflearningproducts(i.e.,timestampedvideoannotationsand
generalvideoannotations),weusedthetwoframeworksandtoolsfortextanalysismostcommonly
usedineducationalpsychology:LinguisticInquiryandWordCount(LIWC)(Tausczik&Pennebaker,
2009)andCohMetrix(McNamara,Graesser,McCarthy,&Cai,2014).FromthesuiteofLIWCvariables,
weadoptedwordcount(WC).Thisvariablewasselected,asitiscommonlyshowntobeagoodproxy
forhigherlevelsofcognitiveprocessing(Joksimović,Gašević,Kovanović,Adesope,&Hatala,2014;
Tausczik&Pennebaker,2009).Moreover,inadditiontothecountsofannotations,thisvariablewasalso
animportantindicatorofthesustaineduseofthetool(especiallyrelevantforresearchquestion2).That
is,ameasureoftheeffortputintothecreationoftheannotationscanbederivedfromthelengthof
text(wordcount).Inaddition,wealsowantedtostudythelengthoftheselfassessmentsineach
courseasaratioofwordcountsperselfassessmentannotation(WC/Ann).AssuggestedbyGašević,
Mirriahi,andDawson(2014),thisratiocanprovideaninsightintothequalityofindividualannotations
ratherthantheentiretextofallannotationstogether.
11
CohMetrixisawellknowntoolkitbuiltonthecomputationallinguistictechniques(Graesser,
McNamara,&Kulikowich,2011;McNamaraetal.,2014).Cohmetrixisusedfortheanalysisof
characteristicsoflanguageanddiscourse.NumerousstudieshaveshownthatCohMetrixmeasurescan
beappliedtoidentifyqualitativedifferencesinformaltextualdocumentsanddiscourse(McNamaraet
al.,2014).TheCohMetrixoffersover100measuresthatcoverdifferentdimensionsoflanguageand
discoursesuchasgenre,cohesion,andsyntax.Measuresoflinguisticcomplexity,characteristicsof
words,andreadabilityscoresarealsoavailableinCohMetrix.GiventhelargenumberoftheCohMetrix
measures,aprincipalcomponentanalysiswasappliedtothe53measuresofCohMetrixinastudyof
37,520textsavailableintheTouchstoneAppliedScienceAssociationcorpus(Graesseretal.,2011).The
principalcomponentanalysisrevealedthateightprincipalcomponentsexplained67.3%ofthevariance.
Thetopfiveprincipalcomponentsexplainedover50%ofthevariability.Thezscoresofprincipal
componentsarecommonlyusedintheliterature.Thecomponentsidentifiedarewellassociatedwith
multileveltheoreticalframeworksofcognitionandcomprehension(Graesser&McNamara,2011;
Kintsch,1998;Perfetti,2000;Snow,2002).Thismakesthecomponentssuitableforresearchinlearning
relatedstudies.Inthisstudy,weusedthefollowingfiveprincipalcomponentsofCohMetrix(Dowell,
Cade,Tausczik,Pennebaker,&Graesser,2014,pp.126–127):
Narrativity.Theextenttowhichthetextisinthenarrativegenre,whichconveysastory,a
procedure,orasequenceofepisodesofactionsandeventswithanimatebeings.Informationaltexts
onunfamiliartopicsareattheoppositeendofthecontinuum.
DeepCohesion.Theextenttowhichtheideasinthetextarecohesivelyconnectedatadeeper
conceptuallevelthatsignifiescausalityorintentionality.
ReferentialCohesion.Theextenttowhichexplicitwordsandideasinthetextareconnectedwith
eachotherasthetextunfolds.
SyntacticSimplicity.Sentenceswithfewwordsandsimple,familiarsyntacticstructures.Polar
oppositearestructurallyembeddedsentencesthatrequirethereadertoholdmanywordsandideas
intheirworkingmemory.
WordConcreteness.Theextenttowhichcontentwordsareconcrete,meaningful,andevokemental
imagesasopposedtoabstractwords.”
Thevaluesofthesefivevariableswerefirstcomputedforeachindividualannotation.Next,we
calculatedameanvalueforeachofthesefivevariablesforeverystudentenrolledinthecourse.The
calculatedmeanvaluesforeachstudentineachcoursewereusedintheanalyses.
Learningstrategy.Toevaluatestudents’learningstrategieswhenusingavideoannotationtoolforself
assessment,wecreatedtransitiongraphsbasedonthetracedataoftherecordedlearningactivities
withinCLAS(Hadwin,Nesbit,JamiesonNoel,Code,&Winne,2007;Malmbergetal.,2014;Winne,
Gupta,&Nesbit,1994).Transitionsgraphswerecreatedbasedonacontingencymatrixwhererowsand
columnsaccountedforallpossibleevents.Therowsrepresentedthestartandthecolumnsrepresented
theendpointsofthetransitionedges.Originally,allthecellsinthematrixhadvaluesofzero.Ifan
event,A,wasfollowedbyaneventB,number1wasrecordedinthecellrepresentingtheintersectionof
rowAandcolumnB.Foreachfutureappearanceofthistransition,thenumberinthecellwas
incrementedby1.Inthisway,wecreatedweightedanddirectedtransitiongraphs.
12
Forthepurposesofthestudy,wedistinguishedbetweeneventsbasedonthetemporalpartsofthe
videostheywereassociatedwith.ThiswasconsistentwiththefindingsofGaševićetal.(2014)andMu
(2010)whoreportedthatstudents’operationswereunevenlydistributedacrosseachvideo.Specifically,
wedistinguishedbetweentimestampedevents(annotations,pause,rewind,andfastforward)based
onthequartilesofthevideostheywereassociatedwith.Forexample,ifatimestampedannotationwas
associatedwiththe12secondmarkofa100secondlongvideo,theeventtypewasannotationin
quartile1.Similarly,ifaparticipantrewoundtosecond34ofa100secondlongvideo,theeventtype
wasrewindinquartile2.Accordingtotheserules,wecreatedatransitiongraphforeachstudentinour
sample.Thesegraphsconsistentof19possiblenodesfoureventtypesfortimestampedannotations,
pause,rewind,andforwardevents,oneforgeneralannotation,onefornonstopwatching,andonefor
endofvideowatching.
Thetransitiongraphs,describedabove,wereusedtoinvestigatestudents’selfregulatedlearning
processeswhenusingCLAS.Inthisinstance,wewereparticularlyinterestedinthelevelofstudents’
metacognitivemonitoringoftheirownlearning.Metacognitivemonitoringisthekeyactivityforlearning
success,sinceitiscommonlyusedforevaluationofthelearningproductandlearningstrategy(Winne,
2001;Winne&Hadwin,1998).Azevedoetal.(2008)foundthatthehigherlevelsofmetacognitive
monitoringwereassociatedwithanincreaseoffeelingofknowing,judgmentoflearning,and
monitoringofprogresstowardgoals.Moreover,GreeneandAzevedo(2009)foundthatmonitoring
activitywasa“keySRLprocesswhendevelopinganunderstandingofacomplexsciencetopicusing
hypermedia”(p.18).AccordingtoHadwinetal.(2007)andWinneetal.(1994),agraphtheoretic
measureofdensityaratiobetweentheactualnumberofedgesbetweennodesinagraphandall
possibleedgesinthegraphcanbeusedtoassessmetacognitivemonitoringinlearningstrategies
followedbylearners.Specifically,Hadwinetal.(2007,)positthat“participantswithloweroverall
densitieshaveformedsomedistinctandregularstudyingpatternswhereasparticipantswithhigher
densitiesareexperimentingwithtacticsandstrategies.Thatis,theselatterstudentsareengagedin
moremetacognitivemonitoringand,hence,moreactiveSRL[selfregulatedlearning]”p.114.
2.6 Analysis
Giventhenestedstructureofourdata(studentswithinacourse)andpotentialproblemofcorrelated
dataduetogrouping(Seltman,2012),wereliedonlinearmixedmodelstoaddresstheresearch
questions.Mixedeffectsmodelingprovidesarobustandflexibleapproachthatallowsforawidesetof
correlationpatternstobemodeledandisrecommendedmethodforstudyingsimilardatasets(Pinheiro
&Bates,2009;Seltman,2012).Mixedeffectsmodelsincludeacombinationoffixedandrandomeffects
andcanbeusedtoassesstheinfluenceofthefixedeffectsondependentvariablesafteraccountingfor
anyextraneousrandomeffects.Fixedeffectscorrespondtothenumericalorcategoricalvariablesthat
areofprimaryinterestandrepresentfixed,repeatablelevelsamongwhichcomparisonsaretobemade.
Randomeffectsarecategoricalvariablesthatrepresentvariabilityamongsubjects,arandomselection
fromalargerpopulationtowhichtheresultscanbeextended.
Mixedeffectsmodelingwasusedtoexaminetheassociationbetweenthetwofactors(instructional
conditionsandpreviousexperience)andthedependentvariables.Inordertoassessthisassociationfor
eachofthedependentvariablesaboveandbeyondtherandomeffects,webuiltthreemodelsforeach
13
ofthedependentvariablesnullmodel,fixedmodel,andfinalmodel(Table4ininAppendixAError!
Referencesourcenotfound.).Thenullmodelinitiallyincludedtherandomeffectonly(studentwithina
course).However,insomecases,wewerenotabletofitthemodelwithsuchastructureofrandom
effects.Insuchcases(i.e.videosannotated,pause,rewind,wordcount,wordcountperannotation,and
deepcohesion),wespecifiedstudentasarandomeffectinsteadandwereabletofitthemodel.
Moreover,insomecases(i.e.,timewatched,narrativity,andreferentialcohesion),wecouldnotfitthe
modelevenwiththerevisedrandomeffect.Ontheotherhand,afixedmodelincludedconditionand
experienceasfixedeffects,whilethefinalmodelincludedcondition,experience,andinteraction
betweenconditionandexperience,asfixedeffects.Intraclasscorrelationcoefficient(ICC)(Raudenbush
&Bryk,2002),secondorderAkaikeinformationcriterion(AICc),andlikelihoodratiotest(Hastie,
Tibshirani,&Friedman,2011)wereusedtodecideonthebestfittingmodel(Table4inAppendixA).We
alsoestimatedaneffectsize(R2)foreachmodelasgoodnessoffitmeasure,calculatingthevariance
explainedusingthemethodsuggestedbyXu(2003).
LinearmixedeffectsmodelswereconductedusingRv.3.0.1softwareforstatisticalanalysiswith
packagelme4(Bates,Maechler,Bolker,&Walker,2015).Thehypothesesspecifythedirectionofthe
effect,howevertwotailedtestswereusedforsignificancetestingwithanalphalevelof.05.
3 Results
Theresultsinthissectionareorganizedaccordingtothethreemaindimensionsusedtooperationalize
thetworesearchquestionsthataimedtoinvestigatetheeffectsofinstructionalconditions(research
questionRQ1)andexperience(researchquestionRQ2)ontheadoptionofalearningtool.
3.1 Countsofannotations
Thelikelihoodratiotestforcountsofannotationsmodelsyieldedsignificantlybetterfitofthefinal
model(i.e.,themodelthatincludedfixed,interaction,andrandomeffects)thanthenullandfixed
models.ThemodelshowedsignificanteffectsofpreviousexperienceF(1,114.94)=11.54,p<.001)and
theinteractionofinstructionalconditionandexperience(F(1,114.94)=4.10,p=.045)onthenumberof
thecountsofannotationscreated.Theeffectofinstructionalcondition(F(1,2.96)=2.52,p=.212)was
interestinglynotsignificantthough.Theestimatedmeanvalues,calculatedasaresultofthismodel,are
showninFigure2a.Theseresultsindicatethatthestudentswithpreviousexperiencetendedtocreate
significantlymoreannotationsthanthosestudentswhoencounteredthetoolforthefirsttime.There
wasnosignificantdifferencebetweenthestudentsinthecountsofannotationscreatedwhenthey
wereinthegradedconditionscomperedwhentheywereinthenongradedcondition.However,the
significantinteractioneffectshowedthatthestudentswithandwithoutexperiencehaddifferenttrends
inannotationcountswhentheywereinthenongradedversusgradedconditions.Whiletherewasno
differenceincountsofannotationsbetweenthestudentswithandwithoutexperienceinthegraded
condition,thisdifferencewassignificantbetweenstudentswithandwithoutexperienceinthegraded
condition.
14
a)b)
Figure2.Effectsofinstructionalconditionsandexperiencewiththevideoannotationtooloni)total
countsofannotationscreatedandb)countsofannotationspervideo
Thelikelihoodratiotestforcountsofannotationspervideomodelsyieldedsignificantlybetterfitofthe
finalmodel(i.e.,themodelthatincludedfixed,interaction,andrandomeffects)thanthenullandfixed
models.Thesignificantfixedeffectsinthismodelwereconsistentwiththosefindinthemodelforthe
totalcountofannotation(discussedinthepreviousparagraph).Thatis,theeffectsofexperience(F(1,
26.17)=25.59,p<0.001)andinteractionofexperienceandinstructionalcondition(F(1,26.17)=4.85,p
=0.036)weresignificant,whiletheeffectofinstructionalconditionswasnotsignificant(F(1,2.24)=
12.34,p=0.061).Theestimatedmeanvalues,calculatedasaresultofthismodel,areshowninthe
diagraminFigure2b.Giventhatthesamesignificanteffectswerealsofoundforcountsofannotations,
theinterpretationoftheresultsforcountsofannotationspervideoisthesameasstatedforcountsof
annotationsinthepreviousparagraph.Moreover,asshowninTable4inAppendixAandconsistent
withtheresultsofthefixedeffectsforthecountofannotationspervideomodel,therandomeffect
studentwithinacourseexplainedabout69%,whilethecourseitselfexplainedonly9%ofthevariability
inthemodel.
TheresultsreportedinAppendixBshowedsimilartrendswithothersecondarydependedvariables
aboutamountofoperationsperformedtointeractwithvideoavailableinthevideoannotationtool.
3.2 Qualityoflearningproducts
Thelikelihoodratiotestforthecountofwordsmodelsunveiledasignificantlybetterfitofthefixed
model(i.e.,themodelthatincludedfixedandrandomeffects)thanthenullandfinalmodels.Themodel
showedsignificanteffectsofpreviousexperience(F(1,88.44)=53.60,p<.001)andinstructional
condition(F(1,99.82)=59.90,p<.001)onthenumberwordsperannotation.Theestimatedmean
valuesofcountofwordswrittenbystudentsintheirannotations,calculatedasaresultofthismodel,
areshowninthediagraminFigure3a.Thesefindingsindicatethatthestudentswithprevious
experiencehadsignificantlymorewordsintheirannotationsthanthosestudentswhoencounteredthe
toolforthefirsttime.Thestudentswhowereinthegradedconditionusedsignificantlymorewordsin
theirannotationscomparedtothestudentsinthenongradedcondition.
15
a)Totalcountofwordsinannotations
b)Wordcountperannotation
Figure3.Effectsofinstructionalconditionsandexperiencewiththevideoannotationtoolonword
countsinannotationsandwordsperannotation
Thelikelihoodratiotestforthewordsperannotationmodelsrevealedasignificantlybetterfitofthe
fixedmodel(i.e.,themodelthatincludedfixedandrandomeffects)thanthenullandfinalmodels.The
modelshowedsignificanteffectsofpreviousexperience(F(1,39.73)=7.02,p=.012)andinstructional
condition(F(1,43.70)=19.74,p<.001)onthenumberofwordsperannotation.Theestimatedmean
values,calculatedasaresultofthismodel,areshowninthediagraminFigure3b.Basedonthese
results,itcouldbeconcludedthatthestudentswithpreviousexperiencetendedtousesignificantly
morewordsperannotationthanthosestudentswhoencounteredthetoolforthefirsttime.The
studentswhowereinthegradedconditionusedsignificantlymorewordsperannotationcomparedto
thestudentsinthenongradedcondition.
Thelikelihoodratiotestforthedeepcohesionmodelsshowedasignificantlybetterfitofthefinalmodel
(i.e.,themodelthatincludedfixed,interaction,andrandomeffects)thanthenullandfixedmodels.The
modelshowednonsignificanteffectofinstructionalconditions(F(1,53.78)=1.15,p=.289),while
experience(F(1,54.54)=6.75,p=.012)andinteractionofexperienceandinstructionalcondition(F(1,
111.13)=6.11,p=0.015)hadsignificanteffectsonthescoresofdeepcohesionofthetextinthe
annotations.Theestimatedmeanvaluesofdeepcohesioninstudentsannotations,calculatedasaresult
ofthismodel,areshowninthediagraminFigure4a.Basedontheseresults,itcouldbeconcludedthat
thestudentswithpreviousexperiencetendedtohaveannotationswiththehigherscoresofdeep
cohesionthanthosestudentswhoencounteredthetoolforthefirsttime.Thestudentswhowereinthe
gradedconditionshadnodifferentscoresofdeepcohesionsforannotationcomparedtothestudentsin
thenongradedcondition.Thesignificantinteractioneffectindicateshoweverthattherewasadifferent
patternofdeepcohesioninannotationsbetweenstudentswithandwithoutexperienceindifferent
(gradedvs.nongraded)instructionalconditions.Whiletherewasnodifferenceindeepcohesion
betweenstudentswithandwithoutexperienceinthegradedcondition,thisdifferencebetween
studentswithandwithoutexperienceinthenongradedconditionwassignificant.
Thelikelihoodratiotestforthesyntacticsimplicitymodelsproducedasignificantlybetterfitofthefixed
model(i.e.,themodelthatincludedfixedandrandomeffects)thanthenullandfinalmodels.Themodel
showednonsignificanteffectsofbothinstructionalconditions(F(1,2.19)=3.51,p=.191)and
16
experience(F(1,21.65)=1.54,p=.229)onthescoresofsyntacticsimplicityofthetextinthe
annotations.Theestimatedmeanvaluesforsyntacticsimplicityofthetextinthestudents’annotations,
calculatedasaresultofthismodel,areshowninthediagraminFigure4b.Basedontheseresults,it
couldbeconcludedthatthestudentswithpreviousexperiencedidnothavedifferentannotationswith
respecttosyntacticsimplicitythanthosestudentswhoencounteredthetoolforthefirsttime.The
studentswhowereinthegradedconditionshadnodifferentscoresofsyntacticsimplicityfor
annotationcomparedtothestudentsinthenongradedcondition.
Thelikelihoodratiotestforthewordconcretenessmodelsyieldedasignificantlybetterfitofthefixed
model(i.e.,themodelthatincludedfixedandrandomeffects)thanthenullandfinalmodels.Themodel
showednonsignificanteffectsofbothinstructionalconditions(F(1,3.04)=0.46,p=.544)and
experience(F(1,52.02)=0.03,p=.859)onthescoresofwordconcretenessofthetextinthe
annotations.Theestimatedmeanvalues,calculatedasaresultofthismodel,areshowninthediagram
inFigure4c.Basedontheseresults,itcouldbeconcludedthatthestudentswithpreviousexperience
didnothavedifferentannotationswithrespecttowordconcretenessthanthosestudentswho
encounteredthetoolforthefirsttime.Thestudentswhowereinthegradedconditionshadnodifferent
scoresofwordconcretenessforannotationcomparedtothestudentsinthenongradedcondition.
a)Deepcohesion
b)Syntacticsimplicity
c)Wordconcreteness
Figure4.EffectsofinstructionalconditionsandexperiencewiththevideoannotationtoolontheCoh
Metrixscoresofdeepcohesion,syntacticsimplicityandwordconcreteness
Thelikelihoodratiotestforthereferentialcohesionandnarrativitymodelsdidnotyieldsignificantly
betterfitofthefixedandfinalmodelsthanthenullmodels.
17
3.3 Learningstrategy
Thelikelihoodratiotestforthedensitymodelsshowedasignificantlybetterfitofthefixedmodel(i.e.,
themodelthatincludedfixedandrandomeffects)thanthenullandfinalmodels.Themodelshowed
nonsignificanteffectsofbothinstructionalconditions(F(1,2.97)=6.49,p=.085)andexperience(F(1,
115.95)=0.71,p=.400)onthedensityoftheirtransitiongraphs.Theestimatedmeanvalues,calculated
asaresultofthismodel,areshowninthediagraminFigure5.Basedontheseresults,itcouldbe
concludedthatthestudentswithpreviousexperiencedidnothavedifferentannotationswithrespectto
wordconcretenessthanthosestudentswhoencounteredthetoolforthefirsttime.Thestudentswho
wereinthegradedconditionshadnodifferentscoresofwordconcretenessforannotationcomparedto
thestudentsinthenongradedcondition.Interestinglythough,Moreover,asshowninTable4in
AppendixA,therandomeffectofcourseexplainedabout43.1%ofthevariabilityinthemodel.Thiscan
probablyshedsomelightwhytheestimatedmeanvaluesofdensity(Figure5)weredifferentbetween
thecourseswithgradedvs.nongradedinstructionalconditions.
Figure5.Effectsofinstructionalconditionsandexperiencewiththevideoannotationtoolondensityof
transitiongraphs
4 Discussion
4.1 DiscussionoftheResultsinRelationtotheResearchQuestions
Theresultsreportedintheprevioussectionsupporttheimportanceofintegratingalearningtoolwith
anassessedinstructionalconditionwhenstudentsencounterthetoolforthefirsttime(research
questionRQ1).Thisofferssomeevidenceforourpropositionthatprovidingascaffoldisnecessary
(Azevedo&Hadwin,2005;Beedetal.,1991)forguidingstudentstowardsproficientuseofatoolin
ordertomeetWinne’s(2006)firstthreeconditionsfortooladoptionawarenessofthetool
availability,mappingtoatask,andskilltousethetool.Onlyoncetheseconditionsaremet,studentswill
likelyusethetoolextensively.Therewasanobservedincreaseinthecountsofannotationsbetween
gradedvs.nongradedcondition(Figure2).Althoughitwasanticipatedthatinstructionalconditions
(gradedvs.graded)wouldhaveasignificanteffectonthecountofannotationsproducedbythe
students(RQ1),thiseffectwasnotfoundtobesignificantintheresultsreportedinSection3.1.Rather
andsomewhatcounterintuitively,theeffectsofexperiencewiththetooluseandinteractionof
instructionalconditionsandexperienceweresignificant(RQ2).
18
Theresultsshowedthatexperienceconsistentlyhadasignificanteffectonthequalityoflearning
productsandindicatorsofdeeplearning.Thesignificanteffectsofbothinstructionalconditionsand
experiencewerefoundonthreemeasuresofthequalityoflearningproducts(Section3.2),namely,
countsofwords,wordsperannotation,anddeepcohesion.Theeffectsofthesefindingsareillustrated
inthediagramsshowninFigure3andFigure4.Itisinterestingtoobservethatthestudentsinthe
gradedinstructionalconditionwithnopriorexperienceproducedalmostthesamecountofwordsin
theirannotationsastheircounterpartswithexperienceinthenongradedcondition(seeFigure3a).A
similarpatternisobservedforwordsperannotationasshowninFigure3b.Thisfindingisimportant,
sincewordcounthasalreadybeenreportedasaproxyofdeepengagementintoknowledge
construction.Forexample,Joksimovicetal.(2014)foundthathigherlevelsofcognitivepresencewere
significantlyassociatedwithhighercountsofwordsinmessagesexchangedinasynchronousonline
discussions.Thisfindingisconsistentwithessaygradingresearchthatdemonstratesthattheword
lengthofanessayisthestrongestpredictorofthefinalessaygrade(Page&Petersen,1995).Somewhat
surprisingly,thefindingsfordeepcohesiondidnotrevealanysignificanteffectsofinstructional
condition,ratherthesignificanteffectswerefoundforexperienceandinteractionofexperienceand
instructionalconditions.Namely,thescoresofdeepcohesionforthestudentswithexperienceinthe
nongradedconditionswereevenhigherthanthoseofthestudentswithoutexperienceinthegraded
condition.Anoppositepatternisfoundforstudentswithoutexperienceinthenongradedcondition
andwhohadthelowestscoresofdeepcondition.Scoresofdeepcohesionforstudentsinthegraded
conditionregardlessoftheirexperienceremainedonthesamelevel.Itshouldbenotedthatdeep
cohesionisanindicatorofdeeperconceptuallevelthatsignifiescausalityorintentionality(Graesseret
al.,2011).Moreover,lowscoresofdeepcohesionarealsoasignofalowdegreeofgoaloriented
connectivesofthetextinannotationswithstudents’selfassessments(Graesseretal.,2011).Therefore,
theresultsofthisstudysuggestthatpriorexperiencewiththetooluseisacriticalfactorfor
engagementintodeeplearninglevels.Thisisalsoconsistentwiththepreviouslyreferencedworkby
Malmbergetal.(2014)onlearningstrategyadoptionwhichindicatesthatalthoughthequantityofthe
strategyuseremainsconsistentwithexperience,thequalityofthelearningproductincreases.
Therewasanevident,thoughstatisticallynonsignificant,decline(Section3.3)inthemeasuresof
metacognitivemonitoringwhentheassessedinstructionalconditionwasremoved(seeFigure5).
Accordingtoourtheoreticalbackground,thedecreaseinmetacognitivemonitoringinthisstudywasa
resultofthechangedstandardsdrivenbytheinstructionalconditions.Thelowermetacognitive
monitoringcanbeexplainedbythereducedneedtopayattentiontothedetailsobservedinthevideos
inordertoaccuratelydescribetheobservationsinannotationsasshownbythedropofpause,rewind,
andfastforwardeventsinCourse4.Thismayhaveledtothedecreaseinthedensityofthegraphsused
formodellingthelearningstrategy.However,inspiteofthesimilarnumbersofannotationscreatedand
quantityoftext,thislesseningofmetacognitivemonitoringcouldreducethelevelofstudents’
understandingofthestudytopicsasshownbyGreeneandAzevedo(2009).Therefore,additional
scaffoldingandinstructionalstrategiesarerequiredinordertomaintainthelevelofmetacognitive
monitoring.Researchonexternallyfacilitatedregulatedlearningofferssomeguidanceforhowto
addressthisissue.Forexample,feedbackonstudents’annotationsorrubricsforself,peer,or
19
instructorassessmentcanhelpguidethequalityofannotations.Moreover,sharingtheannotations
withpeersseemstobeanotherpromisinginstructionalstrategy(Hulsman&vanderVloodt,2015).
Incontrasttotheexistingliterature,thecurrentstudyshowedthatgradingofstudentsuseofa
technologyisnotonlyasignificantfactorinfluencinguptakebutalsostimulateslongertermadoption
andapproachestostudying.Theprevalentassumptionthatthetechnologyuseisdrivenbygradingis
alsoconsistentwiththeliteraturethatemphasizesassessmentasthestrongestpromptforlearning
(Boud,1995;Eisner,1993).Thatis,whatwillbeassessedandhowitwillbeassessedguidesstudents’
learningandmotivationinformaleducation.Forexample,WormaldandSchoeman(2009)showedthat
increasingtheassessmentweightoftheanatomycourseinamedicalschoolhadasignificantpositive
impactonstudents’motivationtolearnthesubjectofanatomy.Inthecurrentstudy,weshowedthat
suchassessmentprompts(i.e.,gradedconditioninourstudy)canalsobeusedtoscaffoldstudents’
approachtostudyingandthatsuchinitialscaffoldshavealongtermeffect.Thatis,studentsmaintained
theuseofthetooltoaidtheirlearningeventhoughtheinstructionalconditionschanged(gradedto
nongraded).
4.2 Limitations
Futurestudiesshouldcollectfurtherdemographicdataaboutstudents(e.g.,disciplinarybackground,
ethnicbackground,andlanguageproficiency)andstudywhetherandifso,towhatextentthese
variablesconfoundthefindingsreportedinthisstudy.Futurestudiesshouldalsoaccountfortheeffects
ofindividualdifferencese.g.,motivationtousetechnology,selfefficacyaboutthesubjectmatter
and/ortechnology,achievementgoalorientation(Elliot,Murayama,&Pekrun,2011),approachesto
learning(Biggs,Kember,&Leung,2001),andmetacognitiveawareness(Duncan&McKeachie,2005)
thatarefoundtobesignificantcovariateoftheadoptionoflearningtools(Clareboutetal.,2013).
Anavenueofinvestigationforfuturestudieswouldbetoattempttoreplicatetheextentthefindingsof
thisstudyapplytoothertoolsandtechnologies.Futurestudiescouldexaminetheextenttowhichthe
specificstudytacticthatwassupportedbytheuseofthetoolcanhaveaneffectonfutureadoption.In
thecurrentstudy,welookedathowatechnologycansupportstudentselfassessment.Itcouldbethe
casethattechnologytaskfitplaysacriticalrole(McGill&Klobas,2009)andthefindingsreportedinthis
paperareonlygeneralizabletotheextenttowhichatooleffectivelysupportsastudytacticofhigh
valueforthecompletionofspecificlearningtasks.
4.3 ImplicationsforResearch
Thisstudyprovidesfurtherinsightandevidenceintoexistingbodyofresearchonlearningtooluseasa
selfregulatedlearningprocess.Thetheoreticalmodeladoptedforthestudy(Lustetal.,2013;Winne,
2006)explainstherationaleforthedecisionsthatstudentsmadewhenusingthetoolunderdiffering
instructionalconditions.Moreover,thestudyshowedtheimportanceofhavingmoreadvanced
measuresoflearningprocessesthatcanaccountforimportantfactorsaffectingselfregulationof
students’tooluse.Inparticular,theuseoftheCOPESmodelwasfoundtobehighlybeneficialfor
informingthedefinitionofthetypesofmeasuresusedinthestudy.TheCOPEmodelallowedforthe
theoreticallygroundedinterpretationoftheresultsandtherelationshipsobservedbetweenindividual
variables.Withthesetwotheoreticalgroundings,selfregulatedlearningandCOPES,futurestudies
20
concerningtooluseshouldfocusmoreonthelearningaspectsofthetoolsratherthanjustthevarious
usabilityfactorsthatarecommoninresearchontechnologyacceptance(Davis,1989).Ofcourse,these
factorsarewellestablishedintechnologyacceptanceresearch(e.g.,perceivedeaseofuse,learning,and
usefulness)andarepotentiallyimportantinternalconditionsaspertheCOPESmodel.However,their
rolecanandshouldbemorecloselyinvestigatedaccordingtothemodeltheorizedinthispaper.
Likewise,theeffectsofotherindividualdifferences(e.g.,selfefficacytouseatool,achievementgoal
orientationorepistemicbeliefs)underalternateinstructionalconditionsareanotherimportantavenue
forfutureresearch.
Toadvanceourunderstandingoflearningtooluseasaselfregulatedlearningprocess,itisimportantto
developmeasuresthatcanallowforthestudyoflearningproductsandlearningstrategiesaswellas
theirassociationwithconditionsandstandardsassociatedwithspecificlearningsituations.Textanalysis
(e.g.,CohMetrix)andtheanalysisoftemporalassociationsbetweeneventsofdifferentlearning
operationsarehighlyrelevant(e.g.,transitiongraphsusedinthispaperorprocessandsequencemining
suggestedbyReimann,Markauskaite,&Bannert(2014)andWinne(2014).