Monster Mischief: A Game-Based Assessment of Selective Sustained Attention in Young Children

Abstract

Selective sustained attention, or the ability to allocate perceptual and mental resources to a single object or event, is an important cognitive ability widely assumed to be required for learning. Assessing young children's selective sustained attention is challenging due to the limited number of sensitive and developmentally appropriate performance-based measures. Furthermore, administration of existing assessments is difficult, as children's engagement with such tasks wanes quickly. One potential solution is to provide assessments within an engaging environment, such as a video game. This chapter reports the design and psychometric validation of a video game (Monster Mischief) designed to assess selective sustained attention in preschool children. In a randomized controlled trial, the authors demonstrate that Monster Mischief is significantly correlated with an existing measure of selective sustained attention (rs ≥ 0.52), and more motivating for young children as almost three times more children preferred Monster Mischief to the existing measure.

Full text

18 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
Keywords: Assessments,Children,ControlledTrial,Games,Psychometrics,SelectiveSustainedAttention
ABSTRACT
Selectivesustainedattention,ortheabilitytoallocateperceptualandmentalresourcestoasingleobjector
event,isanimportantcognitiveabilitywidelyassumedtoberequiredforlearning.Assessingyoungchildren’s
selectivesustainedattentionischallengingduetothelimitednumberofsensitiveanddevelopmentallyap-
propriateperformance-basedmeasures.Furthermore,administrationofexistingassessmentsisdifcult,as
children’sengagementwithsuchtaskswanesquickly.Onepotentialsolutionistoprovideassessmentswithin
anengagingenvironment,suchasavideogame.Thispaperreportsthedesignandpsychometricvalidation
ofavideogame(MonsterMischief)designedtoassessselectivesustainedattentioninpreschoolchildren.In
arandomizedcontrolledtrial,theauthorsdemonstratethatMonsterMischiefissignicantlycorrelatedwith
anexistingmeasureofselectivesustainedattention(rs≥0.52),andmoremotivatingforyoungchildrenas
almost3timesmorechildrenpreferredMonsterMischieftotheexistingmeasure.
Monster Mischief:
Designing a Video Game to Assess
Selective Sustained Attention
KarrieE.Godwin,CarnegieMellonUniversity,Pittsburgh,PA,USA
DerekLomas,CarnegieMellonUniversity,Pittsburgh,PA,USA
KenR.Koedinger,CarnegieMellonUniversity,Pittsburgh,PA,USA
AnnaV.Fisher,CarnegieMellonUniversity,Pittsburgh,PA,USA
INTRODUCTION
Attention is multidimensional and encompasses a diverse set of psychological constructs, includ-
ing (but not limited to) orienting, selection, shifting, maintenance, and executive attention (see
Colombo & Cheatham, 2006; Gitelman, 2003; Fisher & Kloos, in press; Posner & Petersen,
1990). The present work focuses on selective sustained attention which is defined as: “a state
of engagement that involves narrowed selectivity and increased commitment of energy and
resources on the targeted activity … and that primarily enhances information processing in that
system” (Setliff & Courage, 2011, p. 613). Selective sustained attention is important because
the ability to selectively allocate attentional resources is commonly hypothesized to aid learn-
DOI: 10.4018/IJGCMS.2015100102

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 19
ing (e.g., Carroll 1963; Bloom, 1976; Oakes, Kannass, & Shaddy, 2002). As stated by Oakes
and colleagues (2002), “if attention were constantly reoriented to every new event, it would be
difficult ... to learn about any single object or event” (p.1644). In accordance with this assertion,
prior research has implicated selective sustained attention in task performance (e.g., Choudhury
& Gorman, 2000; DeMarie-Dreblow & Miller, 1988), academic achievement (e.g., Duncan et al.
2007; for review see Goodman, 1990), and learning outcomes (e.g., Fisher, Thiessen, Godwin,
Dickerson, & Kloos, 2013; Fisher, Godwin, & Seltman, 2014; Yu & Smith, 2012).
Theoretical Perspectives on Selective Sustained Attention
There has been a long tradition in the cognitive psychology literature of distinguishing between
two modes of attention regulation. One mode refers to regulation that is said to be top-down,
controlled, and endogenous (i.e., arising from within the organism); whereas the other mode is
commonly referred to as bottom-up, automatic (or in some cases automatized through extensive
practice), and exogenous (i.e., arising from outside the organism) (e.g., Jonides, 1981; Miller &
Cohen, 2001; Posner, 1980; Pashler, Johnston, & Ruthruff, 2001; Schneider & Shiffrin, 1977). In
the present paper, we are adopting the developmental framework of attention regulation put forth
by Ruff and Rothbart (1996). This framework builds upon the dual-mode of attention regulation
suggested by the above theories. Specifically, according to Ruff and Rothbart (1996) the state of
selective sustained attention can be obtained through two distinct systems: an orienting system
and an executive control system. Under the orienting system, selective sustained attention is
driven by exogenous factors such as the saliency of the stimuli including characteristics such as
the brightness or contrast of the stimuli as well as motion. Thus, exogenously driven selective
sustained attention is considered largely an automatic process driven by the physical character-
istics of the environment or the stimulus (Bornstein, 1990; Ruff & Rothbart, 1996). Under the
executive control system, selective sustained attention is driven by endogenous factors; in other
words the focus of attention is directed internally based on the individual’s desires, interests,
and goals (Colombo & Cheatham, 2006; Posner & Petersen, 1990; Posner & Rothbart, 2007).
The two systems are said to be neurally and anatomically distinct, with the orienting system
involving areas within the parietal lobe, superior colliculus, and lateral pulvinar nucleus of the
thalamus, and the executive system involving areas within the prefrontal cortex (PFC) and an-
terior cingulate gyrus (fore review see Fisher & Kloos, in press).
In the present paper we focus on endogenously driven selective sustained attention as prior
research suggests that endogenously regulated selective sustained attention may be particu-
larly important for learning in education settings (Erickson, Thiessen, Godwin, Dickerson, &
Fisher, 2014). In education settings children need to attend to an externally-prescribed learning
objective, but often face multiple sources of distraction (Godwin, Almeda, Petroccia, Baker,
& Fisher, 2013). Children who are better able to endogenously regulate their attention may be
more likely to maintain focus on the learning objective in the face of distractions and thus have
better learning outcomes.
Development of Selective Sustained Attention
The orienting and executive control systems follow different developmental trajectories. The
orienting system matures during infancy. In contrast, the executive control system follows a more
protracted developmental course continuing to mature into adolescence (Diamond, 2002; Luna,
2009; Posner & Rothbart, 2007). Thus, early in life selective sustained attention is driven by
exogenous factors, later in development endogenous factors become more instrumental (Diamond,
2006; Colombo & Cheatham, 2006; Oakes, Kannass, & Shaddy, 2002; Ruff & Rothbart, 1996).

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
20 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Marked improvements in selective sustained attention are seen throughout early childhood:
For example, 21-month-old children can only engage in episodes of selective sustained attention
for approximately 2 minutes, in contrast children 5 to 6 years of age are able to engage in epi-
sodes of selective sustained attention for 9 minutes or more (Choudhury & Gorman, 2000; Ruff
& Lawson, 1990; Sarid & Breznitz, 1997; for review see Fisher & Kloos, in press). However,
these estimates likely reflect the upper limit of children’s ability to maintain attention on a single
activity, as selective sustained attention was assessed during free play (or other self-directed
activities), rather than within structured learning contexts where a child’s attention span may be
far shorter (e.g., Geary, 2011). As children advance through early and middle-childhood, selec-
tive sustained attention continues to improve and children become less susceptible to distraction
(Ruff & Rothbart, 1996; Ruff & Capozzoli, 2003) and they develop the ability to produce and
utilize effective attention strategies (DeMarie-Dreblow & Miler, 1988).
Children’s ability to endogenously regulate selective sustained attention is hypothesized
to be related to two core executive functions, namely inhibitory control and working memory
(Miyake, Friedman, Emerson, Witzki, Howerter, & Wager, 2000). Endogenously driven selec-
tive sustained attention is related to inhibitory control, as the successful completion of tasks
often requires inhibiting extraneous information or events (Ruff & Rothbart, 1996). Similarly,
endogenously driven selective sustained attention requires sufficient working memory in order
to maintain an active representation of task goals (Colombo & Cheatham, 2006).
Measurement of Selective Sustained Attention
Selective sustained attention in infants is typically assessed using eye gaze and physiological
measures such as heart rate (for review see Fisher & Kloos, in press; Richards, 2003). To measure
selective sustained attention in toddlers and young children, researchers commonly utilize play-
based measures of selective sustained attention (see Ruff & Rothbart, 1996). Studies relying on
play-based measures of selective sustained attention use extensive coding protocols in which the
child’s eye gaze, facial expression, posture, and activity level are coded. For example, selective
sustained attention entails several behavioral signatures such as looking at the object of interest
with an intent facial expression, leaning toward the object, and engaging in minimal extraneous
body movements (Choudhury & Gorman, 2000; Oakes, at al., 2002; Ruff & Capozzoli, 2003;
Ruff & Rothbart, 1996; Tellinghuisen, Oakes, & Tjebkes, 1999).
Performance-based measures of selective sustained attention are commonly administered to
older children and adults (Fisher & Kloos, in press). The most widely used performance-based
assessment of selective sustained attention is the Continuous Performance Test (CPT; Rosvold,
Mirsky, Sarason, Bransome, & Beck, 1956). In the CPT, participants are presented with a stream
of visual stimuli over a prolonged period of time (e.g., up to 40 minutes). Participants are asked
to respond by pressing a button when the target appears and withhold a response for non-targets.
Targets occur infrequently, and thus require the participant to maintain a state of prolonged
vigilance. The CPT is considered inappropriate for use with preschool children due to the long
task durations and limited familiarity with the task stimuli (e.g., letters and numbers). Child-
friendly adaptions to the CPT have been pursued in which the task duration is shortened and the
stimuli are altered to include highly familiar objects. However, these adaptions have met with
limited success as recent research has documented that young children typically fail to reach
the minimum performance criteria (for review see Fisher & Kloos, in press). Thus, preschool
children are in a measurement gap as they are too old for the physiological measures utilized
with infants and toddlers and too young for the performance-based measures typically employed
with older children and adults (Fisher & Kloos, in press).

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 21
In order to mitigate the measurement gap, Fisher, Thiessen, Godwin, Kloos, and Dickerson
(2013) recently developed Track-It, a computer-based task designed to assess selective sustained
attention in young children. Although Track-It provides a sensitive and developmentally appro-
priate measure of selective sustained attention in children three to five years of age, children’s
engagement with the task wanes quickly. Therefore, we sought to investigate whether it was
possible to redesign Track-It as an engaging video game that would also maintain sufficient
validity as an assessment while increasing children’s motivation.
While games have been used extensively to provide intrinsic motivation for learning (Malone,
1981; for review and meta-analysis see Clark et al., 2013), there have been comparatively few
studies investigating the validity of games as assessments or empirical validations demonstrating
that game-based assessments improve children’s intrinsic motivation. MacPherson and Burns
(2008) explicitly compared game-like versions of assessments of working memory and processing
speed to traditional assessments; but they did not assess whether the games improved player affect
or motivation. Similarly, Delgado et al. (2014) created a battery of 10 game-based assessments for
a range of neuropsychological constructs. They found that the games were reliable assessments
(Cronbach alpha > 0.77) that, in many cases, highly correlated with traditional neuropsychological
assessments. But, again, no comparison was made to determine if the game-based assessments
enhanced player affect or motivation. However, work by Attali and Arieli-Attali (2015) found
that the addition of game-like points to an online math assessment somewhat improved task lik-
ability for middle school students while having no effect on the accuracy of responses. A related
line of research has investigated how game data can be used to generate evidence of ability, as
in Shute’s (2011) notion of “stealth assessments” or other investigations of the psychometrics of
games (GlassLab, 2014). Altogether, the prior research suggests that game-based assessments
have the potential to be both valid and more enjoyable than traditional assessments.
Track-It Assessment of Selective Sustained Attention in Young Children
Track-It (Fisher et al., 2013) is a performance-based measure of selective sustained attention
that was designed to be an isomorphic yet developmentally appropriate version of the Multiple
Object Tracking (MOT) task (Pylyshyn & Storm, 1988; Yantis, 1992). Performance on the
Track-It task is dependent upon a subject’s ability to sustain attention to a target object over
time while ignoring distractors.
The Track-It assessment (see Figure 1) consists of a set of distractor objects and a target that
move along random trajectories across a computer screen for a pre-determined period of time
(prior studies have used trial durations of 10 and 30 seconds) and then disappear. The subject’s
task is to identify the location where the target object disappeared. Prior research utilizing Track-
It revealed that performance on this assessment was significantly related to children’s learning
scores in a classroom-like setting (Fisher et al., 2013). Specifically, children who were better
able to track a target object amidst distractors were also likely to obtain higher scores on the
classroom learning task (r = 0.53, p = 0.01). This finding highlights the important contribution
of selective sustained attention on children’s academic performance.
Track-It is unique in its ability to assess within a single task both exogenous and endog-
enously-driven selective sustained attention by manipulating the different types of distractors
deployed in the task (e.g., homogenous and heterogeneous distractors). In the heterogeneous
distractors condition the distractors are all distinct from the target and from each other (e.g., red
triangle, green diamond, blue square). In this condition, Track-It performance is hypothesized to
reflect endogenously-driven selective sustained attention. The task requires effortful control of
attention as the target and distractors are equally salient and thus there is no contextual support.

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
22 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
In the homogenous distractor condition, the distractors are distinct from the target (e.g., yellow
circle) but identical to one another (e.g., red triangles). In this condition, Track-It performance
is hypothesized to reflect both endogenous as well as exogenously-driven selective sustained
attention as the task still requires effortful control; however, there is contextual support within
the task as the target is unique from all distractors and therefore more salient. Due to our interest
in endogenously-driven selective sustained attention, we utilized the heterogeneous distractors
condition in the present study.
DESIGN OF MONSTER MISCHIEF
The version of the MonsterMischief game described in this paper is a performance-based mea-
sure of endogenously regulated selective sustained attention. MonsterMischief was designed to
closely mirror the original heterogeneous condition of the Track-It assessment while providing
motivational design elements that were intended to sustain students’ intrinsic motivation to
participate and play the game. MonsterMischief (See Figure 2) features a set of colorful “mon-
sters” that run around various settings (e.g., a play room filled with toys) and hide behind an
Figure1.SchematicoftheTrack-ItGame

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 23
assortment of objects (e.g., a toy chest). This design feature stands in contrast to Track-It which
utilized simple shapes which were presented on a 3x3 grid.
The target character is indicated at the beginning of the trial by a glowing circle that encom-
passes the target. Children must click on the target character to initiate the trial. Then, the target
and distractor monsters run around the screen until they hide behind an array of objects in the
room (such as a toy chest or a rocking horse). If the player successfully clicks on the target’s
hiding spot, the character reveals itself and gives the player a jewel. The present study seeks to
determine whether the design elements of MonsterMischief increase children’s engagement and
motivation to play the instructional game, while still maintaining the validity of the measure.
The MonsterMischief game was designed to have a parallel structure to the Track-It assess-
ment (see below for details). Thus, we expected the following pattern of results: Performance
on MonsterMischief should be statistically equivalent to an existing assessment of selective
sustained attention (i.e., Track-It; Hypothesis 1), and Performance on MonsterMischief should
be significantly correlated with an existing assessment of selective sustained attention (i.e.,
Track-It; Hypothesis 2).
In order to create the parallel structure between MonsterMischief and Track-It, we engaged
in an iterative design process that was driven by the goal of having MonsterMischief require
equivalent cognitive processes as Track-It. This equivalency should occur when both tasks can
be described by the same abstract description: a target object was visually indicated to the user
Figure2.SchematicoftheMonsterMischiefvideogame.Thetargetmonsterishighlightedin
yellowatthebeginningofeachtrial

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
24 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
among 3 other objects; all objects then moved randomly about the screen for a period of time
and then disappeared. The user was then required to identify the target object’s location and to
discriminate the target object from other objects. Any task that is reasonably designed to fit this
description might be expected to have the same face validity. This abstracted equivalent task
structure is akin to having the same game mechanic, with different game aesthetics (Hunicke et
al., 2004). As a step in our design process, we articulated the mapped relations between the game
and the assessment. The key components of the Track-It task were incorporated into the Monster
Mischief game to ensure alignment and to equate Track-It and MonsterMischief on the level
of task difficulty. For example, the games were equated as closely as possible on the following
parameters: the number and type of distractors, the movement paths of the target and distractors,
the number of test trials, the number of test locations, and the number of lures included in the
memory check; see Table 1 for additional details on the alignment between Track-It and Mon-
sterMischief. It is important to note that one parameter, trial duration, was not directly aligned
between Track-It and MonsterMischief. Specifically, Monster Mischief had a slightly longer
trial duration than Track-It. The longer duration of the test trials in MonsterMischief compared
to Track-It was a design choice. This decision was based on the concern that MonsterMischief
might not be as attentionally demanding as Track-It due to the changes in the surface features
of the task which were intended to make MonsterMischief more engaging for young children.
Consequently a slightly longer trial duration was selected for MonsterMischief.
Engagement and Motivation
The primary design objective of MonsterMischief was to retain the features of the Track-It task
within a video game design that would be more engaging and motivating for the children to play;
see Table 2 for the full list of the motivational design elements. Thus, children should choose
to play and report liking MonsterMischief more than the existing selective sustained attention
assessment (i.e., Track-It; Hypothesis 3). Video games are intrinsically motivating (Ryan, Rigby,
& Przybylski, 2006); thus, incorporating game design features into an experimental task was
hypothesized to increase student engagement and motivation. In MonsterMischief, we sought to
increase the attractiveness of the game as well as its capacity to support child engagement over
time through intrinsic fantasy, narrative-driven curiosity, as well as achievement motivation.
Table1.AlignmentbetweenTrack-ItandMonsterMischief
Track-It Monster
Mischief
Distractors
(Number and Type)
3 unique distractors 
Motion
(Target/Distractor motion path)
Randomized path 
Trials
(Number)
5 trials
(1 practice trial and 4 test trials)

Trial Duration Approximately 10s Approximately 20s
Test Locations
(Hiding locations)
9 potential locations (3x3 grid) 
Memory Assessment
(Number of lures)
3

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 25
Prior research has demonstrated that introducing fantasy elements can increase the “motiva-
tional appeal” of an activity (Malone, 1981, Parker & Lepper, 1992). The MonsterMischief game
presents the assessment task as an intrinsic part of a fantasy context: the point of the tracking
task is to retrieve jewels stolen by the monsters. The game narrative is hypothesized to increase
players’ curiosity to discover what will happen over time (Malone, 1981; Loewenstein, 1994),
thus helping to maintain children’s engagement.
The Monster Mischief game also supports achievement motivation (Eccles & Wigfield,
2002; Ryan, Rigby, & Przybylski, 2006) by providing players with a clear goal (viz., click on
the hiding location of the monster that stole the jewel), a visual reward for attaining the goal
(viz., an animation of a magical jewel flying into the score board) and a small cost for failing to
attain the goal (viz., seeing an animation of the mischievous monster emerging from its hiding
location and giggling). In contrast, Track-It does not provide children with positive or negative
feedback that might support achievement motivation.
The three aforementioned hypotheses were tested in an experimental study of Monster
Mischief. Specifically, after designing a functional version of the MonsterMischief game, we
conducted an experimental study to assess the validity of the MonsterMischief game as an assess-
ment of endogenously driven selective sustained attention by comparing children’s performance
on Monster Mischief to their performance on the Track-It assessment in the Heterogeneous
Distractors condition. Additionally, we examined whether design elements of MonsterMischief
increased children’s engagement and motivation to play the video game.
Table2.MotivationalDesignElementsofMonsterMischiefcomparedtoTrack-It
Monster Mischief Track-It
Targets Cute animated “monsters” Simple shapes
Background The game takes place in colorful settings that are
familiar to young children (e.g., a play room)
9x9 grid
Test locations Characters hide behind the objects in the room
(e.g., the characters disappear behind the toys in the
play room)
The shapes disappear in one of the
9 squares
Feedback If the child finds the hiding location of the target
character, the character emerges from behind the object
and gives the child a jewel. If the child selects the
wrong location, the target character emerges from the
correct location and giggles.
No feedback is provided
Backstory At the beginning of the game the mischievous
monsters take jewels from the castle. The child’s task
is to find where the characters are hiding in order to
collect all of the jewels.
No backstory
Overarching
Goal
Collect all of the jewels at each level of the game.
Within each level, the number of jewels the child has
earned is displayed at the top of the screen. A map
marks the child’s progress across the game levels.
None

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
26 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
EXPERIMENTAL EVALUATION OF MONSTER MISCHIEF
Method
Participants
Thirty preschool children were recruited to participate in the present study. Three children were
ultimately excluded from the study: 1 child was excluded due to non-compliance with experimenter
instructions, 1 child was excluded due to an interruption in the testing session (i.e., a fire drill),
and 1 child was excluded due to experimenter error. The final sample included 27 children. The
mean age was 4.50 years (SD = 0.31 years). Children ranged in age from 4.08 years to 5.00 years.
Study Design
Children were tested individually by the first author of this paper or a hypothesis blind research
assistant in a room adjacent to the child’s classroom. The entire testing session lasted approxi-
mately 15 minutes. The testing session was divided into three phases. In Phase 1, children played
5 trials of either Track-It or MonsterMischief. In Phase 2, children played 5 trials of the alternate
game. In Phase 3, children were given a choice of playing either Track-It or MonsterMischief.
Children’s game choice in Phase 3 was recorded as an index of children’s motivation. Game
choice has been successfully used as a measure of motivation in the prior literature (Parker &
Lepper, 1992). Additionally, after each game children were asked to rate how much they liked
each game using a 5-point scale (i.e., the SmileScale; see appendix). The order in which the
games were presented was counterbalanced across participants (i.e., whether Track-It or Monster
Mischief was presented first).
Measures
Track-It Task: Children completed 5 trials of the heterogeneous condition of the Track-It task,
1 practice trial and 4 test trials. Each trial lasted for approximately 10 seconds. On each
trial, children saw four objects (e.g., simple shapes) moving on a computer screen: a target
object and three unique distractors. The objects move across a 3x3 gird and land on one of
nine locations. Each grid location was marked in a pastel color to assist children in reporting
the last location visited by the target. For each trial, the target object and the distractors are
randomly selected from an array of simple shapes. Children are asked to watch a particular
object (i.e., target object) while ignoring the rest of the objects (i.e., the distractors). When the
objects stop moving and disappear from the computer screen, children are asked to identify
the location last visited by the target object; see Figure 1. After each trial, a memory check
is administered to ensure that children were tracking the intended target. In the memory
check, children are presented with an array of four objects (i.e., the target and three lures)
and asked to identify the specific object they were tracking. For both the experiment proper
and the memory check, children can provide a verbal or non-verbal response (i.e., point to
the location/object). Overall, Track-It provides two performance indices: Attention (indicated
by the accuracy of identifying the last location visited by the target) and Memory (indicated
by the accuracy of identifying the target object on the memory check).
Monster Mischief Game: Children completed 5 trials of the MonsterMischiefGame, 1 practice
trial and 4 test trials. Each trial lasted for approximately 20 seconds. In the MonsterMischief
Game, children see a set of cute and friendly monsters (one target and three distractors).
During the game, the characters run around various settings (e.g., a play room) and hide

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 27
behind common objects; see Figure 2. At the beginning of the trial, one of the characters
is identified as the target (i.e., the target character is encircled by a glowing ring). At the
end of the trial, all of the characters hide behind various objects (e.g., disappear behind a
toy chest, rocking horse, etc.). If the child correctly identifies the target character’s hiding
spot (by clicking on the appropriate object), the target character will reveal itself and the
child will earn a jewel. At the end of each trial a memory check is administered. Children
are presented with an array of four characters (i.e., the target and three lures) and asked to
identify the specific character they were tracking. For both the experiment proper and the
memory check, children can provide a verbal or non-verbal response (i.e., point to the location/
character). Similar to Track-It, Monster Mischief provides Attention and Memory measures.
Due to constraints relating to school policies regarding how long children may be absent from
the classroom while participating in research, administering additional test trials was not possible
within the allotted time frame. Thus, within a single testing session children were only able to
complete 10 trials, 5 trials of MonsterMischief and 5 trials of Track-It.
Smile Scale: Before starting the experiment proper, children were introduced to the SmileScale.
The SmileScale is a child friendly version of a 5-point likert scale (Read & MacFarlane,
2006). The SmileScale includes five faces (images were obtained from the Google search
engine) that exhibit a range of facial expressions from a big frown to a big smile. Children
were given three practice items (e.g., “WhichfaceshouldIpointtoifItoldyouIreallylike
toJumpRope”). The practice items were included to ensure that the children understood
how to use the scale. See the appendix for the scale, script, and practice trials that were used
to familiarize children with the SmileScale.
Game Choice: After playing both games, children were presented with a piece of paper that
included a screen shot of each game. The screen shots served as a memory cue for the
children. Children were told that they only had a few minutes left before they would return
to class and so the child could choose which game he or she wanted to play. After children
made their selection, they were asked to provide a rationale for their choice (e.g., “Great,
whydidyouchoosethe_____game?”). The child’s response was recorded. Then, the child
played one trial of the game he or she selected.
Results
On both TrackIt and MonsterMischief, children obtained high Memory scores (M = 0.91, SD
= 0.19, range 0.25-1.00 and M = 0.86, SD = 0.21, range 0.25-1.00, respectively). This suggests
that children accurately encoded the identity of the target object, which is a necessary precondi-
tion for successfully tracking a target object moving amidst distractors. Children also achieved
relatively high Attention scores (M = 0.74, SD = 0.26, range 0.25-1.00 and M = 0.73, SD = 0.28
range 0.00-1.00, for Track-It and MonsterMischief respectively); see Figure 3.
Concurrent Validity of Monster Mischief
In order to determine the validity of MonsterMischief and thereby test Hypotheses 1 and 2, we
examined the performance alignment between MonsterMischief and the Track-It task. Children’s
performance on the memory check for the Track-It task and the MonsterMischief game were
not significantly different from each other (paired samples t(26) = 1.22, p = 0.23 ns). A test of
equivalence was also conducted using Weber and Popova’s (2012) Paired-Samples Equivalence

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
28 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Procedure in order to ascertain whether memory performance was statistically equivalent in
MonsterMischief and Track-It. The minimum substantial effect (Δ = 0.5) was selected based on
Cohen’s (1988) guidelines for a medium effect. The equivalence test was statistically significant
suggesting that the memory demands of the two games are comparable; t(26) = 1.22, p = 0.025.
Statistical equivalence could not be confirmed using more conservative (i.e., smaller) effect size
levels (Δ = 0.3 or 0.1 both ps > 0.27). However, the failure to find evidence of equivalence at
these more conservative levels should be interpreted with caution due to the small sample size
utilized in the present paper and the known power problems of equivalence tests (see Weber
& Popova, 2012). Children’s memory check scores on the two games were also found to be
significantly correlated (r = 0.52, p = 0.006), suggesting that these measures tapped reliable
individual differences in memory encoding.
Similarly, there was no significant difference on the Attention scores of Track-It and Monster
Mischief (paired samples t(26) = 0.20, p = 0.84 ns). A test of equivalence was also performed
in order to determine whether children’s attention scores in MonsterMischief and Track-It were
statistically equivalent. The minimum substantial effect (Δ = 0.5) was again selected based on
Cohen’s (1988) guidelines for a medium effect. Children exhibited equivalent attention perfor-
mance indicating that the difficulty level of the two games is comparable; t(26) = -0.20, p =
0.001. Akin to the results for children’s memory performance statistical equivalence could not
be confirmed using more conservative effect size levels (Δ = 0.3 or 0.1, ps = 0.053 and 0.35
Figure3:MeanAttentionandMemory scoresfor theTrack-Ittaskandthe MonsterMischief
Game.Errorbarsrepresentthestandarderrorsofthemeans.Standarderrorsofthemeansare
asfollows,Attention:Track-It=0.05,MonsterMischief=0.053;MemoryCheck:Track-It=
0.035,MonsterMischief=0.041

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 29
respectively). Children’s attention scores on the two games were also significantly correlated
(r = 0.62, p = 0.001), suggesting that these measures tapped reliable individual differences in
endogenously driven selective sustained attention.
The results of the paired sample t-tests, equivalence tests, significant correlations between
MonsterMischief and Track-It, along with the face validity of MonsterMischief (due to the high
degree of similarity across the two tasks) provide converging evidence for the validity of Monster
Mischief as an assessment of endogenously driven selective sustained attention.
Comparison of Enjoyment & Motivation
Smile Scale: The SmileScale was included in the present study to test Hypothesis 3 by quantify-
ing how much children enjoyed each game. We hypothesized that that children should report
liking MonsterMischief more than the existing selective sustained attention assessment and
thus the SmileScale scores should be higher for Monster Mischief than Track-It.
The SmileScale was incorporated after data collection commenced. As a result, data for 18 of the
27 children were obtained. From this limited dataset, children tended to report that they liked both
the Track-It task and the MonsterMischief game. For the Track-It task the average score on the
SmileScale was 4.50 (SD = 0.79). Similarly, on the MonsterMischief game, children’s average
score on the SmileScale was 4.44 (SD = 0.86); paired sample t(17) = 0.22, p = 0.83 ns. Failure
to observe any significant differences on the SmileScale may be due to social desirability effects,
namely children may have been hesitant to report that they did not like either one of the games.
Additionally, children only completed 5 trials of each game. It is possible that differences may
emerge if playtime were extended. For example, with more prolonged exposure the novelty of
both tasks may wane. However, children’s engagement in MonsterMischief may be maintained
due to the incorporation of the motivational design elements. In contrast, engagement in Track-
It, which lacks these game features, may decline. In general, children did not tend to utilize the
entire SmileScale, resulting in a truncated range of scores. For both games, the scores on the
SmileScale ranged from 3 (i.e., “thegameisjustokay”) to 5 (i.e., “Ireallylikedthegame”).
Enjoyment was not found to be related to children’s performance on Track-It or MonsterMischief,
as SmileScale ratings were not significantly correlated with children’s attention performance
scores (both rs < 0.28, ps > 0.26 ns). This pattern of result may be due in part to the truncated
range in children’s SmileScale scores as the limited variability in children’s SmileScale ratings
makes it more difficult to detect an association between enjoyment and children’s performance.
Game Choice: To further test Hypothesis 3, children’s game choice patterns were analyzed
to determine if children would choose to play Monster Mischief more than the existing
selective sustained attention assessment. After playing both games children were asked to
choose which game they wanted to play for the remainder of the testing session (Track-It
or MonsterMischief).
The vast majority of children selected the MonsterMischief game as their free-choice game
option. In fact 74% of children (20 out of 27) selected Monster Mischief while only 26% of
children (7 out of 27) selected the Track-It task. Children selected MonsterMischief as their free-
choice game more than would be expected by chance (0.50). The cumulative binomial probability
that 20 or more of the 27 children would select MonsterMischief as their free-choice game was

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
30 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
0.0095. Additionally, children’s game choice was not related to their attention performance on
Track-It or MonsterMischief (both rs < 0.115, ps > 0.567 ns).
After children made their selection, they were asked to provide a rationale for their choice
(e.g., “Great,whydidyouchoosethe_____game?”). The child’s response was recorded and
the child then played one trial of the game they selected. Children’s responses were analyzed.
Many children had trouble providing a rationale for their game selection. For the 7 children who
selected Track-It as their free-choice, 1 child gave no response, 1 child indicated that they did not
know why they selected the game, and 3 children reported that they selected Track-It because
they simply “liked” the game. The remaining 2 children reported that their game selection was
based on aspects of the game (e.g., “they [the shapes] move”; “becausewedidn’tdoallofthe
shapes”). Of the 20 children who selected MonsterMischief, 13 children stated that they selected
the game simply because they “likedit” or reported that the game was “fun”. The remaining 7
children indicated that they selected MonsterMischief due to novelty or various game design
elements (e.g., “becausethey [the monsters] popoutatyou”; “becauseIlikecastles”; “because
it’ssomuchfun–themonstershide”; “becauseweonlyplayed2or3monstersandIwantto
playthepurpleone”).
Finally, we examined whether children’s free-choice was influenced by the order in which
the tasks were presented. Recall that the presentation order of Track-It and MonsterMischief was
counter-balanced, such that some children played Track-It first and some children played Monster
Mischief first. It is possible that children may exhibit a preference to switch from the game that
they had just played; therefore, we examined whether children were more or less likely to exhibit
this ‘switch preference’ when the last game they played was MonsterMischief vs. Track-It. When
MonsterMischief was the second game children played, children were unlikely to exhibit a switch
preference towards Track-It: only 36% (5 of 14) of the children chose to switch to Track-It, while
64% (9 of 14) of children went against switching and chose to play MonsterMischief again. In
contrast, when Track-It was the second game children played, only 15% (2 of 13) of the children
chose to play Track-It again, while the majority of children exhibited a preference to switch to
MonsterMischief (85% or 11 of 13); see Figure 4. The association between switch preference
and game type (Track-It or Monster Mischief) was statistically significant; Fisher’s Exact test
p = 0.0183. Overall, this analysis indicates that children were less likely to exhibit a preference
to switch away from the game they just played when MonsterMischief was administered last,
but more likely to show a switch preference when Track-It was administered last. This finding
points to MonsterMischief as the more appealing choice over Track-It.
Learning Curve Analysis
A learning curve analysis was conducted to examine whether the task provides some context for
learning. Learning curves are defined as changes in task performance over opportunities (Ritter
& Schooler, 2001). In Figure 5, the average success rates for the memory check and locating the
target are presented over the four experimental trials in both Track-It and MonsterMischief. For
reasons discussed in the Method section, only four test trials of each game were administered.
With only four trials, there is little data that might demonstrate a significant learning curve.
Regular improvements in performance over each trial are observed in children’s MonsterMischief
performance; however, in a single factor regression model predicting student performance, the
trial opportunity number was an insignificant predictor of success (Track-It:p = 0.70; Monster
Mischief:p = 0.33). Future studies involving a greater number of trials will be necessary to
identify significant learning curves, if any.

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 31
CONCLUSION
In conclusion, this study designed and evaluated Monster Mischief, a video game for the as-
sessment of selective sustained attention. Motivational design elements were incorporated to
create a video game that was engaging and motivating to young children. We then conducted
an experimental study to test the validity of MonsterMischief. A valid game-based assessment
of endogenously driven selective sustained attention would show an equivalent level of per-
formance to an existing assessment, would be significantly correlated with performance on the
existing assessment, and be more enjoyable than the existing assessment. The findings from the
present experiment largely align with the expected pattern of results. Specifically, the results
of the present study suggest that we successfully created a video game that was of comparable
difficulty level to the Track-It task. Additionally, performance on Track-It and MonsterMischief
was significantly correlated. Children also exhibited a preference for Monster Mischief over
Track-It when asked what game they would like to play again, presumably due to the addition
of motivational design elements. However, there were no significant difference in enjoyment
observed from the results of the SmileScale.
Limitations
Overall, the MonsterMischief video game shows potential as an engaging assessment of chil-
dren’s selective sustained attention; however, it is important to note several limitations of the
Figure4.Children’sfree-choicegameselectionasafunctionoftheorderinwhichtheassess-
mentswerepresented(i.e.,MonsterMischieffirstfollowedbyTrack-ItorTrack-Itfirstfollowed
byMonsterMischief)

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
32 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
present study. First, we examined the relationship between children’s performance in Monster
Mischief and the heterogeneous condition of Track-It. Currently, it is unknown whether Monster
Mischief can be adapted to measure exogenously driven selective sustained attention as well.
Akin to Track-It, the ideal assessment would enable the measurement of both endogenous and
exogenous factors within a single task or game to provide a more comprehensive analysis of the
development and contribution of each type of selective sustained attention. Second, although we
compared MonsterMischief to Track-It, it would also be beneficial to determine if children’s
performance on MonsterMischief is related to other measures of attention and if it is predictive
of other cognitive abilities (e.g., inhibitory control).
A related issue of particular importance concerns the strength of the correlation between
MonsterMischief and Track-It. Although the correlation between the two attention performance
measures is respectable (r = 0.623), the two measures are not highly correlated, suggesting
that additional information is contained within these scores. Thus, future research will need to
distinguish among three different possibilities: (1) Track-It measures selective sustained atten-
tion better than MonsterMischief, (2) MonsterMischief measures selective sustained attention
better than Track-It, or (3) Both tasks measure selective sustained attention to a similar degree
but each task reflects variability associated with task-specific features (e.g., longer trial duration
in MonsterMischief). No task is able to provide a pure measure of the construct of interest, a
problem known as “task impurity” in the literature (Miyake, Friedman, Emerson, Witzki, How-
erter, & Wager, 2000). Thus, it is likely that performance on MonsterMischief (and Track-It)
reflects selective sustained attention as well as other cognitive abilities and task artifacts. Thus,
future research examining the validity of these measures may help adjudicate between the pos-
sibilities listed above.
Third, extensions to this work are warranted as it is critical to assess children’s performance
and engagement in MonsterMischief over more extended periods of play and across time. With
Figure5.LearningCurveAnalyses fortheTrack-Ittask(PanelA) and theMonsterMischief
Game(PanelB).Inbothpanels,thetoplineshowsthesuccessrateforidentifyingthecorrect
locationofthehiddentargetacrosseachtaskopportunity(i.e.,Attentionscore),whilethebot-
tomlineshowsthesuccessrateforcorrectlyrememberingwhichtargetwasbeingtracked(i.e.,
MemoryCheck).Errorbarsrepresentthestandarderrorsofthemean

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 33
this groundwork in place, we can also begin to collect additional data on the psychometric util-
ity of a broader range of difficulty factors in the game. Lastly, additional research is needed to
fully validate MonsterMischief as a reliable measurement tool. For example, the reliability of
the game should be assessed either through test-retest reliability (comparing the correlation of
student performance over multiple testing sessions) or by introducing multiple items with vari-
able difficulty and reporting on the reliability of the different items (i.e., Cronbach’s Alpha).
Difficulty can be manipulated in MonsterMischief and TrackIt, by varying several parameters
including: speed, number of distractors, number of hiding locations, and the trial duration (i.e.,
the amount of time before the target hides/disappears).
Future Directions
During the development of MonsterMischief, we also produced three additional game mechanics
that could be employed in future research. These game mechanics were intended to increase the
difficulty of the game in order to use the game with a wider age range of children and to explore
the possibility of using MonsterMischief as an instructional tool that may help train children’s
selective sustained attention capacity. Additionally, the game mechanics afford an opportunity to
explore whether MonsterMischief could be adapted to evaluate additional aspects of attention such
as divided attention. The additional game mechanics include: coins, fireworks, and cloud mode.
Each mechanic is discussed briefly below. When coins are turned on, coins appear at random for
a random period of time. If a player clicks on a coin, it is added to their score. The rationale for
this mechanic is to divide a player’s attention between the primary task (tracking the monster that
stole the jewel) and a secondary task (collecting coins). When the firework mechanic is turned
on, small red fireworks appear at random and then pop with a shower of sparks. This mechanic
was produced because young children often have trouble reorienting to a task and returning to
a state of focused attention after attention has been disrupted (DiLalla & Watson, 1988). When
cloud mode is turned on, there are no specific objects behind which the monsters hide – they
simply disappear. Then, players have to click on the area where the target monster disappeared
without the aid of visual landmarks. In this mode, we can measure the visual-spatial accuracy
of a player’s response (where accuracy is defined as 100% minus the percent error, where per-
cent error is calculated as the distance between x,y coordinates of the clicked location and the
actual location, divided by the x,y dimensions of the screen). The rationale for this mechanic is
to provide a more nuanced measure of success (degree of accuracy instead of correct/incorrect)
and to increase difficulty by removing memorable landmarks.
Future work should also explore the game as an instructional tool that may help train children’s
selective sustained attention capacity and reduce their susceptibility to distractions. Presently,
it is an open question whether a cognitive skills game for children could both assess selective
sustained attention and also help improve it, akin to the cognitive skills training games that are
currently popular for adults (e.g., Lumosity & Posit Science). The makers of cognitive skills
training games claim that improving performance on an assessment of a cognitive skill could
broadly transfer to other activities (e.g., one could improve working memory by practicing a
game which tests one’s working memory). Although it is well documented that transfer of skills
from one context to another is quite difficult (e.g., Perkins & Solomon, 1988), the potential gains
that could be achieved from such an intervention warrant further empirical work. Ultimately,
a video game like Monster Mischief might be used to test the hypothesis that cognitive skills
practice can increase the attention span of young children and decrease their susceptibility to
distractions. Accordingly, it is of interest to explore whether interventions could be created to
help support the development of selective sustained attention in young children.

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
34 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
ACKNOWLEDGMENT
We would like to thank Tara Helfer and Jeremy Galante for their help with illustrations and ani-
mation. We also thank Sharan Shodhan for game programming. We thank the children, parents,
teachers, and staff at the Children’s School and Baldwin United Methodist Church Preschool
who made this work possible. This work was supported in part by a Graduate Training Grant by
the Department of Education (R305B090023).
REFERENCES
Attali, Y., & Arieli-Attali, M. (2015). Gamification in assessment: Do points affect test performance?
Computers&Education, 83, 57–63. doi:10.1016/j.compedu.2014.12.012
Bloom, B. S. (1976). HumanCharacteristicsandSchoolLearning. New York: McGraw-Hill.
Bornstein, M. H. (1990). Attention in infancy and the prediction of cognitive capacities in childhood. In J.
Enns (Ed.), DevelopmentofAttention:ResearchandTheory. Elsevier. doi:10.1016/S0166-4115(08)60448-3
Carroll, J. B. (1963). A model of School Learning. TeachersCollegeRecord, 64, 723–733.
Choudhury, N., & Gorman, K. (2000). The relationship between attention and problem solving in 17–24 month
old children. InfantandChildDevelopment, 9, 127–146. doi:10.1002/1522-7219(200009)9:3<127::AID-
ICD225>3.0.CO;2-5
Clark, D., Tanner-Smith, E., Killingsworth, S., & Bellamy, S. (2013). DigitalGamesforLearning:ASys-
tematicReviewandMeta-Analysis(ExecutiveSummary). Menlo Park, CA: SRI International.
Colombo, J., & Cheatham, C. L. (2006). The emergence and basis of endogenous attention in infancy and
early childhood. In R. Kail (Ed.), AdvancesinChildDevelopmentandBehavior (Vol. 34, pp. 283–310).
Oxford: Academic Press. doi:10.1016/S0065-2407(06)80010-8
DeMarie-Dreblow, D., & Miler, P. H. (1988). The development of children’s strategies for selective at-
tention: Evidence for a transitional period. ChildDevelopment, 59(6), 1504–1513. doi:10.2307/1130665
PMID:3208562
Diamond, A. (2002). Normal development of prefrontal cortex from birth to young adulthood: Cognitive
functions, anatomy, and biochemistry. In D. T. Stuss & R. T. Knight (Eds.), Principlesoffrontallobefunction
(pp. 466–503). London, UK: Oxford University Press. doi:10.1093/acprof:oso/9780195134971.003.0029
Diamond, A. (2006). The early development of executive functions. In E. Bialystok & F. Craik (Eds.),
Lifespan Cognition: Mechanisms of Change (pp. 7-95). Oxford University Press. doi:10.1093/acprof:o
so/9780195169539.003.0006
DiLalla, L. F., & Watson, M. W. (1988). Differentiation of fantasy and reality: Preschoolers’ reactions to
interruptions in their play. DevelopmentalPsychology, 24(2), 286–291. doi:10.1037/0012-1649.24.2.286
Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., & Japel, C.
et al. (2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428–1446.
doi:10.1037/0012-1649.43.6.1428 PMID:18020822
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. AnnualReviewofPsychology,
53(1), 109–132. doi:10.1146/annurev.psych.53.100901.135153 PMID:11752481
Erickson, L. C., Thiessen, E. D., Godwin, K. E., Dickerson, J. P., & Fisher, A. V. (2014). Endogenously- but
not Exogenously-driven Selective Sustained Attention is Related to Learning in a Classroom-like Setting in
Kindergarten Children. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedingsofthe36th
AnnualConferenceoftheCognitiveScienceSociety (pp. 457-462). Austin, TX: Cognitive Science Society.

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 35
Fisher, A. V., Godwin, K. E., & Seltman, H. (2014). Visual environment, attention allocation, and
learning: When too much of a good thing may be bad. Psychological Science, 25(7), 1362–1370.
doi:10.1177/0956797614533801 PMID:24855019
Fisher, A. V., & Kloos, H. (in press). Development of selective sustained attention: The Role of executive
functions. In L. Freund, P. McCardle, & J. Griffin (Eds.), Executive functioninpreschoolage children:
Integratingmeasurement,neurodevelopmentandtranslationalresearch. APA Press.
Fisher, A. V., Thiessen, E., Godwin, K., Kloos, H., & Dickerson, J. P. (2013). Mechanisms of focused at-
tention in 3- to 5-year-old children: Evidence from a new object tracking task. JournalofExperimental
ChildPsychology, 114(2), 275–294. doi:10.1016/j.jecp.2012.07.006 PMID:23022318
Geary, K. E. (2011). Theimpactofchoiceonchildsustainedattentioninthepreschoolclassroom (Unpub-
lished Thesis). Louisiana State University, Baton Rouge, Louisiana.
Gitelman, D. R. (2003). Attention and its disorders: Imaging in clinical neuroscience. British Medical
Bulletin, 65(1), 21–34. doi:10.1093/bmb/65.1.21 PMID:12697614
Glasslab. (2014). Pyschometric Considerations in Game-Based Assessments. White Paper Released by
Glasslab, 160.
Godwin, K. E., Almeda, M. V., Petroccia, M., Baker, R. S., & Fisher, A. V. (2013). Classroom activities
and off-task behavior in elementary school children. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth
(Eds.), Proceedings of the 35th Annual Conferenceof the Cognitive Science Society (pp. 2428-2433).
Austin, TX: Cognitive Science Society.
Goodman, L. (1990). Timeandlearninginthespecialeducationclassroom. SUNY Press.
Hunicke, R., LeBlanc, M., & Zubek, R. (2004). MDA: A formal approach to game design and game re-
search. In Proceedings of the WorkshoponChallengesinGameAI,19thNationalConferenceonArtificial
Intelligence, AAAI Press, San Jose, California.
Jonides, J. (1981). Voluntary vs. Automatic control over the mind’s eye’s movement. In J. B. Long & A. D.
Baddeley (Eds.), AttentionandPerformanceIX. Hillsdale, N.J.: Lawrence Erlbaum Associates.
Loewenstein, G. (1994). The Psychology of curiosity: A review and reinterpretation. PsychologicalBul-
letin, 116(1), 75–98. doi:10.1037/0033-2909.116.1.75
Luna, B. (2009). Developmental changes in cognitive control through adolescence. Advances in Child
DevelopmentandBehavior, 37, 233–278. doi:10.1016/S0065-2407(09)03706-9 PMID:19673164
Malone, T. (1981). Toward a theory of intrinsically motivating instruction. CognitiveScience, 5(4), 333–369.
doi:10.1207/s15516709cog0504_2
McPherson, J., & Burns, N. R. (2008). Assessing the validity of computer-game-like tests of processing
speed and working memory. BehaviorResearchMethods, 40(4), 969–981. doi:10.3758/BRM.40.4.969
PMID:19001388
Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. AnnualReviewof
Neuroscience, 24(1), 167–202. doi:10.1146/annurev.neuro.24.1.167 PMID:11283309
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The
unit and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent
variable analysis. CognitivePsychology, 41(1), 49–100. doi:10.1006/cogp.1999.0734 PMID:10945922
Oakes, L., Kannass, K. N., & Shaddy, D. J. (2002). Developmental changes in endogenous control of at-
tention: The role of target familiarity on infants’ distraction latency. ChildDevelopment, 73(6), 1644–1655.
doi:10.1111/1467-8624.00496 PMID:12487484
Parker, L., & Lepper, M. (1992). Effects of fantasy contexts on children’s learning and motivation: Mak-
ing learning more fun. JournalofPersonalityandSocialPsychology, 62(4), 625–633. doi:10.1037/0022-
3514.62.4.625 PMID:1583588

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
36 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Pashler, H., Johnston, J. C., & Ruthruff, E. (2001). Attention and performance. AnnualReviewofPsychol-
ogy, 52(1), 629–651. doi:10.1146/annurev.psych.52.1.629 PMID:11148320
Perkins, D. N., & Salomon, G. (1988). Teaching for transfer. EducationalLeadership, 46(1), 22–32.
Posner, M. I. (1980). Orienting of attention. TheQuarterlyJournal ofExperimentalPsychology, 32(1),
3–25. doi:10.1080/00335558008248231 PMID:7367577
Posner, M. I., & Petersen, S. E. (1990). The attention system of the human brain. AnnualReviewofNeu-
roscience, 13(1), 25–42. doi:10.1146/annurev.ne.13.030190.000325 PMID:2183676
Posner, M. I., & Rothbart, K. R. (2007). Research on attention networks as a model for the integration of psy-
chological science. AnnualReviewofPsychology, 58(1), 1–23. doi:10.1146/annurev.psych.58.110405.085516
PMID:17029565
Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a parallel
tracking mechanism. SpatialVision, 3(3), 179–197. doi:10.1163/156856888X00122 PMID:3153671
Read, J. C., & MacFarlane, S. (2006) Using the fun toolkit and other survey methods to gather opinions in
child computer interaction. Proceedingofthe2006conferenceonInteractiondesignandchildren-IDC
’06, 81. doi:10.1145/1139073.1139096
Richards, J. E. (2003). The development of visual attention and the brain. In De Haan & Johnson (Eds.),
The cognitive neuroscience of development. East Sussex, UK: Psychology Press.
Ritter, F., & Schooler, L. (2001) The learning curve. International Encyclopedia of the Social and Behav-
ioral Sciences, 8602–8605.
Rosvold, H. E., Mirsky, A. F., Sarason, I., Bransome, E. D., & Beck, L. H. (1956). A Continuous perfor-
mance test of brain damage. JournalofConsultingPsychology, 20(5), 343–350. doi:10.1037/h0043220
PMID:13367264
Ruff, A. H., & Lawson, K. R. (1990). Development of sustained, focused attention in young children during
free play. DevelopmentalPsychology, 26(1), 85–93. doi:10.1037/0012-1649.26.1.85
Ruff, H., & Capozzoli, M. (2003). Development of attention and distractibility in the first 4 years of life.
DevelopmentalPsychology, 39(5), 877–890. doi:10.1037/0012-1649.39.5.877 PMID:12952400
Ruff, H. A., & Rothbart, M. K. (1996). Attentioninearlydevelopment:Themesandvariations. New York,
NY: Oxford University Press.
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A self-determination
theory approach. MotivationandEmotion, 30(4), 347–365. doi:10.1007/s11031-006-9051-8
Sarid, M., & Breznitz, Z. (1997). Developmental aspects of sustained attention among 2- to 6-year-old
children. InternationalJournalofBehavioralDevelopment, 21(2), 303–312. doi:10.1080/016502597384884
Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detec-
tion, search, and attention. PsychologicalReview, 84(1), 1–66. doi:10.1037/0033-295X.84.1.1
Setliff, A. E., & Courage, M. L. (2011). Background television and infants allocation of their attention
during toy play. Infancy, 16(6), 611–639. doi:10.1111/j.1532-7078.2011.00070.x
Shute, V. J. (2011). Stealth Assessment in Computer-Based Games To Support Learning. ComputerGames
andInstruction, 503–524.
Tellinghuisen, D. J., Oakes, L. M., & Tjebkes, T. L. (1999). The influence of attentional state and stimulus
characteristics on infant distractibility. Cognitive Development, 14(2), 199–213. doi:10.1016/S0885-
2014(99)00002-7
Tenorio Delgado, M., Arango Uribe, P., Aparicio Alonso, A., & Rosas Díaz, R. (2014). TENI: A compre-
hensive battery for cognitive assessment based on games and technology. ChildNeuropsychology, 1–16.
doi:10.1080/09297049.2014.977241 PMID:25396766

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 37
Weber, R., & Popova, L. (2012). Testing equivalence in communication research: Theory and application.
CommunicationMethodsandMeasures, 6(3), 190–213. doi:10.1080/19312458.2012.703834
Yantis, S. (1992). Multielement visual tracking: Attention and perceptual organization. CognitivePsychol-
ogy, 24(3), 295–340. doi:10.1016/0010-0285(92)90010-Y PMID:1516359
Yu, C., & Smith, L. B. (2012). Embodied attention and word learning by toddlers. Cognition, 125(2),
244–262. doi:10.1016/j.cognition.2012.06.016 PMID:22878116

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
38 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
APPENDIX
Introduction of the Smile Scale
“DuringthisgameIwillaskyouacouplequestionsusingaspecialscale.Hereishowitworks.
Seeallthefaces [point to the faces on the scale]?Seehowsomeofthefacesarefrowningand
somearesmiling?Lookatthisface [point to face 1]. Thisfaceismakinga big frown-thatmeans
Ireally don’t likesomething.Thisfacehasa little frown [point to face 2] –thatmeansI dislike
somethinga little bit. Lookatthisface [point to face 5]. Thisfacehasa big smile-thatmeansI
really like something.Thisfacehasa little smile [point to face 4] –thatmeansIlikesomething
a little bit. Lookatthisface [point to face 3], thisfaceis not smiling or frowningsothatmeans
Ithinksomethingis just okay.”
Practice Items
1. “WhichfaceshouldIpointtoifItoldyouthatI really don’t like brusselssprouts?”
[Wait for child’s response]
“Yes,Iwouldpointtothefacemakingabigfrown [point to face 1] becauseIreallydon’t
likebrussels sprouts.” [Or: “That’sagoodguess,but Iwouldpointto thefacemakinga big
frown [point to face 1] becauseIreallydon’tlikebrusselssprouts.”]
2. “Okay, try this one - which face should I point to if I told you that I really like to jump
rope?”
[Wait for child’s response]
“Yes,Iwouldpointtothefacemakingabigsmile [point to face 5] becauseIreallyliketo
jumprope.” [Or: “That’sagoodguess,butIwouldpointtothefacemakingabigsmile [point
to face 5] becauseIreallyliketojumprope.”]
3. “Okay let’s do one more. Which face should I point to if I told you I think the color green
is just okay?”
[Wait for child’s response]
“Yes,Iwouldpointtothisfacethatisnotsmilingorfrowning [point to face 3] becausethat
meansIthinksomethingisjustokay.” [Or: “That’sagoodguess,butIwouldpointtothisfacethat
isnotsmilingorfrowning [point to face 3] becausethatmeansIthinksomethingisjustokay.”]
Figure6.Smilescale

Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 39
“All right we will use this again in a little bit but now we are going to play some computer
games”
Rating Each Game using the Smile Scale
“Okay,nowIwantyoutopointto thefacethatshowsmewhatyouthoughtaboutthisgame:
Rememberthisfacemeansyou really don’t like thisgame [point to 1], thisfacemeansyou dis-
liked thisgame a little [point to 2], thisfacemeansyouthoughtthegamewas just okay [point
to 3], thisfacemeansyou liked thegame a little [point to 4], andthisfacemeansyou really like
the game [point to 5]. Okaywhatdidyouthinkaboutthegame-whateveryouthinkisfine.”