18 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
Designing a Video Game to Assess
Selective Sustained Attention
Attention is multidimensional and encompasses a diverse set of psychological constructs, includ-
ing (but not limited to) orienting, selection, shifting, maintenance, and executive attention (see
Colombo & Cheatham, 2006; Gitelman, 2003; Fisher & Kloos, in press; Posner & Petersen,
1990). The present work focuses on selective sustained attention which is defined as: “a state
of engagement that involves narrowed selectivity and increased commitment of energy and
resources on the targeted activity … and that primarily enhances information processing in that
system” (Setliff & Courage, 2011, p. 613). Selective sustained attention is important because
the ability to selectively allocate attentional resources is commonly hypothesized to aid learn-
Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 19
ing (e.g., Carroll 1963; Bloom, 1976; Oakes, Kannass, & Shaddy, 2002). As stated by Oakes
and colleagues (2002), “if attention were constantly reoriented to every new event, it would be
difficult ... to learn about any single object or event” (p.1644). In accordance with this assertion,
prior research has implicated selective sustained attention in task performance (e.g., Choudhury
& Gorman, 2000; DeMarie-Dreblow & Miller, 1988), academic achievement (e.g., Duncan et al.
2007; for review see Goodman, 1990), and learning outcomes (e.g., Fisher, Thiessen, Godwin,
Dickerson, & Kloos, 2013; Fisher, Godwin, & Seltman, 2014; Yu & Smith, 2012).
Theoretical Perspectives on Selective Sustained Attention
There has been a long tradition in the cognitive psychology literature of distinguishing between
two modes of attention regulation. One mode refers to regulation that is said to be top-down,
controlled, and endogenous (i.e., arising from within the organism); whereas the other mode is
commonly referred to as bottom-up, automatic (or in some cases automatized through extensive
practice), and exogenous (i.e., arising from outside the organism) (e.g., Jonides, 1981; Miller &
Cohen, 2001; Posner, 1980; Pashler, Johnston, & Ruthruff, 2001; Schneider & Shiffrin, 1977). In
the present paper, we are adopting the developmental framework of attention regulation put forth
by Ruff and Rothbart (1996). This framework builds upon the dual-mode of attention regulation
suggested by the above theories. Specifically, according to Ruff and Rothbart (1996) the state of
selective sustained attention can be obtained through two distinct systems: an orienting system
and an executive control system. Under the orienting system, selective sustained attention is
driven by exogenous factors such as the saliency of the stimuli including characteristics such as
the brightness or contrast of the stimuli as well as motion. Thus, exogenously driven selective
sustained attention is considered largely an automatic process driven by the physical character-
istics of the environment or the stimulus (Bornstein, 1990; Ruff & Rothbart, 1996). Under the
executive control system, selective sustained attention is driven by endogenous factors; in other
words the focus of attention is directed internally based on the individual’s desires, interests,
and goals (Colombo & Cheatham, 2006; Posner & Petersen, 1990; Posner & Rothbart, 2007).
The two systems are said to be neurally and anatomically distinct, with the orienting system
involving areas within the parietal lobe, superior colliculus, and lateral pulvinar nucleus of the
thalamus, and the executive system involving areas within the prefrontal cortex (PFC) and an-
terior cingulate gyrus (fore review see Fisher & Kloos, in press).
In the present paper we focus on endogenously driven selective sustained attention as prior
research suggests that endogenously regulated selective sustained attention may be particu-
larly important for learning in education settings (Erickson, Thiessen, Godwin, Dickerson, &
Fisher, 2014). In education settings children need to attend to an externally-prescribed learning
objective, but often face multiple sources of distraction (Godwin, Almeda, Petroccia, Baker,
& Fisher, 2013). Children who are better able to endogenously regulate their attention may be
more likely to maintain focus on the learning objective in the face of distractions and thus have
better learning outcomes.
Development of Selective Sustained Attention
The orienting and executive control systems follow different developmental trajectories. The
orienting system matures during infancy. In contrast, the executive control system follows a more
protracted developmental course continuing to mature into adolescence (Diamond, 2002; Luna,
2009; Posner & Rothbart, 2007). Thus, early in life selective sustained attention is driven by
exogenous factors, later in development endogenous factors become more instrumental (Diamond,
2006; Colombo & Cheatham, 2006; Oakes, Kannass, & Shaddy, 2002; Ruff & Rothbart, 1996).
Copyright © 2015, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.
20 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Marked improvements in selective sustained attention are seen throughout early childhood:
For example, 21-month-old children can only engage in episodes of selective sustained attention
for approximately 2 minutes, in contrast children 5 to 6 years of age are able to engage in epi-
sodes of selective sustained attention for 9 minutes or more (Choudhury & Gorman, 2000; Ruff
& Lawson, 1990; Sarid & Breznitz, 1997; for review see Fisher & Kloos, in press). However,
these estimates likely reflect the upper limit of children’s ability to maintain attention on a single
activity, as selective sustained attention was assessed during free play (or other self-directed
activities), rather than within structured learning contexts where a child’s attention span may be
far shorter (e.g., Geary, 2011). As children advance through early and middle-childhood, selec-
tive sustained attention continues to improve and children become less susceptible to distraction
(Ruff & Rothbart, 1996; Ruff & Capozzoli, 2003) and they develop the ability to produce and
utilize effective attention strategies (DeMarie-Dreblow & Miler, 1988).
Children’s ability to endogenously regulate selective sustained attention is hypothesized
to be related to two core executive functions, namely inhibitory control and working memory
(Miyake, Friedman, Emerson, Witzki, Howerter, & Wager, 2000). Endogenously driven selec-
tive sustained attention is related to inhibitory control, as the successful completion of tasks
often requires inhibiting extraneous information or events (Ruff & Rothbart, 1996). Similarly,
endogenously driven selective sustained attention requires sufficient working memory in order
to maintain an active representation of task goals (Colombo & Cheatham, 2006).
Measurement of Selective Sustained Attention
Selective sustained attention in infants is typically assessed using eye gaze and physiological
measures such as heart rate (for review see Fisher & Kloos, in press; Richards, 2003). To measure
selective sustained attention in toddlers and young children, researchers commonly utilize play-
based measures of selective sustained attention (see Ruff & Rothbart, 1996). Studies relying on
play-based measures of selective sustained attention use extensive coding protocols in which the
child’s eye gaze, facial expression, posture, and activity level are coded. For example, selective
sustained attention entails several behavioral signatures such as looking at the object of interest
with an intent facial expression, leaning toward the object, and engaging in minimal extraneous
body movements (Choudhury & Gorman, 2000; Oakes, at al., 2002; Ruff & Capozzoli, 2003;
Ruff & Rothbart, 1996; Tellinghuisen, Oakes, & Tjebkes, 1999).
Performance-based measures of selective sustained attention are commonly administered to
older children and adults (Fisher & Kloos, in press). The most widely used performance-based
assessment of selective sustained attention is the Continuous Performance Test (CPT; Rosvold,
Mirsky, Sarason, Bransome, & Beck, 1956). In the CPT, participants are presented with a stream
of visual stimuli over a prolonged period of time (e.g., up to 40 minutes). Participants are asked
to respond by pressing a button when the target appears and withhold a response for non-targets.
Targets occur infrequently, and thus require the participant to maintain a state of prolonged
vigilance. The CPT is considered inappropriate for use with preschool children due to the long
task durations and limited familiarity with the task stimuli (e.g., letters and numbers). Child-
friendly adaptions to the CPT have been pursued in which the task duration is shortened and the
stimuli are altered to include highly familiar objects. However, these adaptions have met with
limited success as recent research has documented that young children typically fail to reach
the minimum performance criteria (for review see Fisher & Kloos, in press). Thus, preschool
children are in a measurement gap as they are too old for the physiological measures utilized
with infants and toddlers and too young for the performance-based measures typically employed
with older children and adults (Fisher & Kloos, in press).
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 21
In order to mitigate the measurement gap, Fisher, Thiessen, Godwin, Kloos, and Dickerson
(2013) recently developed Track-It, a computer-based task designed to assess selective sustained
attention in young children. Although Track-It provides a sensitive and developmentally appro-
priate measure of selective sustained attention in children three to five years of age, children’s
engagement with the task wanes quickly. Therefore, we sought to investigate whether it was
possible to redesign Track-It as an engaging video game that would also maintain sufficient
validity as an assessment while increasing children’s motivation.
While games have been used extensively to provide intrinsic motivation for learning (Malone,
1981; for review and meta-analysis see Clark et al., 2013), there have been comparatively few
studies investigating the validity of games as assessments or empirical validations demonstrating
that game-based assessments improve children’s intrinsic motivation. MacPherson and Burns
(2008) explicitly compared game-like versions of assessments of working memory and processing
speed to traditional assessments; but they did not assess whether the games improved player affect
or motivation. Similarly, Delgado et al. (2014) created a battery of 10 game-based assessments for
a range of neuropsychological constructs. They found that the games were reliable assessments
(Cronbach alpha > 0.77) that, in many cases, highly correlated with traditional neuropsychological
assessments. But, again, no comparison was made to determine if the game-based assessments
enhanced player affect or motivation. However, work by Attali and Arieli-Attali (2015) found
that the addition of game-like points to an online math assessment somewhat improved task lik-
ability for middle school students while having no effect on the accuracy of responses. A related
line of research has investigated how game data can be used to generate evidence of ability, as
in Shute’s (2011) notion of “stealth assessments” or other investigations of the psychometrics of
games (GlassLab, 2014). Altogether, the prior research suggests that game-based assessments
have the potential to be both valid and more enjoyable than traditional assessments.
Track-It Assessment of Selective Sustained Attention in Young Children
Track-It (Fisher et al., 2013) is a performance-based measure of selective sustained attention
that was designed to be an isomorphic yet developmentally appropriate version of the Multiple
Object Tracking (MOT) task (Pylyshyn & Storm, 1988; Yantis, 1992). Performance on the
Track-It task is dependent upon a subject’s ability to sustain attention to a target object over
time while ignoring distractors.
The Track-It assessment (see Figure 1) consists of a set of distractor objects and a target that
move along random trajectories across a computer screen for a pre-determined period of time
(prior studies have used trial durations of 10 and 30 seconds) and then disappear. The subject’s
task is to identify the location where the target object disappeared. Prior research utilizing Track-
It revealed that performance on this assessment was significantly related to children’s learning
scores in a classroom-like setting (Fisher et al., 2013). Specifically, children who were better
able to track a target object amidst distractors were also likely to obtain higher scores on the
classroom learning task (r = 0.53, p = 0.01). This finding highlights the important contribution
of selective sustained attention on children’s academic performance.
Track-It is unique in its ability to assess within a single task both exogenous and endog-
enously-driven selective sustained attention by manipulating the different types of distractors
deployed in the task (e.g., homogenous and heterogeneous distractors). In the heterogeneous
distractors condition the distractors are all distinct from the target and from each other (e.g., red
triangle, green diamond, blue square). In this condition, Track-It performance is hypothesized to
reflect endogenously-driven selective sustained attention. The task requires effortful control of
attention as the target and distractors are equally salient and thus there is no contextual support.
22 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
In the homogenous distractor condition, the distractors are distinct from the target (e.g., yellow
circle) but identical to one another (e.g., red triangles). In this condition, Track-It performance
is hypothesized to reflect both endogenous as well as exogenously-driven selective sustained
attention as the task still requires effortful control; however, there is contextual support within
the task as the target is unique from all distractors and therefore more salient. Due to our interest
in endogenously-driven selective sustained attention, we utilized the heterogeneous distractors
condition in the present study.
DESIGN OF MONSTER MISCHIEF
The version of the MonsterMischief game described in this paper is a performance-based mea-
sure of endogenously regulated selective sustained attention. MonsterMischief was designed to
closely mirror the original heterogeneous condition of the Track-It assessment while providing
motivational design elements that were intended to sustain students’ intrinsic motivation to
participate and play the game. MonsterMischief (See Figure 2) features a set of colorful “mon-
sters” that run around various settings (e.g., a play room filled with toys) and hide behind an
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 23
assortment of objects (e.g., a toy chest). This design feature stands in contrast to Track-It which
utilized simple shapes which were presented on a 3x3 grid.
The target character is indicated at the beginning of the trial by a glowing circle that encom-
passes the target. Children must click on the target character to initiate the trial. Then, the target
and distractor monsters run around the screen until they hide behind an array of objects in the
room (such as a toy chest or a rocking horse). If the player successfully clicks on the target’s
hiding spot, the character reveals itself and gives the player a jewel. The present study seeks to
determine whether the design elements of MonsterMischief increase children’s engagement and
motivation to play the instructional game, while still maintaining the validity of the measure.
The MonsterMischief game was designed to have a parallel structure to the Track-It assess-
ment (see below for details). Thus, we expected the following pattern of results: Performance
on MonsterMischief should be statistically equivalent to an existing assessment of selective
sustained attention (i.e., Track-It; Hypothesis 1), and Performance on MonsterMischief should
be significantly correlated with an existing assessment of selective sustained attention (i.e.,
Track-It; Hypothesis 2).
In order to create the parallel structure between MonsterMischief and Track-It, we engaged
in an iterative design process that was driven by the goal of having MonsterMischief require
equivalent cognitive processes as Track-It. This equivalency should occur when both tasks can
be described by the same abstract description: a target object was visually indicated to the user
24 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
among 3 other objects; all objects then moved randomly about the screen for a period of time
and then disappeared. The user was then required to identify the target object’s location and to
discriminate the target object from other objects. Any task that is reasonably designed to fit this
description might be expected to have the same face validity. This abstracted equivalent task
structure is akin to having the same game mechanic, with different game aesthetics (Hunicke et
al., 2004). As a step in our design process, we articulated the mapped relations between the game
and the assessment. The key components of the Track-It task were incorporated into the Monster
Mischief game to ensure alignment and to equate Track-It and MonsterMischief on the level
of task difficulty. For example, the games were equated as closely as possible on the following
parameters: the number and type of distractors, the movement paths of the target and distractors,
the number of test trials, the number of test locations, and the number of lures included in the
memory check; see Table 1 for additional details on the alignment between Track-It and Mon-
sterMischief. It is important to note that one parameter, trial duration, was not directly aligned
between Track-It and MonsterMischief. Specifically, Monster Mischief had a slightly longer
trial duration than Track-It. The longer duration of the test trials in MonsterMischief compared
to Track-It was a design choice. This decision was based on the concern that MonsterMischief
might not be as attentionally demanding as Track-It due to the changes in the surface features
of the task which were intended to make MonsterMischief more engaging for young children.
Consequently a slightly longer trial duration was selected for MonsterMischief.
Engagement and Motivation
The primary design objective of MonsterMischief was to retain the features of the Track-It task
within a video game design that would be more engaging and motivating for the children to play;
see Table 2 for the full list of the motivational design elements. Thus, children should choose
to play and report liking MonsterMischief more than the existing selective sustained attention
assessment (i.e., Track-It; Hypothesis 3). Video games are intrinsically motivating (Ryan, Rigby,
& Przybylski, 2006); thus, incorporating game design features into an experimental task was
hypothesized to increase student engagement and motivation. In MonsterMischief, we sought to
increase the attractiveness of the game as well as its capacity to support child engagement over
time through intrinsic fantasy, narrative-driven curiosity, as well as achievement motivation.
(Number and Type)
3 unique distractors
(Target/Distractor motion path)
(1 practice trial and 4 test trials)
Trial Duration Approximately 10s Approximately 20s
9 potential locations (3x3 grid)
(Number of lures)
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 25
Prior research has demonstrated that introducing fantasy elements can increase the “motiva-
tional appeal” of an activity (Malone, 1981, Parker & Lepper, 1992). The MonsterMischief game
presents the assessment task as an intrinsic part of a fantasy context: the point of the tracking
task is to retrieve jewels stolen by the monsters. The game narrative is hypothesized to increase
players’ curiosity to discover what will happen over time (Malone, 1981; Loewenstein, 1994),
thus helping to maintain children’s engagement.
The Monster Mischief game also supports achievement motivation (Eccles & Wigfield,
2002; Ryan, Rigby, & Przybylski, 2006) by providing players with a clear goal (viz., click on
the hiding location of the monster that stole the jewel), a visual reward for attaining the goal
(viz., an animation of a magical jewel flying into the score board) and a small cost for failing to
attain the goal (viz., seeing an animation of the mischievous monster emerging from its hiding
location and giggling). In contrast, Track-It does not provide children with positive or negative
feedback that might support achievement motivation.
The three aforementioned hypotheses were tested in an experimental study of Monster
Mischief. Specifically, after designing a functional version of the MonsterMischief game, we
conducted an experimental study to assess the validity of the MonsterMischief game as an assess-
ment of endogenously driven selective sustained attention by comparing children’s performance
on Monster Mischief to their performance on the Track-It assessment in the Heterogeneous
Distractors condition. Additionally, we examined whether design elements of MonsterMischief
increased children’s engagement and motivation to play the video game.
Monster Mischief Track-It
Targets Cute animated “monsters” Simple shapes
Background The game takes place in colorful settings that are
familiar to young children (e.g., a play room)
Test locations Characters hide behind the objects in the room
(e.g., the characters disappear behind the toys in the
The shapes disappear in one of the
Feedback If the child finds the hiding location of the target
character, the character emerges from behind the object
and gives the child a jewel. If the child selects the
wrong location, the target character emerges from the
correct location and giggles.
No feedback is provided
Backstory At the beginning of the game the mischievous
monsters take jewels from the castle. The child’s task
is to find where the characters are hiding in order to
collect all of the jewels.
Collect all of the jewels at each level of the game.
Within each level, the number of jewels the child has
earned is displayed at the top of the screen. A map
marks the child’s progress across the game levels.
26 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
EXPERIMENTAL EVALUATION OF MONSTER MISCHIEF
Thirty preschool children were recruited to participate in the present study. Three children were
ultimately excluded from the study: 1 child was excluded due to non-compliance with experimenter
instructions, 1 child was excluded due to an interruption in the testing session (i.e., a fire drill),
and 1 child was excluded due to experimenter error. The final sample included 27 children. The
mean age was 4.50 years (SD = 0.31 years). Children ranged in age from 4.08 years to 5.00 years.
Children were tested individually by the first author of this paper or a hypothesis blind research
assistant in a room adjacent to the child’s classroom. The entire testing session lasted approxi-
mately 15 minutes. The testing session was divided into three phases. In Phase 1, children played
5 trials of either Track-It or MonsterMischief. In Phase 2, children played 5 trials of the alternate
game. In Phase 3, children were given a choice of playing either Track-It or MonsterMischief.
Children’s game choice in Phase 3 was recorded as an index of children’s motivation. Game
choice has been successfully used as a measure of motivation in the prior literature (Parker &
Lepper, 1992). Additionally, after each game children were asked to rate how much they liked
each game using a 5-point scale (i.e., the SmileScale; see appendix). The order in which the
games were presented was counterbalanced across participants (i.e., whether Track-It or Monster
Mischief was presented first).
Track-It Task: Children completed 5 trials of the heterogeneous condition of the Track-It task,
1 practice trial and 4 test trials. Each trial lasted for approximately 10 seconds. On each
trial, children saw four objects (e.g., simple shapes) moving on a computer screen: a target
object and three unique distractors. The objects move across a 3x3 gird and land on one of
nine locations. Each grid location was marked in a pastel color to assist children in reporting
the last location visited by the target. For each trial, the target object and the distractors are
randomly selected from an array of simple shapes. Children are asked to watch a particular
object (i.e., target object) while ignoring the rest of the objects (i.e., the distractors). When the
objects stop moving and disappear from the computer screen, children are asked to identify
the location last visited by the target object; see Figure 1. After each trial, a memory check
is administered to ensure that children were tracking the intended target. In the memory
check, children are presented with an array of four objects (i.e., the target and three lures)
and asked to identify the specific object they were tracking. For both the experiment proper
and the memory check, children can provide a verbal or non-verbal response (i.e., point to
the location/object). Overall, Track-It provides two performance indices: Attention (indicated
by the accuracy of identifying the last location visited by the target) and Memory (indicated
by the accuracy of identifying the target object on the memory check).
Monster Mischief Game: Children completed 5 trials of the MonsterMischiefGame, 1 practice
trial and 4 test trials. Each trial lasted for approximately 20 seconds. In the MonsterMischief
Game, children see a set of cute and friendly monsters (one target and three distractors).
During the game, the characters run around various settings (e.g., a play room) and hide
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 27
behind common objects; see Figure 2. At the beginning of the trial, one of the characters
is identified as the target (i.e., the target character is encircled by a glowing ring). At the
end of the trial, all of the characters hide behind various objects (e.g., disappear behind a
toy chest, rocking horse, etc.). If the child correctly identifies the target character’s hiding
spot (by clicking on the appropriate object), the target character will reveal itself and the
child will earn a jewel. At the end of each trial a memory check is administered. Children
are presented with an array of four characters (i.e., the target and three lures) and asked to
identify the specific character they were tracking. For both the experiment proper and the
memory check, children can provide a verbal or non-verbal response (i.e., point to the location/
character). Similar to Track-It, Monster Mischief provides Attention and Memory measures.
Due to constraints relating to school policies regarding how long children may be absent from
the classroom while participating in research, administering additional test trials was not possible
within the allotted time frame. Thus, within a single testing session children were only able to
complete 10 trials, 5 trials of MonsterMischief and 5 trials of Track-It.
Smile Scale: Before starting the experiment proper, children were introduced to the SmileScale.
The SmileScale is a child friendly version of a 5-point likert scale (Read & MacFarlane,
2006). The SmileScale includes five faces (images were obtained from the Google search
engine) that exhibit a range of facial expressions from a big frown to a big smile. Children
were given three practice items (e.g., “WhichfaceshouldIpointtoifItoldyouIreallylike
toJumpRope”). The practice items were included to ensure that the children understood
how to use the scale. See the appendix for the scale, script, and practice trials that were used
to familiarize children with the SmileScale.
Game Choice: After playing both games, children were presented with a piece of paper that
included a screen shot of each game. The screen shots served as a memory cue for the
children. Children were told that they only had a few minutes left before they would return
to class and so the child could choose which game he or she wanted to play. After children
made their selection, they were asked to provide a rationale for their choice (e.g., “Great,
whydidyouchoosethe_____game?”). The child’s response was recorded. Then, the child
played one trial of the game he or she selected.
On both TrackIt and MonsterMischief, children obtained high Memory scores (M = 0.91, SD
= 0.19, range 0.25-1.00 and M = 0.86, SD = 0.21, range 0.25-1.00, respectively). This suggests
that children accurately encoded the identity of the target object, which is a necessary precondi-
tion for successfully tracking a target object moving amidst distractors. Children also achieved
relatively high Attention scores (M = 0.74, SD = 0.26, range 0.25-1.00 and M = 0.73, SD = 0.28
range 0.00-1.00, for Track-It and MonsterMischief respectively); see Figure 3.
Concurrent Validity of Monster Mischief
In order to determine the validity of MonsterMischief and thereby test Hypotheses 1 and 2, we
examined the performance alignment between MonsterMischief and the Track-It task. Children’s
performance on the memory check for the Track-It task and the MonsterMischief game were
not significantly different from each other (paired samples t(26) = 1.22, p = 0.23 ns). A test of
equivalence was also conducted using Weber and Popova’s (2012) Paired-Samples Equivalence
28 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Procedure in order to ascertain whether memory performance was statistically equivalent in
MonsterMischief and Track-It. The minimum substantial effect (Δ = 0.5) was selected based on
Cohen’s (1988) guidelines for a medium effect. The equivalence test was statistically significant
suggesting that the memory demands of the two games are comparable; t(26) = 1.22, p = 0.025.
Statistical equivalence could not be confirmed using more conservative (i.e., smaller) effect size
levels (Δ = 0.3 or 0.1 both ps > 0.27). However, the failure to find evidence of equivalence at
these more conservative levels should be interpreted with caution due to the small sample size
utilized in the present paper and the known power problems of equivalence tests (see Weber
& Popova, 2012). Children’s memory check scores on the two games were also found to be
significantly correlated (r = 0.52, p = 0.006), suggesting that these measures tapped reliable
individual differences in memory encoding.
Similarly, there was no significant difference on the Attention scores of Track-It and Monster
Mischief (paired samples t(26) = 0.20, p = 0.84 ns). A test of equivalence was also performed
in order to determine whether children’s attention scores in MonsterMischief and Track-It were
statistically equivalent. The minimum substantial effect (Δ = 0.5) was again selected based on
Cohen’s (1988) guidelines for a medium effect. Children exhibited equivalent attention perfor-
mance indicating that the difficulty level of the two games is comparable; t(26) = -0.20, p =
0.001. Akin to the results for children’s memory performance statistical equivalence could not
be confirmed using more conservative effect size levels (Δ = 0.3 or 0.1, ps = 0.053 and 0.35
Figure3:MeanAttentionandMemory scoresfor theTrack-Ittaskandthe MonsterMischief
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 29
respectively). Children’s attention scores on the two games were also significantly correlated
(r = 0.62, p = 0.001), suggesting that these measures tapped reliable individual differences in
endogenously driven selective sustained attention.
The results of the paired sample t-tests, equivalence tests, significant correlations between
MonsterMischief and Track-It, along with the face validity of MonsterMischief (due to the high
degree of similarity across the two tasks) provide converging evidence for the validity of Monster
Mischief as an assessment of endogenously driven selective sustained attention.
Comparison of Enjoyment & Motivation
Smile Scale: The SmileScale was included in the present study to test Hypothesis 3 by quantify-
ing how much children enjoyed each game. We hypothesized that that children should report
liking MonsterMischief more than the existing selective sustained attention assessment and
thus the SmileScale scores should be higher for Monster Mischief than Track-It.
The SmileScale was incorporated after data collection commenced. As a result, data for 18 of the
27 children were obtained. From this limited dataset, children tended to report that they liked both
the Track-It task and the MonsterMischief game. For the Track-It task the average score on the
SmileScale was 4.50 (SD = 0.79). Similarly, on the MonsterMischief game, children’s average
score on the SmileScale was 4.44 (SD = 0.86); paired sample t(17) = 0.22, p = 0.83 ns. Failure
to observe any significant differences on the SmileScale may be due to social desirability effects,
namely children may have been hesitant to report that they did not like either one of the games.
Additionally, children only completed 5 trials of each game. It is possible that differences may
emerge if playtime were extended. For example, with more prolonged exposure the novelty of
both tasks may wane. However, children’s engagement in MonsterMischief may be maintained
due to the incorporation of the motivational design elements. In contrast, engagement in Track-
It, which lacks these game features, may decline. In general, children did not tend to utilize the
entire SmileScale, resulting in a truncated range of scores. For both games, the scores on the
SmileScale ranged from 3 (i.e., “thegameisjustokay”) to 5 (i.e., “Ireallylikedthegame”).
Enjoyment was not found to be related to children’s performance on Track-It or MonsterMischief,
as SmileScale ratings were not significantly correlated with children’s attention performance
scores (both rs < 0.28, ps > 0.26 ns). This pattern of result may be due in part to the truncated
range in children’s SmileScale scores as the limited variability in children’s SmileScale ratings
makes it more difficult to detect an association between enjoyment and children’s performance.
Game Choice: To further test Hypothesis 3, children’s game choice patterns were analyzed
to determine if children would choose to play Monster Mischief more than the existing
selective sustained attention assessment. After playing both games children were asked to
choose which game they wanted to play for the remainder of the testing session (Track-It
The vast majority of children selected the MonsterMischief game as their free-choice game
option. In fact 74% of children (20 out of 27) selected Monster Mischief while only 26% of
children (7 out of 27) selected the Track-It task. Children selected MonsterMischief as their free-
choice game more than would be expected by chance (0.50). The cumulative binomial probability
that 20 or more of the 27 children would select MonsterMischief as their free-choice game was
30 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
0.0095. Additionally, children’s game choice was not related to their attention performance on
Track-It or MonsterMischief (both rs < 0.115, ps > 0.567 ns).
After children made their selection, they were asked to provide a rationale for their choice
(e.g., “Great,whydidyouchoosethe_____game?”). The child’s response was recorded and
the child then played one trial of the game they selected. Children’s responses were analyzed.
Many children had trouble providing a rationale for their game selection. For the 7 children who
selected Track-It as their free-choice, 1 child gave no response, 1 child indicated that they did not
know why they selected the game, and 3 children reported that they selected Track-It because
they simply “liked” the game. The remaining 2 children reported that their game selection was
based on aspects of the game (e.g., “they [the shapes] move”; “becausewedidn’tdoallofthe
shapes”). Of the 20 children who selected MonsterMischief, 13 children stated that they selected
the game simply because they “likedit” or reported that the game was “fun”. The remaining 7
children indicated that they selected MonsterMischief due to novelty or various game design
elements (e.g., “becausethey [the monsters] popoutatyou”; “becauseIlikecastles”; “because
Finally, we examined whether children’s free-choice was influenced by the order in which
the tasks were presented. Recall that the presentation order of Track-It and MonsterMischief was
counter-balanced, such that some children played Track-It first and some children played Monster
Mischief first. It is possible that children may exhibit a preference to switch from the game that
they had just played; therefore, we examined whether children were more or less likely to exhibit
this ‘switch preference’ when the last game they played was MonsterMischief vs. Track-It. When
MonsterMischief was the second game children played, children were unlikely to exhibit a switch
preference towards Track-It: only 36% (5 of 14) of the children chose to switch to Track-It, while
64% (9 of 14) of children went against switching and chose to play MonsterMischief again. In
contrast, when Track-It was the second game children played, only 15% (2 of 13) of the children
chose to play Track-It again, while the majority of children exhibited a preference to switch to
MonsterMischief (85% or 11 of 13); see Figure 4. The association between switch preference
and game type (Track-It or Monster Mischief) was statistically significant; Fisher’s Exact test
p = 0.0183. Overall, this analysis indicates that children were less likely to exhibit a preference
to switch away from the game they just played when MonsterMischief was administered last,
but more likely to show a switch preference when Track-It was administered last. This finding
points to MonsterMischief as the more appealing choice over Track-It.
Learning Curve Analysis
A learning curve analysis was conducted to examine whether the task provides some context for
learning. Learning curves are defined as changes in task performance over opportunities (Ritter
& Schooler, 2001). In Figure 5, the average success rates for the memory check and locating the
target are presented over the four experimental trials in both Track-It and MonsterMischief. For
reasons discussed in the Method section, only four test trials of each game were administered.
With only four trials, there is little data that might demonstrate a significant learning curve.
Regular improvements in performance over each trial are observed in children’s MonsterMischief
performance; however, in a single factor regression model predicting student performance, the
trial opportunity number was an insignificant predictor of success (Track-It:p = 0.70; Monster
Mischief:p = 0.33). Future studies involving a greater number of trials will be necessary to
identify significant learning curves, if any.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 31
In conclusion, this study designed and evaluated Monster Mischief, a video game for the as-
sessment of selective sustained attention. Motivational design elements were incorporated to
create a video game that was engaging and motivating to young children. We then conducted
an experimental study to test the validity of MonsterMischief. A valid game-based assessment
of endogenously driven selective sustained attention would show an equivalent level of per-
formance to an existing assessment, would be significantly correlated with performance on the
existing assessment, and be more enjoyable than the existing assessment. The findings from the
present experiment largely align with the expected pattern of results. Specifically, the results
of the present study suggest that we successfully created a video game that was of comparable
difficulty level to the Track-It task. Additionally, performance on Track-It and MonsterMischief
was significantly correlated. Children also exhibited a preference for Monster Mischief over
Track-It when asked what game they would like to play again, presumably due to the addition
of motivational design elements. However, there were no significant difference in enjoyment
observed from the results of the SmileScale.
Overall, the MonsterMischief video game shows potential as an engaging assessment of chil-
dren’s selective sustained attention; however, it is important to note several limitations of the
32 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
present study. First, we examined the relationship between children’s performance in Monster
Mischief and the heterogeneous condition of Track-It. Currently, it is unknown whether Monster
Mischief can be adapted to measure exogenously driven selective sustained attention as well.
Akin to Track-It, the ideal assessment would enable the measurement of both endogenous and
exogenous factors within a single task or game to provide a more comprehensive analysis of the
development and contribution of each type of selective sustained attention. Second, although we
compared MonsterMischief to Track-It, it would also be beneficial to determine if children’s
performance on MonsterMischief is related to other measures of attention and if it is predictive
of other cognitive abilities (e.g., inhibitory control).
A related issue of particular importance concerns the strength of the correlation between
MonsterMischief and Track-It. Although the correlation between the two attention performance
measures is respectable (r = 0.623), the two measures are not highly correlated, suggesting
that additional information is contained within these scores. Thus, future research will need to
distinguish among three different possibilities: (1) Track-It measures selective sustained atten-
tion better than MonsterMischief, (2) MonsterMischief measures selective sustained attention
better than Track-It, or (3) Both tasks measure selective sustained attention to a similar degree
but each task reflects variability associated with task-specific features (e.g., longer trial duration
in MonsterMischief). No task is able to provide a pure measure of the construct of interest, a
problem known as “task impurity” in the literature (Miyake, Friedman, Emerson, Witzki, How-
erter, & Wager, 2000). Thus, it is likely that performance on MonsterMischief (and Track-It)
reflects selective sustained attention as well as other cognitive abilities and task artifacts. Thus,
future research examining the validity of these measures may help adjudicate between the pos-
sibilities listed above.
Third, extensions to this work are warranted as it is critical to assess children’s performance
and engagement in MonsterMischief over more extended periods of play and across time. With
Figure5.LearningCurveAnalyses fortheTrack-Ittask(PanelA) and theMonsterMischief
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 33
this groundwork in place, we can also begin to collect additional data on the psychometric util-
ity of a broader range of difficulty factors in the game. Lastly, additional research is needed to
fully validate MonsterMischief as a reliable measurement tool. For example, the reliability of
the game should be assessed either through test-retest reliability (comparing the correlation of
student performance over multiple testing sessions) or by introducing multiple items with vari-
able difficulty and reporting on the reliability of the different items (i.e., Cronbach’s Alpha).
Difficulty can be manipulated in MonsterMischief and TrackIt, by varying several parameters
including: speed, number of distractors, number of hiding locations, and the trial duration (i.e.,
the amount of time before the target hides/disappears).
During the development of MonsterMischief, we also produced three additional game mechanics
that could be employed in future research. These game mechanics were intended to increase the
difficulty of the game in order to use the game with a wider age range of children and to explore
the possibility of using MonsterMischief as an instructional tool that may help train children’s
selective sustained attention capacity. Additionally, the game mechanics afford an opportunity to
explore whether MonsterMischief could be adapted to evaluate additional aspects of attention such
as divided attention. The additional game mechanics include: coins, fireworks, and cloud mode.
Each mechanic is discussed briefly below. When coins are turned on, coins appear at random for
a random period of time. If a player clicks on a coin, it is added to their score. The rationale for
this mechanic is to divide a player’s attention between the primary task (tracking the monster that
stole the jewel) and a secondary task (collecting coins). When the firework mechanic is turned
on, small red fireworks appear at random and then pop with a shower of sparks. This mechanic
was produced because young children often have trouble reorienting to a task and returning to
a state of focused attention after attention has been disrupted (DiLalla & Watson, 1988). When
cloud mode is turned on, there are no specific objects behind which the monsters hide – they
simply disappear. Then, players have to click on the area where the target monster disappeared
without the aid of visual landmarks. In this mode, we can measure the visual-spatial accuracy
of a player’s response (where accuracy is defined as 100% minus the percent error, where per-
cent error is calculated as the distance between x,y coordinates of the clicked location and the
actual location, divided by the x,y dimensions of the screen). The rationale for this mechanic is
to provide a more nuanced measure of success (degree of accuracy instead of correct/incorrect)
and to increase difficulty by removing memorable landmarks.
Future work should also explore the game as an instructional tool that may help train children’s
selective sustained attention capacity and reduce their susceptibility to distractions. Presently,
it is an open question whether a cognitive skills game for children could both assess selective
sustained attention and also help improve it, akin to the cognitive skills training games that are
currently popular for adults (e.g., Lumosity & Posit Science). The makers of cognitive skills
training games claim that improving performance on an assessment of a cognitive skill could
broadly transfer to other activities (e.g., one could improve working memory by practicing a
game which tests one’s working memory). Although it is well documented that transfer of skills
from one context to another is quite difficult (e.g., Perkins & Solomon, 1988), the potential gains
that could be achieved from such an intervention warrant further empirical work. Ultimately,
a video game like Monster Mischief might be used to test the hypothesis that cognitive skills
practice can increase the attention span of young children and decrease their susceptibility to
distractions. Accordingly, it is of interest to explore whether interventions could be created to
help support the development of selective sustained attention in young children.
34 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
We would like to thank Tara Helfer and Jeremy Galante for their help with illustrations and ani-
mation. We also thank Sharan Shodhan for game programming. We thank the children, parents,
teachers, and staff at the Children’s School and Baldwin United Methodist Church Preschool
who made this work possible. This work was supported in part by a Graduate Training Grant by
the Department of Education (R305B090023).
Attali, Y., & Arieli-Attali, M. (2015). Gamification in assessment: Do points affect test performance?
Computers&Education, 83, 57–63. doi:10.1016/j.compedu.2014.12.012
Bloom, B. S. (1976). HumanCharacteristicsandSchoolLearning. New York: McGraw-Hill.
Bornstein, M. H. (1990). Attention in infancy and the prediction of cognitive capacities in childhood. In J.
Enns (Ed.), DevelopmentofAttention:ResearchandTheory. Elsevier. doi:10.1016/S0166-4115(08)60448-3
Carroll, J. B. (1963). A model of School Learning. TeachersCollegeRecord, 64, 723–733.
Choudhury, N., & Gorman, K. (2000). The relationship between attention and problem solving in 17–24 month
old children. InfantandChildDevelopment, 9, 127–146. doi:10.1002/1522-7219(200009)9:3<127::AID-
Clark, D., Tanner-Smith, E., Killingsworth, S., & Bellamy, S. (2013). DigitalGamesforLearning:ASys-
tematicReviewandMeta-Analysis(ExecutiveSummary). Menlo Park, CA: SRI International.
Colombo, J., & Cheatham, C. L. (2006). The emergence and basis of endogenous attention in infancy and
early childhood. In R. Kail (Ed.), AdvancesinChildDevelopmentandBehavior (Vol. 34, pp. 283–310).
Oxford: Academic Press. doi:10.1016/S0065-2407(06)80010-8
DeMarie-Dreblow, D., & Miler, P. H. (1988). The development of children’s strategies for selective at-
tention: Evidence for a transitional period. ChildDevelopment, 59(6), 1504–1513. doi:10.2307/1130665
Diamond, A. (2002). Normal development of prefrontal cortex from birth to young adulthood: Cognitive
functions, anatomy, and biochemistry. In D. T. Stuss & R. T. Knight (Eds.), Principlesoffrontallobefunction
(pp. 466–503). London, UK: Oxford University Press. doi:10.1093/acprof:oso/9780195134971.003.0029
Diamond, A. (2006). The early development of executive functions. In E. Bialystok & F. Craik (Eds.),
Lifespan Cognition: Mechanisms of Change (pp. 7-95). Oxford University Press. doi:10.1093/acprof:o
DiLalla, L. F., & Watson, M. W. (1988). Differentiation of fantasy and reality: Preschoolers’ reactions to
interruptions in their play. DevelopmentalPsychology, 24(2), 286–291. doi:10.1037/0012-1622.214.171.1246
Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., & Japel, C.
et al. (2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428–1446.
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. AnnualReviewofPsychology,
53(1), 109–132. doi:10.1146/annurev.psych.53.100901.135153 PMID:11752481
Erickson, L. C., Thiessen, E. D., Godwin, K. E., Dickerson, J. P., & Fisher, A. V. (2014). Endogenously- but
not Exogenously-driven Selective Sustained Attention is Related to Learning in a Classroom-like Setting in
Kindergarten Children. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedingsofthe36th
AnnualConferenceoftheCognitiveScienceSociety (pp. 457-462). Austin, TX: Cognitive Science Society.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 35
Fisher, A. V., Godwin, K. E., & Seltman, H. (2014). Visual environment, attention allocation, and
learning: When too much of a good thing may be bad. Psychological Science, 25(7), 1362–1370.
Fisher, A. V., & Kloos, H. (in press). Development of selective sustained attention: The Role of executive
functions. In L. Freund, P. McCardle, & J. Griffin (Eds.), Executive functioninpreschoolage children:
Integratingmeasurement,neurodevelopmentandtranslationalresearch. APA Press.
Fisher, A. V., Thiessen, E., Godwin, K., Kloos, H., & Dickerson, J. P. (2013). Mechanisms of focused at-
tention in 3- to 5-year-old children: Evidence from a new object tracking task. JournalofExperimental
ChildPsychology, 114(2), 275–294. doi:10.1016/j.jecp.2012.07.006 PMID:23022318
Geary, K. E. (2011). Theimpactofchoiceonchildsustainedattentioninthepreschoolclassroom (Unpub-
lished Thesis). Louisiana State University, Baton Rouge, Louisiana.
Gitelman, D. R. (2003). Attention and its disorders: Imaging in clinical neuroscience. British Medical
Bulletin, 65(1), 21–34. doi:10.1093/bmb/65.1.21 PMID:12697614
Glasslab. (2014). Pyschometric Considerations in Game-Based Assessments. White Paper Released by
Godwin, K. E., Almeda, M. V., Petroccia, M., Baker, R. S., & Fisher, A. V. (2013). Classroom activities
and off-task behavior in elementary school children. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth
(Eds.), Proceedings of the 35th Annual Conferenceof the Cognitive Science Society (pp. 2428-2433).
Austin, TX: Cognitive Science Society.
Goodman, L. (1990). Timeandlearninginthespecialeducationclassroom. SUNY Press.
Hunicke, R., LeBlanc, M., & Zubek, R. (2004). MDA: A formal approach to game design and game re-
search. In Proceedings of the WorkshoponChallengesinGameAI,19thNationalConferenceonArtificial
Intelligence, AAAI Press, San Jose, California.
Jonides, J. (1981). Voluntary vs. Automatic control over the mind’s eye’s movement. In J. B. Long & A. D.
Baddeley (Eds.), AttentionandPerformanceIX. Hillsdale, N.J.: Lawrence Erlbaum Associates.
Loewenstein, G. (1994). The Psychology of curiosity: A review and reinterpretation. PsychologicalBul-
letin, 116(1), 75–98. doi:10.1037/0033-2909.116.1.75
Luna, B. (2009). Developmental changes in cognitive control through adolescence. Advances in Child
DevelopmentandBehavior, 37, 233–278. doi:10.1016/S0065-2407(09)03706-9 PMID:19673164
Malone, T. (1981). Toward a theory of intrinsically motivating instruction. CognitiveScience, 5(4), 333–369.
McPherson, J., & Burns, N. R. (2008). Assessing the validity of computer-game-like tests of processing
speed and working memory. BehaviorResearchMethods, 40(4), 969–981. doi:10.3758/BRM.40.4.969
Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefrontal cortex function. AnnualReviewof
Neuroscience, 24(1), 167–202. doi:10.1146/annurev.neuro.24.1.167 PMID:11283309
Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The
unit and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent
variable analysis. CognitivePsychology, 41(1), 49–100. doi:10.1006/cogp.1999.0734 PMID:10945922
Oakes, L., Kannass, K. N., & Shaddy, D. J. (2002). Developmental changes in endogenous control of at-
tention: The role of target familiarity on infants’ distraction latency. ChildDevelopment, 73(6), 1644–1655.
Parker, L., & Lepper, M. (1992). Effects of fantasy contexts on children’s learning and motivation: Mak-
ing learning more fun. JournalofPersonalityandSocialPsychology, 62(4), 625–633. doi:10.1037/0022-
36 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Pashler, H., Johnston, J. C., & Ruthruff, E. (2001). Attention and performance. AnnualReviewofPsychol-
ogy, 52(1), 629–651. doi:10.1146/annurev.psych.52.1.629 PMID:11148320
Perkins, D. N., & Salomon, G. (1988). Teaching for transfer. EducationalLeadership, 46(1), 22–32.
Posner, M. I. (1980). Orienting of attention. TheQuarterlyJournal ofExperimentalPsychology, 32(1),
3–25. doi:10.1080/00335558008248231 PMID:7367577
Posner, M. I., & Petersen, S. E. (1990). The attention system of the human brain. AnnualReviewofNeu-
roscience, 13(1), 25–42. doi:10.1146/annurev.ne.13.030190.000325 PMID:2183676
Posner, M. I., & Rothbart, K. R. (2007). Research on attention networks as a model for the integration of psy-
chological science. AnnualReviewofPsychology, 58(1), 1–23. doi:10.1146/annurev.psych.58.110405.085516
Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a parallel
tracking mechanism. SpatialVision, 3(3), 179–197. doi:10.1163/156856888X00122 PMID:3153671
Read, J. C., & MacFarlane, S. (2006) Using the fun toolkit and other survey methods to gather opinions in
child computer interaction. Proceedingofthe2006conferenceonInteractiondesignandchildren-IDC
’06, 81. doi:10.1145/1139073.1139096
Richards, J. E. (2003). The development of visual attention and the brain. In De Haan & Johnson (Eds.),
The cognitive neuroscience of development. East Sussex, UK: Psychology Press.
Ritter, F., & Schooler, L. (2001) The learning curve. International Encyclopedia of the Social and Behav-
ioral Sciences, 8602–8605.
Rosvold, H. E., Mirsky, A. F., Sarason, I., Bransome, E. D., & Beck, L. H. (1956). A Continuous perfor-
mance test of brain damage. JournalofConsultingPsychology, 20(5), 343–350. doi:10.1037/h0043220
Ruff, A. H., & Lawson, K. R. (1990). Development of sustained, focused attention in young children during
free play. DevelopmentalPsychology, 26(1), 85–93. doi:10.1037/0012-16126.96.36.199
Ruff, H., & Capozzoli, M. (2003). Development of attention and distractibility in the first 4 years of life.
DevelopmentalPsychology, 39(5), 877–890. doi:10.1037/0012-16188.8.131.527 PMID:12952400
Ruff, H. A., & Rothbart, M. K. (1996). Attentioninearlydevelopment:Themesandvariations. New York,
NY: Oxford University Press.
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A self-determination
theory approach. MotivationandEmotion, 30(4), 347–365. doi:10.1007/s11031-006-9051-8
Sarid, M., & Breznitz, Z. (1997). Developmental aspects of sustained attention among 2- to 6-year-old
children. InternationalJournalofBehavioralDevelopment, 21(2), 303–312. doi:10.1080/016502597384884
Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information processing: I. Detec-
tion, search, and attention. PsychologicalReview, 84(1), 1–66. doi:10.1037/0033-295X.84.1.1
Setliff, A. E., & Courage, M. L. (2011). Background television and infants allocation of their attention
during toy play. Infancy, 16(6), 611–639. doi:10.1111/j.1532-7078.2011.00070.x
Shute, V. J. (2011). Stealth Assessment in Computer-Based Games To Support Learning. ComputerGames
Tellinghuisen, D. J., Oakes, L. M., & Tjebkes, T. L. (1999). The influence of attentional state and stimulus
characteristics on infant distractibility. Cognitive Development, 14(2), 199–213. doi:10.1016/S0885-
Tenorio Delgado, M., Arango Uribe, P., Aparicio Alonso, A., & Rosas Díaz, R. (2014). TENI: A compre-
hensive battery for cognitive assessment based on games and technology. ChildNeuropsychology, 1–16.
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 37
Weber, R., & Popova, L. (2012). Testing equivalence in communication research: Theory and application.
CommunicationMethodsandMeasures, 6(3), 190–213. doi:10.1080/19312458.2012.703834
Yantis, S. (1992). Multielement visual tracking: Attention and perceptual organization. CognitivePsychol-
ogy, 24(3), 295–340. doi:10.1016/0010-0285(92)90010-Y PMID:1516359
Yu, C., & Smith, L. B. (2012). Embodied attention and word learning by toddlers. Cognition, 125(2),
244–262. doi:10.1016/j.cognition.2012.06.016 PMID:22878116
38 International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015
Introduction of the Smile Scale
Seeallthefaces [point to the faces on the scale]?Seehowsomeofthefacesarefrowningand
somearesmiling?Lookatthisface [point to face 1]. Thisfaceismakinga big frown-thatmeans
Ireally don’t likesomething.Thisfacehasa little frown [point to face 2] –thatmeansI dislike
somethinga little bit. Lookatthisface [point to face 5]. Thisfacehasa big smile-thatmeansI
really like something.Thisfacehasa little smile [point to face 4] –thatmeansIlikesomething
a little bit. Lookatthisface [point to face 3], thisfaceis not smiling or frowningsothatmeans
Ithinksomethingis just okay.”
1. “WhichfaceshouldIpointtoifItoldyouthatI really don’t like brusselssprouts?”
[Wait for child’s response]
“Yes,Iwouldpointtothefacemakingabigfrown [point to face 1] becauseIreallydon’t
likebrussels sprouts.” [Or: “That’sagoodguess,but Iwouldpointto thefacemakinga big
frown [point to face 1] becauseIreallydon’tlikebrusselssprouts.”]
2. “Okay, try this one - which face should I point to if I told you that I really like to jump
[Wait for child’s response]
“Yes,Iwouldpointtothefacemakingabigsmile [point to face 5] becauseIreallyliketo
jumprope.” [Or: “That’sagoodguess,butIwouldpointtothefacemakingabigsmile [point
to face 5] becauseIreallyliketojumprope.”]
3. “Okay let’s do one more. Which face should I point to if I told you I think the color green
is just okay?”
[Wait for child’s response]
“Yes,Iwouldpointtothisfacethatisnotsmilingorfrowning [point to face 3] becausethat
meansIthinksomethingisjustokay.” [Or: “That’sagoodguess,butIwouldpointtothisfacethat
isnotsmilingorfrowning [point to face 3] becausethatmeansIthinksomethingisjustokay.”]
International Journal of Gaming and Computer-Mediated Simulations, 7(4), 18-39, October-December 2015 39
“All right we will use this again in a little bit but now we are going to play some computer
Rating Each Game using the Smile Scale
Rememberthisfacemeansyou really don’t like thisgame [point to 1], thisfacemeansyou dis-
liked thisgame a little [point to 2], thisfacemeansyouthoughtthegamewas just okay [point
to 3], thisfacemeansyou liked thegame a little [point to 4], andthisfacemeansyou really like
the game [point to 5]. Okaywhatdidyouthinkaboutthegame-whateveryouthinkisfine.”