Content uploaded by Derek Lomas
Author content
All content in this area was uploaded by Derek Lomas on Sep 28, 2015
Content may be subject to copyright.
Monster Mischief: Designing a Video Game to Assess Selective
Sustained Attention
Karrie Godwin, Derek Lomas, Kenneth Koedinger, Anna Fisher
Carnegie Mellon University
(Prepublication Draft)
Abstract
Selective sustained attention, or the ability to allocate perceptual and mental resources to
a single object or event, is an important cognitive ability widely assumed to be required for
learning. Assessing young children’s selective sustained attention is challenging due to the
limited number of sensitive and developmentally appropriate performance-based measures.
Furthermore, administration of existing assessments is difficult, as children’s engagement with
such tasks wanes quickly. One potential solution is to provide assessments within an engaging
environment, such as a video game. This paper reports the design and psychometric validation of
a video game (Monster Mischief) designed to assess selective sustained attention in preschool
children. In a randomized controlled trial, we demonstrate that Monster Mischief is significantly
correlated with an existing measure of selective sustained attention (rs ≥ 0.52), and more
motivating for young children as almost 3 times more children preferred Monster Mischief to the
existing measure.
Keywords
Assessments, games, selective sustained attention, psychometrics, children, controlled trial.
Introduction
Attention is multidimensional and encompasses a diverse set of psychological constructs,
including (but not limited to) orienting, selection, shifting, maintenance, and executive attention
(see Colombo & Cheatham, 2006; Gitelman, 2003; Fisher & Kloos, in press; Posner & Petersen,
1990). The present work focuses on selective sustained attention which is defined as: “a state of
engagement that involves narrowed selectivity and increased commitment of energy and
resources on the targeted activity … and that primarily enhances information processing in that
system” (Setliff & Courage, 2011, p. 613). Selective sustained attention is important because the
ability to selectively allocate attentional resources is commonly hypothesized to aid learning
(e.g., Carroll 1963; Bloom, 1976; Oakes, Kannass, & Shaddy, 2002). As stated by Oakes and
colleagues (2002), “if attention were constantly reoriented to every new event, it would be
difficult ... to learn about any single object or event” (p.1644). In accordance with this assertion,
prior research has implicated selective sustained attention in task performance (e.g., Choudhury
& Gorman, 2000; DeMarie-Dreblow & Miller, 1988), academic achievement (e.g., Duncan et al.
2007; for review see Goodman, 1990), and learning outcomes (e.g., Fisher, Thiessen, Godwin,
Dickerson, & Kloos, 2013; Fisher, Godwin, & Seltman, 2014; Yu & Smith, 2012).
Theoretical Perspectives on Selective Sustained Attention
There has been a long tradition in the cognitive psychology literature of distinguishing
between two modes of attention regulation. One mode refers to regulation that is said to be top-
down, controlled, and endogenous (i.e., arising from within the organism); whereas the other
mode is commonly referred to as bottom-up, automatic (or in some cases automatized through
extensive practice), and exogenous (i.e., arising from outside the organism) (e.g., Jonides, 1981;
Miller & Cohen, 2001; Posner, 1980; Pashler, Johnston, & Ruthruff, 2001; Schneider & Shiffrin,
1977). In the present paper, we are adopting the developmental framework of attention
regulation put forth by Ruff and Rothbart (1996). This framework builds upon the dual-mode of
attention regulation suggested by the above theories. Specifically, according to Ruff and
Rothbart (1996) the state of selective sustained attention can be obtained through two distinct
systems: an orienting system and an executive control system. Under the orienting system,
selective sustained attention is driven by exogenous factors such as the saliency of the stimuli
including characteristics such as the brightness or contrast of the stimuli as well as motion. Thus,
exogenously driven selective sustained attention is considered largely an automatic process
driven by the physical characteristics of the environment or the stimulus (Bornstein, 1990; Ruff
& Rothbart, 1996). Under the executive control system, selective sustained attention is driven by
endogenous factors; in other words the focus of attention is directed internally based on the
individual’s desires, interests, and goals (Colombo & Cheatham, 2006; Posner & Petersen, 1990;
Posner & Rothbart, 2007). The two systems are said to be neurally and anatomically distinct,
with the orienting system involving areas within the parietal lobe, superior colliculus, and lateral
pulvinar nucleus of the thalamus, and the executive system involving areas within the prefrontal
cortex (PFC) and anterior cingulate gyrus (fore review see Fisher & Kloos, in press).
In the present paper we focus on endogenously driven selective sustained attention as
prior research suggests that endogenously regulated selective sustained attention may be
particularly important for learning in education settings (Erickson, Thiessen, Godwin, Dickerson,
& Fisher, 2014). In education settings children need to attend to an externally-prescribed
learning objective, but often face multiple sources of distraction (Godwin, Almeda, Petroccia,
Baker, & Fisher, 2013). Children who are better able to endogenously regulate their attention
may be more likely to maintain focus on the learning objective in the face of distractions and
thus have better learning outcomes.
Development of Selective Sustained Attention
The orienting and executive control systems follow different developmental trajectories.
The orienting system matures during infancy. In contrast, the executive control system follows a
more protracted developmental course continuing to mature into adolescence (Diamond, 2002;
Luna, 2009; Posner & Rothbart, 2007). Thus, early in life selective sustained attention is driven
by exogenous factors, later in development endogenous factors become more instrumental
(Diamond, 2006; Colombo & Cheatham, 2006; Oakes, Kannass, & Shaddy, 2002; Ruff &
Rothbart, 1996).
Marked improvements in selective sustained attention are seen throughout early
childhood: For example, 21-month-old children can only engage in episodes of selective
sustained attention for approximately 2 minutes, in contrast children 5 to 6 years of age are able
to engage in episodes of selective sustained attention for 9 minutes or more (Choudhury &
Gorman, 2000; Ruff & Lawson, 1990; Sarid & Breznitz, 1997; for review see Fisher & Kloos, in
press). However, these estimates likely reflect the upper limit of children’s ability to maintain
attention on a single activity, as selective sustained attention was assessed during free play (or
other self-directed activities), rather than within structured learning contexts where a child’s
attention span may be far shorter (e.g., Geary, 2011). As children advance through early and
middle-childhood, selective sustained attention continues to improve and children become less
susceptible to distraction (Ruff & Rothbart, 1996; Ruff & Capozzoli, 2003) and they develop the
ability to produce and utilize effective attention strategies (DeMarie-Dreblow & Miler, 1988).
Children’s ability to endogenously regulate selective sustained attention is hypothesized
to be related to two core executive functions, namely inhibitory control and working memory
(Miyake, Friedman, Emerson, Witzki, Howerter, & Wager, 2000). Endogenously driven
selective sustained attention is related to inhibitory control, as the successful completion of tasks
often requires inhibiting extraneous information or events (Ruff & Rothbart, 1996). Similarly,
endogenously driven selective sustained attention requires sufficient working memory in order to
maintain an active representation of task goals (Colombo & Cheatham, 2006).
Measurement of Selective Sustained Attention
Selective sustained attention in infants is typically assessed using eye gaze and
physiological measures such as heart rate (for review see Fisher & Kloos, in press; Richards,
2003). To measure selective sustained attention in toddlers and young children, researchers
commonly utilize play-based measures of selective sustained attention (see Ruff & Rothbart,
1996). Studies relying on play-based measures of selective sustained attention use extensive
coding protocols in which the child’s eye gaze, facial expression, posture, and activity level are
coded. For example, selective sustained attention entails several behavioral signatures such as
looking at the object of interest with an intent facial expression, leaning toward the object, and
engaging in minimal extraneous body movements (Choudhury & Gorman, 2000; Oakes, at al.,
2002; Ruff & Capozzoli, 2003; Ruff & Rothbart, 1996; Tellinghuisen, Oakes, & Tjebkes, 1999).
Performance-based measures of selective sustained attention are commonly administered
to older children and adults (Fisher & Kloos, in press). The most widely used performance-based
assessment of selective sustained attention is the Continuous Performance Test (CPT; Rosvold,
Mirsky, Sarason, Bransome, & Beck, 1956). In the CPT, participants are presented with a stream
of visual stimuli over a prolonged period of time (e.g., up to 40 minutes). Participants are asked
to respond by pressing a button when the target appears and withhold a response for non-targets.
Targets occur infrequently, and thus require the participant to maintain a state of prolonged
vigilance. The CPT is considered inappropriate for use with preschool children due to the long
task durations and limited familiarity with the task stimuli (e.g., letters and numbers). Child-
friendly adaptions to the CPT have been pursued in which the task duration is shortened and the
stimuli are altered to include highly familiar objects. However, these adaptions have met with
limited success as recent research has documented that young children typically fail to reach the
minimum performance criteria (for review see Fisher & Kloos, in press). Thus, preschool
children are in a measurement gap as they are too old for the physiological measures utilized
with infants and toddlers and too young for the performance-based measures typically employed
with older children and adults (Fisher & Kloos, in press).
In order to mitigate the measurement gap, Fisher, Thiessen, Godwin, Kloos, and
Dickerson (2013) recently developed Track-It, a computer-based task designed to assess
selective sustained attention in young children. Although Track-It provides a sensitive and
developmentally appropriate measure of selective sustained attention in children three to five
years of age, children’s engagement with the task wanes quickly. Therefore, we sought to
investigate whether it was possible to redesign Track-It as an engaging video game that would
also maintain sufficient validity as an assessment while increasing children’s motivation.
While games have been used extensively to provide intrinsic motivation for learning
(Malone, 1981; for review and meta-analysis see Clark et al., 2013), there have been
comparatively few studies investigating the validity of games as assessments or empirical
validations demonstrating that game-based assessments improve children’s intrinsic motivation.
MacPherson and Burns (2008) explicitly compared game-like versions of assessments of
working memory and processing speed to traditional assessments; but they did not assess
whether the games improved player affect or motivation. Similarly, Delgado et al. (2014) created
a battery of 10 game-based assessments for a range of neuropsychological constructs. They
found that the games were reliable assessments (Cronbach alpha > 0.77) that, in many cases,
highly correlated with traditional neuropsychological assessments. But, again, no comparison
was made to determine if the game-based assessments enhanced player affect or motivation.
However, work by Attali and Arieli-Attali (2015) found that the addition of game-like points to
an online math assessment somewhat improved task likability for middle school students while
having no effect on the accuracy of responses. A related line of research has investigated how
game data can be used to generate evidence of ability, as in Shute’s (2011) notion of “stealth
assessments” or other investigations of the psychometrics of games (GlassLab, 2014).
Altogether, the prior research suggests that game-based assessments have the potential to be both
valid and more enjoyable than traditional assessments.
Track-It Assessment of Selective Sustained Attention in Young Children
Track-It (Fisher et al., 2013) is a performance-based measure of selective sustained
attention that was designed to be an isomorphic yet developmentally appropriate version of the
Multiple Object Tracking (MOT) task (Pylyshyn & Storm, 1988; Yantis, 1992). Performance on
the Track-It task is dependent upon a subject’s ability to sustain attention to a target object over
time while ignoring distractors.
The Track-It assessment (see Figure 1Figure 1) consists of a set of distractor objects and
a target that move along random trajectories across a computer screen for a pre-determined
period of time (prior studies have used trial durations of 10 and 30 seconds) and then disappear.
The subject’s task is to identify the location where the target object disappeared. Prior research
utilizing Track-It revealed that performance on this assessment was significantly related to
children’s learning scores in a classroom-like setting (Fisher et al., 2013). Specifically, children
who were better able to track a target object amidst distractors were also likely to obtain higher
scores on the classroom learning task (r = 0.53, p = 0.01). This finding highlights the important
contribution of selective sustained attention on children’s academic performance.!
Track-It is unique in its ability to assess within a single task both exogenous and
endogenously-driven selective sustained attention by manipulating the different types of
distractors deployed in the task (e.g., homogenous and heterogeneous distractors). In the
heterogeneous distractors condition the distractors are all distinct from the target and from each
other (e.g., red triangle, green diamond, blue square). In this condition, Track-It performance is
hypothesized to reflect endogenously-driven selective sustained attention. The task requires
effortful control of attention as the target and distractors are equally salient and thus there is no
contextual support. In the homogenous distractor condition, the distractors are distinct from the
target (e.g., yellow circle) but identical to one another (e.g., red triangles). In this condition,
Track-It performance is hypothesized to reflect both endogenous as well as exogenously-driven
selective sustained attention as the task still requires effortful control; however, there is
contextual support within the task as the target is unique from all distractors and therefore more
salient. Due to our interest in endogenously-driven selective sustained attention, we utilized the
heterogeneous distractors condition in the present study.
Figure 1: Schematic of the Track-It Game.
Design of Monster Mischief
The version of the Monster Mischief game described in this paper is a performance-based
measure of endogenously regulated selective sustained attention. Monster Mischief was designed
to closely mirror the original heterogeneous condition of the Track-It assessment while providing
motivational design elements that were intended to sustain students' intrinsic motivation to
participate and play the game. Monster Mischief (See Figure 2) features a set of colorful
"monsters" that run around various settings (e.g., a play room filled with toys) and hide behind
an assortment of objects (e.g., a toy chest). This design feature stands in contrast to Track-It
which utilized simple shapes which were presented on a 3x3 grid.
Figure 2: Schematic of the Monster Mischief video game. The target monster is highlighted in
yellow at the beginning of each trial.
The target character is indicated at the beginning of the trial by a glowing circle that
encompasses the target. Children must click on the target character to initiate the trial. Then, the
target and distractor monsters run around the screen until they hide behind an array of objects in
the room (such as a toy chest or a rocking horse). If the player successfully clicks on the target's
hiding spot, the character reveals itself and gives the player a jewel. The present study seeks to
determine whether the design elements of Monster Mischief increase children’s engagement and
motivation to play the instructional game, while still maintaining the validity of the measure.
The Monster Mischief game was designed to have a parallel structure to the Track-It
assessment (see below for details). Thus, we expected the following pattern of results:
Performance on Monster Mischief should be statistically equivalent to an existing assessment of
selective sustained attention (i.e., Track-It; Hypothesis 1), and Performance on Monster Mischief
should be significantly correlated with an existing assessment of selective sustained attention
(i.e., Track-It; Hypothesis 2).
In order to create the parallel structure between Monster Mischief and Track-It, we
engaged in an iterative design process that was driven by the goal of having Monster Mischief
require equivalent cognitive processes as Track-It. This equivalency should occur when both
tasks can be described by the same abstract description: a target object was visually indicated to
the user among 3 other objects; all objects then moved randomly about the screen for a period of
time and then disappeared. The user was then required to identify the target object's location and
to discriminate the target object from other objects. Any task that is reasonably designed to fit
this description might be expected to have the same face validity. This abstracted equivalent task
structure is akin to having the same game mechanic, with different game aesthetics (Hunicke et
al., 2004). As a step in our design process, we articulated the mapped relations between the game
and the assessment. The key components of the Track-It task were incorporated into the Monster
Mischief game to ensure alignment and to equate Track-It and Monster Mischief on the level of
task difficulty. For example, the games were equated as closely as possible on the following
parameters: the number and type of distractors, the movement paths of the target and distractors,
the number of test trials, the number of test locations, and the number of lures included in the
memory check; see Table 1 Error! Reference source not found. for additional details on the
alignment between Track-It and Monster Mischief. It is important to note that one parameter,
trial duration, was not directly aligned between Track-It and Monster Mischief. Specifically,
Monster Mischief had a slightly longer trial duration than Track-It. The longer duration of the
test trials in Monster Mischief compared to Track-It was a design choice. This decision was
based on the concern that Monster Mischief might not be as attentionally demanding as Track-It
due to the changes in the surface features of the task which were intended to make Monster
Mischief more engaging for young children. Consequently a slightly longer trial duration was
selected for Monster Mischief.
Engagement and Motivation
The primary design objective of Monster Mischief was to retain the features of the Track-
It task within a video game design that would be more engaging and motivating for the children
to play; see Table 2 Error! Reference source not found. for the full list of the motivational
design elements. Thus, children should choose to play and report liking Monster Mischief more
than the existing selective sustained attention assessment (i.e., Track-It; Hypothesis 3). Video
games are intrinsically motivating (Ryan, Rigby, & Przybylski, 2006); thus, incorporating game
design features into an experimental task was hypothesized to increase student engagement and
motivation. In Monster Mischief, we sought to increase the attractiveness of the game as well as
its capacity to support child engagement over time through intrinsic fantasy, narrative-driven
curiosity, as well as achievement motivation.
Prior research has demonstrated that introducing fantasy elements can increase the
“motivational appeal” of an activity (Malone, 1981, Parker & Lepper, 1992). The Monster
Mischief game presents the assessment task as an intrinsic part of a fantasy context: the point of
the tracking task is to retrieve jewels stolen by the monsters. The game narrative is hypothesized
to increase players’ curiosity to discover what will happen over time (Malone, 1981;
Loewenstein, 1994), thus helping to maintain children’s engagement.
The Monster Mischief game also supports achievement motivation (Eccles & Wigfield,
2002; Ryan, Rigby, & Przybylski, 2006) by providing players with a clear goal (viz., click on the
hiding location of the monster that stole the jewel), a visual reward for attaining the goal (viz., an
animation of a magical jewel flying into the score board) and a small cost for failing to attain the
goal (viz., seeing an animation of the mischievous monster emerging from its hiding location and
giggling). In contrast, Track-It does not provide children with positive or negative feedback that
might support achievement motivation.
The three aforementioned hypotheses were tested in an experimental study of Monster
Mischief. Specifically, after designing a functional version of the Monster Mischief game, we
conducted an experimental study to assess the validity of the Monster Mischief game as an
assessment of endogenously driven selective sustained attention by comparing children’s
performance on Monster Mischief to their performance on the Track-It assessment in the
Heterogeneous Distractors condition. Additionally, we examined whether design elements of
Monster Mischief increased children’s engagement and motivation to play the video game.
Experimental Evaluation of Monster Mischief
Method
Participants
Thirty preschool children were recruited to participate in the present study. Three
children were ultimately excluded from the study: 1 child was excluded due to non-compliance
with experimenter instructions, 1 child was excluded due to an interruption in the testing session
(i.e., a fire drill), and 1 child was excluded due to experimenter error. The final sample included
27 children. The mean age was 4.50 years (SD = 0.31 years). Children ranged in age from 4.08
years to 5.00 years.
Study Design
Children were tested individually by the first author of this paper or a hypothesis blind
research assistant in a room adjacent to the child’s classroom. The entire testing session lasted
approximately 15 minutes. The testing session was divided into three phases. In Phase 1,
children played 5 trials of either Track-It or Monster Mischief. In Phase 2, children played 5
trials of the alternate game. In Phase 3, children were given a choice of playing either Track-It or
Monster Mischief. Children’s game choice in Phase 3 was recorded as an index of children’s
motivation. Game choice has been successfully used as a measure of motivation in the prior
literature (Parker & Lepper, 1992). Additionally, after each game children were asked to rate
how much they liked each game using a 5-point scale (i.e., the Smile Scale; see appendix). The
order in which the games were presented was counterbalanced across participants (i.e., whether
Track-It or Monster Mischief was presented first).
Measures
Track-It Task. Children completed 5 trials of the heterogeneous condition of the Track-
It task, 1 practice trial and 4 test trials. Each trial lasted for approximately 10 seconds. On each
trial, children saw four objects (e.g., simple shapes) moving on a computer screen: a target object
and three unique distractors. The objects move across a 3x3 gird and land on one of nine
locations. Each grid location was marked in a pastel color to assist children in reporting the last
location visited by the target.!For each trial, the target object and the distractors are randomly
selected from an array of simple shapes. Children are asked to watch a particular object (i.e.,
target object) while ignoring the rest of the objects (i.e., the distractors). When the objects stop
moving and disappear from the computer screen, children are asked to identify the location last
visited by the target object; see Figure 1. After each trial, a memory check is administered to
ensure that children were tracking the intended target. In the memory check, children are
presented with an array of four objects (i.e., the target and three lures) and asked to identify the
specific object they were tracking. For both the experiment proper and the memory check,
children can provide a verbal or non-verbal response (i.e., point to the location/object). Overall,
Track-It provides two performance indices: Attention (indicated by the accuracy of identifying
the last location visited by the target) and Memory (indicated by the accuracy of identifying the
target object on the memory check).!
Monster Mischief Game. Children completed 5 trials of the Monster Mischief Game, 1
practice trial and 4 test trials. Each trial lasted for approximately 20 seconds. In the Monster
Mischief Game, children see a set of cute and friendly monsters (one target and three distractors).
During the game, the characters run around various settings (e.g., a play room) and hide behind
common objects; see Figure 2. At the beginning of the trial, one of the characters is identified as
the target (i.e., the target character is encircled by a glowing ring). At the end of the trial, all of
the characters hide behind various objects (e.g., disappear behind a toy chest, rocking horse,
etc.). If the child correctly identifies the target character’s hiding spot (by clicking on the
appropriate object), the target character will reveal itself and the child will earn a jewel. At the
end of each trial a memory check is administered. Children are presented with an array of four
characters (i.e., the target and three lures) and asked to identify the specific character they were
tracking. For both the experiment proper and the memory check, children can provide a verbal or
non-verbal response (i.e., point to the location/character). Similar to Track-It, Monster Mischief
provides Attention and Memory measures.
Due to constraints relating to school policies regarding how long children may be absent
from the classroom while participating in research, administering additional test trials was not
possible within the allotted time frame. Thus, within a single testing session children were only
able to complete 10 trials, 5 trials of Monster Mischief and 5 trials of Track-It.
Smile Scale. Before starting the experiment proper, children were introduced to the Smile
Scale. The Smile Scale is a child friendly version of a 5-point likert scale (Read & MacFarlane,
2006). The Smile Scale includes five faces (images were obtained from the Google search
engine) that exhibit a range of facial expressions from a big frown to a big smile. Children were
given three practice items (e.g., “Which face should I point to if I told you I really like to Jump
Rope”). The practice items were included to ensure that the children understood how to use the
scale. See the appendix for the scale, script, and practice trials that were used to familiarize
children with the Smile Scale.
Game Choice. After playing both games, children were presented with a piece of paper
that included a screen shot of each game. The screen shots served as a memory cue for the
children. Children were told that they only had a few minutes left before they would return to
class and so the child could choose which game he or she wanted to play. After children made
their selection, they were asked to provide a rationale for their choice (e.g., “Great, why did you
choose the _____ game?”). The child’s response was recorded. Then, the child played one trial
of the game he or she selected.
Results
On both Track It and Monster Mischief, children obtained high Memory scores (M =
0.91, SD = 0.19, range 0.25-1.00 and M = 0.86, SD = 0.21, range 0.25-1.00, respectively). This
suggests that children accurately encoded the identity of the target object, which is a necessary
precondition for successfully tracking a target object moving amidst distractors. Children also
achieved relatively high Attention scores (M = 0.74, SD = 0.26, range 0.25-1.00 and M = 0.73,
SD = 0.28 range 0.00-1.00, for Track-It and Monster Mischief respectively); see Figure 3.
Figure 3: Mean Attention and Memory scores for the Track-It task and the Monster Mischief
Game. Error bars represent the standard errors of the means. Standard errors of the means are as
0
0.2
0.4
0.6
0.8
1
Attention Memory Check
Accuracy
Track-It
Monster Mischief
follows, Attention: Track-It = 0.05, Monster Mischief = 0.053; Memory Check: Track-It = 0.035,
Monster Mischief = 0.041.!
Concurrent Validity of Monster Mischief
In order to determine the validity of Monster Mischief and thereby test Hypotheses 1 and
2, we examined the performance alignment between Monster Mischief and the Track-It task.
Children’s performance on the memory check for the Track-It task and the Monster Mischief
game were not significantly different from each other (paired samples t(26) = 1.22, p = 0.23 ns).
A test of equivalence was also conducted using Weber and Popova’s (2012) Paired-Samples
Equivalence Procedure in order to ascertain whether memory performance was statistically
equivalent in Monster Mischief and Track-It. The minimum substantial effect (Δ = 0.5) was
selected based on Cohen’s (1988) guidelines for a medium effect. The equivalence test was
statistically significant suggesting that the memory demands of the two games are comparable;
t(26) = 1.22, p = 0.025. Statistical equivalence could not be confirmed using more conservative
(i.e., smaller) effect size levels (Δ = 0.3 or 0.1 both ps > 0.27). However, the failure to find
evidence of equivalence at these more conservative levels should be interpreted with caution due
to the small sample size utilized in the present paper and the known power problems of
equivalence tests (see Weber & Popova, 2012). Children’s memory check scores on the two
games were also found to be significantly correlated (r = 0.52, p = 0.006), suggesting that these
measures tapped reliable individual differences in memory encoding.
Similarly, there was no significant difference on the Attention scores of Track-It and
Monster Mischief (paired samples t(26) = 0.20, p = 0.84 ns). A test of equivalence was also
performed in order to determine whether children’s attention scores in Monster Mischief and
Track-It were statistically equivalent. The minimum substantial effect (Δ = 0.5) was again
selected based on Cohen’s (1988) guidelines for a medium effect. Children exhibited equivalent
attention performance indicating that the difficulty level of the two games is comparable; t(26) =
-0.20, p = 0.001. Akin to the results for children’s memory performance statistical equivalence
could not be confirmed using more conservative effect size levels (Δ = 0.3 or 0.1, ps = 0.053 and
0.35 respectively). Children’s attention scores on the two games were also significantly
correlated (r = 0.62, p = 0.001), suggesting that these measures tapped reliable individual
differences in endogenously driven selective sustained attention.
The results of the paired sample t-tests, equivalence tests, significant correlations
between Monster Mischief and Track-It, along with the face validity of Monster Mischief (due to
the high degree of similarity across the two tasks) provide converging evidence for the validity
of Monster Mischief as an assessment of endogenously driven selective sustained attention.
Comparison of Enjoyment & Motivation
Smile Scale. The Smile Scale was included in the present study to test Hypothesis 3 by
quantifying how much children enjoyed each game. We hypothesized that that children should
report liking Monster Mischief more than the existing selective sustained attention assessment
and thus the Smile Scale scores should be higher for Monster Mischief than Track-It.
The Smile Scale was incorporated after data collection commenced. As a result, data for
18 of the 27 children were obtained. From this limited dataset, children tended to report that they
liked both the Track-It task and the Monster Mischief game. For the Track-It task the average
score on the Smile Scale was 4.50 (SD = 0.79). Similarly, on the Monster Mischief game,
children’s average score on the Smile Scale was 4.44 (SD = 0.86); paired sample t(17) = 0.22, p
= 0.83 ns. Failure to observe any significant differences on the Smile Scale may be due to social
desirability effects, namely children may have been hesitant to report that they did not like either
one of the games. Additionally, children only completed 5 trials of each game. It is possible that
differences may emerge if playtime were extended. For example, with more prolonged exposure
the novelty of both tasks may wane. However, children’s engagement in Monster Mischief may
be maintained due to the incorporation of the motivational design elements. In contrast,
engagement in Track-It, which lacks these game features, may decline. In general, children did
not tend to utilize the entire Smile Scale, resulting in a truncated range of scores. For both games,
the scores on the Smile Scale ranged from 3 (i.e., “the game is just okay”) to 5 (i.e., “I really
liked the game”). Enjoyment was not found to be related to children’s performance on Track-It
or Monster Mischief, as Smile Scale ratings were not significantly correlated with children’s
attention performance scores (both rs < 0.28, ps > 0.26 ns). This pattern of result may be due in
part to the truncated range in children’s Smile Scale scores as the limited variability in children’s
Smile Scale ratings makes it more difficult to detect an association between enjoyment and
children’s performance.
Game Choice. To further test Hypothesis 3, children’s game choice patterns were
analyzed to determine if children would choose to play Monster Mischief more than the existing
selective sustained attention assessment. After playing both games children were asked to choose
which game they wanted to play for the remainder of the testing session (Track-It or Monster
Mischief).
The vast majority of children selected the Monster Mischief game as their free-choice
game option. In fact 74% of children (20 out of 27) selected Monster Mischief while only 26% of
children (7 out of 27) selected the Track-It task. Children selected Monster Mischief as their free-
choice game more than would be expected by chance (0.50). The cumulative binomial
probability that 20 or more of the 27 children would select Monster Mischief as their free-choice
game was 0.0095. Additionally, children’s game choice was not related to their attention
performance on Track-It or Monster Mischief (both rs < 0.115, ps > 0.567 ns).
After children made their selection, they were asked to provide a rationale for their
choice (e.g., “Great, why did you choose the _____ game?”). The child’s response was recorded
and the child then played one trial of the game they selected. Children’s responses were
analyzed. Many children had trouble providing a rationale for their game selection. For the 7
children who selected Track-It as their free-choice, 1 child gave no response, 1 child indicated
that they did not know why they selected the game, and 3 children reported that they selected
Track-It because they simply “liked” the game. The remaining 2 children reported that their
game selection was based on aspects of the game (e.g., “they [the shapes] move”; “because we
didn’t do all of the shapes”). Of the 20 children who selected Monster Mischief, 13 children
stated that they selected the game simply because they “liked it” or reported that the game was
“fun”. The remaining 7 children indicated that they selected Monster Mischief due to novelty or
various game design elements (e.g., “because they [the monsters] pop out at you”; “because I
like castles”; “because it’s so much fun – the monsters hide”; “because we only played 2 or 3
monsters and I want to play the purple one”).
Finally, we examined whether children’s free-choice was influenced by the order in
which the tasks were presented. Recall that the presentation order of Track-It and Monster
Mischief was counter-balanced, such that some children played Track-It first and some children
played Monster Mischief first. It is possible that children may exhibit a preference to switch from
the game that they had just played; therefore, we examined whether children were more or less
likely to exhibit this ‘switch preference’ when the last game they played was Monster Mischief
vs. Track-It. When Monster Mischief was the second game children played, children were
unlikely to exhibit a switch preference towards Track-It: only 36% (5 of 14) of the children
chose to switch to Track-It, while 64% (9 of 14) of children went against switching and chose to
play Monster Mischief again. In contrast, when Track-It was the second game children played,
only 15% (2 of 13) of the children chose to play Track-It again, while the majority of children
exhibited a preference to switch to Monster Mischief (85% or 11 of 13); see Figure 4. The
association between switch preference and game type (Track-It or Monster Mischief) was
statistically significant; Fisher’s Exact test p = 0.0183. Overall, this analysis indicates that
children were less likely to exhibit a preference to switch away from the game they just played
when Monster Mischief was administered last, but more likely to show a switch preference when
Track-It was administered last. This finding points to Monster Mischief as the more appealing
choice over Track-It.
Figure 4. Children’s free-choice game selection as a function of the order in which the
assessments were presented (i.e., Monster Mischief first followed by Track-It or Track-It first
followed by Monster Mischief).
Learning Curve Analysis
A learning curve analysis was conducted to examine whether the task provides some context for
learning. Learning curves are defined as changes in task performance over opportunities (Ritter
& Schooler, 2001). In Figure 5, the average success rates for the memory check and locating the
0
2
4
6
8
10
12
14
16
Monster Mischeif -> Track-It Track-It -> Monster Mischief
Number of Children
Chose Track-It
Chose Monster Mischief
Free-Choice Game
The Order of Assessments
target are presented over the four experimental trials in both Track-It and Monster Mischief. For
reasons discussed in the Method section, only four test trials of each game were administered.
With only four trials, there is little data that might demonstrate a significant learning curve.
Regular improvements in performance over each trial are observed in children’s Monster
Mischief performance; however, in a single factor regression model predicting student
performance, the trial opportunity number was an insignificant predictor of success (Track-It: p
= 0.70; Monster Mischief: p = 0.33). Future studies involving a greater number of trials will be
necessary to identify significant learning curves, if any.
Figure 5: Learning Curve Analyses for the Track-It task (Panel A) and the Monster Mischief
Game (Panel B). In both panels, the top line shows the success rate for identifying the correct
location of the hidden target across each task opportunity (i.e., Attention score), while the bottom
line shows the success rate for correctly remembering which target was being tracked (i.e.,
Memory Check). Error bars represent the standard errors of the mean.
Conclusion
In conclusion, this study designed and evaluated Monster Mischief, a video game for the
assessment of selective sustained attention. Motivational design elements were incorporated to
create a video game that was engaging and motivating to young children. We then conducted an
experimental study to test the validity of Monster Mischief. A valid game-based assessment of
endogenously driven selective sustained attention would show an equivalent level of
performance to an existing assessment, would be significantly correlated with performance on
the existing assessment, and be more enjoyable than the existing assessment. The findings from
the present experiment largely align with the expected pattern of results. Specifically, the results
of the present study suggest that we successfully created a video game that was of comparable
difficulty level to the Track-It task. Additionally, performance on Track-It and Monster Mischief
was significantly correlated. Children also exhibited a preference for Monster Mischief over
Track-It when asked what game they would like to play again, presumably due to the addition of
Success Rate
Memory Check
50%
60%
70%
80%
90%
60%
70%
80%
90%
100%
1
2
3
4
Trial Number
Success Rate
Memory Check
50%
60%
70%
80%
90%
60%
70%
80%
90%
100%
1
2
3
4
Trial Number
(A)!
(B)!
Track-It
Monster Mischief
motivational design elements. However, there were no significant difference in enjoyment
observed from the results of the Smile Scale.
Limitations
Overall, the Monster Mischief video game shows potential as an engaging assessment of
children’s selective sustained attention; however, it is important to note several limitations of the
present study. First, we examined the relationship between children’s performance in Monster
Mischief and the heterogeneous condition of Track-It. Currently, it is unknown whether Monster
Mischief can be adapted to measure exogenously driven selective sustained attention as well.
Akin to Track-It, the ideal assessment would enable the measurement of both endogenous and
exogenous factors within a single task or game to provide a more comprehensive analysis of the
development and contribution of each type of selective sustained attention. Second, although we
compared Monster Mischief to Track-It, it would also be beneficial to determine if children’s
performance on Monster Mischief is related to other measures of attention and if it is predictive
of other cognitive abilities (e.g., inhibitory control).
A related issue of particular importance concerns the strength of the correlation between
Monster Mischief and Track-It. Although the correlation between the two attention performance
measures is respectable (r = 0.623), the two measures are not highly correlated, suggesting that
additional information is contained within these scores. Thus, future research will need to
distinguish among three different possibilities: (1) Track-It measures selective sustained attention
better than Monster Mischief, (2) Monster Mischief measures selective sustained attention better
than Track-It, or (3) Both tasks measure selective sustained attention to a similar degree but each
task reflects variability associated with task-specific features (e.g., longer trial duration in
Monster Mischief). No task is able to provide a pure measure of the construct of interest, a
problem known as “task impurity” in the literature (Miyake, Friedman, Emerson, Witzki,
Howerter, & Wager, 2000). Thus, it is likely that performance on Monster Mischief (and Track-
It) reflects selective sustained attention as well as other cognitive abilities and task artifacts.
Thus, future research examining the validity of these measures may help adjudicate between the
possibilities listed above.
Third, extensions to this work are warranted as it is critical to assess children’s
performance and engagement in Monster Mischief over more extended periods of play and
across time. With this groundwork in place, we can also begin to collect additional data on the
psychometric utility of a broader range of difficulty factors in the game. Lastly, additional
research is needed to fully validate Monster Mischief as a reliable measurement tool. For
example, the reliability of the game should be assessed either through test-retest reliability
(comparing the correlation of student performance over multiple testing sessions) or by
introducing multiple items with variable difficulty and reporting on the reliability of the different
items (i.e., Cronbach’s Alpha). Difficulty can be manipulated in Monster Mischief and Track It,
by varying several parameters including: speed, number of distractors, number of hiding
locations, and the trial duration (i.e., the amount of time before the target hides/disappears).
Future Directions
During the development of Monster Mischief, we also produced three additional game
mechanics that could be employed in future research. These game mechanics were intended to
increase the difficulty of the game in order to use the game with a wider age range of children
and to explore the possibility of using Monster Mischief as an instructional tool that may help
train children’s selective sustained attention capacity. Additionally, the game mechanics afford
an opportunity to explore whether Monster Mischief could be adapted to evaluate additional
aspects of attention such as divided attention. The additional game mechanics include: coins,
fireworks, and cloud mode. Each mechanic is discussed briefly below. When coins are turned on,
coins appear at random for a random period of time. If a player clicks on a coin, it is added to
their score. The rationale for this mechanic is to divide a player’s attention between the primary
task (tracking the monster that stole the jewel) and a secondary task (collecting coins). When the
firework mechanic is turned on, small red fireworks appear at random and then pop with a
shower of sparks. This mechanic was produced because young children often have trouble
reorienting to a task and returning to a state of focused attention after attention has been
disrupted (DiLalla & Watson, 1988). When cloud mode is turned on, there are no specific objects
behind which the monsters hide – they simply disappear. Then, players have to click on the area
where the target monster disappeared without the aid of visual landmarks. In this mode, we can
measure the visual-spatial accuracy of a player’s response (where accuracy is defined as 100%
minus the percent error, where percent error is calculated as the distance between x,y coordinates
of the clicked location and the actual location, divided by the x,y dimensions of the screen). The
rationale for this mechanic is to provide a more nuanced measure of success (degree of accuracy
instead of correct/incorrect) and to increase difficulty by removing memorable landmarks.
Future work should also explore the game as an instructional tool that may help train
children’s selective sustained attention capacity and reduce their susceptibility to distractions.
Presently, it is an open question whether a cognitive skills game for children could both assess
selective sustained attention and also help improve it, akin to the cognitive skills training games
that are currently popular for adults (e.g., Lumosity & Posit Science). The makers of cognitive
skills training games claim that improving performance on an assessment of a cognitive skill
could broadly transfer to other activities (e.g., one could improve working memory by practicing
a game which tests one’s working memory). Although it is well documented that transfer of
skills from one context to another is quite difficult (e.g., Perkins & Solomon, 1988), the potential
gains that could be achieved from such an intervention warrant further empirical work.
Ultimately, a video game like Monster Mischief might be used to test the hypothesis that
cognitive skills practice can increase the attention span of young children and decrease their
susceptibility to distractions. Accordingly, it is of interest to explore whether interventions could
be created to help support the development of selective sustained attention in young children.
Acknowledgements
We would like to thank Tara Helfer and Jeremy Galante for their help with illustrations and
animation. We also thank Sharan Shodhan for game programming. We thank the children,
parents, teachers, and staff at the Children’s School and Baldwin United Methodist Church
Preschool who made this work possible. This work was supported in part by a Graduate Training
Grant by the Department of Education (R305B090023).
References
Attali, Y., & Arieli-Attali, M. (2015). Gamification in assessment: Do points affect test
performance? Computers & Education, 83, 57–63.
Bloom, B.S. (1976). Human Characteristics and School Learning. McGraw-Hill; New York.
Bornstein, M.H. (1990). Attention in infancy and the prediction of cognitive capacities in
childhood. In J. Enns (Ed.) Development of Attention: Research and Theory. Elsevier.
Carroll , J. B. (1963). A model of School Learning. Teachers College Record, 64, 723–733.
Choudhury N., Gorman K. (2000). The relationship between attention and problem solving in
17–24 month old children. Infancy and Child Development, 9, 127–146.
Clark, D., Tanner-Smith, E., Killingsworth, S ., Bellamy, S. (2013). Digital Games for Learning:
A Systematic Review and Meta-Analysis (Executive Summary). Menlo Park, CA: SRI
International.
Colombo, J. & Cheatham, C.L. (2006). The emergence and basis of endogenous attention in
infancy and early childhood. In R. Kail (ed.), Advances in Child Development and
Behavior, Volume 34, (pp. 283-310). Oxford: Academic Press.
DeMarie-Dreblow, D. & Miler, P. H. (1988). The development of children’s strategies for
selective attention: Evidence for a transitional period. Child Development, 59, 1504-1513.
DiLalla, L. F. & Watson, M. W. (1988). Differentiation of fantasy and reality: Preschoolers’
reactions to interruptions in their play. Developmental Psychology, 24(2), 286-291.
Diamond, A. (2002). Normal development of prefrontal cortex from birth to young adulthood:
Cognitive functions, anatomy, and biochemistry. In D.T. Stuss & R.T. Knight (eds.),
Principles of frontal lobe function (pp. 466-503). London, UK: Oxford University Press.
Diamond, A. (2006). The early development of executive functions. In E. Bialystok & F. Craik
(Eds.), Lifespan Cognition: Mechanisms of Change (pp. 7-95). Oxford University Press.
Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., Pagani,
L.S., Feinstein, L., Engel, M., Brooks-Gunn, J., Sexton, H., Duckworth, K., & Japel, C.
(2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428-
1446.
Eccles, J. S. & Wigfield, A. (2002) Motivational beliefs, values, and goals. Annual review of
psychology, 53, 109–32.
Erickson, L. C., Thiessen, E. D., Godwin, K. E., Dickerson, J. P., & Fisher, A. V. (2014).
Endogenously- but not Exogenously-driven Selective Sustained Attention is Related to
Learning in a Classroom-like Setting in Kindergarten Children. In P. Bello, M. Guarini,
M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Conference of the
Cognitive Science Society (pp. 457-462). Austin, TX: Cognitive Science Society.
Fisher, A. V., Godwin, K. E., & Seltman, H. (2014). Visual environment, attention allocation,
and learning: When too much of a good thing may be bad. Psychological Science, 25(7),
1362-1370.
Fisher, A.V. & Kloos, H. (in press). Development of selective sustained attention: The Role of
executive functions. In L. Freund, P. McCardle, and J. Griffin (Eds.), Executive function
in preschool age children: Integrating measurement, neurodevelopment and translational
research. APA Press.
Fisher, A. V., Thiessen, E., Godwin, K., Kloos, H., & Dickerson, J. P. (2013). Mechanisms of
focused attention in 3- to 5-year-old children: Evidence from a new object tracking task.
Journal of Experimental Child Psychology, 114(2), 275-294.
Geary, K. E. (2011). The impact of choice on child sustained attention in the preschool
classroom (Unpublished Thesis). Louisiana State University, Baton Rouge, Louisiana.
Gitelman, D. R. (2003). Attention and its disorders: Imaging in clinical neuroscience. British
Medical Bulletin, 65(1), 21-34.
Glasslab. (2014). Pyschometric Considerations in Game-Based Assessments. White Paper
Released by Glasslab, 160.
Godwin, K. E., Almeda, M. V., Petroccia, M., Baker, R. S., & Fisher, A. V. (2013). Classroom
activities and off-task behavior in elementary school children. In M. Knauff, M. Pauen,
N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the
Cognitive Science Society (pp. 2428-2433). Austin, TX: Cognitive Science Society.
Goodman, L. (1990). Time and learning in the special education classroom. SUNY Press.
Hunicke, R., LeBlanc, M. and Zubek, R. (2004). MDA: a formal approach to game design and
game research. In Proceedings of the Workshop on Challenges in Game AI, 19th National
Conference on Artificial Intelligence, AAAI Press, San Jose, California.Jonides, J.
(1981). Voluntary vs. Automatic control over the mind's eye's movement. In J.B. Long &
A.D. Baddeley (Eds.) Attention and Performance IX. Hillsdale, N.J.:Lawrence Erlbaum
Associates.
Loewenstein, G. (1994) The Psychology of curiosity: A review and reinterpretation.
Psychological bulletin, 116, 1, 75–98.
Luna, B. (2009). Developmental changes in cognitive control through adolescence. Advances in
Child Development and Behavior, 37, 233-278.
McPherson, J., & Burns, N. R. (2008). Assessing the validity of computer-game-like tests of
processing speed and working memory. Behavior Research Methods, 40(4), 969–981.
Malone, T. (1981) Toward a theory of intrinsically motivating instruction. Cognitive Science, 5,
4, 333–369.
Miller, E.K & Cohen, J.D. (2001). An integrative theory of prefrontal cortex function. Annual
Review of Neuroscience, 24, 167-202.
Miyake, A., Friedman, N.P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D.
(2000). The unit and diversity of executive functions and their contributions to complex
“frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49-100.
Oakes, L., Kannass, K.N., & Shaddy, D.J. (2002). Developmental changes in endogenous control
of attention: The role of target familiarity on infants' distraction latency. Child
Development, 73, 1644-1655.
Parker, L. & Lepper, M. (1992) Effects of fantasy contexts on children’s learning and
motivation: Making learning more fun. Journal of Personality and Social Psychology, 62,
4, 625–633.
Pashler, H., Johnston, J. C., & Ruthruff, E. (2001). Attention and performance. Annual Review
of Psychology, 52, 629-651.
Perkins, D. N., & Salomon, G. (1988). Teaching for transfer. Educational Leadership, 46 (1):
22-32.
Posner, M. I. (1980). Orienting of attention. The Quarterly Journal of Experimental Psychology
32 (1): 3–25.
Posner, M.I. & Petersen, S.E. (1990). The attention system of the human brain. Annual Review
of Neuroscience, 13:25–42.
Posner, M.I. & Rothbart, K.R. (2007). Research on attention networks as a model for the
integration of psychological science. Annual Review of Psychology, 58:1–23.
Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a
parallel tracking mechanism. Spatial Vision, 3, 179–197.
Read, J.C. and MacFarlane, S. (2006) Using the fun toolkit and other survey methods to gather
opinions in child computer interaction. Proceeding of the 2006 conference on Interaction
design and children - IDC ’06, 81.
Richards, J.E. (2003). The development of visual attention and the brain. In De Haan & Johnson
(Eds.), The cognitive neuroscience of development. East Sussex, UK: Psychology Press.
Ritter, F. & Schooler, L. (2001) The learning curve. International Encyclopedia of the Social and
Behavioral Sciences, 8602–8605.
Rosvold, H.E., Mirsky, A.F., Sarason, I., Bransome, E.D., Beck, L.H. (1956). A Continuous
performance test of brain damage. Journal of Consulting Psychology, 20, 343-350.
Ruff, A.H. & Lawson, K.R. (1990). Development of sustained, focused attention in young
children during free play. Developmental Psychology, 26, 85–93.
Ruff, H. A. & Rothbart, M. K. (1996). Attention in early development: Themes and variations.
New York, NY: Oxford University Press.
Ruff, H. & Capozzoli, M. (2003) Development of attention and distractibility in the first 4 years
of life. Developmental Psychology, 39, 877-890.
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A
self-determination theory approach. Motivation and Emotion, 30, 347-365.
Sarid, M., & Breznitz, Z. (1997). Developmental aspects of sustained attention among 2- to 6-
year-old children. International Journal of Behavioral Development, 21, 303-312.
Setliff, A. E., & Courage, M. L. (2011). Background television and infants allocation of their
attention during toy play. Infancy, 16(6), 611-639.
Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information
processing: I. Detection, search, and attention. Psychological Review, 84(1), 1-66.
Shute, V. J. (2011). Stealth Assessment in Computer-Based Games To Support Learning.
Computer Games and Instruction, 503–524.
Tellinghuisen, D.J., Oakes, L.M., & Tjebkes, T.L. (1999). The influence of attentional state and
stimulus characteristics on infant distractibility. Cognitive Development, 14, 199-213.
Tenorio Delgado, M., Arango Uribe, P., Aparicio Alonso, A., & Rosas Díaz, R. (2014). TENI: A
comprehensive battery for cognitive assessment based on games and technology. Child
Neuropsychology, 1–16.
Weber R. & Popova, L. (2012). Testing equivalence in communication research: Theory and
application. Communication Methods and Measures, 6(3), 190-213.
Yantis, S. (1992). Multielement visual tracking: Attention and perceptual organization. Cognitive
Psychology, 24, 295–340.
Yu, C., & Smith, L.B. (2012). Embodied attention and word learning by toddlers. Cognition,
244–262.
!
!
!
!
!
Table 1. Alignment between Track-It and Monster Mischief
Track-It
Monster
Mischief
Distractors
(Number and Type)
3 unique distractors
✔
Motion
(Target/Distractor motion path)
Randomized path
✔
Trials
(Number)
5 trials
(1 practice trial and 4 test trials)
✔
Trial Duration
Approximately 10s
Approximately 20s
Test Locations
(Hiding locations)
9 potential locations (3x3 grid)
✔
Memory Assessment
(Number of lures)
3
✔
Table 2. Motivational Design Elements of Monster Mischief compared to Track-It
Monster Mischief
Track-It
Targets
Cute animated “monsters”
Simple shapes
Background
The game takes place in colorful settings that are
familiar to young children (e.g., a play room)
9x9 grid
Test
locations
Characters hide behind the objects in the room
(e.g., the characters disappear behind the toys in the
play room)
The shapes disappear in
one of the 9 squares
Feedback
If the child finds the hiding location of the target
character, the character emerges from behind the
object and gives the child a jewel. If the child selects
the wrong location, the target character emerges
from the correct location and giggles.
No feedback is provided
Backstory
At the beginning of the game the mischievous
monsters take jewels from the castle. The child’s task
is to find where the characters are hiding in order to
collect all of the jewels.
No backstory
Overarching
Goal
Collect all of the jewels at each level of the game.
Within each level, the number of jewels the child has
earned is displayed at the top of the screen. A map
marks the child’s progress across the game levels.
None
Appendix
Introduction of the Smile Scale:
“During this game I will ask you a couple questions using a special scale. Here is how it works. See all
the faces [point to the faces on the scale]? See how some of the faces are frowning and some are smiling?
Look at this face [point to face 1]. This face is making a big frown - that means I really don't like
something. This face has a little frown [point to face 2] – that means I dislike something a little bit. Look
at this face [point to face 5]. This face has a big smile - that means I really like something. This face has a
little smile [point to face 4] – that means I like something a little bit. Look at this face [point to face 3],
this face is not smiling or frowning so that means I think something is just okay.”
Practice items:
1. “Which face should I point to if I told you that I really don't like brussels sprouts?”
[Wait for child’s response]
“Yes, I would point to the face making a big frown [point to face 1] because I really don't like brussels
sprouts.” [Or: “That’s a good guess, but I would point to the face making a big frown [point to face 1]
because I really don't like brussels sprouts.”]
2. “Okay, try this one - which face should I point to if I told you that I really like to jump rope?”
[Wait for child’s response]
“Yes, I would point to the face making a big smile [point to face 5] because I really like to jump rope.”
[Or: “That’s a good guess, but I would point to the face making a big smile [point to face 5] because I
really like to jump rope.”]
3. “Okay let's do one more. Which face should I point to if I told you I think the color green is just
okay?”
[Wait for child’s response]
1"
2"
3"
4"
5"
“Yes, I would point to this face that is not smiling or frowning [point to face 3] because that means I think
something is just okay.” [Or: “That’s a good guess, but I would point to this face that is not smiling or
frowning [point to face 3] because that means I think something is just okay.”]
“All right we will use this again in a little bit but now we are going to play some computer games”
Rating each game using the Smile Scale:
“Okay, now I want you to point to the face that shows me what you thought about this game: Remember
this face means you really don’t like this game [point to 1], this face means you disliked this game a little
[point to 2], this face means you thought the game was just okay [point to 3], this face means you liked
the game a little [point to 4], and this face means you really like the game [point to 5]. Okay what did you
think about the game - whatever you think is fine.”
