Content uploaded by H. van Oostendorp
Author content
All content in this area was uploaded by H. van Oostendorp
Content may be subject to copyright.
Pieter Wouters
Erik van der Spek
Herre van Oostendorp
Institute of Information and Computing Sciences, Utrecht University, the Netherlands
ABSTRACT
Despite scant empirical substantiation, serious games are in widespread use. We review 28
studies with empirical data from a learning outcome perspective to outline the effectiveness of
serious games (compared to other learning approaches and specific game features). We conclude
that serious games potentially improve the acquisition of knowledge and cognitive skills.
Moreover, they seem to be promising for the acquisition of fine-grid motor skills and to
accomplish attitudinal change. However, not all game features increase the effectiveness of the
game. To further advance game research we propose recommendations including the alignment
of learning outcome(s) and game type, the alignment of the game complexity and human
cognitive processes, attention for cognitive and motivational processes, research on specific
mitigating factors like gender on game effectiveness and, finally, developing new ways of
assessing game effectiveness.
Keywords: computer games; video games; PC games; digital games; serious games; virtual
reality; instructional technology; learning outcome; computer-based training; cognitive theory;
cognitive skills, attitudinal change; game types; simulations; game-based learning
INTRODUCTION
The use of games in learning and instruction, often referred to as serious games, has been
propagated by many researchers. Serious games are hypothesized to address both the cognitive
and affective dimensions of learning (O’Neil, Wainess & Baker, 2005), to enable learners to
adapt learning to their cognitive needs and to provide motivation for learning (Malone, 1981).
However, reviews focusing on serious games have revealed little substantiation for these claims
(Fletcher & Tobias, 2008; Kirriemuir & McFarlane, 2004; Leemkuil, de Jong & Ootes, 2000;
O’Neil et al; Vogel et al., 2006).
This review focuses on the learning outcomes for two reasons. First, typically serious games
aim at specific learning goals and consequently for specific learning outcomes as well. Therefore
it seems obvious to conduct the review from this perspective. Secondly, many studies on serious
games have focused particularly on cognitive learning outcomes: learning of knowledge and
problem solving skills. Consequently, previous reviews have focused on particular types of
learning outcomes and have neglected other types. We contend that a comprehensive taxonomy
of learning outcomes will not only reveal in which situations serious games improve learning, but
also uncover dimensions of learning that have been neglected thus far in reviews.
In the remainder of this chapter we first define games. Next, we present a taxonomy of
learning outcomes. For each learning outcome we then review relevant studies and draw some
conclusions. Finally, we present some directions for future research and draw a final conclusion.
WHAT ARE GAMES?
A serious game is a computer based game with a primary purpose other than entertainment,
ranging from anywhere between advertisements to military training exercises (Michael & Chen,
2005). Naturally in this review we will concern ourselves mainly with games that aim at the
aforementioned learning outcomes. Many definitions exist that describe a game (cf. Garris,
Ahlers & Driskell, 2002; Vogel et al., 2006), but mostly a definition along the following lines is
chosen: that it is goal-directed, a competitive activity (against the computer, another player, or
oneself) and conducted within a framework of agreed rules (Lindley, 2004). In addition, games
constantly provide feedback to enable players to monitor their progress towards the goal
(Prensky, 2001).
A TAXONOMY OF LEARNING OUTCOMES IN SERIOUS GAMES
There are many classifications of learning outcomes. Traditionally, researchers have focused on
the cognitive dimension of learning outcomes (Bloom, 1956; Gagné, 1977). Others have included
affect-oriented objectives such as appreciation (Krathwohl, Bloom & Massai, 1964). More recent,
other classifications have emerged identifying factors such as collaboration/teamwork,
communication and self-regulation as potential outcomes of learning (Baker & Mayer, 1999). An
interesting classification of learning outcomes has been provided by Kraiger, Ford and Salas
(1993), who distinguish between cognitive outcomes (e.g., problem solving), skill-based
outcomes concerning the development of technical or motor skills, and affective outcomes
including attitude and motivation. Drawing from the two latter classification schemes, we propose
a taxonomy consisting of four categories of learning outcomes: cognitive, motor skills, affective
and communicative. Figure 1 presents an overview of these learning outcomes and their
constituent parts. Insert figure 1 here
Cognitive learning outcomes can be divided into knowledge and cognitive skills. Knowledge
refers to encoded knowledge reflecting both text-oriented (e.g., verbal knowledge) and non text-
oriented knowledge (e.g., knowledge in the form of an image). Several types of encoded
knowledge can be discerned such as declarative (explicit knowledge of facts) and procedural
(knowledge of how to perform a task). A cognitive skill pertains to more complex cognitive
processes. In problem solving, for example, learners have to apply knowledge and rules to solve
new problems. In complex and dynamic situations people are sometimes forced to make decisions
under time-pressure. Such decision making skills require situational awareness, that is, the ability
to attend to and perceive the relevant information in a situation, comprehend this information and
predict how the situation may develop (O’Brien & O’Hare, 2007).
The second type of learning outcome, learning motor skills, involves several stages. Initially a
learner has to acquire the skill by making a transition from declarative knowledge to procedural
knowledge. In subsequent stages the learner practices the motor behavior and in this way
compiles the behavior, that is, make the motor behavior faster, less error-prone and independent
of verbal rehearsal.
With affective learning outcomes we can differentiate two subtypes. To start with, learning
may focus on a change in the attitude of the learner. Attitudes refer to internal states that
influence the choices or actions of an individual (Gagné, 1977). This may pertain to a change
from a negative to a positive learning attitude towards (subjects at) school, but also to a change in
behavior that is exhibited in daily live (e.g., driving cautiously) or for therapeutic purposes (e.g.,
overcoming fear of spiders). The second subtype, motivation, is a prerequisite for learning to
commence. Motivation reflects the willingness to pay attention to learning material and to spent
cognitive resources to process information.
The last type comprises communicative learning outcomes. Although collaborative learning is
claimed to lead to a deeper level of understanding and long term retention of the learned material,
it also emphasizes the opportunities for developing social and communication skills, and building
social relationships and group cohesion (Kreijns, Kirschner & Jochems, 2003). In environments
where teams have to work together on tasks that go beyond the capabilities of one individual
(e.g., firefighters, cockpit crews), the training of communication and collaboration skills can be
the primary purpose of an instructional intervention.
In general the performance of (complex) tasks will involve different types of learning
outcomes. For example, learning to drive may comprise knowledge (e.g., traffic rules), motor
skills (e.g., changing gears) and attitudinal change (e.g., driving cautiously). This example also
illustrates that the learning outcomes are sometimes hierarchical: before the changing gears motor
skill can be performed the procedural knowledge has to be learned.
THE REVIEW
For the purpose of this review we searched several databases (PsychINFO, ERIC) with terms
including ‘game-based learning’, ‘PC games’, ‘video game’, ‘computer video game’, ’serious
games’, ‘educational games’, ‘simulation games’, ‘virtual environments’, ‘virtual reality’. If
necessary these terms were combined with ‘learning’, ‘instruction’ and ‘training’. In addition, we
consulted the proceedings of relevant conferences (CHI, AERA and so forth) with the same
terms. The review was conducted in Summer 2008 and covered the last 10 years. Studies were
only considered when empirical data were available. Table 1 classifies the serious game studies in
the taxonomy of learning outcomes. We classified each study according to the learning outcome
that the research was primarily aimed at.
Insert table 1 here
First we discuss the relevant studies for each type of learning outcome. We discuss studies that
compare serious games with other instructional methods as well as studies that investigate the
effectiveness of specific game features.
Cognitive learning outcomes
Encoded knowledge
First, studies comparing a game group with a control group are discussed. In the multi-user
virtual environment River City two gaming groups (with a focus on respectively ‘learning-by-
doing’ and modeling) were compared with a control condition (Dede, Clarke, Ketelhut, Nelson &
Bowman, 2005). The task for all participants was to discover why residents of a virtual town were
getting ill. In this process they learned about biology. The results showed that the two gaming
groups gained more biology content knowledge than the control group. In learning about
electromagnetism Squire, Barnett, Grant and Higginbotham (2004) compared learners engaging
in the Supercharged game with a group engaging in guided inquiry. The game places students in
a three dimensional environment where they must navigate a spaceship by controlling the electric
charge of the ship. Learners in the Supercharged game performed better on knowledge than
guided inquiry learners. In training Navy electronic technicians, Parchman, Ellis, Christinaz and
Vogel (2000) found that trainees engaging in an adventure game (King’s Quest V-based) in which
they had to visit a series of compartments in a battleship and perform exercises, were
outperformed by trainees who received a computer-based practice-and-drill or an enhanced
computer-based instruction on a posttest measuring knowledge of definitions. However, no
differences were found on knowledge of symbols. In the domain of biology the game Metalloman
was compared with a hypertext and a text instruction group in knowledge on physiology concepts
(Wong et al., 2007). It was observed that the game and the hypertext group yielded higher
learning gains than the text instruction group.
The effectiveness of the game features for relevance of information, interactivity, instructional
guidelines, level of stress, game task and game type have been investigated as well. In the game
America’s Army people learn about the values and history of the army. A study with military
academy recruits revealed that information that was relevant for the task was better recalled than
non-relevant information (Belanich, Sibley & Orvis, 2004). Regarding the game feature for
interactivity Wong et al. (2007) surprisingly found that an interactive version of Metalloman
yielded no higher learning gains than a non-interactive version of the game. In KMQuest, learners
run a commercial organization in which they have to make decisions in order to make the
organization more efficient. Leemkuil (2006) investigated several variations of the game feature
for instructional guidelines (advice vs. no advice, extra assignments vs. no extra assignments,
advice with hints vs. advice without hints vs. no advice), however, no difference on the
knowledge learning outcome was found in a transfer test that was administered after the training
in KMQuest. In a training with the game Delta Force no effect of the game feature for level of
stress was found on recall of knowledge of military tactics content (e.g., use of equipment)
between trainees who were exposed to either a low or a high level of stress (Morris, Hancock &
Shirkey, 2004). In geography Virvou, Katsionis and Manos (2005) investigated the effect of the
game feature for game task. For this purpose they compared a group engaging in a 3D virtual
reality game (based on Doom) with a group working in a hypertext environment. Both groups
differed in the fact that the virtual reality group had a game task, that is, they had to navigate
through a virtual 3D world with the explicit mission to find the missing pages of the ‘book of
wisdom’. The game group yielded higher learning gains than the hypertext group. Closely related
to the game task is the type of game that is used. In the game Re-Mission young people engage in
missions in 3-D virtual bodies of cancer patients and learn about the mechanisms underlying
cancer. Participants engaging in Re-Mission showed larger knowledge gains than participants
who received a commercial adventure game (Beale, Kato, Marin-Bowling, Guthrie & Cole,
2007).
The fact that three out of four studies report higher performance for groups learning with
games provides some evidence that the new generation of serious games support the acquisition
of knowledge. The effectiveness of specific game features is mixed. The game features for
relevance of information, game task and game type ameliorate the acquisition of knowledge,
whereas the game features for level of stress, instructional guidelines, and interactivity failed to
have an impact on learning.
Cognitive skills
First, we describe studies comparing a game group with a control group. In the multi-user
environment QuestAtlantis (QA) learners travel to virtual worlds and engage in educational
activities (quests). It was used by Barab, Warren and Ingram-Goble (2006) to compare the
performance in writing skills of a group receiving traditional instruction with a group working as
an investigative reporter for the local newspaper in QA. In writing tasks similar to the tasks
performed during instruction the QA group showed significantly more improvement. In the
aforementioned River City environment cognitive skills were measured in two ways: with open-
ended questions and with a letter to the mayor in which the participants discussed their
hypothesis, the results and the interpretations. Interestingly only a difference in favor of the game
groups was found when the cognitive skills were measured by the letter to the mayor (Dede et al.,
2005). In the military, games have been used to investigate the effect on learning specific
cognitive skills during a flight mission such as task management, decision making, and assessing
the situation. In the training of cockpit crews, commonly known as crew resource management
(CRM), the comparison of a group training on a PC-based simulator and a control group revealed
that the former group performed better on task management and situational awareness, but not on
other cognitive skills, such as decision making and planning (Nullmeyer, Spiker, Golas, Logan &
Clemons, 2006). Problematic in this study was that no information was provided with respect to
the training in the control condition. In the aforementioned training of Navy electronic
technicians, Parchman et al. (2000) found that the two computer-based groups performed better
on the application of principles than the game and classical instruction groups. For the application
of rules no differences between the groups were found. Finally, Ke and Grabowski (2007)
compared two game groups (students were either assigned to a cooperative or competitive version
of the game ASTRA Eagle) with a no game group on the acquisition of mathematical problem
solving skills. It was found that both game groups outperformed the no game group on
mathematical skills.
Other researchers have studied the effect of the game features for instructional guidelines,
level of stress and game type. Leemkuil (2006) varied the guideline advice (advice vs. no advice
and advice with hints vs. advice without hints vs. no advice) in the KMQuest environment, but
found no differences in a transfer task that was administered after the learning phase in KMQuest.
Nelson (2007) used River City to investigate whether guidance would improve science inquiry
and hypothesis formation skills. It appeared that groups with either extensive or moderate
guidance did not use this guidance and consequently did not gain a better command of these
cognitive skills than a group without such guidance. However, further analyses revealed higher
learning gains for learners who did use the provided guidance. Also the effect of extra
assignments was studied in the KMQuest learning environment, but no differences were found on
the transfer task that was administered afterwards (Leemkuil, 2006). The game feature for level of
stress was investigated in the game Delta Force in which the participants had to engage in an
arctic mission that required cognitive skills. The trainees in the group with a high level of stress
during training were more successful in completing the mission than trainees who were exposed
to a low level of stress (Morris et al., 2004). Finally, the impact of the game feature for game
type was investigated by comparing the effect of an action game (Medal of Honor: Pacific
Assault) with a non action game (Balance) on spatial cognition (Feng, Spence & Pratt, 2007). It
was found that only the action game enhanced spatial cognition. Interestingly, they also found
that the initial superiority of males over females in spatial cognition was much reduced after
working with the action game.
Although four out of five studies substantiate the claim that serious games are more effective
in training cognitive skills than traditional instructional methods, the results in the River City
study also pose the question how cognitive skills should be measured. The contextualized type of
learning that takes place in serious games may not be detected by traditional measurements, but
all the more with alternative measurements (i.e., essays). Also for learning cognitive skills the
game feature for instructional guidelines failed to be effective.
Motor skill learning outcomes
Much research has focused on the effect of video game experience on screen-mediated surgery
skills (i.e., the surgeon operates via a monitor). Researchers have compared groups that practiced
with games with groups that did not practice with games (Rosenberg, Landsittel & Averch, 2005;
Waxberg, Schwaitzberg & Cao, 2005). These studies did not show that video game experience
yielded better surgery skills. For instance, in laparoscopic surgery Waxberg et al. (2005)
hypothesized that practicing with the video game James Bond 007: Goldeneye
for a week
would
lead to better performance on several tasks on a surgery skills trainer. The results showed that this
was true for some tasks, but that the no-game group performed better on other tasks.
Other researchers have correlated video game experience with surgery skills and reported
that
video game experience predicted surgery performance (Enochsson
et al., 2004; Grantcharov,
Bardram, Funch-Jensen & Rosenberg, 2003; Rosser et al., 2007). Rosser et al. (2007), for
instance, showed that surgeons with video games experience made less errors and showed faster
completion times in a learning environment for laparoscopic surgery. Moreover, the video games
skills of these surgeons, demonstrated during three different video games, appeared to be
significant predictors of laparoscopic surgery skills.
The results in the domain of surgery are still inconclusive: whereas experimental designs fail
to show a beneficial effect, correlation studies seem to confirm the predictive power of game
experience. Given the ample evidence indicating that experience in video games enhances the
ability for visual search (e.g., Castel, Pratt & Drummond, 2005; Green & Bavelier, 2003, 2006)
and dual-tasking (Satyen & Ohtsuka, 2001), we believe that the use of serious games is promising
in learning fine-grid motor skills that require excellent hand-eye coordination.
Affective learning outcomes
Attitude
Increasingly, virtual reality systems are used to support people in desensitizing a large range of
fears and phobias by allowing them to confront frightening situations without the danger of
possible (physical) harm. Since these systems are rather expensive, researchers have investigated
whether realistic games can be used for this purpose as well. Their focus was on the effect of
games on attitude and not the comparison of a game therapy with a traditional therapy. Bouchard,
Côte, St-Jacques, Robillard and Renaud (2006) designed a therapy requiring participants with fear
of spiders to engage in a Half-Life based environment where they were increasingly exposed to
spiders. Before the treatment, the majority stayed 2 metres from a bowl with spiders. After the
treatment the majority was able to stand next to the bowl. A similar positive effect was obtained
with participants who were diagnosed as having an accident phobia. Before and after the
intervention comprising a 12 hours program of game racing (e.g., London Racer) they were
assessed with several ratings such as distress and severity of fear of driving. It was found that the
participants showed posttest reductions on all measures (Walshe, Lewis, Kim, O’Sullivan &
Wiederhold, 2003).
The effectiveness of serious games on attitudinal change towards school topics was
investigated in the aforementioned study of Ke and Grabowski (2007), which revealed that
students engaging in a cooperative game developed a more positive attitude towards mathematics
than students who engaged in a competitive game or in paper-and-pencil lessons. This effect was
stronger for socio-economically disadvantaged students.
Regarding attitudinal change the effect of the features game task and game type have been
studied. In a simulator environment for training driving skills a group receiving a game task (i.e.,
they had to follow an ambulance without losing sight of it) was compared with a group without
such game task. It was found that the group with the game task showed safer traffic behavior on
dimensions such as looking in rearview windows and lane changing than the group without a
game task (Backlund, Engström, Johanneson & Lebram, 2007). The effect of the feature game
type on attitudinal change was investigated in a series of experiments conducted by Fischer,
Kubitzki, Guter and Frey (2007). They found that participants who played race games (e.g.,
Burnout) exhibited a less cautious driving behavior in terms of risk-taking and excitement than
participants who played neutral games (e.g., Fifa 2005). Apparently, not only is engaging in a
game task important, but so is the game type in which the task is performed.
Tentatively it can be concluded that serious games facilitate attitudinal change. Game features,
such as a game task and the game type also have an impact on attitude. The findings of both race
games studies also confirm that individual characteristics should be taken into account when
using serious games for attitudinal change. The traumatized participants in the Walshe et al.
(2003) study benefitted from the race game and became less fearful for driving, whereas the non-
traumatized participants in the Fisher et al. (2007) study became more reckless drivers after
engaging in the race game.
Motivation
Previous reviews have claimed that serious games motivate players to continue and subsequently
it is alleged that this feature can be useful for the purpose of learning (cf. Garris et al., 2002), but
recent research on motivation is scant. Dede et al. (2005) argued that the large drop in absentee
rate (50%) during learning in River City environment may have indicated an increased
engagement during the implementation of River City. However, the absentee rate for the
traditional instruction group was not reported. In the aforementioned training of military cadets in
the Parchman et al. study (2000) a motivation questionnaire based on Keller’s ARCS model was
used to compare trainees’ motivation in the four groups (a game group, classical instruction, and
computer-based practice-and-drill or enhanced instruction). Although the game group participants
were more attentive to the contents than the classical instruction and computer-based practice-
and-drill group, no differences were observed between the game and computer-based enhanced
instruction groups. A qualification of the motivational aspects of games comes from Tuzun,
Yilmaz Soylu, Karakus, Inal and Kizilkaya (2008) who compared a game group (Quest Atlantis)
with traditional school learning on intrinsic and extrinsic motivation. They found some evidence
that students in the game group were more intrinsically motivated, whereas students in the
traditional school setting were more extrinsically motivated.
In short, no recent convincing evidence was found for the assumed motivational pull of
serious games.
Given the popularity of playing games among adolescents, it seems obvious that
games are motivating. It is not clear to what extent this pertains to serious games. In Quest
Atlantis, for example, Lim, Nonis and Hedberg (2006) reported that learners were less motivated
than the researchers had expected. It is remarkable that characteristics of games such as
immersion and interactivity that are considered motivating in entertainment games, refrained
students in Quest Atlantis from full engagement in the learning task. This indicates that a better
understanding is required about the underlying motivational processes in serious games. We will
return to this issue in the Discussion section at the end of the chapter.
Communicative learning outcomes
In training communicative skills of cockpit crews (CRM training), Brannick, Prince and Salas
(2005) compared the communication skills of trainees receiving CRM-simulator training with a
group receiving group exercises and video games (Asteroids). The CRM-simulator group showed
better communicative skills than the group with exercises and video games in an assessment task
requiring the trainees to contact the air traffic controller (ATC) in order to obtain the information
that was deliberately omitted by the ATC. However, these results were not confirmed in a similar
study comparing a PC-based simulator and a control group (Nullmeyer et al., 2006), although a
problem with this study was that it did not report what kind of training the control group received.
The impact of a game feature called the awareness tool was investigated in the game
SpaceMiners where dyads have to collect minerals located in asteroids by launching drones and
bring them to a space station. The players can use tools to manipulate the direction of the drone,
but they have to negotiate where to position these tools in space. An awareness tool helps players
to understand the activities of other players in the game. It appeared that pairs with an awareness
tool outperformed pairs without an awareness tool in collecting minerals (Nova, Dillenbourg,
Wehrle, Goslin & Bourquin, 2003). The game feature for level of immersion was investigated in
the game DOOM II in which dyads had to collaborate in order to find their way through a virtual
maze. It appeared that the quality of collaboration in the immersive (head-mounted display) and
the nonimmersive (monitor) conditions was comparable, with the exception that the immersive
group took more time to complete the task (Galimberti, Ignazi, Vercesi, & Riva, 2001).
Research investigating the effect of serious games on communicative skills is still
undeveloped. Recently, massive multiplayer online games (MMOGs) have become very popular.
These MMOGs (e.g., World of Warcraft) are graphical 3D videogames allowing players, by
means of self-created digital characters or ‘avatars’, to interact with the game world and with
other players’ avatars as well. Research on interaction and collaboration in these games is very
limited. There is some research on social interaction in these MMOGs, but these studies focus on
the characteristics of game players (e.g., Seay, Jerome, Lee & Kraut, 2004). Hopefully, the
increasing popularity of MMOGs will become an encouragement for more research into the
impact of online gaming on communicative skills.
DISCUSSION: FUTURE RESEARCH DIRECTIONS
It should be noted that the number of studies is too low to make definite conclusions. In order to
substantiate the claims regarding the learning potential of serious games more research is required
and, even more important, data and results have to be reported. We recommend five directions for
follow-on research that may further advance learning with serious games.
1. Alignment of game type and learning outcome
First, the mixed results of this review pose the question whether the appropriate game design was
selected for obtaining the specified learning outcome(s). As different game types can elicit very
different cognitive and affective responses in the player (e.g., Ravaja et al., 2004), designers of
serious games should carefully consider the implications of the game design on the possible
learning outcomes. For this reason, we propose a framework that categorizes the games according
to their level of cognitive and affective complexity (CALC). Most taxonomies that have been
introduced (cf. Lindley, 2003; Björk, Lundgren, & Holopainen, 2003) approach the categorization
from a design standpoint. As our review approached the research from a user-centered viewpoint,
namely the learning outcomes, we also propose a more user-centered framework for the
categorization of games (Figure 2). Insert Figure 2 here
The scale we currently propose consists of four different layers, although in the future more
differentiation is possible. Furthermore, a level of cognitive and affective complexity is defined as
the corresponding layer together with the previous underlying layers. As increasing levels of
complexity open up new possibilities for training while maintaining those of previous levels, the
levels are said to work cumulatively in possible learning outcomes.
The first level comprises games that are textual or symbolic in outlook often with simple and
explicit mechanics. In cognitive terms, players have to create a mental model of the game rules
and consequences of the actions they perform. At this basic level, games can be used to train
problem solving skills, decision making and teach verbal and conceptual knowledge.
The second level comprises games that are situated in a spatial environment. Here spatial
dimensions have to be interpreted and the spatial interrelationships between different objects are
thus added to the mental model constructed in the first layer. Some basic situational awareness
can be trained with these kinds of games, for instance in assessing the distance and route to the
nearest exit in case of fire, as well as hand-eye coordination and motor skills.
A layer on top of this is the presence layer, where players not only have to navigate a virtual
world, but strongly feel that they are a person immersed in this world. This feeling of ‘presence’
opens up a range of affective responses that could be part of a training exercise, for example
stress control or anxiety alleviation. Different presentation types generate different degrees of
presence (Nunez & Blake, 2003), but First Person 3D games are probably best suited.
Lastly, because the feeling of being in an environment opens up possibilities for social
interactions with other beings inside the virtual world, and these virtual communities add to the
complexity of the game, multiplayer or MMO games make up the top level on our cognitive and
affective complexity scale. These games can be used to study and train a person’s social skills in
a group or large scale community setting.
As all taxonomies trying to cope with the highly diffuse area of games, it is not perfect; a
3D game with no social interaction may be more perceptually rich, and therefore cognitively
demanding, than a text based multi-user dungeon. However, while tentative, we maintain that
it can provide a guideline for choosing the right game design to achieve the desired learning
goals; situational awareness may not transfer well when trained with a 2D game, while overly
complex designs may compromise the learning outcomes that can also be achieved with
simpler games.
2. The role of human cognitive architecture
The second recommendation pertains to the question how to (further) optimize the effectiveness
of serious games. As this review has shown game features can be manipulated to improve the
effectiveness of the serious game. In Table 2 an overview is presented of the game features that
were discussed in four types of learning outcomes. It shows that the investigated instructional
guidelines failed to increase the effectiveness of the serious game. In other cases the effectiveness
was only increased for one type of learning outcome, but not for another learning outcome.
Insert Table 2 here
Playing a serious game is a complex task, even when an appropriate design was chosen for the
intended learning outcomes: Players have to visually attend different locations on the screen,
coordinate this with mouse or joystick movement, interpret verbal cues, and solve problems that
occur during the game play. We contend that the effectiveness of a game feature is contingent on
the ability of designers to align the complexity of the serious game with the limitations of human
processing capacity. From a cognitive theory perspective it can be argued that without support
novice game players can easily become overwhelmed by all the information that has to be
processed. For example, a relevant game task may limit the amount of irrelevant information that
the player has to process. In this way cognitive capacity can be effectively used for processing
information that fosters learning from the serious game.
It would be interesting to see whether instructional guidelines that have been successful in
learning from animations pertain to the design of serious games as well (cf. Wouters, Tabbers &
Paas, 2007). Potential instructional guidelines that may reduce information overload in serious
games include pacing (i.e., regulating the speed of information presentation), focusing attention
and activating relevant domain knowledge (e.g., by providing knowledge gaps). The challenge for
designers would be to implement these instructional guidelines without losing the power of
attraction that games have.
The purpose of these guidelines is to enable learners to engage in cognitive processes that
contribute to learning. However, little is known about the types of cognitive processes that occur
during serious gaming. Therefore we recommend more research be carried out that extends the
understanding of effective and ineffective cognitive processes in learning with serious games.
For
example, cognitive theories consider the use of trail-and-error methods in learning how to
solve problems to generate ineffective cognitive processes. It would be valuable to see under
which conditions such ineffective cognitive processes occur. One of these conditions, the
game structure, was investigated by Pillay (2002) who observed that linear cause-and-effect
oriented games yielded a trial-and-error problem solving behavior in the game, whereas
adventure games encouraged more inferential and proactive thinking. Apart from the
implication for cognitive processes, the Pillay study also emphasizes the importance of the
structure of serious games.
3. The role of mitigating factors
The third recommendation is related to the lack of understanding on factors that mitigate the
effect of serious games on learning, and three factors in particular. The first factor pertains to the
gender of learners. Some researchers have reported notable differences in results between male
and female participants. For example, in the River City study Nelson (2007) found girls to be
more effective in the use of guidance and Feng et al. (2007) reported that on spatial cognition
female students benefitted more from action games than male students.
The second factor concerns training time. If it is true that players immerse themselves in
games and consequently spend more time on the task, then the question arises whether the higher
performance can be ascribed to the extra time spent on the task in the game or to the
characteristics of the games that support learning.
The last mediating factor to be discussed is age. One of the central findings in cognitive aging
research is that the efficiency of working memory deteriorates with aging. This may be particular
relevant for complex serious games. Elderly learners may have problems with discerning between
relevant and irrelevant information in the game or their processing speed can not keep up with the
progress in the game. Without instructional support, a serious game that may be effective for
young learners may be ineffective for elderly learners.
4. Understanding motivation(al) processes
The fourth recommendation concerns the assumed motivational impact of games. Apart from
theoretical accounts of game characteristics that motivate players to sustain playing a game, we
also need a better understanding of the psychological mechanisms underlying these motivation
processes. Of particular note for this type of study are those conducted by Ryan, Rigby and
Przybylski (2006). Drawing from self-determination theory, they hypothesized that perceived
autonomy (i.e., feeling uncontrolled when pursuing an activity) and competence (i.e., a need for
challenge and feelings of effectance) would enhance motivation to play games. In the studies
participants played games like Super Mario 64, Zelda and A Bugs Life. The results showed that
experiences of competence and autonomy while playing accounted for gaming motivation and
enjoyment. Another promising avenue of research is the relation between ‘flow’ and learning
from serious games. Flow has been described as such an extent of involvement in a task that
nothing else seems to matter (Garris et al., 2002). Although it is undeniable that players fully
engage in popular games and forget the real world around them, it is still unclear how such full
engagement relates to learning. With this focus research may begin to establish links between
game features, motivational processes, and learning outcomes.
5. Assessment of learning outcomes
The final recommendation concerns the validity of the learning outcomes, that is, did the
assessment test that was used really measure the learning outcome that was aimed at? Dede et al.
(2005) demonstrated a better command of cognitive skills for a game group when measured with
an evaluation letter to the mayor, but not with traditional test items. Most serious games are
situated in specific contexts that may yield learning outcomes that are contextualized as well.
Assessment methods that take the context of learning into account (e.g., an evaluation letter to the
mayor) may reveal differences in performance that would be undisclosed with traditional
assessment methods.
An additional argument for reconsidering the traditional assessment methods follows from the
results of Belanich et al. (2004) who found that items with visual information were better recalled
than written information. Video games are highly visual and may favor the acquisition of visually
encoded knowledge. In that case visually-oriented assessment may reveal learning of knowledge
that would probably not have been found with a text-based assessment method.
Another promising direction for assessment in serious games comes from a study by Day,
Arthur and Gettman (2001) who measured the learning of complex skills with the game Space
Fortress by assessing the knowledge structures that the players constructed. In knowledge
structures the information is mentally organized in concepts, the features that define them and the
relationships between the concepts. It appeared that the degree of similarity between the
knowledge structures of trainees and those of experts was correlated with complex skill
acquisition and a good predictor of skill retention and transfer. Altogether, future research on the
effectiveness of serious games should also consider other techniques to measure learning.
CONCLUSION
We gave an outline of the current practices in serious games research by reviewing 28 studies
with empirical data from the perspective of learning outcomes. We discerned cognitive, motor
skills, affective and communicative learning outcomes. In general, serious games seem to be
effective when it comes to cognitive learning outcomes. Serious games for training motor skills
and attitudinal change is promising. Finally, little recent substantiation was found for the
effectiveness on motivation and communicative learning outcomes. With respect to the
effectiveness of game features, especially, the implementation of the investigated instructional
guidelines did not improve learning. Although
the number of studies is too low to make definite
conclusions, the review provides an indication of the current practices. We believe that
serious games are promising, but that more research is required that should also consider the
alignment of learning outcomes and game type, the limited cognitive capacity, specific mitigating
factors (e.g., gender), motivational processes and new assessment methods.
AUTHORS NOTE
This research has been supported by the GATE project, funded by the Netherlands Organization
for Scientific Research (NWO) and the Netherlands ICT Research and Innovation Authority (ICT
Regie)
REFERENCES
Backlund, P., Engström, H., Johannesson, M., & Lebram, M. (2007). Games and traffic safety –
an experimental study in a game-based simulation environment. Proceedings of the International
Conference Information Visualization, 11, 908–916.
Baker, E. L., & Mayer, R. E. (1999). Computer-based assessment of problem solving. Computers
in Human Behavior, 15, 269–282.
Barab, S., Warren S., & Ingram-Goble, A. (2006, April). Academic play space: Designing games
for education. Paper presented at the meeting of the American Educational Research Association,
San Francisco, CA.
Beale, I. L., Kato, P. M., Marin-Bowling, V. M., Guthrie, N., & Cole, S. W. (2007). Improvement
in cancer-related knowledge following use of a psychoeducational video game for adolescents
and young adults with cancer. Journal of Adolescent Health, 41, 263–270.
Belanich, J., Sibley, D. E., & Orvis, K. L. (2004). Instructional characteristics and motivational
features of a PC-based game (Research Report No. 1822). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences.
Björk, S., Lundgren, S., & Holopainen, J. (2003). Game Design Patterns. In M. Copier and J.
Raessens (Eds.), Proceedings of the Level Up: Digital Games Research Conference. Utrecht, The
Netherlands.
Bloom, B. (1956). Taxonomy of educational objectives: The cognitive domain. New York:
Donald McKay.
Bouchard, S., Côte, S., St-Jacques, J., Robillard, G., & Renaud, P. (2006). Effectiveness of virtual
reality exposure in the treatment of arachnophobia using 3D games. Technology and Health Care,
14, 19–27.
Brannick, M. T., Prince, C., & Salas, E. (2005). Can PC-based systems enhance teamwork in the
cockpit? The International Journal of Aviation Psychology, 15, 173–187.
Castel, A. D., Pratt, J., & Drummond, E. (2005). The effects of action video game experience on
the time course of inhibition of return and the efficiency of visual search. Acta Psychologica, 119,
217–230.
Day, E. A., Arthur Jr., W., & Gettman, D. (2001). Knowledge structures and the acquisition of a
complex skill. Journal of Applied Psychology, 86, 1022–1033.
Dede, C., Clarke, J., Ketelhut, D. J., Nelson, B., & Bowman, C. (2005, April). Students’
motivation and learning of science in a multi-user virtual environment. Paper presented at the
meeting of the American Educational Research Association, Montréal, Quebec.
Enochsson, L., Isaksson, B., Tour, R., Kjellin, A., Hedman, L., Wredmark, T., & Tsai-Fellander,
L. (2004). Visuospatial skills and computer game experience influence the performance of virtual
endoscopy. Journal of Gastrointestinal Surgery, 8, 874–880.
Feng, J., Spence, I., & Pratt, J. (2007). Playing an action video game reduces gender differences
in spatial cognition. Psychological Science, 18, 850–855.
Fischer, P., Kubitzki, J., Guter, S., & Frey, D. (2007) Virtual driving and risk taking: Do racing
games increase risk-taking cognitions, affect, and behaviors? Journal of Experimental
Psychology: Applied, 13, 22–31.
Fletcher, J.D., & Tobias, S. (2008). What research has to say (thus far) about designing
computer games for learning. Paper presented at the American Educational Research
Association, New York, NY.
Gagné, R. M. (1977). The conditions of learning. New York: Holt, Rinehart & Winston.
Galimberti, C., Ignazi, S., Vercesi, P., & Riva, G. (2001). Communication and cooperation in
networked environments: An Experimental Analysis. CyberPsychology & Behavior, 4, 131–146.
Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and
practice model. Simulation & Gaming, 33, 441–467.
Grantcharov,T. P. , Bardram, L., Funch-Jensen, P., & Rosenberg, J. (2003). Impact of hand
dominance, gender, and experience with computer games on performance in virtual reality
laparoscopy. Surgical Endoscopy, 17, 1082–1085.
Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention.
Nature, 423, 534–537.
Green, C. S., & Bavelier, D. (2006). Effect of action video games on the spatial distribution of
visuospatial attention. Journal of Experimental Psychology: Human Perception and
Performance, 32, 1465–1478.
Ke, F., & Grabowski, B. (2007). Gameplaying for maths learning: cooperative or not? British
Journal of Educational Technology, 38, 249–259.
Kirriemuir, J., & McFarlane, A. (2004). Literature review in games and learning (Futurelab Series
Report 8). Retrieved March 17, 2008 from Futurelab Website
http://www.nestafuturelab.org/research/reviews/08_01.htm.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective
theories of learning outcomes to new methods of training evaluation. Journal of Applied
Psychology, 78, 311–328.
Krathwohl, D. R., Bloom, B. S., & Massai, B. B. (1964). Taxonomy of educational objectives:
The classification of educational goals. White Plains, NY: Longman.
Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction
in computer-supported collaborative learning environments: A review of the research. Computers
in Human Behavior, 19, 335–353.
Leemkuil, H. (2006). Is it all in the game? Learner support in an educational knowledge
management simulation game. Unpublished doctoral thesis. University of Twente, Enschede, The
Netherlands.
Leemkuil, H., de Jong, T., & Ootes, S. (2000). Review of educational use of games and
simulations (IST-1999-13078 Deliverable D1).University of Twente, Enschede, The
Netherlands.
Lim, C. P., Nonis, D., & Hedberg, J. (2006). Gaming in a 3D multiuser virtual environment:
engaging students in Science lessons. British Journal of Educational Technology, 37, 211–231.
Lindley, C.A. (2003). Game Taxonomies: A High Level Framework for Game Analysis and
Design. Retrieved March 17, 2008 from Gamasutra Website.
http://www.gamasutra.com/features/20031003/lindley_01.shtml.
Lindley, C.A. (2004). Narrative, game play, and alternative time structures for virtual
environments. In S. Göbel, U. Spierling, A. Hoffman, I. Iurgel, O. Schneider, J. Dechau & A.
Feix (Eds.), Lecture Notes in Computer Science: Vol. 3105. Technologies for Interactive Digital
Storytelling and Entertainment (pp. 183-194). Heidelberg: Springer Berlin.
Malone, T. (1981). Toward a theory of intrinsically motivating instruction. Cognitive Science, 4,
333–369.
Michael, D., & Chen, S. (2006). Serious games: Games that educate, train, and inform. Boston,
MA.: Thomson Course Technology.
Morris, C. S., Hancock, P. A., & Shirkey, E. C. (2004). Motivational effects of adding context
relevant stress in PC-based game training. Military Psychology, 16, 135–147.
Nelson, B. C. (2007). Exploring the use of individualized reflective guidance in an educational
multiuser virtual environment. Journal of Science Education and Technology, 16, 83–97.
Nova, N., Dillenbourg, P., Wehrle, T., Goslin, J., & Bourquin, Y. (2003). The impacts of
awareness tools on mutual modelling in a collaborative video-game. In J. Favela & D.
Decouchant (Eds.), Lecture Notes in Computer Science: Vol. 2806. Groupware Design,
Implementation, and Use (pp. 99–108). Heidelberg: Springer Berlin.
Nullmeyer, R. T., Spiker, V. A., Golas, K. C., Logan R. C., & Clemons, L. (2006, December).
The effectiveness of a PC based C-130 crew resource management aircrew device. Paper
presented at the meeting of the Interservice/Industry training, simulation, and Education
Conference, Orlando, FL.
Nunez, D. & Blake, E. (2003). A direct comparison of presence levels in text-based and graphics-
based virtual environments. Proceedings of the 2nd international Conference on Computer
Graphics, Virtual Reality, Visualisation and Interaction in Africa, 53-56.
O’Brien, K. S., & O’Hare, D. (2007). Situational awareness ability and cognitive skills training
in a complex real-world task. Ergonomics, 50, 1064–1091.
O’Neil, H. F., Wainess, R., & Baker, E. L. (2005). Classification of learning outcomes: Evidence
from the computer games literature. The Curriculum Journal, 16, 455–474.
Parchman, S. W., Ellis, J. A., Christinaz, D., & Vogel, M. (2000). An evaluation of three
computer-based instructional strategies in basic electricity and electronic. Military
Psychology, 12, 73–87.
Pillay, H. (2002). An investigation of cognitive processes engaged in by recreational
computer game players: An implication for skills of the future. Journal of Research on
Technology in Education, 34, 336–350.
Prensky, M. (2001). Digital Game-Based Learning. New York: McGraw-Hill
.
Ravaja, N., Salminen, M., Holopainen, J., Saari, T., Laarni, J., & Järvinen, A. (2004). Emotional
response patterns and sense of presence during video games: Potential criterion variables for
game design. Proceedings of the Third Nordic Conference on Human-Computer interaction,
339
–
347.
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A
self-determination theory approach. Motivation and Emotion, 30, 347–363.
Rosenberg, B. H., Landsittel, D., & Averch, T. D. (2005). Can video games be used to predict or
improve laparoscopic skills? Journal of Endourology, 19, 372–376.
Rosser, J. C., Lynch, P. J., Cuddihy, L., Gentile, D. A., Klonsky, J., & Merell, R. (2007). The
impact of video games on training surgeons in the 21
st
century. Archives of Surgery, 142, 181–
186.
Satyen, L., & Ohtsuka, K. (2001). Strategies to develop divided attention skills through video
game training. In D. Harris (Ed.), Engineering psychology and cognitive ergonomics (Vol. 6) (pp.
191 – 199). Aldershot, UK: Ashgate.
Seay, A. F., Jerome, W. J., Lee, K. S., & Kraut, R. E. (2004). Project massive: A study of online
gaming communities. Extended abstracts of the Conference on Human Factors in Computing
Systems (pp. 1421-1424), Vienna, Austria.
Squire, K., Barnett, B., Grant, J. M., & Higginbotham, T. (2004). Electromagnetism
Supercharged! Learning physics with digital simulation games. Proceedings of the international
conference on Learning sciences, 6, 513–520.
Tuzun, H., Yilmaz Soylu, M., Karakus, T., Inal Y., & Kizilkaya, G. (2008, March). The effects of
computer games on primary school students’ achievements and motivation in geography
learning. Paper presented at the meeting of the American Educational Research Association, New
York, New York.
Virvou, M., Katsionis, G., & Manos, K. (2005). Combining software games with education:
Evaluation of its educational effectiveness. Educational Technology & Society, 8(2), 64–65.
Vogel, J. J., Vogel, D.S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006).
Computer gaming and interactive simulations for learning: A meta-analysis. Journal of
Educational Computing Research, 34, 229–243.
Walshe, D. G., Lewis, E. J., Kim, S. I., O’Sullivan, K., & Wiederhold, B. K. (2003). Exploring
the use of computer games and virtual reality in exposure therapy for fear of driving following a
motor vehicle accident. CyberPsychology & Behavior, 6, 329–334.
Waxberg, X. L., Schwaitzberg, S. D., & Cao, C. G. L. (2005). Effects of videogame experience
on laparoscopic skill acquisition. Proceedings of the Human Factors and Ergonomics Society, 49,
1047–1051.
Wong, W. L., Shen, C., Nocera, L., Carriazo, E., Tang, F., Bugga, S., & Narayanan, H. (2007).
Serious video game effectiveness. Proceedings of the international conference on Advances in
computer entertainment technology, 49–55.
Wouters, P., Tabbers, H. K., & Paas, F. (2007). Interactivity in video-based models. Educational
Psychology Review, 27, 327–342.
Table 1. Classification of studies in the taxonomy of learning outcomes
Learning outcome Study Game Domain Results Effect
Cognition
Knowledge
Game vs. control group)
Dede et al. (2005) River City Biology Game > Text +
Squire et al. (2004) Supercharged Science Game > Guided inquiry +
Parchman et al. (2000) King’s Quest V Electronics Game < Computer-based +/–
Wong et al. (2007) Metalloman Biology Game and Hypertext > Text +
Game features
Belanich et al. (2004) America’s Army Military Relevant knowledge better recalled +
Wong et al. (2007) Metalloman Biology Interactivity has no effect –
Leemkuil (2006) KMQuest Economy Instructional support not effective –
Morris et al. (2004) DeltaForce Military Stress level has no effect on recall –
Virvou et al. (2004) VR Engage Geography Game mission more effective +
Beale et al. (2007) Re-Mission Medicine Game type has an effect +
Cognitive skills
Game vs. control group
Barab et al. (2006) QuestAtlantis Writing Game > Traditional +
Dede et al. (2005) River City Biology Game > Text +/
–
Nullmeyer et al. (2006) Unknown CRM Game > Control +/
–
Parchman et al. (2000) King’s Quest V Electronics Computer-based > Game = Text +/–
Ke et al. (2007) Astra Eagle Math Game > No game +
Game features
Leemkuil (2006) KMQuest Economy Instructional support not effective –
Nelson (2007) River City Biology Guidance not effective –
Morris et al. (2004) Delta Force Military High stress yields better mission success +
Feng et al. (2007) MoH, Balance Spatial Cogn. Action game yields better spatial cognition +
Motor skills
Game effect
Rosenberg et al. (2005) Top Spin Surgery Game experience has no effect –
Waxberg et al. (2005 Goldeneye Surgery Game experience has no effect –
Enochsson et al. (2004) Unknown Surgery Game experience yields better performance +
Grantcharov et al. (2003) Unknown Surgery Game experience yields better performance +
Rosser et al. (2007) Silent Scope Surgery Game experience yields better performance +
Learning outcome Study Game Domain Results Effect
Affective
Attitude
Game effect
Bouchard et al. (2006) Half-Life Phobia Reduction of fear of spiders +
Walshe et al. (2003) London Racer Phobia Reduction on fear of driving +
Ke et al. (2007) Astra Eagle Math Cooperative game > Competitive/ No game +
Game features
Backlund et al. (2007) Unknown Driving Safe driving behavior +
Fischer et al. (2007) Burnout/Fifa Driving Race gamers less cautious than non race gamers +
Motivation
Game vs. control group
Clarke et al. (2006) River City Biology Game > traditional +
Parchman et al. (2000) King’s Quest V Electronics Enhanced = Game > Drill-practice = Text +/–
Tuzun et al. (2008) Quest Atlantis Geography Intrinsic: Game > traditional +
Extrinsic: Game < Traditional
Communicative
Game vs. control group
Branninck et al. (2005) Asteroids CRM PC based simulator > Game + Exercises –
Nullmeyer et al. (2006) unknown CRM Game = Control –
Game features
Nova et al. (2003) Spaceminers Science task performance better with awareness tool +
Galimberti et al. (2001) DOOM II Maze Collaboration: Immersion = nonimmersion +/–
Note: + = an effect (positive or negative) is reported, – = no effect is reported, +/– = results are inconclusive
Table 2. Overview of results by game feature
Game feature Description Study Results Effect
Awareness tool Helping player
understanding
activities of other
players
Nova et al. (2003) An awareness tool
yields higher task
performance than no
awareness tool
+
Game task A specific task or
mission is involved
or not
Virvou et al. (2005)
Backlund et al.
(2007)
The task/mission has
a positive effect on
knowledge or attitude
+
Game type The specific game
type that is involved
(e.g., an action
game or not)
Feng et al. (2007)
Fischer et al. (2007)
Beale et al. (2007)
Game type causes
cognitive processes
to occur or not
+
Instructional
Guidelines
Advice/Guidance Advice or guidance
with or without
hints is offered
Leemkuil (2006)
Nelson (2007) Advice or guidance
have no effect
without additional
support
–
Assignments Extra assignments
(tasks) are
implemented or not
Leemkuil (2006) Extra assignments
have no effect –
Interaction Game allows
choices of the
player or not
Wong et al. (2007) Interaction has no
effect
–
Level of immersion Effect of head-
mounted display vs.
monitor
Galimberti et al.
(2001) No effect, except
immersion took more
time for the task
+/–
Level of stress A high or low level
of stress is brought
into the game
Morris et al. (2004) The level of stress
has a positive effect
on cognitive skills,
but no effect on
knowledge
+/–
Relevance The information is
relevant or not for
the game
Belanich et al.
(2004)
Relevant information
has a positive effect +
Note: + = an effect (positive or negative) is reported, – = no effect is reported,
+/– = results are inconclusive
Figure 1. A taxonomy of learning outcomes.
Learning outcomes
Cognitive
knowledge
- textual
- non-textual
skills
- problem solving
- decision making
- situational awareness
Motor skills
acquisition
compilation
Affective
attitude
motivation
Communicative
communicate
cooperate
negotiate
Figure 2. Cognitive and Affective Level of Complexity (CALC).
