Problem Solving and Game-Based Learning: Effects of Middle Grade Students' Hypothesis Testing Strategies on Learning Outcomes

Abstract

Targeted as a highly desired skill for contemporary work and life, problem solving is central to game-based learning research. In this study, middle grade students achieved significant learning gains from gameplay interactions that required solving a science mystery based on microbiology content. Student trace data results indicated that effective exploration and navigation of the hypothesis space within a science problem-solving task was predictive of student science content learning and in-game performance. Students who selected a higher proportion of appropriate hypotheses demonstrated greater learning gains and completed more in-game goals. Students providing correct explanations for hypothesis selection completed more in-game goals; however, providing the correct explanation for hypothesis selection did not account for greater learning gains. From the analysis, we concluded that hypothesis testing strategies play a central role in game-based learning environments that involve problem-solving tasks, thereby demonstrating strong connections to science content learning and in-game performance.

Full text

Problem Solving and Game-Based Learning
!
1
Problem Solving and Game-Based Learning: Effects of Middle Grade Students’
Hypothesis Testing Strategies on Learning Outcomes
Hiller A. Spires, Jonathan P. Rowe, Bradford W. Mott, and James C. Lester
North Carolina State University
Author Note
Hiller A. Spires, Friday Institute for Educational Innovation, North Carolina State
University; Jonathan P. Rowe, Bradford W. Mott, and James C. Lester, Department of
Computer Science, North Carolina State University
This research was supported by the National Science Foundation under Grants
REC-0632450 and DRL-0822200. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the National Science Foundation.
Correspondence concerning this article should be addressed to Hiller A. Spires,
Friday Institute for Educational Innovation, North Carolina State University, Raleigh, NC
27695. Email: hiller_spires@ncsu.edu

Problem Solving and Game-Based Learning
!
2
Abstract
Targeted as a highly desired skill for contemporary work and life, problem
solving is central to game-based learning research. In this study, middle grade students
achieved significant learning gains from gameplay interactions that required solving a
science mystery based on microbiology content. Student trace data results indicated that
effective exploration and navigation of the hypothesis space within a problem-solving
task was predictive of student learning and in-game performance. Students who selected
a higher proportion of inappropriate hypotheses demonstrated smaller learning gains and
completed fewer in-game goals. Although there was no relationship observed between
providing correct explanations for hypothesis selections and learning gains, students
providing incorrect explanations completed fewer goals within the game. Finally, there
was no significant gender effect observed on the relationship between hypothesis testing
strategies and learning or in-game performance. Hypothesis testing strategies play a
central role in narrative-centered learning environments, demonstrating their connections
to learning gains and problem solving in gameplay.
Keywords: game-based learning, problem solving, hypothesis testing strategies,
narrative-centered learning

Problem Solving and Game-Based Learning
!
3
Problem Solving and Game-Based Learning: Effects of Middle Grade Students’
Hypothesis Testing Strategies on Learning Outcomes
In 2006, the educational research community received a strong mandate from the
National Summit on Educational Games to ramp up empirical investigations targeting
how and under what conditions games can be used to maximize learning potential. One
of the assumptions that members of the Summit made was that students “acquire new
knowledge and complex skills from game play, suggesting gaming could help address
one of the nation’s most pressing needs–strengthening our system of education and
preparing workers for 21st century jobs” (Federation of American Scientists, 2006, p. 3).
Earlier work by Levy and Murnane (2004) supports this assumption. They conclude that
the nation’s challenge is to prepare youth for the high-wage/high-skilled jobs that are
rapidly growing in number—jobs that involve expert problem solving skills and complex
communication. Games are heralded as important tools for teaching an array of 21st
century skills because: 1) they accommodate various learning styles and promote a
complex decision-making context (Squire, 2006); 2) skill sets and dispositions embedded
in well-designed games are a good match for the contemporary, technology-rich worlds
students inhabit (Gee, 2003; Spires, 2008); and 3) games have the potential to promote
the 21st century skills recognized as critical for all citizens (NRC, 2010). Increasingly,
educators, private industry as well as policy makers agree that the bottom line for success
in contemporary life and work is the ability “to learn rapidly and efficiently and to go into
almost any situation and figure out what has to be learned” (Morrison, 2001). Games can
be viewed as potential tools for learning since they can simulate real-world complexity

Problem Solving and Game-Based Learning
!
4
and fast-paced processing in ways that traditional school learning scenarios cannot
approximate.
Empirical support for how games can improve learning outcomes for academic
content in schools has been slow to emerge (Hayes, 2005). Mayer & Johnson (2010)
conducted a recent review where they synthesized learning outcome research in three
categories: “cognitive consequences, media comparisons, and value added” (p. 246).
They concluded that the value added research, (i.e., when researchers ask which features
add value in terms of educational effectiveness of the game) is most important when the
goal is academic learning. Their research synthesis, providing a much needed utilitarian
analysis of game effectiveness, is critical to providing definition and articulation of
learning outcomes, as the field of game-based learning continues to evolve.
Historically, games have been associated with play, but in light of low testing
scores and increasing rates of high school dropouts, many are looking to games for
educational benefits in schools. Pelletier (2009) has challenged educators to resist the
urge to frame games as a way to salvage education by claiming that the value of games
should be re-thought in terms of “the situated signification of ‘game’ rather than games
causing learning” (p. 83). Klopfer, Osterweil, and Salen in Moving Games Forward:
Obstacles, Opportunities, & Openness (2009), on the other hand, provide a conceptual
path for people and organizations interested in fostering the development of games for
learning purposes. Bypassing the mutual exclusion argument, they make “a case for
learning games grounded in the principles of good fun and good learning” (p. 1) and
devote their efforts to motivating and informing educators and researchers who want to
constructively participate, as creators as well as consumers, in the gaming domain.

Problem Solving and Game-Based Learning
!
5
One of the most promising goals for games is to foster problem-solving skills
related to science content. In contrast to textbook treatments of problem solving as
isolated investigations, games can engage students in dimensional problem solving
related to scientific inquiry. For example, Chris Dede and his colleagues have used multi-
user virtual environments (MUVEs) as a pedagogical apparatus to teach science concepts
to middle grade students. They argue that rather than learning by listening to lectures or
reading textbooks, students can learn science by exploring and solving problems in
realistic environments. Their early MUVE, River City, is set in a historical town in the
late 1800's, where residents are becoming ill. The students take on the role of 21st century
scientists who travel back in time in order to help the mayor identify the cause of the
illness. Kettlehut (2007) found that embedding science inquiry curricula in River City
could act as a catalyst for change in middle grade students’ self-efficacy and learning
processes. Nelson (2007) found differences in overall learning outcomes by gender and in
patterns of guidance use by boys and girls, with girls outperforming boys across a
spectrum of guidance system use. Dede’s latest project, EcoMUVE is designed as a
collaborative, inquiry-based, simulated ecosystem experience to support learners
developing an understanding of complex causality (Metcalf, Dede, Grotzer, &
Kamarainen, 2010).
Similar to River City in terms of a problem solving focus, CRYSTAL ISLAND is an
example of an academic innovation that targets science education for 8th grade middle
school students. Taking their cues from Bruner (1990, p. 35), who observed that the way
people organize their experience and knowledge with the social world “is narrative rather
than conceptual,” the CRYSTAL ISLAND designers devised a narrative centered learning

Problem Solving and Game-Based Learning
!
6
environment for students to explore concepts related to microbiology (Mott & Lester,
2006).
The Case of CRYSTAL ISLAND
CRYSTAL ISLAND (See Figure 1) is a narrative-centered learning environment built
on Valve Software’s SourceTM engine, the 3D game platform for Half-Life 2. The
curriculum underlying CRYSTAL ISLAND’s mystery narrative is aligned with North
Carolina’s standard course of study for eighth-grade microbiology. Students play the role
of the protagonist, Alyx, who is attempting to discover the identity and source of an
infectious disease plaguing a research station. Several of the team members have fallen
gravely ill, and it is the student’s task to discover the nature and cause of the outbreak.
Insert Figure 1 here
CRYSTAL ISLAND’s narrative takes place in a small research camp situated on a
recently discovered tropical island. As students explore the camp, they investigate the
island’s spreading illness by forming questions, generating hypotheses, collecting data,
and testing hypotheses. Throughout their investigations, students interact with non-player
characters offering clues and relevant microbiology facts via multimodal “dialogues”
delivered by characters through student menu choices and characters’ spoken language.
The dialogues’ content is supplemented with virtual books, posters, and other resources
encountered in several of the camp’s locations. As students gather useful information,
they have access to a personal digital assistant to take and review notes, consult a

Problem Solving and Game-Based Learning
!
7
microbiology field manual, communicate with characters, and report progress in solving
the mystery. To solve the mystery, students complete a diagnostic worksheet to manage
their working hypotheses and record findings about patients’ symptoms and medical
history, as well as any findings from tests conducted in the camp’s laboratory. Once a
student enters a hypothesized diagnosis, cause of illness, and treatment plan into the
diagnosis worksheet, the findings are submitted to the camp nurse for review and possible
revision.
Research Support for CRYSTAL ISLAND
Over the past several years, CRYSTAL ISLAND has been the subject of multiple
lines of investigation, including explorations of student learning and engagement in
narrative-centered learning environments, as well as the development of advanced game-
based learning technologies. Previous studies have investigated the impact of narrative on
learning and presence (McQuiggan, Rowe, Lee, & Lester, 2008), as well as the
relationship between learning and engagement in narrative-centered learning
environments (Rowe, Shores, Mott, & Lester, 2010b). Recent work has explored
individual differences in students’ gameplay behaviors and learning (Rowe, Shores, Mott,
& Lester, 2010a), student note-taking practices (McQuiggan, Goth, Ha, Rowe, & Lester,
2008), and off-task behavior (Rowe, McQuiggan, Robison, & Lester, 2009). An
additional line of investigation has pursued the incorporation of intelligent tutoring
system technologies into narrative-centered learning environments. This has included
work devising and evaluating empathetic virtual agents (McQuiggan, Rowe, & Lester,
2008; Robison, McQuiggan, & Lester, 2009), predictive models of student affect and
self-efficacy (McQuiggan, Mott, & Lester, 2008), models for interactive narrative

Problem Solving and Game-Based Learning
!
8
generation (Mott and Lester, 2006a; Mott and Lester, 2006b) and models for automatic
recognition of student goals (Mott, Lee, & Lester, 2006).
Theoretical Framework: Blending Narrative, Social, and Cognitive Assumptions for
Learning Within CRYSTAL ISLAND’s Gameplay
Three theories provide the foundation for the development of CRYSTAL ISLAND
and the specific study at hand: narrative-centered learning, activity theory, and cognitive
load theory. These theories are discussed briefly in relationship to the game world of
CRYSTAL ISLAND and specifically the problem-solving task learners engage in to solve
the mystery. We propose a blending of these three theories to illustrate how within the
problem-solving task learners simultaneously engage the narrative, navigate mediating
artifacts, and use tools to offset cognitive load during processing as they test their
hypotheses about what is causing people to become sick.
Narrative Centered Learning
Mott et al. (1999) introduced the theory of narrative-centered learning to game-
based learning environments and virtual worlds by building on Gerrig’s (1993) two
principles of cognitive processes in narrative comprehension. First, readers are
transported, i.e., they are somehow taken to another place and time in a manner that is so
compelling it seems real. Second, they perform the narrative. Simulating actors in a play,
readers actively draw inferences and experience emotions prompted from interactions
with the narrative text, or what Deslandes (2004) refers to as emoting by proxy. In the
same way that good readers employ a particular stance in order to achieve their reading

Problem Solving and Game-Based Learning
!
9
purpose and goals, a game player may employ a stance, not unlike those in the efferent-
aesthetic continuum in Rosenblatt’s transactional theory (2004), in order to “read” and
participate successfully in the game (Spires, Turner, Rowe, Mott & Lester, 2010). With
the recent explosion in game-based learning environments and virtual world creation,
narrative is being appropriated as a dynamic tool for exploring the structure and
processes of game-based learning related to engagement and meaning creation (Mott &
Lester, 2006).
Activity Theory
Activity theory suggests that learning is shaped by practice, with people and
artifacts mediating the learner’s relationship with reality (Vygotsky, 1978; Cole &
Engeström, 1993; Kaptelinin & Nardi, 2006). Derived from Vygotsky’s cultural-
historical approach to learning and applied to distributed learning, activity theory
purports learning as “expanding involvement,” (social as well as intellectual) with other
people and cultural tools. Meaning and subsequent learning are produced in the
enactment of activity with other people and things, rather than being something confined
to individual mental processes alone. Activity theory’s meditational triangle (Engeström,
1987; Cole & Engeström, 1993) provides a theoretical lens for gameplay, specifically the
tools available to complete the activity and the relationships within the game that create
the interactivity leading to action and subsequent meaning. The basic triangle consists of
the following elements: subject, object, and mediating tools--all in the service of learning
outcomes. Activity theory and the meditational triangle can be used to evaluate students’
problem solving paths and specifically the architecture of how and under what conditions
students test hypotheses to solve the mystery.

Problem Solving and Game-Based Learning
!
10
Cognitive Load Theory
Building on Miller’s theory (1956) showing that short-term memory is limited in
the number of elements it can handle concomitantly, Sweller (1998) proposed cognitive
load theory, which treats schemata as the cognitive structures that make up an
individual's knowledge base. The contents of long-term memory are "sophisticated
structures that permit us to perceive, think, and solve problems," (p. 20) rather than
isolated facts (Sweller, 2005). These structures, referred to as schemata, are what permit
the learner to treat multiple elements as a single element. Mayer (2005) related cognitive
load (i.e., extraneous processing) to multimedia learning in games by illustrating that
extraneous processing does not enhance the desired instructional objective, and in fact
competes with cognitive processing resources.
Blended Theoretical Assumptions for Problem Solving Within CRYSTAL ISLAND
Following is a description of selected pedagogical supports for problem solving in
CRYSTAL ISLAND. The interplay of the three theories outlined above is illustrated in
relationship to the problem-solving task, and specifically the potential hypothesis testing
paths employed by learners. Initially, in terms of narrative-centered learning, the story
line creates a sense of urgency by drawing the learner into the problem as explained by
the character Kim, the nurse, who says: “Thank goodness, you're here. I'm Kim, the
camp nurse. People on the island are getting sick, and we don’t know why. Please, can
you help us?” This interaction quickly prompts the student to a level of action.
Throughout the game Kim serves the function of increasing intensity within the narrative
as she directs the learner to possible avenues to gather pertinent information. A dialogue
branch listing the sub-activities necessary for completing the task includes: “You can

Problem Solving and Game-Based Learning
!
11
gather clues by talking to other team members, exploring the camp, and using the
laboratory’s testing equipment. Complete a diagnosis worksheet, then come talk to me.”
At this stage, students are free to engage the resources within the game: they may
choose to speak with other characters, explore the island and read books and posters, or
begin conducting tests in the laboratory. Activity theory is enacted in the problem space
as students become involved in shared tasks with cultural tools that potentially can
produce learning, i.e., in this context, microbiology content. The activity system
produced an opportunity for learning through the provision of additional resources;
Vygotsky (1978) called these types of opportunities “zones of proximal development,”
which he defined as the difference between what one could do alone and what one could
do with assistance. Additional pedagogical supports for problem solving in the character
dialogues are provided by Elise, the lab technician character. Activity theory also comes
in to play when something unanticipated happens within the problem space. For example,
some students take a scatter shot approach to testing hypotheses about what is causing the
illness rather than a strategic approach; this choice is a result of social and cultural
practices related to personal learning trajectories that students bring to the problem
solving context.
At the next stage of the game, students are encouraged to use the lab’s testing
equipment to investigate objects for contaminants, and to speak with Elise if any
contaminated objects are found. Hypothesis options include: 1) I believe this object is
contaminated with Pathogens; 2) I believe this object is contaminated with Mutagens; 3) I
believe this object is contaminated with Carcinogens. (See options in Figure 2).
Insert Figure 2 here

Problem Solving and Game-Based Learning
!
12
If the students have exhausted all five of the initial ‘tests’ on the equipment, they
received a phone call from the camp nurse. The nurse offers an opportunity to earn back
more tests, but requires that they answer microbiology quiz questions in order to
demonstrate that they are able to make informed decisions about which tests to run in the
future. Students complete the diagnosis worksheet, which scaffolds problem solving by
explicitly outlining four sub-components of the game’s overall task: recorded symptoms,
laboratory testing results, beliefs about various candidate diagnoses, and a final diagnosis.
The worksheet serves as a tool to offset cognitive load as the students are making
decisions about which objects to test as part of their hypothesis strategy. In essence the
tool was designed to help reduce working memory load and increase schema construction
and automation. As a result of employing their hypothesis testing strategy, ideally
students should understand that the team members are sick from a pathogen, and more
specifically are suffering from a bacterial infection that was spread via the milk. Next,
they must determine a specific diagnosis and treatment plan, again using the worksheet as
a tool to assist with cognitive load processing.
Problem solving in CRYSTAL ISLAND relies on students’ abilities to apply their
knowledge about science and microbiology concepts (e.g., definition of pathogen, sizes
and structures of bacteria and viruses, treatments, scientific method, symptoms of various
diseases, etc.). By blending three theories—narrative, activity, and cognitive load—the
game design promotes engagement, understanding, motivation, and interaction, by
immersing players in a complex, feedback-rich problem space.

Problem Solving and Game-Based Learning
!
13
Research Context
Since problem solving is a targeted 21st century skill that is highly sought after in
contemporary society, we were curious how well students who played CRYSTAL ISLAND
could problem solve within the game. To eliminate potential confounds and to ensure a
clear analysis with the trace data, we focused on one aspect of the problem-solving
process, namely, students’ strategies for hypothesis testing. Because of the level of
complex processing required, we were specifically interested in the relationship between
students’ strategies for testing food items for contamination and their learning outcomes.
The study addressed the following question: What is the relationship between students’
hypothesis testing strategies and their learning gains in a game environment?
Method
Participants and Design
Participants who interacted with the CRYSTAL ISLAND environment consisted of
153 eighth grade students ranging in age from 12 to 15 (M = 13.3, SD = 0.48). Since
eight of the participants were eliminated due to incomplete data, and eight additional
students were eliminated because they had prior experience with CRYSTAL ISLAND, the
number used for data analysis was 137 (77 males and 60 females). Approximately 3% of
the participants were American Indian or Alaska Native, 2% were Asian, 32% were
African American, 13% were Hispanic or Latino, and 50% were White. The study was
conducted prior to students’ exposure to the microbiology curriculum unit of the North
Carolina state standard course of study in their regular classes. The study took place in a

Problem Solving and Game-Based Learning
!
14
public magnet school, in which 35% of the students received free or reduced lunches; the
school is located in a large urban district in NC.
Procedure
Students had completed the majority of pre-test materials one week prior to the
intervention. Participants were randomly assigned to treatments before they entered the
intervention classroom. Each participant was seated in front of a laptop computer with
headphones as they participated in a 3-part introductory session that lasted twenty
minutes. First, students participated in a brief demonstration session, which included
general details about the CRYSTAL ISLAND mystery and game controls. The demonstration
was conducted by one of the researchers. Second, students completed the remaining pre-
test materials. Third, students were provided with several CRYSTAL ISLAND
supplementary documents that could be referred to during gameplay. These materials
consisted of a CRYSTAL ISLAND backstory and task description, a character handout, a
map of the island, and an explanation of the game’s controls.
After the introductory session was completed, participants were given sixty
minutes to play the game and solve the mystery. Solving the mystery consisted of several
objectives, including the following: learning about pathogens, viruses, and bacteria;
compiling the symptoms and recent history of the sick researchers; recording details
about diseases believed to be afflicting the team members; and formulating and testing a
variety of hypotheses concerning sources for the disease. The final task was to report the
solution to the mystery (which included the cause, source, and treatment) to the camp
nurse. After the designated amount of time had elapsed (60 minutes) or the students had
completed their interaction, participants were instructed to move on to the post-

Problem Solving and Game-Based Learning
!
15
intervention phase where they completed assessments that included multiple-choice
content questions. The total duration of the introduction, gameplay, and post-test sessions
did not exceed 120 minutes. Students who finished before the allotted time were directed
to complete a related work sheet; all students left the testing area at the same time.
Dependent Measures
Multiple-Choice Content Questions. The pre- and post-intervention content test
consisted of 16 questions designed by an interdisciplinary team of researchers and
curriculum specialists. Two eighth-grade science teachers critiqued the content test to
establish content validity. There were 8 factual questions that were designed to be direct
and literal in nature and 8 application questions that required an application of knowledge
to a situation.
In-Game Performance. A significant feature of the research is the analysis of
game-based performance traces to measure students’ behavioral engagement. Using
automated logging facilities students’ problem-solving activities were monitored during
gameplay and logged to a database for post-hoc analysis. The logging facility recorded
student actions at a level of granularity that provided detailed representations of all
navigation and artifact manipulation behaviors within the learning environment at
millisecond intervals. The present study examined one aspect of these traces, the number
of goals completed. To complete CRYSTAL ISLAND, participants had to complete eleven
goals; however, not all students completed all the goals in the 60 minutes allotted.
Therefore, the number of goals completed could range from 0 to 11.

Problem Solving and Game-Based Learning
!
16
Results
Several analyses were conducted in order to investigate relationships between
students’ hypothesis testing strategies, problem solving, and learning. The results of a
one-tailed paired samples t-test showed that students achieved significant learning gains
from their interactions with CRYSTAL ISLAND, (t(136) = 9.34, p < .0001). Students
answered an average of 2.26 more questions correctly on the post-test (M = 8.60) than the
pre-test (M = 6.34). The additional analyses are organized in three categories and
demonstrate the relationship between: 1) hypothesis testing strategies and learning gains,
2) hypothesis testing strategies and in-game performance, and 3) gender and learning
gains and in-game performance.
Results for Hypothesis Testing Strategies and Learning Gains
The association between in-game problem solving and learning on the content test
was investigated. In-game problem solving was measured in terms of the number of in-
game goals students completed during interactions with CRYSTAL ISLAND. Multiple
regression analysis was performed in order to determine the association between number
of goals completed and learning outcomes. The post-test score was used as the dependent
variable and both pre-test score and the number of goals completed were treated as
independent variables. The model explained a significant proportion of the variance in
post-test scores, F(2, 134) = 33.17, p < .0001. R2 for the model was .33, and adjusted R2
was .32. Both pre-test score (
β
= .323, p < .0001) and number of goals completed (
β
=
.426, p < .0001) were observed to be significantly predictive of post-test score.

Problem Solving and Game-Based Learning
!
17
Insert Table 1 here.
A regression analysis was conducted in order to investigate the association
between students’ hypothesis selection practices and learning. Students conducted tests in
the laboratory using a virtual computer terminal shown in Figure 2. Before each test,
students would determine whether to test an object for pathogens, mutagens, or
carcinogens, and select a short justification for the test they were about to conduct. Early
in the mystery, students were expected to infer that the sick team members were suffering
from some form of pathogen that had contaminated the camp’s food supply. One measure
of students’ ability to navigate the space of hypotheses in CRYSTAL ISLAND is the
proportion of laboratory tests conducted that hypothesize an item is contaminated with
mutagens or carcinogens rather than pathogens. Conducting laboratory tests to investigate
mutagens or carcinogens on objects represents a portion of the hypothesis space that
students should be able to eliminate before conducting any tests, and therefore an
inefficient navigation of the hypothesis space. The proportion of inappropriate
hypotheses was calculated for each student by counting the number of tests conducted for
mutagens and carcinogens, divided by the total number of tests conducted. Multiple
regression analysis was performed with post-test score as the dependent variable, and
pre-test score and proportion of inappropriate hypotheses as independent variables. Both
variables explained a significant proportion of the variance in post-test scores, R2 = .20,
F(2, 124) = 15.93, p < .0001. The adjusted R2 was .19. Pre-test score had a significant
positive association (
β
= .382, p < .0001) with post-test score, whereas proportion of
erroneous hypotheses exhibited a significant negative association (
β
= -.209, p < .01)

Problem Solving and Game-Based Learning
!
18
with post-test score. The full regression model is shown in Table 1. The finding indicates
that students who tested a higher proportion of inappropriate hypotheses exhibited
reduced learning gains.
A similar analysis was conducted to investigate the number of incorrect
explanations provided for laboratory tests. Students could justify a test by selecting one
of five possible explanations: the sick team members recently ate the item, the sick team
members recently drank the item, the sick team members recently touched the item, the
item usually carries disease, or the item looks dirty (see Figure 2). The only explanations
that were consistent with the virtual characters’ dialogues were the first two; the latter
three were considered incorrect for the scenario. The proportion of incorrect
explanations was calculated for each student by counting the number of tests conducted
with incorrect explanations, divided by the total number of tests conducted. Multiple
regression analysis was performed with post-test score as the dependent variable, and
pre-test score and proportion of incorrect explanations as independent variables. While
the model explained a significant proportion of the variance in post-test score, R2 = .16,
F(2, 124) = 12.02, p < .0001, only pre-test score was found to be a significant predictor
of post-test score (
β
= .399, p < .0001). Proportion of incorrect explanations was not a
significant predictor of post-test (
β
= -.038, p > .60).
Results for Hypothesis Testing Strategies and In-Game Performance
Analyses were conducted to investigate the association between hypothesis
testing strategies and in-game performance. A multiple regression analysis was
performed with number of in-game goals completed as the dependent variable, and pre-

Problem Solving and Game-Based Learning
!
19
test score and proportion of inappropriate hypotheses as independent variables. The
model significantly predicted number of goals completed, F(2, 124) = 8.17, p < .001. R2
for the model was .12, and the adjusted R2 was .10. Investigating each of the individual
predictors, proportion of inappropriate hypotheses had a significant negative association
with number of goals completed, (
β
= -.292, p < .001). Pre-test score had a marginally
significant positive association with number of goals completed, (
β
= .151, p < .08).
Multiple regression analysis was performed to investigate the association between
proportion of incorrect explanations and in-game performance. The dependent variable
was number of in-game goals completed, and the independent variables were pre-test
score and proportion of incorrect explanations. The model significantly predicted number
of goals completed, F(2, 124) = 4.12, p < .02. R2 for the model was .06, and the adjusted
R2 was .05. Both independent variables were significant predictors of number of goals
completed. Pre-test score had a marginally significant positive association with the
dependent variable, (
β
= .169, p < .06), whereas proportion of incorrect explanations had
a significant negative association, (
β
= -.175, p < .05).
Results for Gender
Additional analyses were conducted to investigate whether there was an effect of
gender on learning or in-game performance. A two-tailed t-test was performed to
investigate the association between gender and post-test score. The analysis failed to
observe a significant effect of gender, t(135) = .05, p > .95. A two-tailed t-test was also
performed to investigate the association between gender and number of in-game goals
completed. A marginal effect of gender on number of goals completed was observed,

Problem Solving and Game-Based Learning
!
20
t(135) = 1.81, p < .08. On average, males (M = 9.47) accomplished a slightly greater
number of goals than females (M = 8.87).
Gender was added to each of the regression models investigating hypothesis
testing strategies and learning gains, as well as the models investigating hypothesis
testing strategies and in-game problem solving. However, gender was not observed to
contribute significantly to any of the four models. This suggests that gender may not be a
mediating factor between hypothesis testing behavior and learning or in-game
performance.
Discussion
Interpretation and Design Implications
The study addressed the question: What is the relationship between students’
hypothesis testing strategies and their learning gains in a game environment? Given that,
overall, the intervention yielded learning gains as measured by post-test scores
controlling for pre-test scores, it is important to observe that in-game performance is
predictive of learning gains. The findings of the study revealed three families of
relationships:
1. Hypothesis Testing Strategies and Learning:
• Hypothesis Selection: Students who selected a higher proportion of
inappropriate hypotheses during gameplay exhibited smaller learning gains.
• Explanation: Students providing incorrect explanations for laboratory tests
was not observed to have a significant association with their posttest
performance.

Problem Solving and Game-Based Learning
!
21
2. Hypothesis Testing Strategies and In-game Performance:
• Hypothesis Selection: Student selection of inappropriate hypotheses had a
strong negative association with their goal achievement in the game.
• Explanation: Students providing incorrect explanations for laboratory tests
had a negative association with their goal achievement in the game.
3. Gender Effects
• Post-Test Scores: There was no observed effect of gender on post-test scores.
• In-Game Performance: Gender had only a marginal effect on in-game
performance.
• Gender as a Mediating Factor: Gender was not observed to be a mediating
factor between hypothesis testing strategies and learning, and gender was not
observed to be a mediating factor between in-game performance and learning.
Hypothesis testing strategies and learning findings. Selection of inappropriate
hypotheses is associated with students’ ability to apply content knowledge. Specifically,
hypothesis selection is associated with students’ ability to distinguish between the
characteristics of pathogens, mutagens and carcinogens. If students know the
characteristics of pathogens, mutagens and carcinogens, and have inferred that the
spreading illness on CRYSTAL ISLAND stems from a pathogen, then they should be able to
infer that they need only conduct laboratory tests for pathogen-based contaminants. This
inference requires precisely the type of content knowledge that is assessed by the post-

Problem Solving and Game-Based Learning
!
22
test. It is plausible that students who have the ability to apply their content knowledge to
effectively select hypotheses during laboratory tests have a stronger mastery of the
relevant content than students who are unable to apply content knowledge to effectively
select hypotheses during laboratory tests. It is therefore likely that students who are
unable to effectively select hypotheses will exhibit decreased performance on the post-
test.
Providing incorrect explanations is associated with students’ ability to apply
knowledge about the scenario. Specifically, providing explanations for laboratory tests is
associated with students’ ability to apply knowledge about the virtual characters’ prior
histories and eating habits. Students need only conduct tests on objects from which the
virtual characters recently ate or drank. None of the characters specifically report
touching objects, and explanations about objects looking dirty or objects often carrying
diseases are weak. Knowledge about the virtual characters’ prior histories is distinct from
the content knowledge that is assessed by the post-test. One would not expect to observe
that students’ ability to apply such knowledge would transfer to the post-test and yield
increased learning gains. Therefore, it is likely that providing incorrect explanations
would have no effect on post-test performance.
The activity system within the game offered an opportunity for learning through
the provision of additional resources; specifically, reading books and posters and
interacting with knowledgeable characters using multiple modalities was available for
students to acquire and review the necessary microbiology concepts. These embedded
supports were designed to induce “zones of proximal development,” so that the students
could process the microbiology content with appropriate assistance. The level of support

Problem Solving and Game-Based Learning
!
23
and the incentives for taking advantage of the supports possibly were not adequate for the
students to navigate successfully the problem solving space.
Hypothesis testing strategies and in-game problem-solving findings. Students
interacted with CRYSTAL ISLAND for a fixed amount of time (approximately sixty
minutes). Students who selected greater proportions of inappropriate hypotheses and
provided greater proportions of incorrect explanations were likely to be less efficient at
identifying the source of CRYSTAL ISLAND’s spreading illness. Identifying the source of
CRYSTAL ISLAND’S illness required students to test a specific food object for pathogens
that the sick team members recently consumed. This was the only laboratory test that
would yield a positive result for contaminants. Students who practiced less effective
hypothesis testing strategies, and were therefore less efficient at identifying the source of
the spreading illness, would have less time available to complete other goals in the game
(e.g., goals related to reporting their final diagnosis). Therefore, it is likely that less
efficient hypothesis testing strategies would be associated with a decreased numbers of
goals completed in the game.
An important element in game design centers on creating an appropriate balance
between discovery elements and enough embedded supports to facilitate cognitive
processing of the content. As Mayer (2005) has noted extraneous processing competes
with cognitive processing resources and can be a liability when a learner is acquiring new
content material. Problem solving in CRYSTAL ISLAND relied on students' ability to apply
their knowledge about science and microbiology concepts (e.g., definition of pathogen,
sizes and structures of bacteria and viruses, treatments, scientific method, symptoms of

Problem Solving and Game-Based Learning
!
24
various diseases). The narrative format and accompanying features may have contributed
to learner distractions by competing for valuable processing time.
Gender findings. The effect of gender on in-game problem solving may actually
be better explained by students’ prior game-playing experience. It is likely that female
students had less prior game-playing experience in general, as well as less experience
with the specific game genre exemplified by CRYSTAL ISLAND (first person role-playing
game). It is also plausible that students with less prior game-playing experience would be
less adept at using the game’s controls, navigating the game environment, and
accomplishing in-game goals because they have not yet developed mature game-playing
schemata. Therefore, the effect of gender on number of in-game goals accomplished may
primarily be a function of game-playing experience; gender is simply a mediating
variable in the relationship.
Because computer game playing could be linked to computer literacy, and the
assumption that computer literacy is an important 21st century skill, the gender imbalance
in computer game playing has been a research focus. Gros (2007) observed that gender
differences do not necessarily influence interest in games, but do influence different lines
of play or preferences. For example, when playing The Sims, girls devoted more time to
the decoration of the house and to the determination of the physical aspects of people;
boys initiated the game more quickly. In creating a critical discourse about games,
obviously it is important to encourage innovative and alternative images of men and
women within gameplay that do not reinforce gender stereotypes.

Problem Solving and Game-Based Learning
!
25
Limitations of the Research
There are several limitations to this study. First, there are limitations to using
multiple choice response items to measure complex cognition and inquiry processes
within an environment like CRYSTAL ISLAND. New models for assessment are emerging
and we are interested in adapting more authentic assessments to the CRYSTAL ISLAND
experience in future studies. As Shaffer et al. (2010) assert, “Assessments of digital
learning need to focus on performance in context rather than on tests of abstracted and
isolated skills and knowledge” (p. 34). The capacity to use trace data for analysis
provides future opportunities to use evidence-centered design, in which there is alignment
between learning theory and assessment method. Additionally, transfer measures will
assess how well students can apply the information learned in the game to a new context.
Second, while the game is a narrative-centered learning environment and key features of
narrative are embedded in the pedagogical apparatus, the game play of CRYSTAL ISLAND
does not approximate the action and visual engagement offered by high-end commercial
games. Students are generally accustomed to action and visual stimulation when playing
games and can be disappointed when academic games do not mimic the level of
engagement and entertainment for which they are accustomed. Third, the study was
conducted as an experimental intervention that by and large took place outside of the
instructional context of the student’s class. The curriculum content was aligned with the
North Carolina Standard Course of Study for eighth grade science and was vetted by
science teachers who are part of the project; however, due to logistical issues, students
were not able to play the game at the appropriate juncture as a supplement to enhance
their classroom instruction focused on microbiology. The timing of the study may have

Problem Solving and Game-Based Learning
!
26
potentially impacted the students’ motivation to engage with the academic content since
there was no apparent connection at the time of the intervention between the game and
school requirements.
Conclusion
Results indicated that the effective exploration and navigation of the hypothesis
space in a problem-solving task was predictive of student learning mediated by individual
differences and in-game performance. Specifically, students who selected a higher
proportion of inappropriate hypotheses demonstrated smaller learning gains on the post-
test and completed fewer in-game goals. There was no relationship between providing
incorrect explanations for hypothesis selection and learning gains; additionally, students
providing incorrect explanations completed fewer in-game goals. Finally, there were no
significant gender effects on learning or in-game performance.
Results indicate that hypothesis testing strategies play a central role in narrative-
centered learning environments, demonstrating their connections to learning gains and
problem solving. These results are significant in that they contribute to a growing body of
research that explores the educational benefits, both practical and theoretical, of problem
solving tasks within games in and out of school settings. Given the central role problem
solving has as a 21st century skill combined with the challenges schools encounter as they
implement problem solving curricula, CRYSTAL ISLAND offers one example of how to
engage students in narrative-centered learning that also takes into account social aspects
of learning and cognitive load concerns.
Future research with CRYSTAL ISLAND will focus on more in depth analyses of in-
game performance and its relationship to learning outcomes as well as different

Problem Solving and Game-Based Learning
!
27
pedagogical game features. Plans are underway to create the next iteration of CRYSTAL
ISLAND, including a web-based interface and additional modules related to eighth grade
science curriculum.
No single educational approach, including gaming, is effective across all subjects
and for all students. As members of the 2006 Summit suggest, gaming research needs to
continue to focus on what works with whom and in which context. When the research
community adequately addresses this concern, games will become more compatible with
school learning contexts and potentially have a greater impact on developing students’
21st century skills.

Problem Solving and Game-Based Learning
!
28
References
Chen, H-H., & O’Neil., H. F. (2008). A formative evaluation of the training effectiveness
of a computer game. In H. F. O’Neil & R. S. Perez (Eds.), Computer games and
team and individual learning (pp. 39-54). Amsterdam: Elsevier.
Cole, M., & Engestrom, Y. (1993) A cultural-historical approach to distributed cognition.
In G. Salomon (Ed.), Distributed cognitions: Psychological and educational
considerations. New York: Cambridge University Press.
Davidson, J., & Sternberg, R. (2003). (Eds.). The psychology of problem solving.
Cambridge, UK: Cambridge University Press.
Deslandes, J. (2004). A philosophy of emoting. Journal of Narrative Theory, 34 (3),
335-372.
Engeström, Y. (1987). Learning by expanding: an activity theoretical approach to
developmental research. Helsink: Orienta-Konsultit Oy.
Federation of American Scientists. (2006). Harnessing the power of video game for
learning. Retrieved from http://fas.org/gamesummit/
Gee, J.P. (2003). What video games can teach us about literacy and learning. New York:
Palgrave-McMillan.
Gerrig, R. (1993). Experiencing Narrative Worlds: On the Psychological Activities of
Reading. NewHaven: Yale University Press.
Gros, B. (2007). Digital games in education: Design of games-based learning
environments. Research on Technology in Education, 40(1), 23-38

Problem Solving and Game-Based Learning
!
29
Hays, R. T. (2005). The effectiveness of instructional games: A literature review and
discussion. Naval Air Warfare Center Training Systems Division (No 2005-004).
Retrieved from http://stnet.dtie.mil/oai/
Hong, N.S., Jonassen, D.H., & McGee, S. (2003). Predictors of well-structured and ill-
structured problem solving in an astronomy simulation. Journal of Research in
Science Teaching, 40 (1), 6-33.
Kaptelinin, V. & Nardi, B. (2006). Acting with technology: Activity theory and
interaction design. Cambridge: MIT Press.
Ketelhut, D. J. (2007). The impact of student self-efficacy on scientific inquiry skills: An
exploratory investigation in River City, a multi-user virtual environment. The
Journal of Science Education and Technology (Feb. 2007), 99–111. DOI =
10.1007/s10956-006-9038-y
Klopfer, E., Osterweil, S. & Salen, K. (2009). Moving learning games forward:
Obstacles, opportunities, & openness. The Education Arcade: MIT.
Levy, F. & Murnane, R. (2004). The new division of labor: How computers are creating
the next job market. Princeton, NJ: Princeton University Press.
Mayer, R.E. & Johnson, C. (2010). Adding instructional features that promote learning in
a game-like environment. Journal of Educational Computing Research, 42 (3)
241-265.
Mayer, R.E. (2005). (Ed.). The Cambridge handbook of multimedia learning. New York:
Cambridge University Press.

Problem Solving and Game-Based Learning
!
30
Mayer, R. E., & Wittrock, M. C. (1996). Problem-solving transfer. In R. Calfee & R.
Berliner (Eds.), Handbook of educational psychology (pp. 47-62). New York:
Macmillan.
McQuiggan, S., Rowe, J., Lee, S., & Lester, J. (2008). Story-based learning: the impact
of narrative on learning experiences and outcomes. In Proceedings of the Ninth
International Conference on Intelligent Tutoring Systems (pp. 530-539). Montreal,
Canada.
McQuiggan, S., Goth, J., Ha, E. Y., Rowe, J., & Lester, J. (2008). Student note-taking in
narrative-centered learning environments: Individual differences and learning
outcomes. In Proceedings of the Ninth International Conference on Intelligent
Tutoring Systems (pp. 510-519). Montreal, Canada.
McQuiggan, S., Rowe, J., & Lester, J. (2008). The effects of empathetic virtual characters
on presence in narrative-centered learning environments. In Proceedings of the
2008 SIGCHI Conference on Human Factors in Computing Systems (pp. 1511-
1520). Florence, Italy.
McQuiggan, S., Mott, B., & Lester, J. (2008). Modeling self-efficacy in intelligent
tutoring systems: an inductive approach. User Modeling and User-Adapted
Interaction, 18 (1-2), 81-123.
Metcalf, S. Dede, C., Grotzer, T. Kamarainen, A. (2010). EcoMUVE: Design of virtual
environments to address science learning goals. Paper presented at the American
Educational Research Association (AERA) Conference, Denver, CO.
Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our
capacity for processing information. Psychological Review, 63, 81-97

Problem Solving and Game-Based Learning
!
31
Morrison, T. (2001). The work of nations—Interview with Robert Reich. Aurora Online:
http://aurora.icaap.org/archive/reich.html
Mott, B., Callaway, C., Zettlemoyer, L., Lee., S. and Lester, J. (1999). Towards
narrative-centered learning environments. In working notes of the AAAI Fall
Symposium on Narrative Intelligence, 78-82. Cape Cod, MA.
Mott, B., Lee, S., & Lester, J. (2006). Probabilistic goal recognition in interactive
narrative environments. In Proceedings of the Twenty-First National Conference
on Artificial Intelligence (pp.187-192). Boston, Massachusetts.
Mott, B., & Lester, J. (2006). U-Director: a decision-theoretic narrative planning
architecture for storytelling environments. In Proceedings of the Fifth
International Conference on Autonomous Agents and Multiagent Systems (pp.
977-984). Hakodate, Japan.
Mott, B., & Lester, J. (2006). Narrative-centered tutorial planning for inquiry-based
learning environments. In Proceedings of the Eighth International Conference on
Intelligent Tutoring Systems (pp. 675-684). Jhongli, Taiwan.
National Research Council. (2010). Hilton, M. (Ed.) Exploring the intersection of science
education and 21st century skills: A workshop summary. Washington, D.D.:
National Academy Press.
Nelson, B. (2007). Exploring the use of individualized, reflective guidance in an
educational multi-user virtual environment. The Journal of Science Education and
Technology, 16 (1) 83–97.
Pelletier, C. (2009). Games and learning: What’s the connection? International Journal
of Learning & Media, 1(1), 83-97. MIT. doi:10.1162/ijlm.2009.0006

Problem Solving and Game-Based Learning
!
32
Robison, J., McQuiggan, S., & Lester, J. (2009). Evaluating the consequences of
affective feedback in intelligent tutoring systems. In Proceedings of the
International Conference on Affective Computing & Intelligent Interaction (pp.
37-42). Amsterdam, The Netherlands.
Rosenblatt, L. M. (2004). The transactional model of reading and writing. In R. B.
Ruddell, M. R. Ruddell, & H. Singer (Eds.), Theoretical models and processes of
reading (pp. 1363-1398). Newark, DE: IRA.
Rowe, J., Shores, L., Mott, B., & Lester, J. (2010). Individual differences in game play
and learning: A narrative-centered learning perspective. In Proceedings of the
Fifth International Conference on Foundations of Digital Games (pp. 171-178).
Monterey, California.
Rowe, J., Shores, L., Mott, B., & Lester, J. (2010). Integrating learning and engagement
in narrative-centered learning environments. In Proceedings of the Tenth
International Conference on Intelligent Tutoring Systems (pp. 166-177).
Pittsburgh, Pennsylvania.
Rowe, J., McQuiggan, S., Robison, J., & Lester, J.(2009). Off-task behavior in narrative-
centered learning environments. In Proceedings of the Fourteenth International
Conference on Artificial Intelligence in Education (pp. 99-106). Brighton, U.K.
Schafer, D.W., Hatfield, D., Svarovsky, G., Nash, P., Nulty, A., Bagley, E., Frank, K.,
Rupp, A., Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st
century assessment of learning. International Journal of Learning and Media,
1(2), 33-53. doi: 10.1162/ijlm.2009.0013

Problem Solving and Game-Based Learning
!
33
Spires, H., Turner, K., Rowe, J., Mott, B., & Lester, J. (2010, April/May). Effects of
game-based performance on science learning: A transactional theoretical
perspective. Paper presented at AERA, Denver, CO.
Spires, H.A. (2008). 21st century skills and serious games: Preparing the N generation. In
L.A. Annetta (Ed.), Serious educational games (pp. 13-23). Rotterdam, The
Netherlands: Sense Publishing.
Squire, K. D. (2006). From content to context: Videogames as designed experience.
Educational Researcher, 35(8), 19-29.
Sweller, J., (1988). Cognitive load during problem solving: Effects on learning.
Cognitive Science, 12, 257-285. In R.E.,Mayer, R.E. (2005). (Ed.), The
Cambridge handbook of multimedia learning (pp. 19 –30). New York: Cambridge
University Press.
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design.
Learning and Instruction, 4, 295–312. Doi:10.1016/059-4752(94)90003-5.
Sweller, J. (2005). Impllications of cognitive load theory for multimedia learning. In
Vygotsky, L.S. (1978). Mind in society: The development of higher psychological
processes. Cambridge, MA: Harvard University Press.
Warren, S., Dondlinger, M., Stein, R., & Barab, S. (2009). Educational game as
supplemental learning tool: Benefits, challenges, and tensions arising from use in
tan elementary school classroom. Journal of Interactive Learning Research, 20
(4), 487–505.

Problem Solving and Game-Based Learning
!
34
!
!
Figure 1. CRYSTAL ISLAND learning environment.

Problem Solving and Game-Based Learning
!
35
Figure 2. In-game Hypothesis Testing Tool.

Problem Solving and Game-Based Learning
!
36
Notes: R2 = .20. R2 Adj. = .19. * p < .01. ** p < .001.
Table 1
Linear Regression of Pre-Test Score and Proportion of Inappropriate
Hypotheses on Post-Test Score
!