Serious Game, Serious Results: A Case Study with Evidence

Abstract

This article describes the process used to design and development of a serious game to
teach a difficult topic, statistics. It describes specific decisions made at each stage by answering 12
general questions of game design and evaluation. These questions can be used by game designers to
facilitate choices made at decision points and are applicable to any tutorial game development project.
The resulting game was tested for learning efficacy in a formal quantitative experiment. The game
was found to produce significantly better learning than conventional classroom instruction. However,
student acceptance of learning from the game was mixed. Analysis of player responses to open ended
questions about their experience suggest some specific reasons for this lack of acceptance that are
worthy of further study. Of note, the single largest negative experience reported by players of the game
was the lack of communication with the instructor and other students during game play. Suggestions for
iterative improvement of the game derived from student comments are described.

Full text

13
Computer Games and Communication 2016; 1(1): 13-25
Research Article
Serious Game, Serious Results: A Case
Study with Evidence
James H. Watt
1*
, Mark Hamilton
2
, Kristine L. Nowak
3
,
John L. Christensen
4
1 2 3 4 University of Connecticut, USA
Abstract: This article describes the process used to design and development of a serious game to
teach a dicult topic, statistics. It describes specic decisions made at each stage by answering 12
general questions of game design and evaluation. These questions can be used by game designers to
facilitate choices made at decision points and are applicable to any tutorial game development project.
The resulting game was tested for learning ecacy in a formal quantitative experiment. The game
was found to produce signicantly better learning than conventional classroom instruction. However,
student acceptance of learning from the game was mixed. Analysis of player responses to open ended
questions about their experience suggest some specic reasons for this lack of acceptance that are
worthy of further study. Of note, the single largest negative experience reported by players of the game
was the lack of communication with the instructor and other students during game play. Suggestions for
iterative improvement of the game derived from student comments are described.
Keywords: Serious game design, evidence of learning effectiveness, teaching statistical reasoning,
student reactions to games
*James H. Watt, Department of Communication, University of Connecticut, 337 Manseld Road-Unit
1259 Storrs, CT 06269-1259, USA
Email: James.Watt@uconn.edu
Mark Hamilton, Department of Communication, University of Connecticut, 337 Manseld Road-Unit
1259 Storrs, CT 06269-1259, USA
Email: Mark.Hamilton@uconn.edu
Kristine L. Nowak, Department of Communication, University of Connecticut, 337 Manseld Road-Unit
1259 Storrs, CT 06269-1259, USA
Email: Kristine.Nowak@uconn.edu
John L. Christensen, Department of Communication, University of Connecticut, 337 Manseld Road-
Unit 1259 Storrs, CT 06269-1259, USA
Email: John.Christensen@uconn.edu
Computer Games and Communication 2016; 1(1): 13-25
DOI: 10.15340/2148188111925
This article describes, in a case study format, a two-year project to develop a serious pedagogical
game. The stages of the design process are presented as a series of questions designed to be
used as a template for the game design process relevant to any pedagogical game. This paper
describes what happened at each stage in the design of a specic game teaching statistics.
1. What’s the Problem?
The rst step in creating a serious game is to identify the problem it could solve or the need
to be met. Developing new ways of doing things starts with a kernel of dissatisfaction with the
old ways. In the project described in this article, the old way was conventional instruction and
its discouraging results in a dicult subject area: understanding and using basic statistics. This
skill is central to general information literacy as well as professional practice in many scientic,
business, political, health, and manufacturing professions. In the general population, public policy
and personal life-style choices are guided by polls, reports of clinical studies, and other media
reporting that uses statistical evidence. Unfortunately, far too few people -- including college
graduates -- know how to properly collect, compute, interpret, and use even basic statistical
information (cf. National Science Board 2004; van Buuren 2006).

14
Computer Games and Communication 2016; 1(1): 13-25
This lack of basic statistical knowledge is not because high schools, community colleges, and
universities have ignored the subject. The most recent survey of undergraduate programs in
mathematical sciences, which includes courses in introductory statistics in a variety of disciplinary
areas inside and outside mathematics departments, provides data that generates an estimate of
more than 350,000 students per year enrolled in these courses in various academic departments
at the post-secondary level (Conference Board of the Mathematical Sciences, 2013). Yet it is evident
that conventional approaches to teaching reasoning with statistics have not been particularly
successful.
Conventional pedagogy in statistics falls short in several respects. The rst major problem is
that undergraduate students do not like statistics and react negatively to its delivery in standard
classroom lecture format. This format has been shown to evoke anxiety and a negative attitude
toward content (Ruggeri, Dempster, Hanna, & Cleary, 2008). Students often see statistics as
dicult, tedious, and irrelevant to their interests or the skills they expect to need in professional
life. We have seen this in our own academic department. In a recent internal study, 140 seniors
evaluated the courses they had taken in their Communication curriculum. One was a required
course in basic statistics and research methods taught as a conventional course with a standard
lecture-exercises format. This course was rated at the very bottom of all courses in student
satisfaction, signicantly lower than any other course. But an earlier alumni survey of graduates
who had been in the work force at least three years listed this same statistics course as one of
the most useful they had taken in college. So we know the course is valuable, and given the
importance of student engagement for learning, we see a clear need to present the course
material using modes that will effectively engage students. A related pressing challenge is to
convince students that learning statistics will be useful to them in their careers by addressing the
“why do I need to learn this?” question.
A second documented weakness of conventional courses involves the diculty students
have in applying the statistical concepts they have apparently mastered in class to situations and
problems outside that narrow context (Ruggeri, Dempster, Hanna, & Cleary, 2008). A student may
be able to compute a t-value for the difference between the means of two groups of individuals,
but can’t then use that skill to answer a question like “do males differ signicantly from females in
the amount of time spent in social networking sites?”
This problem can be partially addressed in course design by situating the statistical and
research examples as well as the exercises in an area of interest to the student, particularly if this
area is related to a future career. This allows a student to see how the content relates to his/her
future career goals, which has been shown to improve learning (Lovett and Chang 2007; Singley
and Anderson 1989; van Buuren 2006). Providing realistic problems and situations in which to
apply statistical reasoning may also counteract the perception of the topic’s irrelevance; the
context should be clearly created so as to closely resemble those students expect to see when
they step out into “real life.”
2. Is a Game Appropriate?
We needed to nd a better medium for delivering this challenging material to students. Some
simple observations of student behavior led us to an obvious conclusion: digital games is a
medium that strongly engages students. They focus on playing them for extended periods of
time and enjoy them. Lenhart, et al. (2008) estimated that 97% of American teens from across the
socioeconomic spectrum frequently play video games. So it seemed self-evident that students
would enjoy learning from a game. This was a naïve assumption, as we will see, but it provided
the initial motivation to develop a game as an alternative to conventional classroom instruction.
Empirical support exists for the use of gaming for instruction. Many elds of study have been
experimenting with digital games to deliver instruction. These games are now generally referred
to as “serious games” and they are an emerging eld of study (cf. Rittereld, Cody and Vorderer
2009). These are not the rather trivial “skill-and-drill” computer games of the past, but games that
combine realistic challenges and rewards across a complex set of simulated situations played
out in virtual worlds. The games also have added entertainment aspects that involve competitive
challenges designed to hold students’ attention and motivate them to spend time playing and
learning. Research evidence indicates that these serious games are an effective pedagogic tool,
although most of these studies have been done with K-12 students (Grechus 1999; Keeble 2008;
Pange 2003). Surprisingly little research has evaluated the potential of serious games in higher
education.
When applied to the teaching of introductory statistics, games have been shown to enhance
two key competencies. Games increase (1) the comprehension of statistical concepts and (2)

15
Computer Games and Communication 2016; 1(1): 13-25
statistical self-ecacy, the perception that one is competent to understand and make decisions
based on statistics (Swingler, Bishop, & Swingler, 2009). The game-based approach requires
active manipulation and reasoning with statistical tools rather than the more passive “information
intake” of conventional courses (Gareld, Hogg, Schau, & Whittinghill, 2002). This active learning
theoretically should lead to better comprehension of concepts and acquisition of skills in statistical
reasoning.
While these are common justications for using a game, there is very little rigorous formal
research support for the superiority of games as a mode of instruction in higher education. This
points to the need to include a formal assessment of research components to verify the “common
sense” presumption that a course delivered within a game leads to more engagement and
improved learning outcomes. We treated this presumption as a research hypothesis and included
a formal test of our game’s teaching ecacy as a central part of this project.
3. What Do You Want To Accomplish with a Game?
This is a critical question that game designers too often only vaguely specify. It requires dening
what you want the game to achieve, which in turn requires a measure of the game’s success in
accomplishing it. In our project, the answer to this question reduced to two general goals and
four specic requirements.
General Goal 1. First and foremost, the game should produce better learning of statistical
concepts and reasoning skills than conventional classroom instruction.
General Goal 2. The game should be seen by the students as interesting and even approach (as
nearly possible) actually being fun to play. In short, it should be a good game. Students should see
it as a desirable alternative to conventional instruction, which they view negatively.
Requirement 1. The game must deliver the same curricular content as a conventional
introductory course. The game should not be a narrow special-topics supplement to regular
instruction; it should deliver a complete learning environment within the game itself by essentially
following a standard introductory course syllabus embedded within a game narrative. The game
should integrate learning resources like video demonstrations and textbook and supplementary
reading material. The game narrative should require the student to use these resources in order
to progress in the game.
Requirement 2. The game should provide an extensive amount of active challenges that
require students to apply concepts, master computational skills, and draw conclusions from
evidence and should provide motivation to succeed. This requirement parallels the oft dreaded
“problem sets” of the conventional methods course, but is woven as seamlessly as possible
into the game play. In standard game play fashion, these challenge problems should result in
immediate feedback showing success or failure, provide students an opportunity to “succeed
by failing” (by being able to try the challenge again to correct their mistakes before going on).
Instead of evaluating learning and providing student feedback only several times a semester with
exams or problem sets, this iterative process should be continuous: challenge problemstudent
responseimmediate evaluation and feedbacktry again. This requires immediate automated
“grading” of student player responses to challenge problems, but should motivate the student to
keep trying until he or she succeeds. This is a dicult task in test materials design and technical
implementation, but if done properly it can provide very detailed and immediate information
about individual students’ performance, areas of weakness, etc., to both the student and the
course instructor.
Requirement 3. The game narrative should approximate the interest area of the target student
population. As the research literature indicated, this should help to establish the relevance of
learning the material to the student and result in more enthusiastic involvement, which in turn
should produce better learning.
Requirement 4. The digitally delivered game should capitalize on capabilities that go beyond
those available in a conventional classroom. For the instructor, this means the ability to add to
or modify learning and test materials used by the game without requiring that the game be
recompiled and redistributed to the student players. It also means that the instructor should
have the ability to closely monitor individual student progress on a very detailed basis and be
provided with automatic “red ags” when students are apparently having diculty with material
or are not participating in the game. For the student, it means having access to digital resources
that provide hands-on interactive experiences. These include data visualization affordances in
the game interface coupled with the ability to do trial-and-error data manipulation. Interacting
with data and seeing changes in statistics that describe that data are crucial. The student should

16
Computer Games and Communication 2016; 1(1): 13-25
also have (1) calculation technology that is very easy to use, simulating what will be available
in professional practice, and (2) access to relevant information and examples, via integrated
multimedia documents and video available at a click.
4. What Technology?
Ease of use of the game for the student was critically important. This strongly suggests a
Web browser game that could be played by students at any time that ts their schedule and
at any place where there is access to the Internet. Early feedback from students indicated that
they would greatly appreciate such spatio-temporal exibility. We had earlier experience with
games that required installation on widely varying student computer systems. This experience
discouraged us from distributing a static game installation. Although installed games can be
made more powerful than Web-based games, the diculty in providing technical support for the
large number of variations in students’ computer operating systems, computer age and speed,
memory, networking, etc., not to mention the variation in users’ technical skills in installing and
updating the game, outweighs this advantage by a large amount.
We decided to develop the game using the Unity game platform (http://unity3d.com/) which
provided very good 3D animation in Web browsers and has extensive programming resources.
Importantly, Unity was available in a limited version as a free download. This zero-cost alternative
allowed us to test it before committing to use it. Programming Unity character actions and
interfaces was done in C#, a version of which is included in the Unity engine.
There were signicant programming hurdles to link a Web server to the individual students’
game and prole information. These hurdles stemmed from the requirements that the game
download challenge problems, video, and text, as well as upload and store student performance
data. An initial test with Python programming using the Microsoft IIS server was not successful.
The combination worked but the system was extremely clunky and hard to administer. In the
end, we decided to invest the time to write original code for a special-purpose Web server for our
game. The code was written with Delphi 2010 (http://www.embarcadero. com/), using Web server
components provided in that development platform. It worked seamlessly, recording student
logins, time spent with the game and tutorial materials, etc. for over 50 students during the week
of gameplay used in the game evaluation experiment.
5. What Human Resources Are Required?
As relative newcomers to game design, we rst spent time familiarizing ourselves with the process
of game development. Reviewing the goals and their associated tasks and the time required to
complete them, we determined that we needed expertise in the following areas:
1. Curriculum Content (convert the standard course syllabus to game levels).
2. Teaching Materials (identify supporting video and text materials, develop game level
challenge problems).
3. Game Designer (write game narrative script implementing the curriculum content, dene
game mechanics).
4. Graphic Artist (create character appearance, design the virtual world of the game).
5. Animator (implement character movement, virtual world interactions with objects).
6. Programmers (develop and implement statistical algorithms, narrative branching logic,
server/game information transfer, instructor course management software).
In the perfect world of well-nanced projects, we would have turned much of this work (with
the exception of curriculum content and teaching materials) over to professional subcontractors
or to a full service game development studio. However, we work at an academic institution with
limited internal funding opportunities. This meant that the six functions were actually carried out
internally by a relatively small number of people whose project duties often spanned several of
the areas.
6. What Financial Resources are Available?
Our game had a demanding set of goals and nding the resources required to accomplish them
was perhaps the most dicult task in this game development project. The game could have been

17
Computer Games and Communication 2016; 1(1): 13-25
quite expensive to produce. Limited resources meant a lot of trade-offs had to be made at this
stage to balance the aspirations outlined in the goals with the resources available to achieve
them.
We proposed doing a proof-of-concept prototype project to our university’s Research
Foundation, which provides seed funds for faculty projects. We were fortunate to receive sucient
funding to establish a game design team of four digital media students who were employed 10
hours per week each, paid with student labor funds. The total funding was meager by professional
game development standards. The funding was, however, enough to allow us to create a game
that was of a high enough technical quality to test student reactions to the game, as well as
their learning. This rst version was limited to a single week’s instruction but included signicant
student learning assessment in order to establish the ecacy of game-based instruction in the
chosen topic for the week.
7. How Do You Go About Designing a Game?
We had an eight-person design team, with two faculty members acting as the primary producers
and writers, two other faculty members advising on pedagogical content and human-computer
interaction, two student programmers, one student animator, and one student graphic artist.
The design team was long on enthusiasm and short on prior experience in game development.
The early stages of the design process provided a steep learning curve for everyone, particularly
the producers. Managing an inexperienced student team when it is also the rst time you have
tackled a task like this is a major challenge.
The producers and students initiated weekly meetings to dene and assign tasks and report
progress. This was truly on-the-job training for all as we moved forward, partially by trial-and-
error. We began by jointly studying a number of excellent game design texts (Rouse III 2005;
Aldrich 2009; Schell 2008), experimenting with software, and brainstorming possible game genre.
Very soon it was clear that we could use professional guidance. We were able to engage an
excellent consultant who conducted a two-day workshop focused on brainstorming game ideas
and characters that would t the topic. We decided to situate the game in a simulated professional
environment and create characters and a narrative that would have some relevance to students
in a pre-professional academic major in Communication. The consultant critiqued (gently) a wide
range of ideas and guided the discussion toward some xed decisions about characters and a
narrative that could be the framework for plot development over a long period of play.
The game characters that emerged were actually based on some Jungian archetypes (Jung,
1990). Whether or not these represent manifestations of the culture or collective unconscious is
an argument for another day, but they do provide a nice set of diverse characteristics on which to
build game characters’ persona. A back story describing the behavioral and visual characteristics
of each character was then created, with each character developed by a different member of
the design team. The nal character archetypes who appear in the game were: The Hero (the
player him/herself who must act to advance the narrative); The Friend (who gives advice and
encouragement); The Leader (who directs work, advances the plot, and evaluates work); The Sage
(who provides information, a library, and a back room with interesting data manipulation gizmos);
The Nemesis (who competes with The Hero, and is rather nasty); and The Trickster (who provides
unexpected actions, humor, and who drops clues to further the plot).
The genre of mystery game with an unknown culprit was chosen for the narrative. The player
must use statistics to investigate different suspects, as well as improve her/his standing with the
Leader by carrying out important data analyses.
A deductive mystery is a natural kind of plot that mirrors the process of statistical reasoning,
which can be seen as searching for hidden truths in a pile of numbers. It is easy to create situations
and tasks that require the player to construct and interpret statistics in order to discover potential
bad actors in the set of game characters and off-stage organizations. A description of the nal
game design is below.
8. How Does the Game Play (Game Narrative and Game Mechanics)?
Good content is critical to the success of a game, as the content and the narrative engage the
users and keep them motivated. To begin the game, the student player takes on the role of a new
intern at StatStar, a company that provides marketing communication research to a major client
who is a manufacturer of pharmaceuticals. This client, PsychoPharma, Inc., has an important new
product coming out. Its success is critical to both StatStar and PsychoPharma.
But something is not right. There are conicting research results coming from different

18
Computer Games and Communication 2016; 1(1): 13-25
people at StatStar and from PsychoPharma’s own research. There may or may not be dangerous
side effects of the new drug. There are also questions about some important advertising appeals
that might be made to different market segments. Are there honest errors in using statistics
being made by the game characters or might there be unethical data manipulation or outright
research fraud going on? People could be harmed if a dangerous drug is advertised using false
research results. The player must prevent these outcomes by nding out what is happening and
who is behind it. In addition to his or her normal job duties as an intern, the player must act
as an undercover investigator to critically evaluate the analyses and conclusions of other game
characters who are suspects. As the game advances, the player will even be given a budget to
commission independent research to identify and target the culprits (and to experience the cost/
benet tradeoffs of a research study). The player’s score, called the Job Evaluation, depends on his
or her success in doing both ordinary work assignments and also uncovering wrongdoing. As the
game progresses, the successful student is promoted to full-time employee and then to project
manager and given more demanding challenges as the game levels (i.e., topics on the syllabus)
progress.
The game characters are woven into the plot as the game unfolds. In addition to advancing
the plot, these characters perform teaching functions. Some characters provide help and tips for
the player, other characters compete with the player, although most characters are suspects in
the mystery at one time or another. The Hero is viewed with a 3
rd
person trailing camera, and can
navigate around the oce, interacting with other game characters, using the data manipulation
gizmos, and accessing text and video informational materials.
The game advances through twelve major levels, each one corresponding to approximately
one week’s topics on a standard syllabus in an introductory research methods course. The topics
of each level are based on the recommendations of a National Science Foundation-funded panel
of experts (Watt, Gourgey, Nambisan, Landau, & Gourgey, 2005). Within each level are a number
of sublevels, each with a specic learning goal and associated player challenge problems.
The challenge problems are presented at the player’s workstation in the virtual StatStar oce.
A screen shot showing a part of the StatStar oce with the Hero at a workstation during game
play is shown in Figure 1.
Figure 1. The StatStar Virtual World
Student Players provide answers to the challenges after conducting analyses and sending
their results, decisions, and recommendations via “e-mail” to game characters. The results are

19
Computer Games and Communication 2016; 1(1): 13-25
immediately evaluated by the character (the grading software) for correctness. Feedback on
performance is given by game characters, usually with encouragement (The Friend) but sometimes
with mild insults if it’s wrong (The Nemesis). Correct and incorrect answers are immediately
shown to players, who have the opportunity to improve their performance. A player can be held
at a sublevel until a minimum performance score that is dened by the instructor is reached. If
a student has diculty in advancing, the game will notify the course instructor automatically by
email, if the instructor desires.
The challenge problems and their grading criteria can be created and modied at any time
by the course instructor with the StatStar problem set editor software. Changes in challenge
problems are made to external les stored on the game server and do not require modifying
the game software itself. This exibility to easily and immediately change evaluation materials to
respond to individual class needs is a major teaching advantage for the instructor, and is a feature
not found in most serious games.
Another important learning aid built into the game allows the player to do Direct Data
Manipulation (DDM) and observe the impact of changes in the data and statistics visually. This
hands-on experience with data helps the player make the connection between observed data
values and statistical descriptions of the data. DDM is also used in the game narrative in challenge
problems. The player experiments with different ways that data might have been changed or
biased by game characters, either through accidental errors or by malicious intent. To do this,
the player uses a ctional construction called the Data Gizmo, located in The Sage’s oce. The
player can experiment with generating different sets of data with specied statistical properties,
different numbers of sample observations, and so forth. The player is encouraged to use the
Gizmo for exploratory learning. This experimentation could be suggested by game characters
if the player is performing poorly, or can be used any time on a player’s own initiative. The Data
Gizmo can also be used by the player to compute statistics to respond to the challenge problems.
The game also provides the student player with conventional textbook material and video
mini-lectures and demonstrations. These can be prepared by the instructor or obtained from
commercial sources. Like the challenges, these les are external to the game and can be added,
deleted, or changed by the instructor at any time without modifying the game itself. The instructor
simply uploads the new video or text les and the materials are instantly available to the players.
This gives the course instructor wide exibility to modify the resources available to the students
during the course of the semester.
The player uses these conventional learning resources in preparing to respond to the
challenge questions. Pilot research with players of the game showed that some students prefer
learning from these familiar sources, whereas others prefer trial-and-error interactions with the
game. Having conventional textbook and video material integrated with the game gives students
the option of learning in the way that they prefer.
9. How Does the Game Perform as a Teaching Method?
Student learning is the most important objective. Developing a game is expensive in time and
expensive in dollars, and these expenses need to be justied by research that assures that they
are a good investment. A successful pedagogical game has to demonstrate relative advantage
over conventional teaching methods, either in student acceptance and engagement or student
in objective performance, but preferably in both. An important part of our project was a formal
classroom experiment to see if we had reached these important general goals.
9.1. The Experimental Procedure
This research used the prototype version of StatStar which fully implemented one week’s
instruction and included excerpts from a published textbook (Watt & van den Berg, 1995) as
well as professionally prepared videos of lectures and demonstrations that covered descriptive
statistics.
The prototype included only Level One of the 12 levels in the full game. This rst level
had ve sublevels, each representing an important topic in basic descriptive statistics (level of
measurement, measures of central tendency, dispersion, etc.). Each sublevel had an associated
challenge problem set that was integrated into the game narrative.
The research was carried out in two Northeastern U.S. universities. One was a large public
institution and the other was a smaller private and highly selective technological university. At the
public institution two classes in Introduction to Communication Research Methods participated.
In one class, students received standard classroom instruction in descriptive statistics for a week.

20
Computer Games and Communication 2016; 1(1): 13-25
In the other, the class did not meet and the students played the StatStar game instead. Both
classes lled out a short questionnaire before the beginning of the week that measured, among
other things, their opinion of the desirability of taking a course based on game rather than a
conventional class. To our surprise, one-third of the students did not like the idea of using a
video game. Comments like “I hate games” and “I never play them” countered our perception of
undergraduate students as a monolithic group of enthusiastic gamers.
Students in both the game and conventional classroom groups took a 15-minute standard
test of statistical knowledge at the end of the experimental week. This test was developed in an
earlier NSF-funded project and covered basic statistical topics recognized as very important by a
national panel of statistics educators (Landau, et al. 2005; Watt, et al. 2005). Items from the full
test that covered basic descriptive statistics were selected for this test. In addition, the course
coordinator added items on topics that were emphasized in the classroom instruction. At the
technological university, students who had completed a course in basic research methods and
statistics in the Department of Cognitive Science took this test, which included the added items,
at the end of the course. This group was used as a comparison group for both the conventional
classroom and game playing classes.
To measure student behavior in the game-playing experimental group, the software provided
detailed recording of player data that included the number of logins, amount of time playing
the game, performance on each of the challenge tasks, and use of the integrated text and
video materials. At the end of game play, the game players group also lled out a questionnaire
evaluating the game and their experience.
Fifty-ve students played the StatStar game and their statistical knowledge at the end of the
game play was compared to that of 32 students in the conventional classroom. The game players
were also compared to the 17 students at the technological university.
9.2. Results
The primary nding was that the game players scored signicantly higher in statistical knowledge
at the end of the week than did students in the classroom comparison group, as seen in Table 1.
Table 1. Comparison of Learning from Game with Conventional Classroom
GROUP N Mean
Std. Error of
Mean
t
Signicance
(one-tail)
Statistics
Knowledge
Score
Game Players 55 13.85 .523 1.954(d.f.=85) p = 0.027
Comparison Class 32 12.19 .667
In addition, in the game playing group, students who spent more time playing the game
showed signicantly higher levels of statistical knowledge (Pearson correlation r = .35, p < .003).
This provides some indication that learning was at least partially the result of playing the game
itself, and was not produced solely by other, non-game, sources of learning and helps emphasize
the importance of making the game fun to engage students and get them to play it.
Table 2 shows that students in the conventional classroom course scored signicantly lower
on statistical knowledge at the end of instruction when compared to the selective technological
institution’s students. This is not unexpected, as the technological institution’s students must score
high on mathematical abilities to be admitted and must navigate a curriculum that is very heavy
on math and science. In contrast, students in the public institution comparison classroom were
a range of liberal arts students who typically score lower at admission on tests of mathematical
abilities and are not nearly as oriented to mathematical subjects.
Table 2. Comparison of Classroom Learning in Two Types of Universities
GROUP N Mean
Std. Error
of Mean
t
Signicance
(two-tail)
Statistics
Knowledge
Score
Public University
Comparison Class
32 12.19 .667
2.441
(d.f.=47)
p = 0.018
Private Tech. University
Comparison Class
17 15.00 .963

21
Computer Games and Communication 2016; 1(1): 13-25
But as Table 3 shows, there was no signicant difference in statistical knowledge between the
game players and the private institution comparison group, although the latter scored somewhat
higher.
Table 3. Comparison of Game and Technological University Classroom
GROUP N Mean
Std. Error
of Mean
t
Signicance
(one-tail)
Statistics
Knowledge
Score
Public U. Game Players 55 13.85 .523
1.059
(d.f.=70)
p = 0.147
(not sig.)
Private U. Comparison Class 17 15.00 .963
Taking these ndings as a whole, we conclude that the game shows promise of improving
student learning, particularly for those not primarily oriented to math and science. The game
tested in this study allowed students who likely began with less initial aptitude to move up
to the levels that were near those of students who were likely more comfortable initially with
mathematics. These are promising preliminary results, particularly for general education or at-
risk students. Hence, we are comfortable concluding that the game met the rst general goal of
improving student learning of statistical material over conventional instruction.
10. How Do Students React to the Game?
Our secondary goal of increasing student acceptance of the course and involvement in learning
received less support. Students actually showed a statistically signicant decrease in their
enthusiasm for a game-delivered course of 30% (on a 10-point Likert scale) after playing the
game. Understanding the nature of their discontent was critical to understanding how to improve
the delivery of this content in game format in the future.
In the post-test questionnaire, game players were asked to describe their opinion of the
most negative aspect of learning from the game. Table 4 shows a summary of their open-ended
comments. Over one-third cited lack of communication with the instructor and other students
as the biggest problem, which is has been a criticism of online learning more generally. Many
students clearly would have preferred more human interaction than the single player game
provided. An improved game needs to have a method for easy and extensive instructor-to-
student, student-to-student and possibly group communication or even competition with others
in a multiplayer format as an integral game feature.
Table 4. Negative Student Reactions to Game
Open-Ended Category:
Single Most Negative Aspect of Prototype Game Pct. of Total
Lack of communication with instructor 34%
Mismatch with preferred learning style 29%
Lack of structure, explanations 16%
Game technical problems 8%
Problems with text or video 8%
Problems with attention, motivation 5%
The next two most frequently mentioned negative categories had to do with the nature
of game delivery versus conventional classrooms. The rst of these, mismatch with preferred
learning style, contained comments about preferring classroom lectures to online instruction.
This perception may have been partially due to the lack of interaction with the instructor and
participation in class discussions, and thus might overlap with the comments in the rst category.
But it may also reect the feelings of the segment of students who simply do not like the idea of
learning from a game, or did not like this game in particular.

22
Computer Games and Communication 2016; 1(1): 13-25
A small percentage of students also wished for more structure, organized notes, and more
detailed explanations of what the student was expected to do. This criticism appears to be a
learning style preference for the more familiar structure of a conventional class. It points to a
need to provide additional game features to help students who prefer this style of learning and
perhaps a better tutorial for the students. It is possible that this is related to the complaint about
a lack of interaction with an instructor, as those not clear on what to do may not have been sure
how to obtain assistance with the game or the task. Whatever the reason for their discontent,
students who do not nd game-based learning particularly attractive must also be served within
the learning environment created by the game.
Technical problems with the prototype made up the reminder of the negative comments.
These problems are an unfortunate reality with the rst version of most software, and particularly
the case with complex programs like games. However, technical issues are usually well dened
(e.g. “the video kept stopping”) and can be addressed successfully with more testing and
debugging in subsequent versions of the game.
On a more encouraging note, students were asked in a similar open-ended question to
identify the most positive thing about their experience playing the game. These open-ended
comments were classied into 7 different categories, with the results shown in Table 5.
Table 5. Positive Student Reactions to Game
Open-Ended Category:
Single Most Positive Aspect of Prototype Game Pct. of Total
Scheduling Flexibility 45%
Interactive, Immediate Feedback, Multiple tries 18%
Videos and Texts Integrated 13%
Cool, fun 8%
Challenge of playing 5%
Hands on and examples 5%
Game learning, easy to use 5%
By far the most frequent plus identied by students was the exibility of studying and
interacting with the game at any time. Almost half the game players cited this as the single most
positive thing about the game. This desire appears to be at odds with the negative comments
about the lack of the ability to interact with instructor and class. Individuals appear to want both
the benets of in-person interaction from a classroom setting and the scheduling exibility of a
single player game, but these characteristics are essentially a zero-sum trade-off. This seeming
contradiction may again point to the presence of two groups of students in the course that have
different expectations and desires in their preferred way of learning, and both of which need to
be accommodated.
To investigate this, a K-Means cluster analysis was conducted. This analysis nds relatively
homogenous patterns of response in groups of individuals based on a set of rating variables (IBM
2015). Two distinctly different groups in terms of importance ratings of scheduling exibility and
communication by the students were found. The results are in Table 6.
The rst group, “Communication Priority” contains about two-thirds of the students. They rate
scheduling exibility as moderately important but give very high importance to communication
with the instructor and classmates. The second group, “Mixed Priorities” considers scheduling
exibility as much more important, but also rates communication with instructor as important.
But this group does not feel interaction with classmates is very important. To reach the rst group
more effectively, the game should have added affordances for communication with both the
instructor and with classmates. This might entail online communications integrated into the game,
face-to-face meetings, student group projects, or occasional scheduled classes combined with the
game. To reach the second group, no class meetings or group work would be necessary, but some
communication channel to the instructor is needed. The addition of integrated communications
into the game might suce.

23
Computer Games and Communication 2016; 1(1): 13-25
Table 6. Scheduling Flexibility and Communication Cluster Results
Mean Importance (10 point scale)
Cluster Group Flexibility
Communication with
Instructor
Communication
with classmates
Communication Priority
(N=26) 5.7 9.3 8.1
Mixed Priorities
(N=10) 7.8 8.2 4.0
Signicance of difference in
means p = .01 p = .06 p < .001
Another thing to consider is that the scheduling advantage would be present in any
asynchronous on-line course, so this is an advantage that is not unique to an interactive digital
game. But the second most mentioned positive category directly identied the interactive and
immediate nature of the game, especially when integrated with supporting text and video
material. Virtually all the other categories also mentioned positives that are unique to the game
delivery of instruction. So a number of students found the benets of a game that are beyond
those provided by an asynchronous on-line course to be important positives.
11. Where Do We Go From Here?
The results of the evaluation research point to the prospect of the full StatStar game being
an effective teaching platform that has the potential to outperform conventional classroom
instruction, but only if the game is improved in several ways. To meet the second important goal
of improving student attitudes toward learning the material would require adding communication
affordances and additional tutorial aids.
The information gained from this assessment underscores the statement that doing extensive
evaluation research as part of the game development process is absolutely essential. We need to
understand not only where we are with this version, but where we must go to improve the game.
It is important to treat pedagogical game development as a continuous process of improvement,
driven not only by objective measurement of learning but also by analysis of what students expect
from the game and how they react to using it.
Finally, we asked students to suggest improvements for the next iteration of the game. Their
comments were very instructive and are summarized in Table 7.
Table 7. Student Suggestions for Game Improvement
Open-Ended Suggestion Category: Improvement of Prototype Game Freq. Pct. of Total
Technical Improvements in game play like character movement, audio 10 25%
Improve instructional aids like notes, examples, feedback 10 25%
Improve audio/text teaching materials 6 15%
Make interaction more “game-like” 5 13%
Add communication with students, instructor 4 10%
Other misc. suggestions e.g. “nothing”; “hate learning online” 3 8%
Simplify challenge problem sets 2 5%
TOTAL 40 100%

24
Computer Games and Communication 2016; 1(1): 13-25
These comments, coupled with those identifying the positive and negative aspects of the
game, lead to three priorities for improvement.
1. Make it a better game, both in the gameplay and narrative, as well as in the technical
quality. Our prototype version worked as a teaching device, but less well as an entertaining
and involving way to learn. We confess that because of budget constraints our focus
in the prototype game was heavily on developing teaching features at the expense of
entertaining game play. The next stage of development will need more emphasis on pure
entertainment and gameplay mechanics that increase student interest, involvement, and
enjoyment. The pure fun of playing a game was mentioned far less frequently than we
anticipated, and this lack needs to be addressed.
2. Add social communication capabilities to remove the sense of isolation for individual
players and to foster peer-to-peer teaching, joint work where appropriate, and instructor-
to-student communication to answer questions and address both technical and learning
problems.
3. Accommodate some students’ desire for more conventional methods of learning by
providing more extensive optional materials like notes, video lectures, and demonstrations
and examples. This might even extend to a syllabus-like guide to allow a student to orient
her/himself within a semester’s instruction. We know that different segments of a class
like to learn in different ways, and the capability of a digital game to deliver a rich array of
materials and experiences should be exploited to provide support for a range of different
learning styles.
12. What Did We Learn?
Much of the description of this case study is unique to the problem being addressed and the
game that we developed. But the 12 general questions comprise a framework that applies to most
tutorial game development projects. A couple of conclusions drawn from this project probably
generalize to most serious games that are used as a central part of teaching.
Not all students like games and we need to nd ways to adapt games that reach these
indviduals too. But we found that students do learn from a game, even if it does not meet their
preferred mode of learning. The learning environment and its constraints also probably affect
acceptance of game-based learning. A game like StatStar would appear to be perfect for online
distance education where conventional classroom instruction is not available, and is perhaps less
suited (at least for some students) for on-campus use.
This experience illustrates how developing a game is dicult, expensive, and a lot of fun. Less
enjoyable is nding continuing funding for the iterative development process that is clearly needed
to make a really good game. At some point, it is necessary to move from the enthusiastic amateur
bootstrap approach that we have been using out of necessity to partnering with professionals in
art, animation, game mechanics, and game programming. Seeking these partners to produce a
fully developed and fully professional game is the current focus of this project.
References
Aldrich, C. (2009). The Complete Guide to Simulations & Serious Games. San Francisco, CA: John Wiley &
Sons.
Conference Board of the Mathematical Sciences. (2013). Statistical Abstract of Undergraduate Programs in
the Mathematical Sciences. Retrieved 5 6, 2014, from Conference Board of the Mathematical Sciences:
http://www.ams.org/profession/data/cbms-survey/cbms2010-work
Gareld, J., Hogg, B., Schau, C., & Whittinghill, D. (2002). First courses in statistical science: The status of
educational reform effort. Journal of Statistics Education, 10(2). Retrieved from http://www.amstat.
org/publications/jse/v10n2/gareld.html
Grechus, M. (1999). The comparison of individualized computer game reinforcement versus peer-
interactive board game reinforcement on nutrition label knowledge retention of fth graders.
Dissertation Abstracts International Section A: Humanities and Social Sciences, 59, 2323.
Jung, C. J. (1990). The archetypes and the collective unconscious. (R. Hull, Trans.) Princeton, NJ: Princeton
University Press.
Keeble, K. (2008). Digital gaming as a pedagogical tool among fourth and fth grade children. Dissertation
Abstracts International Section A: Humanities and Social Sciences, 69, 948.

25
Computer Games and Communication 2016; 1(1): 13-25
Landau, S., Watt, J., Sundararajan, B., & Young, B. (2005). Field trial of an audio/tactile statistics curriculum
for blind and visually impaired students. Tactile Graphics Conference. Birmingham, U.K.
Lenhart, A., Kahne, J., Middaugh, E., Macgill, A., Evans, Cl, & Vitak, J. (2008). Teens, Video Games and
Civics. Pew Research Internet Project. http://www.pewinternet.org/2008/09/16/teens-video-games-
and-civics/
Lovett, M. C., & Chang, N. M. (2007). Data-analysis skills: What and how are students learning? . In M.
Lovett, & P. Shah (Eds.), Thinking with Data (pp. 293-318). New York: Lawrence Erlbaum Associates.
National Science Board. (2004). Science and Engineering Indicators 2004. National Science Foundation,
Division of Science Resources Statistics.
Pange, J. (2003). Teaching probabilities and statistics to preschool children. Information Technology in
Childhood Education Annual, 15, 163-172.
Rittereld, U., Cody, M., & Vorderer, P. (2009). Serious Games: Mechanisms and Effects. New York: Routledge.
Rouse III, R. (2005). Game Design: Theory and Practice, Second Edition. Plano, TX: Wordware Publishing.
Ruggeri, K., Dempster, M., Hanna, D., & Cleary, C. (2008). Experiences and expectations: The real reason
nobody likes stats. Psychology Teaching Review, 14, 75-83.
Schell, J. (2008). The Art of Game Design. Burlington, MA: Elsevier.
Sheldon, L. (2012). The Multiplayer Classroom: Designing Coursework as a Game. Boston, MA: Course
Technology (Cengage Learning).
Sheldon, L. (2014). Character Development And Storytelling For Games, 2nd Edition. Boston, MA: Course
Technology.
Singley, M. K., & Anderson, J. R. (1989). The transfer of cognitive skill. Cambridge, MA: Harvard University
Press.
IBM (2015). SPSS Statistics 20.0.0. http://www-01.ibm.com/support/knowledgecenter/ SSLVMB_22.0.0/
com.ibm.spss.statistics.help/spss/base/idh_quic.htm
Swingler, M. V., Bishop, P., & Swingler, K. M. (2009). SUMS: A exible approach to the teaching and
learning of statistics. Psychology Learning & Teaching, 8, 39-45.
van Buuren, H. (2006). Teaching statistics and research methods: An integrated approach. 7th International
Conference on Teaching Statistics,, Brazil. Salvador, Bahia, Brazil.
Watt, J. H., & van den Berg, S. (1995). Research methods for communication science. Boston: Allyn & Bacon.
Watt, J., Gourgey, K., Nambisan, P., Landau, S., & Gourgey, A. (2005). A model for enhancing graphical
learning for students with print disabilities: An audio/tactile statistics curriculum. Human Computer
Interaction International 2005. Las Vegas, NV.