Content uploaded by Elena Novak
Author content
All content in this area was uploaded by Elena Novak on Jan 31, 2019
Content may be subject to copyright.
Effects of Game-Based Learning on Students’ Mathematics Achievement:
A Meta-analysis
Umit Tokaca, Elena Novakb, and Christopher G. Thompsonc
aCollege of Nursing, University of Missouri – St. Louis, St. Louis, MO
bSchool of Teaching, Learning and Curriculum Studies, Kent State University, Kent, OH
cEducational Psychology, Texas A&M University, College Station, TX
How to cite this article:
Tokac U., Novak E., & Thompson C.G. (2019), Effects of game‐based learning on students' mathematics achievement: A meta‐
analysis. Journal of Computer Assisted Learning.1–14. https://doi.org/10.1111/jcal.12347
Statement on conflict of interest
We declare that there is no conflict of interest with respect to the research, authorship and/or publication of this work.
Abstract
This meta-analysis investigated the effects of learning video games on mathematics achievement of PreK-12th grade students
compared to traditional classroom instructional methods. Results from the 24 collected studies showed heterogeneity among effect
sizes, both in magnitude and direction. Using a random-effects model, a small but marginally significant overall effect (
) suggested that mathematics video games contributed to higher learning gains as compared to traditional instructional
methods. In addition, moderator analyses were mixed in terms of statistical significance and explored effect-size heterogeneity across
effects using grade level, instrument type, length of game-based intervention, country, publication type and study year characteristics.
Overall findings indicate that video games are a slightly effective instructional strategy for teaching mathematics across PreK-12th
grade levels.
Keywords: digital games, mathematics education, academic achievement, meta-analysis
1. Introduction
Learning mathematics presents various challenges for many children due to the difficult and often tedious nature of the subject
(Sedig, 2008). Educational video games have the potential to address these challenges and positively impact mathematics learning
and attitudes. Video games are able to consume children’s attention for hours while providing instruction and an engaging learning
experience. Video games have been used to promote children’s mathematics achievement in various domains including problem-
solving and algebra skills (Abramovich, 2010), strategic and reasoning abilities (Bottino, Ferlino, Ott, & Tavella, 2007), critical
geometry skills (Yang & Chen, 2010), and arithmetic procedures (Moreno & Duran, 2004). Nevertheless, the National Mathematics
Advisory Panel (NMAP, 2008) and others (e.g., Martinez-Garza, Clark, & Nelson, 2013; Pellegrino & Hilton, 2012; Young et al.,
2012) do not provide a direct recommendation for using games “as a potentially useful tool in introducing and teaching specific
subject-matter content to specific populations” (NMAP, 2008; p. 51) due to the limited number of rigorous studies exploring effects of
game-based learning on mathematics skills development. This meta-analysis addresses this concern by systematically examining the
effects of mathematics video games used in PreK-12 curricula on student mathematics achievement compared to traditional, non-
video game-based classroom instructional methods (i.e., media comparisons). In addition, we assess several possible moderating
effects: Grade Level, Instrument Type, Length of Game-Based Intervention, Country, Publication Type and Publication Year.
In spite of the increased popularity of educational video games over the last two decades, empirical research on the effects of
math video gaming on student academic achievement is inconsistent. For example, Kebritchi (2008) found that high school students
who interacted with the mathematics video game DimensionM outperformed their non-gaming peers. Beserra, Nussbaum, Zeni,
Rodriguez, and Wurman (2014) examined third grade students’ arithmetic performance in game-based and traditional classroom
conditions in three different countries (Brazil, Chile, and Costa Rica). The authors found that game-based learning was more effective
than a traditional classroom approach. Conversely, several studies did not show positive benefits of using video games in a
mathematical classroom (e.g., Costabile, De Angeli, Roselli, Lanzilotti, & Plantamura, 2003; Gelman, 2010; Jones, 2011). For
instance, Ferguson (2014) found that traditional instruction was more effective than game-based instruction for high school Algebra 1
students. Swearingen’s (2011) research showed that both game-based and traditional instructional approaches were equally beneficial
for teaching high school mathematics. Moreover, not only did different research teams that used different games for promoting
distinct learning outcomes report mixed results, some findings by the same researchers who used the same math video games were
inconsistent (e.g., Ke, 2008a, 2008b, 2008c; Ke & Grabowski, 2007).
A meta-analytic review that quantitatively integrates findings of math video-gaming studies can provide an understanding of
the effectiveness of game-based learning for student mathematics achievement. Previous computer-assisted instruction (CAI) meta-
analyses focused on a broad range of interactive technologies, including drill-and-practice programs, web-based learning materials,
simulations, virtual reality technologies, digital visualization tools, and video games (e.g., Bayraktar, 2001; Chambers, 2002;
Christmann & Badgett, 2003; Cohen & Decanay, 1992). More recent media comparisons meta-analyses refine and focus on video
games, but due to the paucity of video gaming research, video games are lumped together with simulations and virtual reality
technologies (e.g., Merchant, Goetz, Cifuentes, Keeney-Kennicutt, & Davis, 2014). Like older CAI reviews, these meta-analyses did
not set game-based learning apart from learning with other interactive technologies, such as simulations and virtual reality. Moreover,
due to scarce empirical video gaming research these CAI meta-analyses examined general academic achievement or combined
mathematics and science academic achievement, without explicitly focusing on mathematics education. As such, no findings about
the learning effectiveness of video games in mathematics education were reported.
Several meta-analyses have attempted to quantitatively synthesize findings of empirical research on game-based learning and
academic achievement (e.g., Connolly et al., 2012; Young et al., 2012). However, due to methodological challenges associated with
the shortage of rigorous research in game-based learning, a qualitative synthesis of video gaming studies was implemented instead of
the initially planned meta-analysis. In addition, recent media comparisons meta-analyses (e.g., Clark, Tanner-Smith, & Killingsworth,
2016; Merchant et al., 2014; Vogel, Greenwood-Ericksen, Cannon-Bowers, & Bowers, 2006) spanned multiple content areas without
addressing the specific instructional needs and requirements of a single subject area (such as mathematics).
2. Definition of Learning Video Games
The term "game" refers to a learning video game. Garris et al. (2002) note that there is little consensus in the literature on how
to define educational games. Some consider Caillois’s (1961) definition as the most comprehensive analysis of games, characterizing
a game as “an activity that is voluntary and enjoyable, separate from the real world, uncertain, unproductive in that the activity does
not produce any goods of external value, and governed by rules” (p. 442).
In this meta-analysis, we distinguish between games and simulations, as simulations and games offer different learning
affordances. Simulations model real systems by displaying a procedure or phenomenon (Alessi & Trollip, 2001) and do not
necessarily include learning objectives, but do allow students to interact with the simulation and observe how variable manipulation
affects the observed phenomenon. Games, on the other hand, are designed to motivate students and create game-like learning
experiences. Games often encourage competition by setting clear objectives to score as many points as possible, move up in difficulty
level, or win in general (Young et al., 2012). For this meta-analysis we use Shute and Ke (2012) definition of video games, which
suggests that good games must have interactive problem solving, specific goals/rules, adaptive challenges, control, ongoing feedback,
uncertainty that evokes suspense and player engagement, and sensory stimuli (a combination of graphics, sounds, and/or storyline
used to excite the senses). These gaming characteristics are essential for creating an effective learning environment that enhances
students’ engagement and facilitates knowledge and skills acquisition.
To examine the learning effectiveness of mathematics video games that included game attributes suggested by Shute and Ke
(2012), this meta-analysis focused on modern digital games – the latest generation of learning video games developed using advanced
technologies and recent pedagogical approaches (Kebritchi, Hirumi, & Bai, 2010). Unlike the earlier generation of educational games
created in 1980s and 1990s, modern learning video games offer more advanced graphics and interface design, telecommunication and
networking capabilities, immersive 3D learning environments, visual and audio effects, multi-player options, and a learner-centered
approach, all of which are believed to increase gameplay motivation.
3. Aims of the Meta-analysis
This meta-analysis examined the relationships between game-based learning for mathematics skills development and student
mathematics achievement in PreK-12th grades. Specifically, we assessed the relative effectiveness of game-based interventions
compared to a traditional, non-video game-based classroom instruction. Our work was motivated by previous meta-analyses that
emphasized the importance of studying factors which can influence the relationships among mathematics game-based learning and
academic achievement, including mathematics skills and knowledge promoted in a game, game design elements, and design of game-
based interventions (Clark et al., 2016; de Boer, Donker, & van der Werf, 2014; Sitzmann, 2011).
de Boer et al. (2014)’s meta-analysis revealed that instructional intervention methodology influenced students’ academic
performance. For instance, factors such as cooperative/individual learning and implementer of instructional interventions moderated
the intervention effect. This area of research is certainly relevant and important for mathematics video game research, as the
implementation of game-based interventions can directly affect students’ mathematics achievement and engagement.
We conducted an extensive search of published and unpublished research on mathematics video gaming (described in detail
later) in order to collect studies that compared the effectiveness of video gaming in mathematics with a traditional, non-video game-
based classroom approach. Our initial list of study characteristics included:
study participant characteristics (age, gender, race, learning disabilities, and socio-economic status)
general study characteristics (length of game-based intervention, implementer of game-based intervention, teacher training on
game-based instruction, teacher’s familiarity with the learning game(s), amount of time students spent interacting with the
game(s) before and after the video game intervention)
game characteristics
mathematics instructional approaches (lecture-based, inquiry-based, drill and practice)
mathematics skills and knowledge promoted in the game
publication characteristics (location of study, publication year and type)
research characteristics (assessment type, outcome format, number of test items, instrument reliability)
However, this list of moderator variables was eventually reduced to six: grade level, instrument type, length of game-based
intervention, country, publication year and type. The main reason for this reduction was due to methodological challenges and
partially reported study characteristics and study methods. Specifically, classifying video games and math instructional approaches
with regard to their characteristics and types presented considerable challenges for several reasons. First, many studies reported
longitudinal research interventions that took place over an academic quarter or semester (e.g., Carr, 2012; Din & Caleo, 2000;
Swearingen, 2011; Weiss et al., 2010). These studies employed multiple video games for teaching mathematics and likely used
various mathematics classroom instructional approaches. In addition, studies often failed to describe the type of mathematics
classroom instruction used for the so-called "traditional mathematics classroom instruction.” We also considered classifying video
games as general-purpose commercial games and serious games that are “designed with the intention of improving some specific
aspect of learning” (Derryberry, 2010). However, a vast majority of video games employed in the selected studies were serious games
designed to improve specific math skills. Only three studies used general-purpose commercial games (Brain Age 2 in Gelman, 2010;
My Sims in Hawkins, 2008; Sims 2 in Panoutsopoulos & Sampson, 2012). As such, for a quantitative comparison perspective this
study characteristic was excluded. Furthermore, we attempted to code the studies with regard to math content (e.g., geometry,
arithmetic, algebra). However, due to considerably different school settings and paucity of experimental and quasi-experimental
research on math game-based learning, we were unable to classify math content.
In particular, the present study addressed the following research questions:
RQ1: What is the overall relative learning effectiveness of game-based interventions as compared to a traditional, non-video
game-based classroom instruction for student mathematics achievement in PreK-12th grades?
RQ2: How heterogeneous are results from studies on the overall relative learning effectiveness of game-based interventions as
compared to a traditional, non-video game-based classroom instruction for student mathematics achievement in PreK-12th grades?
RQ3: To what extent do study characteristics, namely grade level, instrument type, length of game-based intervention, country,
publication year and type, moderate the effect?
4. Method
4.1 Literature search
Searches of ERIC, PsycINFO, Wilson, Google Scholar, JSTOR, and ISI Web of Science databases were performed to collect
empirical studies, peer-reviewed journals, book chapters, thesis and dissertations, and conference papers, focusing on the effects of
computer games on student mathematics achievement. The following keywords were used: computer games, electronic games, video
games, computer software, mathematics achievement, mathematics education, number sense, numerical skills, numbers, experiment,
and experimental studies. The initial search located 860 studies.
4.2 Inclusion criteria
All studies from the initial search were examined by two reviewers and assessed for inclusion in the meta-analysis using the
following criteria:
1. Publication year range from 2000-2017
2. Study employed game-based and traditional, non-video game-based classroom instructional interventions
3. Study used at least one gamed-based classroom and one traditional classroom
4. Study participants were PreK-12th grade students
5. Student mathematics achievement was used as an outcome
6. It was possible to infer that the video games could be characterized as “good video games” (Shute & Ke, 2012)
7. Study reported sufficient data to calculate effect sizes
For the mathematics achievement outcomes requirement, eligible studies were required to measure students’ mathematics
performance. Studies focusing on related or general learning outcomes and benefits (e.g., creativity, cognitive strategies, self-efficacy,
work ethics, enjoyment, motivation) were excluded. Upon review of the initial set of 860 studies there were 48 studies that satisfied
the first five inclusion criteria items. However, many studies did not include a clear description of the employed learning games.
These studies were further examined by the first two authors to determine if they could be characterized as learning games and if they
included Shute and Ke’s (2012) attributes of good learning games. The first author’s field of expertise is Educational Research
Methods, Measurement and Statistics, and Instructional Design. The second author specializes in Instructional Design and
Technology with a research focus on video games. The first step in determining whether the employed video games could be
characterized as “good video games” was to search the Web, including video game websites and YouTube videos about the games. If
the initial Web search was insufficient, we searched whether the games were mentioned in the literature with regard to the game
description, development, and research uses.
Moreover, some studies did not report sufficient data to calculate effect sizes. A total of 24 studies satisfied the inclusion criteria
and were included in the meta-analysis. From these 24 studies we extracted 39 statistically independent effect sizes. The reason for
having more effect sizes than the number of studies is that, in nine studies there were multiple groups (i.e., pairs) that produced more
than one effect size. Unlike instances were, say, multiple outcomes are provided by the same sample of students (e.g., one reading
outcome and one math outcome), these nine studies provided statistically independent effect sizes from multiple groups. Figure 1
provides a flowchart of the inclusion and exclusion decisions that lead to the final dataset.
4.3 Selection of variables
Part of our methods included moderator analyses of select study characteristics. The following moderator variables were
categorized based on the common characteristics of the selected studies.
4.3.1 Grade level. Grade Level was a categorical factor consisting of three levels: Preschool-Kindergarten students (or its
equivalent if outside of the United States), 1st-6th grade students, and 7th grade and above. This variable evaluated whether the
effectiveness of mathematics game-based learning varied across different grade levels as to accommodate for a continued increase in
difficulty of mathematics skills and decrease in student motivation to learn from preschool-elementary to middle-high school settings
(Harter, 1981).
4.3.2 Instrument type. The instrument type variable classified the instrument types used in studies. This moderator consisted
of three levels, the first of which was researcher-made scale (surveys, questionnaires, and tests created or partially created by
researchers of the study). If researchers used selected questions or portions from a standardized instrument or large-scale assessment,
this was also considered as a researcher-made instrument. The rationale for this is that once an instrument is altered from its original
form, psychometric qualities often fluctuate and the instrument is no longer presented as intended by the original instrument creator(s).
The other two factor levels were commercial/standardized test and research-based scale. Commercial/standardized tests
consisted of utilitarian standardized instruments and large-scale assessments. These instruments have long-standing validity,
reliability, and psychometric properties which are generally accepted in research literature. Alternatively, research-based scales are
instruments previously used by researchers and are generally accepted and used in studies in their respective field. Overall, the
Instrument Type moderator variable was used in order to examine whether using different types of instruments (research-made,
commercial/standardized, and research-based scales) had an effect on student math achievement and showed significant differences
among studies.
4.3.3 Length of game-based intervention. This moderator represented the duration of the game-based intervention in order to
determine whether the intervention length contributed the effectiveness of game-based learning. Intervention length was categorized
into three levels: up to one hour, between one hour and eight hours, and over eight hours.
4.3.4 Country. A dummy variable was created for the country moderator: Studies completed in the United States and studies
completed outside of the United States. The United States and other countries have different educational systems. This moderator
variable examined the effect of these differences on the study outcomes.
4.3.5 Publication year. We observed a somewhat large increase in the amount of video-gaming research over the last decade.
Thus, the publication year moderator assessed differences among study results over time. We scaled publication year (i.e., first year
of publication within the set of studies we set to zero, then next publication year to one, and so forth) for ease of interpretation.
4.3.6 Publication type. As is often in meta-analysis we assessed any differences of effects based on the publication type
classification of a study. The publication type of a study consisted of two factor levels: journal and thesis/dissertation. Journal articles
were considered published documents and theses and dissertations were considered unpublished documents.
4.4 Effect sizes and variances
The effect-size metric was the standardized mean difference. Some studies had relatively small sample sizes, thus we opted to
use an unbiased version of the standardized mean difference proposed by Hedges (1981). The unbiased sample standardized mean
difference for the th of studies is
(1)
where
and
are respective mean mathematics achievement outcomes for the treatment and control groups,
is a pooled
standard deviation, and
and
are respective treatment and control sample sizes. A positive effect size is interpreted as a mean
difference favoring the treatment group and a negative effect size favoring the control group. In all instances the treatment group
refers to the group of students who received a mathematics video-gaming intervention.
For five effect sizes, insufficient group-mean results were reported. In these cases we computed the standardized mean
difference as
(2)
where is the statistic from a one-way analysis of variance (see Borenstein, 2009). After computing effect-size estimates using (1)
or (2), we calculated the sample variance of the estimate as
(3)
where all terms have been previously defined.
4.5 Effect-size homogeneity
An important step when conducting a meta-analysis is determining the degree of homogeneity of a collection of effects.
Several homogeneity measures were computed to assess the similarity of effects. First we computed the statistic (Hedges, 1982), a
common index for assessing homogeneity in meta-analysis,
(4)
where is the fixed-effect (i.e., inverse-variance weighted) mean. Under the null hypothesis of effect-size homogeneity, follows
an approximate chi-square distribution with degrees of freedom. Larger values correspond to more disagreement among
effect sizes.
The second homogeneity measure was (Higgins & Thompson, 2002; Higgins, Thompson, Deeks, & Altman, 2003),
computed as
(5)
Rough interpretations of are no variation, low variation, moderate variation, and high variation for values 0, 25, 50, and 75,
respectively (Higgins et al., 2003).
4.6 Unconditional random-effects model
Based on the results from homogeneity tests (shown in the Results section), as well as our intention to generalize results to
more than the set of the collected studies, we adopted a random-effects model when making inferences to the overall results. A
benefit of the random-effects model is the incorporation of a non-zero variance parameter, , which represents a between-studies
heterogeneity. This between-studies variance components represents the degree of heterogeneity among the study-specific effects
(i.e., not at the primary-study level). Furthermore, under a random-effects model we allow individual study effects to differ across
studies. The random effects model used in our research distributional form is
(6)
where represents the true value of the th effect size, population parameter of the overall mean of effects, and is the between-
studies variance parameter (estimated using restricted maximum likelihood).
For the overall effect we computed a random-effects mean, its variance, and the associated 95% confidence interval. The
random-effects mean estimate was calculated as
(7)
with an estimated variance
(8)
4.7 Moderator analyses
As a follow-up to our unconditional overall models, we considered two types of conditional random-effects models to explain
systematic effect-size heterogeneity: ANOVA-like models and meta-regression. Both modeling techniques use effect sizes as
dependent variables and study characteristics as independent variables. In total, we examined six study characteristics (Grade Level,
Instrument Type, Length of Game-Based Intervention, Country, Publication Year and Type) as moderators of effect sizes. Of the six
coded variables, five were categorical (analyzed using ANOVA-like models) and one was continuous (analyzed using meta-
regression).
For the ANOVA-like models, we report within-group effect means and their standard errors, as well as 95% confidence
intervals. We also report two forms of chi-square statistics, and . These two measures assess predictor-specific significance in
terms of explaining systematic effect-size heterogeneity ( and within-group variability of effects (). Because is group-
specific, each group within a moderator (i.e., factor level) will have an estimate, while will have a single value for each specific
factor.
The meta-regression results include coefficients and their standard errors, as well as 95% confidence intervals. Similar to the
ANOVA-like modeling, we also provide two chi-square statistics, and . Both are related measures of a model fit, with larger
values of and smaller values of corresponding to greater explanatory power of the effect-size heterogeneity by the set of model
predictors.
4.8 Publication bias
Publication bias concerns studies with statistically significant and/or larger effects having publication preference over smaller
and/or non-statistically significant effects (see Rothstein, Sutton, & Borenstein, 2006). We used several methods to check for
publication bias. First, we provided a funnel plot (Figure 3) and assessed the expected relationship between effect-size magnitudes
and their standard errors. Along with this graphic, we used three quantitative assessment methods: Trim and Fill (Duval & Tweedie,
2000), Egger’s regression test (Egger et al., 1997), and Failsafe-N (Rosenthal, 1979). None of these tools prove the presence or
absence of publication bias, rather they collectively indicate the likelihood of publication bias. All analyses and graphics were
completed in R (R Core Team, 2016) using the meta package (Schwarzer, 2015) and metaphor package (Viechtbauer 2010).
5. Results
The 39 effect sizes in the meta-analysis ranged from -0.73 to 0.87, with roughly 67% of point estimates being positive (i.e.,
favorability to the video-gaming instruction group). Of the 24 unique studies, 9 contributed multiple (but statistically independent)
effect sizes. Primary study sample sizes ranged from 41 to 437 students (combined intervention and control group samples) with
publication dates from 2008 to 2016. Based on point estimates and confidence intervals shown in Figure 2, the variability of effect-
size magnitudes and precision appear somewhat heterogeneous. Not only do the effect-size magnitudes vary, the range spanned both
positive and negative sides of the spectrum, further suggesting a diverse collection of effects. Both statistical assessments of
homogeneity, and , supported parts of our interpretations of Figure 2 regarding the effect-size
heterogeneity. Given these results and our generalizability intentions when answering our research questions, we forwent using a
more restrictive fixed-effect model and adopted random-effects models when making statistical inferences. Overall and moderator
results are shown in Table 1.
The overall random-effects weighted effect size was , with an associated 95% confidence interval of [0.02,
0.24]. The overall effect of video-gaming instruction on mathematical achievement was marginally significant and quite variable, as
denoted by the rather wide confidence interval and relatively large standard error (). Furthermore, Figure 2 suggests that
effects likely vary from study-to-study. This is supported by the between-studies variability estimate of .
When assessing publication bias, all indicators lead to a small likelihood of publication bias. The funnel plot (Figure 3) shows
a moderate amount of effect-size symmetry. Trim-and-Fill results required no imputed effects to achieve asymmetry, which also
aligns with the lack of statistical significance of Egger’s regression test (). Last, the Failsafe-N assessment required
141 additional non-statistically significant effects to be added for our overall mean results for its observed significance level to a non-
statistically significant level (i.e., ).
5.1 Moderator analysis results
In this section we discuss results of moderator analyses. The selected study characteristics had varying degrees of an
explanatory power of the effect-size variability. We discuss each of the six moderators separately.
5.1.1 Grade Level. The grade level moderator did not explain a statistically significant amount effect-size heterogeneity
(). However, variability within groups was statistically significant for two of the three groups. More
specifically, effect sizes varied within the 1st Grade – 6th Grade group () and the 7th grade and above group
(). Effect-size variability was not statistically significant for the Preschool-Kindergarten group (
). This result, in terms of statistical significance is dependent on the choice of Type I error level (i.e., ). In our case we
choose to be more conservative and not assume statistical significance rather than risk the inflation of a result (i.e., assume statistical
significance).
5.1.2 Instrument type. A non-significant amount of the effect-size heterogeneity was explained by the instrument type
moderator (). Thus, the use of different measurement instruments for the assessment of mathematics
achievement did not seem to impact the effect of the intervention. However, two of the three measure types did show significant
within-group variability, specifically both commercial/standardized tests () and researcher-made scales
(). The effect-size variability was not statistically significant for the researcher-based scales (
).
5.1.3 Length of game-based intervention. Like the grade level, country, and instrument type moderators, the length of game-
based intervention did not explain a statistically significant amount of effect-size heterogeneity (). However,
all three groups showed a within-group variability: up to one hour (), between one hour and eight hours
(), and over eight hours ().
5.1.4 Country. Similar to the grade level moderator, we did not find a significant amount of explained effect-size
heterogeneity for the country moderator (). Nevertheless, variability within groups was found to be statistically
significant. For studies from the United States, the homogeneity test statistic was and for studies outside of
the United States the homogeneity statistic was .
5.1.5 Publication type. The publication type moderator was one of the few moderators found to have explained a significant
effect-size heterogeneity (). This indicates that effect sizes varied between the type of study (journal or
thesis/dissertation). The mean effect of the journal group was , while the mean for the thesis/dissertation group was
, showing a disagreement in both magnitude and direction. Moreover, the variance within groups differed between
the source types. While the variance of effects within the thesis/dissertation type was not statistically significant (
), the variance of effects within the journal type was significant ().
5.1.6 Publication year. The publication year variable was scaled so that the year 2000 (i.e., the earliest year of collected
studies) was set to 0, then 2001 set to 1, and so forth. Our analysis revealed that the publication year slightly influenced the amount of
the effect-size variability (). The slope from this regression model was , indicating a small
increase in effect-size magnitude (i.e., an increase in the effect of video-gaming instruction on mathematics achievement) as the
publication year of a study increased. However, the publication year moderator did not explain all effect-size variability (
).
Table 1
Moderator Analysis Results
Moderator
Mean(SE)
95% CI
Overall
39
0.13(0.06)
[0.02, 0.24]
Grade Level [(2) = 4.00, p = .14]
Preschool – Kindergarten
3
0.58(0.25)
[0.09, 1.06]
6.00
1st Grade – 6th Grade
26
0.13(0.07)
[0.00, 0.27]
52.84****
7th grade and Above
10
0.05(0.10)
[-0.14, 0.24]
24.22**
Country [(1) = 0.29, p = .60]
United States
17
0.10(0.08)
[-0.06, 0.26]
35.99**
Other
22
0.16(0.08)
[0.00, 0.32]
54.20***
Instrument Type [(2) = 3.01, p = .22]
Commercial/Standardized Test
10
0.03(0.10)
[-0.17, 0.23]
28.10***
Research-based Scale
11
0.29(0.11)
[0.07, 0.50]
13.62
Researcher-made Scale
18
0.11(0.08)
[-0.05, 0.27]
41.51***
Length of Game-Based Intervention [(2) = 2.51, p = .28]
Up to one hour
11
-0.03(0.12)
[-0.27, 0.21]
25.04**
Between one hour and eight hours
21
0.19(0.08)
[0.05, 0.34]
41.05**
Over eight hours
7
0.12(0.13)
[-0.13, 0.37]
22.92***
Publication Type [(1) = 6.49, p = .01]
Journal
30
0.21(0.06)
[0.10, 0.33]
58.92***
Thesis or Dissertation
9
-0.07(0.09)
[-0.25, 0.11]
11.14
Coefficient(SE)
95% CI
Publication Year [(1) = 4.46, p = .03]
39
0.01(0.01)
[0.00, 0.02]
79.88***
*p < .05; **p < .01; ***p < .001
Note: Means within groups are weighted under the mixed-effects model; indicates a specific group; Regression coefficient is standardized.
6. Discussion
The empirical research on video games in mathematics education remains limited (Connolly et al., 2012) and our present study
has further confirmed the paucity of research in this area. Despite a considerably large number of the reviewed studies (over 800),
only 24 studies that compared mathematics game-based learning with traditional instructional methods were included in the meta-
analysis (see Table 2). To generalize beyond the collection of studies in this meta-analysis, two types of random-effects models were
used. One random-effects model provided an unconditional representation of the overall effect of video-gaming instruction on
mathematics achievement. The second set of random-effects models looked at potential explanatory factors of systematic effect-size
variation.
6.1 Overall Effectiveness
Our first research question was what is the overall relative learning effectiveness of game-based interventions as compared to a
traditional, non-video game-based classroom instruction for student mathematics achievement in PreK-12th grades? A small but
marginally significant overall effect ( ) suggests that mathematics video games contribute to a higher degree of
mathematics achievement compared to traditional instructional methods. Although previous meta-analyses did not focus specifically
on mathematics achievement, our findings converge with previous media comparisons meta-analyses that revealed benefits of CAI
relative to non-CAI conditions (Clark et al., 2016; Merchant et al., 2014; Vogel et al., 2006).
Table 2
Study Characteristics Included in Meta-Analysis
Study
Sample
Size
Grade
Level
Country
Instrument Type
Game(s)
Length of Game-
based
Intervention
Bai et al. (2012)
437
7th-12th
USA
researcher-made
instrument
DimensionM
long
Beserra et al. (2014)
271
1st-6th
Brazil,
Chile, and
Costa Rica
research-based scale
Researcher-developed game
medium
Carr (2012)
104
1st-6th
USA
researcher-made
instrument
iPad math games
long
Chang et al. (2012)
92
1st-6th
Taiwan
commercial/standardized
test
Researcher-developed game
medium
Chang et al. (2015)
306
1st-6th
USA
commercial/standardized
test
Researcher-developed game
medium
Ferguson (2014)
222
7th-12th
USA
researcher-made
instrument
Slope game
medium
Garneli et al. (2016)
80
1st-6th
Greece
commercial/standardized
test
Gem-game
short
Gelman (2010)
80
7th-12th
USA
researcher-made
instrument
Brain Age 2, Nintendo DS
long
Giannakos (2013)
87
7th-12th
Norway
research-based scale
Gem-game
medium
Hall (2015)
405
1st-6th
USA
researcher-made
instrument
iPad multiplication games
medium
Hawkins (2008)
139
1st-6th
USA
researcher-made
instrument
MySims Wii, Nintendo Wii
medium
Hung et al. (2014)
69
1st-6th
China
research-based scale
Researcher-developed Brick
Breaker game
medium
Ke (2008)
358
1st-6th
USA
researcher-made
instrument
ASTRA EAGLE (a series of
web-based computer games;
academic content is based on
the Pennsylvania System of
School Assessment (PSSA)
standards for mathematics)
medium
King (2011)
128
7th-12th
USA
research-based scale
VmathLive (academic content is
based on the National Council
of Teachers of Mathematics
(NCTM) standards)
medium
Lin et al. (2013)
64
1st-6th
Taiwan
research-based scale
Researcher-developed
Monopoly game
short
Panoutsopoulos &
Sampson (2012)
57
1st-6th
Greece
commercial/standardized
test
Sims 2–Open for Business
medium
Pareto et al. (2012)
47
1st-6th
Sweden
researcher-made
instrument
Researcher-developed
Teachable Agents game
long
Ploger & Hecht (2009)
196
1st-6th
USA
research-based scale
Chartworld
medium
Sedig (2008)
59
1st-6th
Canada
research-based scale
Researcher-developed Super
Tangrams game
medium
Shin et al. (2012)
41
1st-6th
USA
research-based scale
Skills Arena
long
Starkey (2013)
168
7th-12th
USA
researcher-made
instrument
Lure of the Labyrinth
medium
Sung, Chang & Lee
(2008)
60
PreK-K
Taiwan
commercial/standardized
test
Researcher-developed game
short
Swearingen (2011)
280
7th-12th
USA
research-based scale
Researcher-developed massive
multiplayer online game
(MMOG)
long
Weiss et al. (2006)
116
1st-6th
Israel
research-based scale
Goldilocks series games
long
EFFECTS OF GAME-BASED LEARNING 31
Related to this, our second research question asked how variable are results from studies
in terms of learning effectiveness of game-based interventions as compared to a traditional, non-
video game-based classroom instruction for student mathematics achievement? Based on our
assessment of effect-size heterogeneity (, , and ), evidence pointed to disagreement among
study results. In other words, not all studies agreed on the effectiveness of video-gaming
instruction compared to traditional instruction with respect to measures of mathematical
achievement.
6.2 Effectiveness by Study Characteristics
To further explore potential reasons for this effect-size heterogeneity, our third research
question asked to what extent did study characteristics (grade level, country, instrument type,
length of game-based intervention, publication type and publication year) moderate the effect.
Of the six moderators used in our study, only two (publication type and publication year) had
significant explanatory power with respect to effect-size heterogeneity.
6.2.1 Grade Level
With regard to grade level, results suggest that mathematics video games were similarly
beneficial for students from various grade levels. These findings converge with Vogel et al.
(2006) media comparisons meta-analysis that examined the effects of games and simulations on
student general performance across various disciplines. Vogel et al. (2006) found that interactive
games and simulations were more beneficial for cognitive gains than traditional teaching
methods for all age groups. Our literature search revealed only three published studies
contrasting game-based learning and traditional classroom instruction in preschool-kindergarten
students. Thus, the findings should be interpreted with caution when generalizing to the PreK-K
EFFECTS OF GAME-BASED LEARNING 32
population since the results were obtained for the most part from studies with 1st-12th grade
students.
6.2.2 Length of Game-Based Intervention
Studies used in the meta-analysis varied considerably in terms of the length of the game-
based interventions (a single game session of 33 minutes as the shortest and multiple game
sessions with a total of 10080 minutes as the longest). In line with previous research (Clark et
al., 2016; Merchant et al., 2014), the length of game-based interventions did not have significant
explanatory power. This finding may appear counterintuitive – one might expect that longer
interventions should be more effective than shorter interventions. However, prior research on
the effects of interventions in mathematics classrooms reports that intervention duration has only
a small impact on students’ academic achievement in both primary and secondary schools
(Dignath & Büttner, 2008).
6.2.3 Instrument Type
Similar to Clark et al. (2016) and Merchant et al. (2014), this meta-analysis revealed non-
significant differences among the studies that utilized researcher-made scales,
commercial/standardized tests, or research-based scales. Although previously validated
instruments such as commercial/standardized tests and research-based scales are usually
associated with higher quality, some video-gaming studies assess learning outcomes for which
validated instruments were unavailable. We note that if a study utilized only selected questions
or portions from standardized instruments or large-scale assessments, this was considered as a
researcher-made instrument in the present meta-analysis. Given that the present and previous
meta-analyses found no relationship among different types of assessment instruments and
academic achievement, we believe that utilizing researcher-made scales when pre-existing
EFFECTS OF GAME-BASED LEARNING 33
validated instruments are unavailable would not necessarily diminish the quality of video game
research.
6.2.4 Publication Year
The publication year of a study had significant explanatory power with respect to effect-
size heterogeneity. Though only to a small degree, the effect of game-based learning on
mathematics achievement increased as the year of publication increased. There are several
possible explanations for this finding. First, because game-based learning is a developing area of
research, it is plausible to assume that game-based interventions utilized in more recent studies
were informed by previous video gaming research and therefore capitalized on past results.
Furthermore, the video games industry is constantly seeking new ways to produce more
engaging and higher quality games. These innovations could possibly contribute to higher
mathematics achievement as well.
6.2.5 Publication Type
In addition to publication year, the publication type moderator demonstrated that the
effect of video-gaming instruction on mathematics achievement differed between published and
unpublished studies. The set of published studies had a statistically significant mean effect size
which was positive and small-to-moderately large in magnitude. Conversely, the collection of
unpublished studies had a mean effect size which was not statistically different from zero. This
implies that published studies tended to claim larger effectiveness of the video-gaming
intervention than their unpublished studies counterparts.
7. Conclusions and Recommendations
This meta-analysis highlighted the need for more empirical research on mathematics
video games in order to deepen our understanding of how video games can enhance mathematics
EFFECTS OF GAME-BASED LEARNING 34
learning. Our initial intention was to examine various factors that could affect the relationship
among mathematics game-based learning and academic achievement, including student
individual differences, video game design characteristics, and attributes of video game-based
interventions. However, most of the identified mathematics video game studies only provided
partial information about the video games and game-based instructional interventions, thus
limiting our ability to systematically examine the effects of several moderator variables. To
advance research on mathematics game-based learning, we urge authors to include more detailed
descriptions of research procedures and assessment instruments, as well as information about
learning game(s) and expected learning outcomes. For example, it is important to report how the
employed game(s) align with the classroom curriculum, the amount of video game training that
teachers received, teacher familiarity with the game(s), how the video game intervention was
implemented and who implemented it, the duration and frequency of video game interventions,
and specific skills and knowledge promoted in the game.
With regard to future meta-analysis research on mathematics video games, there is a need
to examine how video games facilitate acquisition of mathematics skills and concepts within
different mathematical domains (e.g., geometry, arithmetic, algebra). Examining how video
games facilitate acquisition of various skills can advance our understanding of how to select an
optimal video game for enhanced learning of specific mathematics concepts and skills. Thus,
future research should attempt to examine whether mathematics learning tasks can explain the
relationships between video gaming and student achievement.
Clark et al. (2016) emphasized the importance of studying the relationships among game
design and learning outcomes. This is certainly true for mathematics video game research. We
should devote more attention to connecting game design characteristics with specific learning
EFFECTS OF GAME-BASED LEARNING 35
outcomes across various mathematical domains. However, current literature reviews suggest
that this type of research investigation can be a challenging task. Earlier research on educational
video games usually employed a single video game for an instructional intervention, which
allowed for a focus on game design and its impact on learning and engagement. However,
technological advances in digital video games created new opportunities and expectations for
teaching and learning. More recent studies implemented mathematics game-based learning using
a series of video games or multiple video-gaming apps that utilized various game designs and
genres and were played in the same game sessions, thus making the task of examining the role of
game design quite difficult, if not impossible.
Last, studying the impact of game-based intervention attributes on student achievement
can improve the quality of video game research in general and mathematics video gaming in
particular (de Boer et al., 2014). This area of research is limited within the game-based learning
literature. Identifying specific attributes of game-based interventions and examining
relationships among these attributes and learning outcomes would be an important contribution
to the literature on video games.
Funding
This research did not receive any specific grant from funding agencies in the public,
commercial, or not-for-profit sectors.
EFFECTS OF GAME-BASED LEARNING 36
References
References with an asterisk indicate studies included in the meta-analysis.
Abramovich, S. (2010). Topics in mathematics for elementary teachers: A technology-enhanced
experiential approach. Charlotte, NC: Information Age Publishing, Inc.
Alessi, S. M., & Trollip, S. R. (2001). Multimedia for learning: Methods and development. New
Jersey: Pearson.
*Bai, H., Pan, W., Hirumi, A., & Kebritchi, M. (2012). Assessing the effectiveness of a 3-D
instructional game on improving mathematics achievement and motivation of middle
school students. British Journal of Educational Technology, 43(6), 993-1003.
doi:10.1111/j.1467-8535.2011.01269.x
Bayraktar, S. (2001). A meta-analysis of the effectiveness of computer-assisted instruction in
science education. Journal of Research on Technology in Education, 34, 173-188.
*Beserra, V., Nussbaum, M., Zeni, R., Rodriguez, W., & Wurman, G. (2014). Practising
arithmetic using educational video games with an interpersonal computer. Educational
Technology & Society, 17 (3), 343–358.
Borenstein, M. (2009). Effect sizes for continuous data. In H. M. Cooper, L.V. Hedges, & J. C.
Valentine (Eds.), The handbook of research synthesis and meta-analysis. (pp. 221-235).
New York: Russell Sage Foundation.
Bottino, R. M., Ferlino, L., Ott, M., & Tavella, M. (2007). Developing strategic and reasoning
abilities with computer games at primary school level, Computers & Education, 49(4),
1272-1286.
EFFECTS OF GAME-BASED LEARNING 37
Caillois, R. (1961) Man, Play and Games. Translated by Meyer Barash. New York: Free Press.
*Carr, J. M. (2012). Does math achievement h'APP'en when iPads and game-based learning are
incorporated into fifth-grade mathematics instruction? Journal of Information Technology
Education, 11, 269-286.
Chambers, E. A. (2002). Efficacy of educational technology in elementary and secondary
classrooms: A meta-analysis of the research literature from 1992-2002. (PhD
Dissertation), Southern Illinois University, Carbondale.
*Chang, K.-E., Wu, L.-J., Weng, S.-E., & Sung, Y.-T. (2012). Embedding game-based problem-
solving phase into problem-posing system for mathematics learning. Computers &
Education, 58(2), 775-786.
*Chang, M., Evans, M. A., Kim, S., Norton, A., & Samur, Y. (2015). Differential effects of
learning games on mathematics proficiency. Educational Media International, 52(1), 47-
57.
Christmann, E. P., & Badgett, J. L. (2003). A meta-analytic comparison of the effects of
computer-assisted instruction on elementary students’ academic achievement.
Information Technology in Childhood Education Annual, 15, 91-104.
Clark, D. B., Tanner-Smith, E., & Killingsworth, S. (2016). Digital games, design, and learning:
A systematic review and meta-analysis. Review of Educational Research, 86(1), 79-122.
Cohen, P. A., & Dacanay, L. S. (1992). Computer-based instruction and health professions
education: A meta-analysis of outcomes. Evaluation and the Health Professionals, 15(3),
259-281.
EFFECTS OF GAME-BASED LEARNING 38
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic
literature review of empirical evidence on computer games and serious games.
Computers & Education, 59, 661-686.
Costabile, M., De Angeli, A., Roselli, T., Lanzilotti, R., & Plantamura, P. (2003). Evaluating the
educational impact of a tutoring hypermedia for children. Information Technology in
Childhood Education Annual, 15(1), 289-308.
de Boer, H., Donker, A. S., & van der Werf, M. P. C. (2014). Effects of the attributes of
educational interventions on students’ academic performance. Review of Educational
Research, 84(4), 509-545.
*Delacruz, G. C. (2010). Games as formative assessment environments: Examining the impact of
explanations of scoring and incentives on math learning, game performance, and help
seeking. (PhD Dissertation), University of California.
Derryberry, A. (2010). Serious games: Online games for learning.
Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among
students: A meta-analysis on intervention studies at primary and secondary school level.
Metacognition Learning, 3, 231–264.
*Din, F. S., & Caleo, J. (2000). Playing computer games versus better learning. Paper presented
at the Eastern Educational Research Association, Clearwater, FL.
Duval, S., & Tweedie, R. (2000). A nonparametric ‘Trim and Fill’ method of assessing
publication bias in meta-analysis. Journal of the American Statistical Association,
95(449), 89-98.
EFFECTS OF GAME-BASED LEARNING 39
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by
a simple, graphical test. British Medical Journal, 315(7109), 629-634.
*Ferguson, T. (2014). Mathematics achievement with digital game-based learning in high school
algebra 1 classes. (PhD Dissertation), Liberty University.
Fu, K. (1999). My first math adventure – Counting and classification [Computer software].
Taipei: Kuang Fu Publishing Co.
*Garneli, V., Giannakos, M., & Chorianopoulos, K. (early view). Serious games as a malleable
learning medium: The effects of narrative, gameplay, and making on students’
performance and attitudes. British Journal of Educational Technology.
Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and
practice model. Simulation & Gaming, 33(4), 441-467.
*Gelman, A. (2010). Mario math with millennials: the impact of playing the Nintendo DS on
student achievement. (PhD Dissertation), University of Denver.
*Giannakos, M. N. (2013). Enjoy and learn with educational games: Examining factors affecting
learning performance. Computers & Education, 68, 429-439.
*Hall, M. M. (2015). Traditional vs. technology based math fluency practice and its effect on
student achievement and motivation in mathematics. (PhD Dissertation), the University
of St. Francis.
Harter, S. (1981). A new self-report scale of intrinsic versus extrinsic orientation in the
classroom: Motivational and informational components. Developmental Psychology, 17,
300-312.
EFFECTS OF GAME-BASED LEARNING 40
*Hawkins, D. (2008). The application of entertainment video games in elementary mathematics
instruction. (PhD Dissertation), Argosy University.
Hedges, L. V. (1981). Distribution theory for Glass’ estimator of effect size and related
estimators. Journal of Educational Statistics, 6(2), 107-128.
Hedges, L. V. (1982). Estimation of effect size from a series of independent experiments.
Psychological Bulletin, 92(2), 490-499.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic
Press.
Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in meta-analysis.
Statistics in Medicine, 21, 1539-1558.
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring
inconsistency in meta-analysis. British Medical Journal, 327, 557-560.
*Hung, C.-M., Huang, I., & Hwang, G.-J. (2014). Effects of digital game-based learning on
students’ self-efficacy, motivation, anxiety, and achievements in learning mathematics.
Journal of Computers in Education, 1(2), 151-166.
*Jones, V. C. (2011). The effects of computer gaming on student motivation and basic
multiplication fluency. (PhD Dissertation), Teachers College, Columbia University.
*Ke, F. (2006). Computer-Based playing within alternative classroom goal structures on fifth-
grades' math learning outcomes: cognitive, metacognitive, and affective evaluation and
interpretation. (Doctoral dissertation) Retrieved from ProQuest Dissertations and Theses.
(UMI Number 3229015).
EFFECTS OF GAME-BASED LEARNING 41
Ke, F. (2008a). Alternative goal structures for computer game-based learning. Computer-
Supported Collaborative Learning, 3, 429-445.
*Ke, F. (2008b). A case study of computer gaming for math: Engaged learning from gameplay?
Computer & Education, 51, 1609-1620.
Ke, F. (2008c). Computer games application within alternative classroom goal structures:
Cognitive, metacognitive, and affective evaluation. Education Tech Research and
Development, 56, 539-556.
*Ke, F., & Grabowski, B. (2007). Gameplaying for maths learning: cooperative or not? British
Journal of Educational Technology, 38(2), 249-259.
Kebritchi, M. (2008). Effects of a computer game on mathematics achievement and class
motivation: An experimental study. Dissertation Abstracts International Section A:
Humanities and Social Sciences, 69(6-A), 2121.
*Kebritchi, M., Hirumi, A., & Bai, H. (2010). The effects of modern mathematics computer
games on mathematics achievement and class motivation. Computers & Education,
55(2), 427-443.
*King, A. (2011). Using interactive games to improve math achievement among middle school
students in need of remediation. (PhD Dissertation), George Washington University.
Lin, C-H., Liu, E. Z.-F., Chen Y.-L., Liou, P.-Y., Chang, M., Wu, C.-H., Yuan, S.-M. (2013).
Game-based remedial instruction in mastery learning for upper-primary school students.
Educational Technology & Society, 16(2), 271–281.
EFFECTS OF GAME-BASED LEARNING 42
Martinez-Garza, M., Clark, D. B., & Nelson, B. (2013). Digital games and the US National
Research Council‘s science proficiency goals. Studies in Science Education, 49, 170-208.
Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014).
Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12
and higher education: A meta-analysis. Computers & Education, 70, 29-40.
Moreno, R., & Duran, R. (2004). Do multiple representations need explanations? The role of
verbal guidance and individual differences in multimedia mathematics learning. Journal
of Educational Psychology, 96(3), 492-503.
National Mathematics Advisory Panel (2008). Foundations for Success: The Final Report of the
National Mathematics Advisory Panel, U.S. Department of Education: Washington, DC.
*Ortega-Tudela, J. M., & Gomez-Ariza, C. J. (2006) Computer-assisted teaching and
mathematical learning in Down Syndrome children. Journal of Computer Assisted
Learning, 22, 298-307.
*Panoutsopoulos, H., & Sampson, D. G. (2012). A Study on exploiting commercial digital
games into school context. Educational Technology & Society, 15(1), 15–27.
*Pareto, L., Haake, M., Lindström, P., Sjödén, B., & Gulz, A. (2012). A teachable-agent-based
game affording collaboration and competition: evaluating math comprehension and
motivation. Education Tech Research Dev, 60(5), 723-751.
Pellegrino, J., & Hilton, M. (2012). Education for Life and Work: Developing Transferable
Knowledge and Skills in the 21st Century. Washington, D.C. National Research
Academy.
EFFECTS OF GAME-BASED LEARNING 43
*Ploger, D., & Hecht, S. (2009). Enhancing children's conceptual understanding of mathematics
through Chartworld software. Journal of Research in Childhood Education, 23(3), 267-
277.
R Core Team. (2016). R: A language and environment for statistical computing (version 3.3.2).
Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-
project.org
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results. Psychological
Bulletin, 86(3), 638-641.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2006). Publication bias in meta-
analysis: Prevention, assessment and adjustments. John Wiley & Sons.
Schwarzer G, Carpenter JR and Rücker G (2015). Meta-Analysis with R (Use-R!). Springer
International Publishing, Switzerland. http://www.springer.com/gp/book/9783319214153
*Sedig, K. (2008). From play to thoughtful learning: A design strategy to engage children with
mathematical representations. Journal of Computers in Mathematics and Science
Teaching, 27(1), 65-101.
*Shin, N., Sutherland, L. M., Norris, C. A., & Soloway, E. (2012). Effects of game technology
on elementary student learning in mathematics. British Journal of Educational
Technology, 43(4), 540-560.
Shute, V. J., & Ke, F. (2012). Games, learning and assessment. In D. Ifenthaler, D. Eseryel &
Ge, X. (Eds.), Assessment in game-based learning: Foundations, innovations, and
perspectives (pp.43-58) New York, NY: Springer.
EFFECTS OF GAME-BASED LEARNING 44
Siemer, J., & Angelides, M. C. (1995). Evaluating intelligent tutoring with gaming simulations.
Paper presented at the 1995 Winter Simulation Conference.
Sitzmann, T. (2011). A meta-analytic examination of the instructional effectiveness of computer-
based simulation games. Personnel Psychology, 64(2), 489-528.
SMEC (1999). Toby’ IQ training camp – The seed of logic [Computer Software]. Taipei: SMEC
Media Entertainment Corp.
*Starkey, P. L. (2013). The effects of digital games on middle school students’ mathematical
achievement. (PhD Dissertation), Lehigh University.
*Sung, Y.-T., Chang, K.-E., & Lee, M.-D. (2008). Designing multimedia games for young
children’s taxonomic concept development. Computers & Education, 50(3), 1037-1051.
*Swearingen, D. K. (2011). Effect of digital game based learning on ninth grade students’
mathematics achievement. (PhD Dissertation), University of Oklahoma.
*Van Eck, R., & Dempsey, J. (2002). The effect of competition and contextualized advisement
on the transfer of mathematics skills a computer-based instructional simulation game.
Educational Technology Research and Development, 50(3), 23-41.
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of
Statistical Software, 36(3), 1-48.
Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006).
Computer gaming and interactive simulations for learning: A meta-analysis. Journal of
Educational Computing Research, 34(4), 229-243.
EFFECTS OF GAME-BASED LEARNING 45
*Weiss, I., Karamarski, B., & Talis, S. (2006) Effect of multimedia environments on
kindergarten children’s mathematical achievement and style of learning. Educational
Media International, 43(1), 3-17.
Yang, J. C., & Chen, S. Y. (2010). Effects of gender differences and spatial abilities within a
digital pentominoes game. Computers & Education, 55(3), 1220-1233.
Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., et al. (2012). Our princess
is in another castle: A review of trends in serious gaming for education. Review of
Educational Research, 82, 61-89.
Running head: EFFECTS OF GAME-BASED LEARNING 46
Table 1
Moderator Analysis Results
Moderator
Mean(SE)
95% CI
Overall
39
0.13(0.06)
[0.02, 0.24]
Grade Level [(2) = 4.00, p = .14]
Preschool – Kindergarten
3
0.58(0.25)
[0.09, 1.06]
6.00
1st Grade – 6th Grade
26
0.13(0.07)
[0.00, 0.27]
52.84****
7th grade and Above
10
0.05(0.10)
[-0.14, 0.24]
24.22**
Country [(1) = 0.29, p = .60]
United States
17
0.10(0.08)
[-0.06, 0.26]
35.99**
Other
22
0.16(0.08)
[0.00, 0.32]
54.20***
Instrument Type [(2) = 3.01, p = .22]
Commercial/Standardized Test
10
0.03(0.10)
[-0.17, 0.23]
28.10***
Research-based Scale
11
0.29(0.11)
[0.07, 0.50]
13.62
Researcher-made Scale
18
0.11(0.08)
[-0.05, 0.27]
41.51***
Length of Game-Based Intervention [(2) = 2.51, p = .28]
Up to one hour
11
-0.03(0.12)
[-0.27, 0.21]
25.04**
Between one hour and eight hours
21
0.19(0.08)
[0.05, 0.34]
41.05**
Over eight hours
7
0.12(0.13)
[-0.13, 0.37]
22.92***
Publication Type [(1) = 6.49, p = .01]
Journal
30
0.21(0.06)
[0.10, 0.33]
58.92***
Thesis or Dissertation
9
-0.07(0.09)
[-0.25, 0.11]
11.14
Coefficient(SE)
95% CI
Publication Year [(1) = 4.46, p = .03]
39
0.01(0.01)
[0.00, 0.02]
79.88***
*p < .05; **p < .01; ***p < .001
Note: Means within groups are weighted under the mixed-effects model; indicates a specific group; Regression coefficient is
standardized.
EFFECTS OF GAME-BASED LEARNING 47
Table 2
Study Characteristics Included in Meta-Analysis
Study
Sample
Size
Grade
Level
Country
Instrument Type
Game(s)
Length of Game-
based
Intervention
Bai et al. (2012)
437
7th-12th
USA
researcher-made
instrument
DimensionM
long
Beserra et al. (2014)
271
1st-6th
Brazil,
Chile, and
Costa Rica
research-based scale
Researcher-developed game
medium
Carr (2012)
104
1st-6th
USA
researcher-made
instrument
iPad math games
long
Chang et al. (2012)
92
1st-6th
Taiwan
commercial/standardized
test
Researcher-developed game
medium
Chang et al. (2015)
306
1st-6th
USA
commercial/standardized
test
Researcher-developed game
medium
Ferguson (2014)
222
7th-12th
USA
researcher-made
instrument
Slope game
medium
Garneli et al. (2016)
80
1st-6th
Greece
commercial/standardized
test
Gem-game
short
Gelman (2010)
80
7th-12th
USA
researcher-made
instrument
Brain Age 2, Nintendo DS
long
Giannakos (2013)
87
7th-12th
Norway
research-based scale
Gem-game
medium
EFFECTS OF GAME-BASED LEARNING 48
Hall (2015)
405
1st-6th
USA
researcher-made
instrument
iPad multiplication games
medium
Hawkins (2008)
139
1st-6th
USA
researcher-made
instrument
MySims Wii, Nintendo Wii
medium
Hung et al. (2014)
69
1st-6th
China
research-based scale
Researcher-developed Brick
Breaker game
medium
Ke (2008)
358
1st-6th
USA
researcher-made
instrument
ASTRA EAGLE (a series of
web-based computer games;
academic content is based on
the Pennsylvania System of
School Assessment (PSSA)
standards for mathematics)
medium
King (2011)
128
7th-12th
USA
research-based scale
VmathLive (academic content is
based on the National Council
of Teachers of Mathematics
(NCTM) standards)
medium
Lin et al. (2013)
64
1st-6th
Taiwan
research-based scale
Researcher-developed
Monopoly game
short
Panoutsopoulos &
Sampson (2012)
57
1st-6th
Greece
commercial/standardized
test
Sims 2–Open for Business
medium
Pareto et al. (2012)
47
1st-6th
Sweden
researcher-made
instrument
Researcher-developed
Teachable Agents game
long
Ploger & Hecht (2009)
196
1st-6th
USA
research-based scale
Chartworld
medium
Sedig (2008)
59
1st-6th
Canada
research-based scale
Researcher-developed Super
Tangrams game
medium
Shin et al. (2012)
41
1st-6th
USA
research-based scale
Skills Arena
long
Starkey (2013)
168
7th-12th
USA
researcher-made
instrument
Lure of the Labyrinth
medium
Sung, Chang & Lee
(2008)
60
PreK-K
Taiwan
commercial/standardized
test
Researcher-developed game
short
EFFECTS OF GAME-BASED LEARNING 49
Swearingen (2011)
280
7th-12th
USA
research-based scale
Researcher-developed massive
multiplayer online game
(MMOG)
long
Weiss et al. (2006)
116
1st-6th
Israel
research-based scale
Goldilocks series games
long
EFFECTS OF GAME-BASED LEARNING 50
Figures:
Figure 1. Flowchart of inclusion and exclusion decisions.
Figure 2. Forest plot for effect sizes. CI = Confidence Interval.
Figure 3. Funnel plot of effect sizes with 95% confidence interval.
A preview of this full-text is provided by Wiley.
Content available from Journal of Computer Assisted Learning
This content is subject to copyright. Terms and conditions apply.
