Content uploaded by Jamaal R. Young
Author content
All content in this area was uploaded by Jamaal R. Young on Sep 07, 2018
Content may be subject to copyright.
Technology effectiveness in the mathematics classroom:
a systematic review of meta-analytic research
Jamaal Young
1
•Faith Gorumek
2
•
Christina Hamilton
3
Received: 12 January 2018 / Revised: 24 April 2018 / Accepted: 2 May 2018 /
Published online: 12 May 2018
Beijing Normal University 2018
Abstract The purpose of this systematic review was to examine trends in prior
meta-analytic research to provide recommendations for future mathematics educa-
tion research and instructional praxis. The current study aims to contextualize the
effects of technology-enhanced instruction in the mathematics classroom. The
researchers conducted a comprehensive literature search of articles written between
1980 and 2015. The final pool of studies comprised 18 meta-analyses inclusive of
studies conducted between 1986 and 2014, representing 1193 independent effect
sizes. The results suggest that the effects of technology on mathematics achieve-
ment range from small to large. Results suggest that researchers and educators
should consider grade level, duration, and the instructional role of technology as key
components when incorporating technology in the mathematics classroom. Results
also suggest that race, socioeconomic status (SES), and gender did not moderate the
effects of technology integration, although they were examined less frequently
across studies. Implications are provided for practice, and research related to these
results. Because of the chosen research approach, the research results provide rel-
evant and practical implications to support classroom teaching with technology.
This study contributes to the literature on technology-enhanced mathematics
instruction by providing synthesis of 30 years of meta-analytic research.
Keywords Technology-enhanced instruction Mathematics Systematic review
Moderators
&Jamaal Young
Jamaal.young@unt.edu
1
Department of Teacher Education and Administration, University of North Texas, Matthews
Hall 218-G, Denton, TX 76206, USA
2
University of North Texas, Denton, TX, USA
3
Texas A&M Central Texas, Killeen, TX, USA
123
J. Comput. Educ. (2018) 5(2):133–148
https://doi.org/10.1007/s40692-018-0104-2
Introduction
Digital technologies are exciting pedagogical tools that can enhance the delivery,
clarity, and precision of mathematics instruction. Incorporating technology in the
classroom makes an essential contribution to student success in mathematics (Nepo,
2017). Based on this trend, research examining effective use of technology in the
mathematics classroom has grown exponentially. Over the last three decades,
numerous meta-analytic studies have investigated technology’s effects on mathe-
matics achievement and the factors that mediate these effects (Chan and Leung
2014; Li and Ma 2010). These meta-analyses provide summary effect size
estimates, as well as moderators of the effect sizes across studies. Summary effect
sizes are often the focus of traditional meta-analysis, while less emphasis is placed
on the moderators of these effects.
Effect size reporting and its role in meta-analytic thinking are considered
significant concerns in effective mathematics education research consumption and
reporting. The American Psychological Association (APA 2010) and the American
Educational Research Association (AERA 2006) regularly advocate for the
reporting of effect sizes and more recently, considered meta-analytic thinking an
extension to previous reporting practices. Numerous mathematics education
scholars cite the benefits of effect size reporting and meta-analytic thinking through
the presentation and interpretation of confidence intervals (Young et al. 2013;
Young and Young 2016; Cumming 2012; Zientek et al. 2008). Effect sizes and
confidence intervals are organic elements of meta-analytic research and represent
metrics for comparison and summarization of effects across studies. Therefore,
reviewing the trends in previous meta-analytic research on the moderators of the
effects of technology integration on mathematics achievement is vital to the fidelity
of technology integration research in the mathematics classroom.
However, to enhance the teaching and learning of mathematics with technology,
researchers must refine theoretical constructs through empirical specification, which
can, and should guide classroom applications. Moderators are often directly related
to classroom implementation, and can be used to refine theoretical constructs
thereby supporting empirical specification. Unfortunately, moderators of effect sizes
are rarely synthesized in the empirical literature. Synthesizing the moderators of
effect sizes across prior meta-analyses has empirical and practical importance to
effective implementation of technology-enhanced teaching in the mathematics
classroom.
The purpose of this systematic review was to examine the moderator analysis
results for prior meta-analytic research to identify trends in empirical research and
practice. It is our hope that results of this study provide recommendations for future
research and instructional praxis. These results are relevant because they
demonstrate how the expansion of meta-analytic thinking supports effective
classroom teaching with technology.
134 J. Comput. Educ. (2018) 5(2):133–148
123
Literature review
Prior syntheses and meta-analyses combine knowledge from individual studies to
inform the teaching and learning practice with technology. Recent syntheses have
examined the influence of technology-enhanced instruction on learning across a
multitude of disciplines and contexts (Chang et al. 2018; Fu and Hwang 2018;
Wang et al. 2017), however few studies have systematically reviewed prior meta-
analyses to synthesize the results across first-order meta-analysis (Young et al.
2018; Gurevitch et al. 2018; Tamim et al. 2011). Within mathematics education
research, numerous studies have examined the unique influences of specific
technology integration on the teaching and learning of mathematics. Several studies
have examined the relationship between teacher pedagogical beliefs and their use of
technology in the mathematics classroom. The majority of prior meta-analytic
research has focused on the unique effects of integrating specific technological tools
in the mathematics classroom. In the sections that follow, the researchers review the
effects of several common classroom technologies on student achievement in
mathematics.
Computer-assisted instruction
The use of computers to guide and enhance mathematics learning is well
documented. Two of the most common applications of computers in the
mathematics classroom are computer-assisted instruction (CAI) and computer-
based instruction (CBI). CAI and CBI are similar applications of computers in the
classroom, but their instructional purposes are nuanced. Computer-assisted
instruction (CAI) is a more precise term, often referring to the use of computers
in drill and practice, tutorials, or simulation activities offered in substitution or as a
supplement to traditional, teacher-directed instruction (Hicks and Holden 2007),
while computer-based instruction (CBI) is broadly defined as the use of computers
in the delivery of instruction (Kulik 1983). The effects of CAI and CBI on student
achievement in general and in mathematics education specifically have been
examined across a multitude of grade levels and diverse contexts (Yung and Paas
2015). Despite their nuances, CAI and CBI are often operationalized as learning
delivered primarily by means of the computer, which typically incorporate drill and
practice, simulations, and well-defined feedback mechanisms. CBI and CAI have
been used interchangeably within prior meta-analyses in mathematics education
research, and thus, they are discussed as one in the same here.
The results of prior meta-analyses have suggested that the effects of CBI/CAI on
mathematics achievement vary from small to medium based on effect size
benchmarks (Cohen 1992). Prior meta-analyses were conducted across grade levels
and various types of mathematics content (Chadwick 1997; Chen 1994; Hsu 2003;
Larwin and Larwin 2011; Lee 1990). CBI/CAI studies consistently conclude that
duration and mode of instructional use were statistically significant moderators of
study effects. These results are particularly pertinent as they relate to the length of
treatment and the instructional modality necessary to enhance mathematics teaching
J. Comput. Educ. (2018) 5(2):133–148 135
123
and learning. Calculator use has a rich tradition within mathematics education, and
unlike CAI or CBI, calculators are viewed as a more content specific instructional
technology.
Calculators
Many mathematics educators continue to debate when to use calculators in the
mathematics classroom within research and policy documents. The affordances of
calculators as pedagogical tools cannot be denied. The variety of hand-held
calculators continues to evolve. Today, calculators range from simple arithmetic
calculators to scientific calculators, graphing calculators, and symbolic calculators
with a variety of calculating modes, including algebraic systems and spreadsheets
(Close et al. 2012). The National Council of Teachers of Mathematics (NCTM)
contends that calculators are fundamental technologies in mathematics classrooms
that enrich student understanding (NCTM 2000). Given the multiple perspectives on
the use of calculators in the mathematics classroom, the results of a prior meta-
analysis on the effects of calculators on mathematics achievement were instrumental
to the acceptance of calculators as pedagogically meaningful tools.
The results of prior meta-analysis investigating calculator use and mathematics
achievement tend to converge at the moderate level of effectiveness. Statistically,
the significant moderators of calculator effects on mathematics achievement are
grade level and assessment type (Ellington 2006; Hembree and Dessart 1986;
Nikolaou 2001; Tokpah 2008). This is not surprising given that the grade level
remains a point of contention. Many concerns remain regarding early exposure to
calculators in the mathematics classroom, due in part to the inconsistencies in access
during examinations. For instance, the results of the 2009 National Assessment of
Educational Progress (NAEP) indicate that 66% of fourth graders claimed they
never used a calculator for exams or quizzes, compared to only 28% of eighth
graders surveyed (Planty et al. 2009). These results are further substantiated by
trends observed in prior meta-analyses. Hembree and Dessart (1986) conclude,
‘‘average students (except fourth grade) who use calculators in concert with
traditional mathematics instruction improve their basic skills with paper–pencil
tasks, both in computational operations and in problem-solving’’ (p. 96). Therefore,
assessment and grade require additional pedagogical consideration.
Mathematics software and emerging trends in mobile technology
Mathematics software applications vary from general to specific forms, such as
digital geometry software (DGS), and virtual manipulatives. Compared to CAI/CBI
and calculator use in the mathematics classroom these tools are relatively under-
researched. Thus fewer meta-analyses exist. The overall effect sizes for mathemat-
ics software applications range from 0.09 to 1.02 (Chan and Leung 2014; Cheung
and Slavin 2013; Moyer-Packenham and Westenskow 2013; Steenbergen-Hu and
Cooper 2013). The consistent statistically significant moderators of effect sizes
observed in the literature are grade level, duration, and mathematics subject matter
136 J. Comput. Educ. (2018) 5(2):133–148
123
(algebra, geometry, etc.). This indicates that the divergence in effect sizes across
these studies may be attributed to these aforementioned moderators.
Emerging research trends tend to focus on the effects of mobile technology on
the teaching and learning of mathematics. For example, Bano et al. (2018) identified
three themes within the pedagogical approaches present in the mathematics and
science instruction with mobile devices literature. These approaches were collab-
oration, inquiry-based learning, and realistic learning. Fabian et al. (2016) found the
overall mean effect of mobile technology on achievement in elementary
mathematics was .48. The researchers also found that the results of studies in
middle grades classrooms were positive overall, but the effects on high school
environments were mixed. Given that the results of prior meta-analysis provide
credence to the use of technology-enhanced teaching methods in the mathematics
classroom, but lack overarching prescriptive conclusions for general praxis with
technology, a summary of effects across prior meta-analysis is warranted.
Moderators and meta-analytic thinking
A meta-analytic lens may serve as the most suitable empirical tool to identify the
best practices with technology in the mathematics classroom. Meta-analysis is a
research synthesis tool that uses summaries of effect sizes to generate empirical
conclusions from ostensibly similar studies. Meta-analysis involves (1) summariz-
ing several studies regarding effect sizes, and (2) combining the results to make
summative inferences (Cooper 2016). This process involves calculating the average
effect size, testing for homogeneity, detecting moderators, and explaining any
heterogeneity (Hunter and Schmidt 2004). The detection of moderators is the
critical feature of any meta-analytic study; because differences in strength and
direction in effect sizes are identified here. Rosenthal (1991) argues, ‘‘The search for
moderators is not only an exciting intellectual enterprise but indeed…it is the very
heart of scientific enterprise’’ (p. 447). Moderators offer conditions for effects that
are theorized, thus informing researchers of the circumstances in which the effects
under investigation can be reliable (Schmidt and Hunter 2014). This information is
vital to successful implementation of technology in the mathematics classroom
across instructional contexts.
Using the lens of meta-analytic thinking, researchers can make better decisions
about technology integration in the mathematics classroom. Meta-analysis can help
researchers find specific variables that account for the variance in the effectiveness
of technology integration in the mathematics classroom. Moderators quantify
qualitative variables that influence the strength or direction of relationships in meta-
analytic research (Steel et al. 2002). Moderators are also important because they
identify statistical interactions, which do not imply causation but rather add context
to effect size results (Cooper and Patall 2009). Given the distinctions among the
associations they identify, moderators are consistently placed in three categories.
Moderators are categorized as either: (1) methodological variations, (2) theoretical
constructs, or (3) study characteristics (DeCoster 2004).
Methodological variations refer to components of the experimental design such
as sample size, random assignment, or treatment duration. Theoretical constructs are
J. Comput. Educ. (2018) 5(2):133–148 137
123
moderators grounded in theory or based on the application of recognized theoretical
trends. The final category of moderator variables includes study related artifacts,
such as publication status or publication year. Moderators are recognized for their
ability to enhance theory development and increase the general richness of
empirical work (Aguinis et al. 2011). Given the empirical merit of meta-analytic
research and the contextualization offered by moderator analysis, systematically
summarizing results across studies is practically and scientifically necessary. To
examine the effect of technology on achievement in mathematics across multiple
contexts, therefore, a literature survey was conducted to identify and characterize
pertinent moderators of effect size. The present study was guided by the following
research question:
1. How are the moderators of effect size characterized in prior meta-analyses
of technology-enhanced mathematics instruction?
Method
The current systematic review utilized the Preferred Reporting Items for Systematic
Reviews and Meta-analysis (PRISMA) protocol. According to Moher, Liberati,
Tetzlaff, and Altman Moher et al. (2009), PRISMA represents a set of evidence-
based items that represent accepted practices for conducting systematic reviews and
meta-analysis. Eligible studies were limited to meta-analyses written between 1980
and 2015. Due to the focus on meta-analyses, systematic reviews, literature
synthesis, and traditional qualitative or quantitative studies were not included.
Data sources were electronic databases covering education, psychology, and
social sciences. The specific databases included JSTOR, ERIC, EBSCO, Pych
INFO, and ProQuest. In each database, an initial search was performed against the
abstracts using the Boolean search term ‘‘meta-analysis OR systematic review’’
AND ‘‘ mathematics OR STEM’’ AND ‘‘ technology OR digital’’. Whenever possible,
search limiters were used to align the initial search results more closely with the
eligibility criteria. For example, most databases allow limiting the search to a
specific date range. The search was concluded in January of 2016.
Screening process
Figure 1presents the complete study inclusion and exclusion process. The screening
criteria shown in Table 1guided the selection of articles from the initial pool. First,
study titles and abstracts were screened for relevance to the research question and
study topic of interest. Then, the remaining studies were screened against the
criteria provided in Table 1. The initial pool of 42 studies was systematically
screened using this process and reduced to a final pool of 18. As shown in Fig. 1,
most of studies were removed for lack of effect size reporting and the absence of a
digital technology focus. Pertinent data related to the research questions were
extracted from the remaining studies.
138 J. Comput. Educ. (2018) 5(2):133–148
123
ERIC k = 13
PsycINFO k = 16
ProQuest k = 5
JSTOR = 8
Database Search
k = 42
Screened via title,
abstract, & references
k = 42
Total screened studies
k = 45
Met inclusion criteria
k = 26
Reported mean effect
size info
Manuscripts cod ed
k = 18
Sufficient data to
calculate and mean
effect size supplied k =
3
Did not report an
average effect size
k
=
11
Retrieved from
references of screened
studies
k = 3
Excluded studies
Was not a meta-analysis k = 4
Did not address the impact of digital
technology k = 8
Did not use student achievement or
performance as the dependent k = 6
Same dataset presented in multiple studies k
= 1
Fig. 1 Technology meta-analysis study inclusion flowchart
Table 1 Inclusion Screening Process
Criterion Include Exclude
Publication
Year
1980-2015 Before 1980
Language English Non-English
Context Classroom settings Context other than classroom settings
Research
Design
Meta-analysis Other empirical research (quantitative,
qualitative, mixed-methods), secondary data
analysis, and systematic reviews
Participants K-12 and post-secondary students Non-student populations
Relevance Examined the effects of technology-
enhanced instruction on mathematics
achievement
Did not study the effects of technology
enhanced instruction on mathematics
achievement
J. Comput. Educ. (2018) 5(2):133–148 139
123
Data collection and analysis
The extraction protocol presented in Table 2guided data extraction from the
retained articles. Data extracts included citation, purpose, mean effect sizes, number
of independent effect sizes, moderators, and key findings. Extracted data were
stored in a database indexed by article. In addition, Results, Discussion, and
Conclusion sections of each article were extracted and stored in a database for
critical analysis.
To examine the moderators affecting the strength and direction of the results,
each meta-analysis’ methodological, theoretical, and study characteristic modera-
tors, the researchers used a semi-structured coding protocol based on an adapted list
of features and trends found in the systematic review. Moderators were coded
verbatim initially, and then coded categorically after all studies were reviewed.
Moderator categories were based on operational definitions that emerged during the
coding discussions and data extraction process. The researchers assessed coding
reliability by comparing the independent coding results from the studies. The initial
inter-rater agreement was 95%, and we met to resolve the remaining inconsistencies
in the coding results.
Data were analyzed descriptively to best characterize the trends in moderator
influence on effect size variability. Frequency counts for each moderator were
recorded along with the Q
B
statistics, and p-values. Moderators were assigned a
rating of high, medium, or low based on the ratio between the frequencies of
statistically significant observation compared to the total number of observations for
that particular moderator. These data represent a measure of the impact of each
moderator across the studies reviewed in the current study.
Results
The final pool of studies comprised 18 meta-analyses inclusive of studies conducted
between 1986 and 2014, representing 1193 independent effect sizes (Table 3). The
median year of publication was 2007 and the range for the year of publication was
28 years. A complete list of study characteristics is presented in Table 1, which
Table 2 Data Extraction Protocol
Extract Description
Citation Author(s) and publication date
Source Article, conference proceeding, or dissertation
Purpose Purpose, objectives, research questions
KNumber of independent effect sizes included in meta-analysis
ES Mean effect size
Moderators Detailed description of moderators (operationalization, statistical significance, etc.)
Key findings Summary of main findings and conclusions
140 J. Comput. Educ. (2018) 5(2):133–148
123
Table 3 Description of Included Meta-Analysis
Citation Purpose Source kES
Hembree and Dessart
(1986)
Integrate the findings of the research on effects on
students of using calculators in learning
mathematics in Grades K-12
Journal 29 .64
Lee (1990) Determine the Effectiveness of CAI in elementary
and secondary instruction
Dissertation 243 .38
Chen (1994) Synthesize and extract the main findings from
studies on computer-based instruction (CBI) in
mathematics education
Dissertation 76 .50
Chadwick (1997) Examine the effects of CAI in the secondary
mathematics classroom on cognitive and
affective outcomes
Dissertation 41 .51
King (1997) Determine the effect of computer-enhanced
instruction (CEI) on college level mathematics
Dissertation 30 .20
Nickolau (2001) Synthesize the effects of hand-held calculators on
K-12 mathematics achievement
Dissertation 24 .54
Hsu (2003) Examined the effectiveness of Computer-Assisted
Instruction (CAI) instruction in statistics
education
Dissertation 25 .43
Ellington (2006) Examined the effects of calculator use on student
achievement and attitude levels
Journal 54 NA
Schenker (2007) Examine the effectiveness of using technology to
enhance statistics instruction
Dissertation 117 .24
Tokpah 2008) Examined the Effects of Computer Algebra
systems (CAS) on mathematics achievement
Dissertation 102 0.38
Rosen and Salomon
(2007)
Examined the effectiveness of constructivist
technology intensive learning environments
versus traditional learning environments
Journal 32 0.46
Wang et al. (2007) Examined the effect of testing mode (computer vs.
paper and pencil) on mathematics achievement
Journal 14 -.11
Li and Ma (2010) Examined the effects on computer Technology on
mathematics achievement in K-12
Journal 46 0.28
Larwin and Larwin
(2011)
Determine the effectiveness of CAI student
mathematics achievement in post-secondary
statistics courses
Journal 219 .57
Cheung and Slavin
(2013)
Examined the effects of educational technology on
mathematics achievement in K-12 settings
Journal 74 0.16
Steenbergen-Hu and
Cooper (2013)
Examined the effects of intelligent tutoring systems
on K-12 mathematics achievement
Journal 26 0.09
Moyer-Packenham
and Westenskow
(2013)
Synthesize the findings examining the effects of
virtual manipulatives on student achievement
Journal 32 .35
Chan and Leung
(2014)
Evaluate the effects of digital geometry software on
mathematics achievement
Journal 9 1.02
J. Comput. Educ. (2018) 5(2):133–148 141
123
shows that the majority of the meta-analyses were journal articles (10 out of 18) and
the remaining meta-analyses were dissertation studies. All studies except one
included either an overall mean effect size or sufficient data to calculate the overall
effect size. Only one study reported an overall negative effect size, and the overall
effect sizes ranged from -.11 to 1.02 in magnitude.
To answer the research question, the researchers identified 17 specific moderators
by screening the frequency data using IBM SPSS Statistics 20; then, based on
characterizations and definitions presented in prior research narrowed the list of
moderators. The researchers operationalized each moderator for fidelity. Table 4
presents the operational definition of each moderator, frequency of investigation,
and impact based on the ratio between the number of times the moderator was a
Table 4 Moderator Operational Definitions, Frequencies, and Impact
Moderator Operationalization fRank
Grade level Grade spans and other ordinal descriptions of student progress within
educational systems (elementary, secondary, freshman, etc.)
12 High
Role Instructional role of technology. Examples substitute, supplement, etc 10 High
Duration Length of treatment reported in days, weeks, or months 9 High
Ability Non-exceptionality classification of students based on prior
achievement
8 Medium
Mode Instructional modality of technology use in the classroom. ex. drill/
practice, tutorial, simulation, problem-solving, etc
8 High
Assessment Describes the role of technology in the delivery and completion of
mathematics formative and summative assessment. ex. computer-
based assessments, access to calculators during exams, etc
7 High
SES Socioeconomic Status (SES), the social status or class of an individual
or group. Operationalized as low and high based on receipt of free or
reduced lunch
6 Low
Subject-matter Mathematics content strands—Algebra, geometry, fractions, statistics,
calculus, etc
6 High
Gender Reported dichotomous factors. male and female 5 Low
Concentration Time per session expressed in minutes or hours 5 High
Technology
type
Specific category or type of technological tool – graphing vs. basic
calculator, virtual manipulative, computer-assisted instruction, etc
5 Low
Differentiation Student receiving specialized instruction – Special education, ESL,
ELL, GT, etc
4 Medium
Race Student self-reported background – Asian, Black, Latino, White, Other 3 Low
Community Urban, rural, or suburban described by population density 3 Low
Teacher Teacher’s positionality as either facilitator or instructor regarding
technology integration
3 High
Organization Describes whether treatment was delivered individually, in pairs, or in
groups of three or more
3 Low
Access Accessibility of technology reported as low, medium, or high. Pertains
to students’ ability to use technology at will
2 Medium
142 J. Comput. Educ. (2018) 5(2):133–148
123
statistically significant predictor and the moderator identification frequency. Next,
the researchers ranked the moderator impact as either (low \.50), (medium &
.50), or (high [.50) based on the data observed. The researchers listed the
moderators in descending order of frequency of observation for explanatory
transparency.
Grade level was the most frequently observed moderator while access was the
least frequently observed. The remaining moderators in the upper quartile, listed in
descending order, were role, duration, ability, and mode. The lower quartiles of
moderators in ascending order were organization, teacher, community, and race.
Results in Table 4suggest that four out of the five moderators identified in the upper
quartile of frequency had a high impact on the variability of meta-analysis results.
Most of the lower quartile moderators have a low impact on the variability of effect
sizes. The teacher was the only moderator in the lower quartile that had a high
impact on the variability of meta-analysis results. In summary, eight moderators
ranked as high, three as medium, and six as low. In the discussion section that
follows, the researchers provide substantive conclusions and implications for
teachers, administrators, and researchers based on these results.
Discussion
The purpose of this systematic review was to examine the moderator analysis results
for prior meta-analytic research to identify trends in empirical research and practice.
The analysis of this research compared different conceptualizations of learning with
technology examined measurement in mathematics classrooms, and identified
common and generalizable findings across the meta-analyses regarding the
moderators of the effectiveness of technology integration in mathematics class-
rooms. The results suggest that the effects of technology integration on mathematics
achievement vary from negligible to large, but are consistently small. However,
given the practical significance of a small effect size in the mathematics classroom
this finding has educative merit for teachers and administrators (Hill et al. 2008).
Moderators are important tools to use when evaluating and planning technology-
mediated learning in the mathematics classroom. Thus, the focus of this study was
on moderators of the effects of meta-analyses. The 17 moderators investigated in
this study varied in frequency of investigation and impact on effect size differences.
Grade level, role, and duration were the three most investigated moderators, and all
had a high impact on effect size variation across the results of the meta-analyses
examined. In the 12 studies examining the moderator variable grade level, 66%
found that grade level had a statistically significant influence on the student
achievement effects variability when technology was integrated in the mathematics
classroom. Many included studies favored technology in middle and high school
classrooms as opposed to early elementary settings. This conclusion is consistent
with prior studies that found that technology is utilized less in elementary schools
compared to high school classrooms (Brown et al. 2007). Researchers should
continue to assess this phenomenon, and teachers and administrators should
J. Comput. Educ. (2018) 5(2):133–148 143
123
examine the grade level implications of technology integration closely when
designing and applying interventions.
The results of the ten studies examining the instructional role of technology
suggest that technology was a statistically significant moderator of effect sizes in
90% of the meta-analysis examined. Only one study concluded that the instructional
role of technology was not a statistically significant moderator of effect sizes, but
the findings from the other nine studies concluded that effect sizes were larger when
technology was used to supplement or augment instead of substitute or replace
traditional instruction in the classroom. In addition to traditional instructional tools
such as software resources and standard tools such as graphing calculators,
educators are exploiting a variety of technological tools in mathematics instruction,
including cell phones and other mobile technologies (Gay and Burbridge 2016;
Young and Young 2012; Davis 2010; Valk et al. 2010). As studies continue to use
these tools, it will be important to revisit the effects of instructor role on
achievement.
Third, duration was assessed in nine studies and found to be a significant
moderator of effect sizes in 89% of the studies investigated. Most studies found that
at approximately three weeks the effect of a technology intervention weans. Thus,
researchers and educators need to be cognizant of overexposure and the novelty
factor. Other considerable instructional findings were that mode of instruction,
assessment, subject matter, concentration, and teacher instructional orientation all
statistically significantly influenced the variability of effect sizes in the meta-
analysis; however, these moderators were investigated less often across studies.
Such student demographic variables such as race, gender, SES, and community
were not consistent moderators of the effects of technology on mathematics
achievement.
Limitations
Summarizing effects of moderators on the effect sizes across meta-analyses has
several limitations. First, much of the data pertinent to each moderator resides at the
individual study level. This is problematic because a precise estimation of the exact
influences of all moderators assessed in prior meta-analyses would be difficult to
feasibly examine even through second-order meta-analysis (Young 2017). Thus, a
representative sample of moderators that could be assessed at the meta-analysis
rather than at study level was selected for systematic review in the present study.
Conclusion
This study provides a comprehensive systematic review and literature survey of
research conducted from 1985 until 2015. Based on the summary of almost 30 years
of research, this study provides important conclusions related to the effectiveness
and moderators of technology integration in mathematics classrooms. In conclusion,
the results of this systematic review indicate that technology integration supports
144 J. Comput. Educ. (2018) 5(2):133–148
123
mathematics achievement across prior meta-analytic research. However, the
statistically significant moderators of the effects vary across studies.
Based on these results, the researchers recommend that teachers and researchers
continue to implement technology in the mathematics classroom, but emphasize
optimizing the effects grade level, role of technology, and duration. The researchers
also recommend further research into demographic variables, which were inves-
tigated less frequently across studies. In addition, more research is necessary to
capture the unique influences of teachers on the effects of technology integration in
the mathematics classroom. Finally, the researchers recommend further investiga-
tion into variables such as student access to technology at home and the effects of
instructional context regarding the duration effects of technology integration.
Armed with these recommendations, researchers and educators are better equipped
to make informed decisions concerning the when, where, and how of integrating
technology in the mathematics classroom.
Compliance with ethical standards
Conflict of interest All authors declared that they have no conflict of interest
Ethical approval No animals were involved in this project. All procedures performed in studies
involving human participants were in accordance with the ethical standards of the institutional and/or
national research committee and with the 1964 Helsinki declaration and its later amendments or com-
parable ethical standards.
Informed consent Informed consent was obtained from all individual participants included in the study.
References
Aguinis, H., Gottfredson, R. K., & Wright, T. A. (2011). Best-practice recommendations for estimating
interaction effects using meta-analysis. Journal of Organizational Behavior, 32(8), 1033–1043.
American Educational Research Association. (2006). Standards for reporting on empirical social science
research in AERA publications. Educational Researcher, 35(6), 33–40.
American Psychological Association. (2010). Publication manual of the American Psychological
Association (6th ed.). Washington, DC: American Psychological Association.
Bano, M., Zowghi, D., Kearney, M., Schuck, S., & Aubusson, P. (2018). Mobile learning for science and
mathematics school education: A systematic review of empirical evidence. Computers & Education,
121, 30–58.
Brown, E. T., Karp, K., Petrosko, J. M., Jones, J., Beswick, G., Howe, C., et al. (2007). Crutch or catalyst:
Teachers’ beliefs and practices regarding calculator use in mathematics instruction. School Science
and Mathematics, 107(3), 102–116.
Chadwick, D. K. H. (1997). Computer-assisted instruction in secondary mathematics classrooms: A
meta-analysis (Ed.D.). Drake University, United States-Iowa. Retrieved from http://search.proquest.
com/pqdtglobal/docview/304389928/abstract/9D8837D0D8EB4A14PQ/11. Accessed 10 Sept 2015.
Chan, K. K., & Leung, S. W. (2014). Dynamic geometry software improves mathematical achievement:
Systematic review and meta-Analysis. Journal of Educational Computing Research, 51(3),
311–325.
Chang, C. Y., Lai, C. L., & Hwang, G. J. (2018). Trends and research issues of mobile learning studies in
nursing education: A review of academic publications from 1971 to 2016. Computers & Education,
116, 28–48.
J. Comput. Educ. (2018) 5(2):133–148 145
123
Chen, T. Y. (1994). A meta-analysis of effectiveness of computer-based instruction in mathematics
(Ph.D.). The University of Oklahoma, United States - Oklahoma. Retrieved from http://search.
proquest.com/pqdtglobal/docview/304105627/abstract/9D8837D0D8EB4A14PQ/5. Accessed 10
Sept 2015.
Cheung, A. C. K., & Slavin, R. E. (2013). The effectiveness of educational technology applications for
enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research
Review, 9, 88–113.
Close, S., Oldham, E., Shiel, G., Dooley, T., & O’Leary, M. (2012). Effects of calculators on mathematics
achievement and attitudes of ninth-grade students. Journal of Educational Research, 105(6),
377–390.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.
Cooper, H. (2016). Research synthesis and meta-analysis: A step-by-step approach (Vol. 2). Thousand
Oaks: Sage publications.
Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual
participant data versus aggregated data. Psychological Methods, 14(2), 165–176.
Cumming, G. (2012). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-
analysis. Routledge: Routledge publications.
Davis, M. R. (2010). Solving algebra on smartphones. Technology Counts, 29(26), 20–23.
DeCoster, J. (2004). Meta-analysis notes. Retrieved from http://www.stathelp.com/Meta%20analysis%
202009-06-01.pdf. Accessed 7 Nov 2015.
Ellington, A. J. (2006). The effects of non-CAS graphing calculators on student achievement and attitude
levels in mathematics: A meta-analysis. School Science and Mathematics, 106(1), 16–23.
Fabian, K., Topping, K. J., & Barron, I. G. (2016). Mobile technology and mathematics: Effects on
students’ attitudes, engagement, and achievement. Journal of Computers in Education, 3(1),
77–104.
Fu, Q. K., & Hwang, G. J. (2018). Trends in mobile technology-supported collaborative learning: A
systematic review of journal publications from 2007 to 2016. Computers & Education, 119,
129–143.
Gay, A. S., & Burbridge, L. (2016). ‘‘Bring Your Own Device’’ for Formative Assessment. Mathematics
Teacher, 110(4), 310–313.
Gurevitch, J.,Koricheva, J.,Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of
research synthesis. Nature, 555 (7695), Retrieved from https://www.nature.com/articles/
nature25753.pdf. Accessed 15 Mar 2018.
Hembree, R., & Dessart, D. J. (1986). Effects of hand-held calculators in precollege mathematics
education: A meta-analysis. Journal for Research in Mathematics Education, 17, 83–99.
Hicks, D., & Holden, C. (2007). Teaching the global dimension: Key principles and effective practice.
London: Routledge.
Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting
effect sizes in research. Child Development Perspectives, 2(3), 172–177.
Hsu, Y. (2003). The effectiveness of computer-assisted instruction in statistics education: A meta-analysis
(Ph.D.). The University of Arizona, United States - Arizona. Retrieved from http://search.proquest.
com/pqdtglobal/docview/305338759/abstract/9D8837D0D8EB4A14PQ/29. Accessed 1 Aug 2015.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of Meta-Analysis: Correcting Error and Bias in Research
Findings (2nd ed.). Thousand Oaks: Sage.
King H. J. (1997). Effects of computer-enhanced instruction in college-level mathematics as determined
by a meta-analysis (vol. 59, p. 114A). United States: The University of Tennessee.
Kulik, J. A. (1983). Synthesis of research on computer-based instruction. Educational Leadership, 41(1),
19–21.
Larwin, K., & Larwin, D. (2011). A meta-Analysis examining the impact of computer-assisted instruction
on postsecondary statistics education: 40 years of research. Journal of Research on Technology in
Education, 43(3), 253–278.
Lee, W. C. (1990). The effectiveness of computer-assisted instruction and computer programming in
elementary and secondary mathematics: A meta-analysis (Ed.D.). University of Massachusetts
Amherst, United States - Massachusetts. Retrieved from http://search.proquest.com/pqdtglobal/
docview/303852079/abstract/9D8837D0D8EB4A14PQ/18.
Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’
mathematics learning. Educational Psychology Review, 22(3), 215–243.
146 J. Comput. Educ. (2018) 5(2):133–148
123
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for sys- tematic
reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151, 264–269.
https://doi.org/10.7326/0003-4819-151-4-200908180-00135.
Moyer-Packenham, P. S., & Westenskow, A. (2013). Effects of virtual manipulatives on student
achievement and mathematics learning. International Journal of Virtual and Personal Learning
Environments,4(3), 35–50.
National Council of Teachers of Mathematics. (2000). Principles and standards of school mathematics.
Reston: Author.
Nepo, K. (2017). The use of technology to improve education. Child & Youth Care Forum, 46(2),
207–221.
Nikolaou, C. (2001). Hand-held calculator use and achievement in mathematics: A meta analysis (Ph.D.).
Georgia State University, United States - Georgia. Retrieved from http://search.proquest.com/
pqdtglobal/docview/304696658/abstract/9D8837D0D8EB4A14PQ3. Accessed 10 Sept 2015.
Planty, M., Hussar, W., Snyder, T., Kena, G., KewalRamani, A., Kemp, J., Bianco, K., & Dinkes, R.
(2009). The Condition of Education 2009 (NCES 2009-081). National Center for Education
Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC.
Rosen, Y., & Salomon, G. (2007). The differential learning achievements of constructivist technology-
intensive learning environments as compared with traditional ones: A meta-analysis. Journal of
Educational Computing Research, 36(1), 1–14.
Rosenthal, R. (1991). Meta-analytic procedures for social research (rev ed.). Beverly Hills: Sage.
Schenker, J. D. (2007). The effectiveness of technology use in statistics instruction in higher education: A
meta-analysis using hierarchical linear modeling. (Doctoral dissertation). Retrieved from ProQuest
Digital Dissertations. (AAT 3286857).
Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research
findings. Thousand Oaks: Sage publications.
Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring
systems on K–12 students’ mathematical learning. Journal of Educational Psychology,105(4),
970–987.
Steel, P. D., & Kammeyer-Mueller, J. D. (2002). Comparing meta-analytic moderator estimation
techniques under realistic conditions. Journal of Applied Psychology, 87(1), 96–111.
Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years
of research says about the impact of technology on learning: A second-order meta-analysis and
validation study. Review of Educational research, 81(1), 4–28.
Tokpah, C. L. (2008). The effects of computer algebra systems on students’ achievement in mathematics
(Ph.D.). Kent State University, United States - Ohio. Retrieved from http://search.proquest.com/
pqdtglobal/docview/304549974/abstract/9D8837D0D8EB4A14PQ. Accessed 1 Dec 2015.
Valk, J., Rashid, A. T., & Elder, L. (2010). Using mobile phones to improve educational outcomes: An
analysis of evidence from Asia. International Review of Research in Open and Distance Learning,
11(1), 117–140.
Wang, H. Y., Liu, G. Z., & Hwang, G. J. (2017). Integrating socio-cultural contexts and location-based
systems for ubiquitous language learning in museums: A state of the art review of 2009–2014.
British Journal of Educational Technology, 48(2), 653–671.
Wang, S., Jiao, H., Young, M. J., Brooks, T., Olson, J. (2007). A meta-analysis of testing mode effects in
grade K-12 mathematics tests. Educational and Psychological Measurement, 67(2), 219–238
Young, J. (2017). Technology-enhanced mathematics instruction: A second-order meta-analysis of 30
years of research. Educational Research Review, 22, 19–33
Young, J. L., Young, J. R., & Capraro, R. M. (2018). Gazing past the gaps: A growth-based assessment of
the mathematics achievement of black girls. The Urban Review, 50(1), 156–176.
Young, J. R., & Young, J. L. (2012). ‘‘But that’s not fair’’: Teacher technology readiness and African
American Students’. The Journal of the Texas Alliance of Black School Educators, 4(1), 19–32.
Young, J. R., Young, J. L., & Hamilton, C. (2013). The use of confidence intervals as a meta-analytic lens
to summarize the effects of teacher education technology courses on preservice teacher TPACK.
Journal of Research on Technology in Education, 46(2), 149–172.
Young, J. R., & Young, J. L. (2016). Young, black, and anxious: Describing the black student
mathematics anxiety research using confidence intervals. Journal of Urban Mathematics Education,
9(1), 79–93.
Yung, H. I., & Paas, F. (2015). Effects of computer-based visual representation on mathematics learning
and cognitive load. Educational Technology and Society, 18(4), 70–77.
J. Comput. Educ. (2018) 5(2):133–148 147
123
Zientek, L. R., Capraro, M. M., & Capraro, R. M. (2008). Reporting practices in quantitative teacher
education research: One look at the evidence cited in the AERA Panel Report. Educational
Researcher, 37(4), 208–216.
Jamaal Young focuses his attention on culturally responsive mathematics teaching, particularly the
educational needs of African American children, multicultural STEM project-based learning, preparation
of pre-service mathematics teachers to work with diverse learners, literature synthesis, and meta-analysis
methodology. His scholarship seeks to increase the number of underrepresented students in STEM career
fields by leveraging technological pedagogical content knowledge (TPACK), culturally responsive
pedagogies, and research synthesis methodologies to improve teaching and learning in STEM content
areas. He earned his doctorate from the Texas A&M University in Curriculum and Instruction, with an
emphasis in Mathematics education, and he is currently an Assistant Professor at the University of North
Texas.
Faith Gorumek focuses on technology integration pedagogies across K-12 content areas. He is currently
a graduate student at the University of North Texas.
Christina Hamilton focuses her attention on access and equity as it relates to the integration of
technology into the mathematics classroom. She graduated from the University of North Texas and
currently teaches mathematics to pre-service teachers at the Texas A&M Central Texas.
148 J. Comput. Educ. (2018) 5(2):133–148
123
