Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay on May 15, 2018
Content may be subject to copyright.
Creating an Evidence-Based Framework for
Selecting and Evaluating Mathematics Apps
Robin Kay
University of Ontario Institute of Technology
Oshawa, Canada
robin.kay@uoit.ca
Jae Kwak
University of Ontario Institute of Technology
Oshawa, Canada
jae.kwak@uoit.ca
Abstract: Given that there are over 80,000 educational apps in the Apple store alone (Apple, 2017),
teachers need clear guidance on choosing the most useful tools possible. The purpose of this paper is to
provide an evidence-based framework for selecting and evaluating math apps used in elementary school
settings. Based on a comprehensive review of the literature, five app types (instructive, practice-based,
constructive, productive, game-based) will be discussed in detail. Next, eight characteristics (learning
value, content quality, learning goals, usability, engagement, challenge level, feedback, and collaboration)
for evaluating the quality of math apps will be described. Finally, the role of the teacher in integrating math
apps into the classroom will be explained.
Introduction
The National Council of Teachers of Mathematics (NCTM) maintains that technology is essential to math
curricula (NCTM, 2000). However, research the impact of technology on improving mathematics education has
been mixed (Cheung & Slavin, 2013; Hattie, 2012; Murray & Olcese, 2011). Part of the problem is that a number of
studies focus on the technology used, as opposed to the software or pedagogy employed. For example, recent
studies on the use of technology in mathematics have focussed primarily on tablet use and not the type or quality of
math apps employed (e.g., An et al., 2015; Milman et al., 2014; van Deursen et al., 2014). Alon et al. (2015) argue
that the proper selection of mathematics apps is critical for integrating tablets into the classroom. Currently, there
are thousands of math apps available, most are not formally regulated, and few focus on how students actually learn
(Alon et al., 2015; Hirsh-Pasek et al., 2015). Therefore, an evidence-based framework for selecting and evaluating
math apps is required.
To develop an effective framework, over 25 peer-reviewed articles targeting the use of math apps in
elementary school environments were assessed. Thirteen of these articles focussed on either characteristics (Cayton-
Hodge et al., 2015; Falloon, 2013; Falloon, 2014; Handal et al., 2015; Hawkins, Collins, & Flowers, 2017) or type
(Alon et al., 2015; Ebner, 2015; Grandgenett et al. 2011; Handal et al., 2015) of math apps. In addition, 15
empirically-based studies on the use of tablets and math apps in elementary school classrooms were critiqued.
Based on the review of the literature, five types of math apps were identified and are described in Table 1. Next, an
amalgamation of eight characteristics was created to evaluate the quality and effectiveness of math apps (Table 2).
Results
Type of Apps
When selecting math app, the first step is to establish the desired learning outcomes. If the goal is to have
students learn a new concept, then an instructive app might be appropriate. On the other hand, if the intent is to
review concepts recently learned, practice or game-based apps would be a good choice. Constructive apps might be
more advantageous when higher level skills are targeted, and productive apps could be particularly useful for a
culminating task. In addition, some apps may represent multiple types. For example, game-based math apps can
involve considerable practice. Productive apps may incorporate the construction of knowledge and higher-level
skills. Once the type of math app is selected, the teacher can then evaluate it based on the eight characteristics
discussed below. All apps should provide clear learning goals, accurate content based on solid mathematical
principles, and an easy-to-use format. The influence of the other characteristics, though, will vary by app type.
Instructive. The primary purpose of this type of app is to teach a student a new concept or provide
tutoring/training (Ebner, 2015). These apps tend to guide students by providing organized, step-by-step, systematic
-755-
SITE 2018 - Washington, D.C., United States, March 26-30, 2018
scaffolding (Falloon, 2013). In terms of app characteristics, an ideal instructive app would provide emotional and
cognitive engagement, explicit feedback, tracking and progress reports, and a sufficient range of challenge levels
(Handal et al., 2015). While active learning is generally promoted in mathematics (NCTM, 2000), direct instruction
using math apps to learn essential facts, concepts and skills can significantly improve student achievement (Hattie,
2012; Keengwe, 2013; Pitchford, 2014; Riconscente, 2013; Zhang et al., 2015).
Practice-based. Practice-based apps are designed to help students practice new content, concepts, and skills
(Grandgenett et al.,2011). While many teachers are encouraged to promote critical and reflective thinking, a basic
knowledge of content and concepts is required to engage in higher levels of thinking, especially at the primary
school level. Practice-based apps are used to support the acquisition of foundational mathematics knowledge
(Hirsh-Pasek et al., 2015). Important characteristics in a practice app might include high emotional engagement,
timely, corrective feedback, a range of challenge levels, and progress reports.
Constructive. Constructive apps focus on exploration (Handal et al., 2016; Murray, 2011), making sense of new
information, reflection, conjecture (Grandgenett et al., 2011), skill acquisition, data management (Domingo &
Gargante, 2016) and the active manipulation of ideas and concepts (Keenwge, 2013). This type of app could be
particularly useful for applying and extending concepts or skills. The main goal for this type of app is to help
student construct understanding. The structure of a constructive app is more open-ended than practice-based or
instruction apps, and students can experiment with different scenarios. Useful characteristics for these apps would
be authentic content leading to cognitive engagement, along with sufficient control to manipulate a wide range of
parameters.
Table 1 – Types of Math Apps
Type Description Examples Evidence/Research
Instructive Direct instruction, acquiring
information, stand-alone, self-
directed, step-by-step
progression, tutoring
Math42
Number Line
Ebner, 2015; Falloon,
2013; Grandgenett et al.,
2011; Hattie, 2012;
Practice-Based Drill and practice, test-taking,
quizzes, practicing fact or skill-
based knowledge
IXL
Quick Math
Grandgenett et al., 2011;
Hattie, 2012;
Risconscente et al., 2013
Constructive Exploration, elements of
ambiguity, posing a conjecture,
develop argument, categorize,
interpret result, estimate,
compare and contrast, testing
and evaluating solutions,
making sense of new
information, questioning,
reflection
Desmos
Math Gizmos
Grandgenett et al., 2011;
Handal et al., 2015;
Hattie, 2011; Murray &
Olcese, 2011
Productive Demonstrating knowledge,
producing artefacts, creating
representations (graphs, mind
maps, videos)
Coggle (Mind Maps)
SnagIt (Videos)
Grandgenett et al., 2011;
Handal et al., 2015;
Hattie, 2011; Murray &
Olcese, 2011
Game-Based Background story, aesthetically
engaging, progressive challenge,
fantasy, curiosity, active
interactive participation,
continuous feedback loop
Mystery Math Town
Prodigy
Ebner, 2015; Falloon,
2013; Kiili et al., 2014;
Risconscente et al., 2013;
Whitton, 2014
Productive. Productive or tool-based apps (Murray & Olcese, 2011) are used to demonstrate and use knowledge
by creating math artefacts and representations (e.g., graphs, mind maps, videos) (Grandgenett et al., 2011; Handal et
al., 2016). This type of app could be useful in a culminating activity and may involve collaboration. Typical
attributes of a productive app would include an open-ended, tool-based design that is relatively easy to use and
provides numerous options for creating artefacts. Learning goals, cognitive engagement, and collaboration would be
established outside the app by the teacher and students.
-756-
SITE 2018 - Washington, D.C., United States, March 26-30, 2018
Game-based. Game-based apps involve learning and practicing concepts while playing games (Ebner, 2015;
Kiili et al., 2014; Riconscente, 2013). In a typical education-based game, students are exposed to challenging
activities structured with a narrative, rules, goals, progression and rewards (Whitton, 2014). One advantage of a
game-based app is regular interaction and engagement (Kiili et al., 2014). Another benefit is a strong narrative that
can deeply engage students, possibly leading to a state of “flow” (Hirsh-Pasek et al., 2015). However, with some
apps, educational content is artificially introduced into a game creating a quiz-like atmosphere with extrinsic
rewards (Riconscente, 2013). Compelling game-based apps are relatively easy to use and engaging, both
emotionally and cognitively. These apps provide quick corrective feedback and a wide range of challenge levels.
Preferred game-based math apps naturally bring about communication and collaboration regarding strategy
(Risconscente et al., 2013).
Characteristics of Apps
The following eight characteristics emerged from a detailed review of the literature. When evaluating an app,
not all of these characteristics are required for a math app to be useful. For example, learning goals and
collaboration may not be present, but a teacher can augment the process outside of the app by communicating the
learning outcomes and arranging for students to work in teams. Additionally, there is no exact formula for assessing
each characteristic because the learning goals and type of app can vary. Practice-based apps may score high on
usability and engagement but help students to acquire foundational knowledge. Apps for constructing knowledge
may be harder to use and less engaging but offer more extrinsic rewards and a higher challenge-level.
Learning value. The primary goal of any app is to support and promote learning. Not surprisingly, then, one of
the most researched characteristics of math apps is learning value, often measured by assessing teacher or student
perceptions. Key areas of focus are control over learning (Clark & Luckin, 2013), promoting knowledge building
and information searching skills (Chou et al., 2014; Domingo & Gargante, 2016), improving learning outcomes and
achievement (An et al., 2015; Milman et al., 2014; van Deuersen, 2014), and have a positive impact on student
achievement. Promising areas of learning value, not yet formally studied for math apps, include developing
arguments, categorizing, interpretation, comparing and contrasting, making sense of new information and
generalizing relationships (Grandgenett et al., 2011).
Content quality. Limited research has been conducted on the quality of content addressed in math apps (Cayton-
Hodge et al., 2015). Moyer-Packenham et al. (2016) recommend that the concepts and knowledge included in an
app must be faithful to the underlying mathematical properties. Other issues, not yet addressed by researchers of
math apps, include gender, cultural and ethnic bias (Papadakis et al., 2017), authenticity (Boone & Higgins, 2012),
and accuracy (Alon et at., 2015).
Learning goals. A number of theorists have noted the need for clear learning goals to be communicated to
students when they are using math apps (Falloon, 2013, 2014; Wiggins & McTighe, 2005). The absence of learning
goals can discourage students, lead to off-task behaviour or the pursuit of entertainment and gamification (Falloon,
2014). It is worth noting that a teacher can articulate the learning goals of an app if these goals are not explicitly
stated in an app.
Usability. Usability is necessary but not sufficient for a math app to promote meaningful learning. Critical
features identified include using the appropriate language level, user-friendliness, clear instructions, and navigation
(An et al., 2015; Clark & Luckin, 2013; Ebner, 2015; Falloon, 2013, 2014; Handal et al., 2015). The primary goal
of usability is to reduce cognitive load while using math apps, so full attention can be directed toward learning the
targeted knowledge or skills.
Engagement. This characteristic is a highly valued and extensively researched with respect to tablet use. Key
descriptors focus on fun, entertainment, excitement, aesthetics, richness of interactions, pacing, and persistence (An
et al., 2015; Cayton-Hodge et al., 2015; Domingo & Gargante, 2016; Handal et al., 2015; Hattie, 2012; Hawkins et
al., 2017; Keenwge, 2013; Milman et al., 2014; Risconscente et al., 2013). Most studies ask students and teachers
whether working with a tablet and math apps is motivating or engaging, overall (An et al., 2015; Clark and Luckin,
2013; Keenwge, 2013; Kiili et al., 2014; Riconscente, 2014). However, engagement is a more complex concept
consisting of at least three components: behavioural (involved in activities), emotional (positive and negative
reactions), and cognitive (investment in learning) (Fredricks et al., 2004). Math apps may be emotionally engaging
and highly interactive, for example, but they need to be cognitively engaging to support learning (Cayton-Hodges et
al., 2015; Hirsh-Pasek et al., 2015). Even if students appear to be engaged, it is hard to ascertain without direct
observation what they are focusing on (Falloon, 2014).
-757-
SITE 2018 - Washington, D.C., United States, March 26-30, 2018
Table 2 – Characteristics for Evaluating Math Apps
Characteristic Descriptors Evidence/Research
Learning Value Structures, trial and error,
gamification, remembering,
understanding, applying, analysing,
evaluating, creating, achieving
fluency, academic improvement
An et al., 2015; Clark & Luckin, 2013;
Falloon, 2013, 2014; Handal et al., 2015;
Hawkins et al., 2017; Milman et al.,
2014; van Deuersen, 2014
Content Quality Accuracy, faithful to underlying math
principles,
Alon et al., 2015; Cayton-Hodge et al.,
2015; Moyer-Packenham et al., 2016
Learning Goals Clear objectives, personal Falloon, 2013, 2014; Keenwge, 2013;
Wiggins & McTighe, 2005
Usability User-friendly, appropriate language,
distraction-free, clear instructions,
easy to follow, intuitive, navigation
An et al., 2015; Clark & Luckin, 2013;
Ebner, 2015; Falloon, 2013, 2014;
Handal et al., 2015; van Deuersen, 2014
Engagement Emotional (look and feel,
entertainment value, fun, exciting)
Behavioural (rich interactions,
persistence)
Cognitive (pacing, control over
settings, desire to participate)
An et al., 2015; Cayton-Hodge et al.,
2015; Domingo & Gargante, 2016;
Falloon, 2014; Fredricks et al., 2004;
Handal et al., 2015; Hattie, 2012;
Hawkins et al., 2017; Keenwge, 2013;
Kiili et al., 2014; Milman et al., 2014;
Risconscente et al., 2013; Whitton, 2011;
Van Deursen et al., 2014
Challenge Level Adaptability, differentiation, levelling,
independent learning, selecting
content parameters, instructional
pacing
An et al., 2015; Cayton-Hodge et al.,
2015; Falloon, 2013, 2014; Handal et al.,
2015; Hawkins et al., 2017; Milman et
al., 2014
Feedback Scaffolding, hints/corrective,
formative, accommodations, tracking,
progress reports, text vs visual
feedback, intrinsic vs, extrinsic
Cayton-Hodge et al., 2015; Ebner, 2015;
Falloon, 2013, 2014; Hawkins et al.,
2017; Handal et al., 2015; Keengwe,
2013; Kiili et al., 2014; Riconscente
Collaboration Social interaction, sharing Ebner, 2015; Handal et al., 2015; Hattie,
2012; Keenwge, 2013
Challenge level. Challenge level, sometimes referred to as differentiation or the adaptability of an app to adjust
to and meet the learning needs of individual users, is another significant app characteristic (An et al., 2015; Cayton-
Hodges et al., 2015; Milman et al., 2014). Descriptors used by researchers to describe challenge level include
adaptability, differentiation, levelling, independent learning, selecting content parameters, and instructional pacing
(An et al., 2015; Cayton-Hodge et al., 2015; Falloon, 2013, 2014; Handal et al., 2015; Hawkins et al., 2017; Milman
et al., 2014). The fundamental premise is that a responsive math app needs to match students’ personal preferences
(e.g., look and feel, avatars) and/or ability level. Falloon (2104) cautions that if the challenge level does not align
with cognitive ability, elementary school students will experience app fatigue, disengage, or seek entertainment.
Feedback. Different kinds of math app feedback, including rewards and visual progress markers, the status of
the problem being solved, corrective guidance, and conceptual correction can help students understand concepts
better (Cayton-Hodges et al., 2015; Falloon, 2013; Hirsh-Pasek et al., 2015). Tracking student behaviour in the form
of a report can provide useful information for both students and the teacher regarding progress toward learning goals
(Ebner, 2015; Falloon, 2014). Intrinsic rewards can motivate students to learn rote skills that require considerable
practice (Ebner, 2015). However, extrinsic rewards in the form of authentic learning goals with corrective guidance
may be preferable for older students learning higher level concepts.
Collaboration. Hattie (2012) presents several meta-analyses suggesting that discussion, cooperative activities,
and peer tutoring significantly improve student achievement. Hirsh-Pasek et al. (2015) add that working together
toward common learning goals and explaining one’s reasoning to a peer deepens mathematical understanding. Math
apps that allow students to collaborate with, share and co-create knowledge are rare at the elementary school level
(Ebner, 2015). However, the potential of general tools like Google Apps, to aid in the co-construction of
mathematical ideas and artefacts is considerable (Lee et al., 2015; Papadakis et al., 2017). Even though an app may
-758-
SITE 2018 - Washington, D.C., United States, March 26-30, 2018
not be designed to be collaborative, teachers can frame the use of math apps within a co-operative environment
where students share their ideas, thoughts and responses.
Role of the Teacher
The teacher is absolutely fundamental in determining the success or failure of a math app (Risconscente et al.,
2013). For example, many apps do not provide explicit learning goals, nor do they connect the math app to specific
course curricula. Teachers can supplement this process by communicating learning goals to the class and selecting
apps that meet course learning objectives. Additionally, as stated earlier, teachers can optimize the use of
constructive and productive apps using a collaborative, team-based approach. Furthermore, teachers can select a
wide range of math apps to accommodate the ability and interest levels of students (Bouck et al., 2016). Teachers
must also monitor app use during class to ensure the intended learning goals are pursued. It is particularly
challenging to determine whether actual learning is occurring without observing and interacting with students using
the math apps (Falloon, 2014). Finally, it is critical to integrate math apps with the correct teaching strategy.
Matching the right math app to the desired learning goals and appropriate learning approach is a challenging but
necessary process to achieve meaningful learning gains (Handal et al., 2015).
Future Research
The purpose of this paper was to provide an evidence-based framework for selecting and evaluating math apps
used in elementary school settings. Five application types and eight characteristics were presented as a starting
point, based on a detailed review of the current literature on math apps. The next step is to create a metric based on
this framework and test it with small, formative case studies. This type of qualitative analysis can help refine
parameters with practical and real-world feedback. Once the scale is refined, it can then be tested on a larger sample
of students and teachers to establish reliability and validity. Finally, the scale can be employed to evaluate
promising and document math apps for elementary school.
References
Alon, S., An, H., & Fuentes, D. (2015). Teaching mathematics with Tablet PCs: A professional
development program targeting primary school teachers. Christou, G., Maromoustakos, S.,
Mavrou, K., Meletiou-Mavrothers, M. & Stylianou, G. (Eds.), Tablets in K-12 education:
Integrated experiences and implications (pp.175-197). Hershey, PA: IGI Global.
An, H., Alon, S., & Fuentes, D. (2015). iPad implementation approaches in K-12 school environments.
Alon, S., An, H. & Fuentes, D. (Eds.), Tablets in K-12 education: Integrated experiences and
implications (pp. 22-33). Hershey, PA: IGI Global.
Apple (2017). App Store – Education. Retrieved from https://www.apple.com/education/apps-books-and-more/
Boone, R., & Higgins, K. (2012). The software √-list: Evaluating educational software for use by
students with disabilities. Journal of Special Education Technology, 27(1), 50-63.
Bouck, E.C., Satsangi, R. & Flanagan, S. (2016) Focus on inclusive education: Evaluating apps for
students with disabilities. Success, Childhood Education, (92),4, 324-328, doi:
0.1080/00094056.2016.1208014
Cayton-Hodges, G. A., Feng, G., Pan, X. (2015). Tablet-based math assessment: What can we learn
from math apps? Educational Technology & Society, 18 (2), 3–20. Retrieved from
http://www.ifets.info/journals/18_2/2.pdf
Cheung, A.C., & Slavin, R.E. (2011). The effectiveness of educational technology applications for
enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research
Review, 9, 88-113. doi: 10.1016/j.edurev.2013.01.001
Chou, C. C., Block, L., & Jesness, R. (2014). Strategies and challenges in iPad initiative: Lessons learned from year
two. IADIS International Journal on WWW/Internet, 12(2), 85-101.
Clark, W., & Luckin, R. (2013). IPads in the Classroom. London Knowledge Lab,1, 1-31. Retrieved
from http://www.thepdfportal.com/ipads-in-the-classroom-report-lkl_61713.pdf
Domingo, M.G., & Gargante, A.B. (2016). Exploring the use of educational technology in primary
education: Teachers’ perception of mobile technology learning impacts and applications’ use in
the classroom. Computers in Human Behavior, 56, 21-28. doi: 10.1016/j.chb.2015.11.023
Ebner, M. (2015). Mobile applications for math education – how should they be done? In Crompton,
H., & Traxler, J. (Eds.). Mobile learning and mathematics. Foundations, design, and case studies
(pp. 20.32), New York: Routledge.
-759-
SITE 2018 - Washington, D.C., United States, March 26-30, 2018
Falloon, G. (2013). Young students using iPads: App design and content influences on their learning
pathways. Computers & Education, 68, 505-521. doi: 10.1016/j.compedu.2013.06.006
Falloon, G. (2014). Researching young students’ learning pathways using iPads: What's going on
behind the screens? Journal of Computer Assisted Learning, 30(4), 318–336.
doi: 10.1111/jcal.12044
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the
concept, state of the evidence. Review of Educational Research, 74(1), 59–109.
doi:10.3102/00346543074001059
Grandgenett, N., Harris, J., & Hofer, M. (2011). An activity-based approach to technology integration
in the mathematics classroom. NCSM Journal of Mathematics Education Leadership, 13(1), 19–
28.
Handal, B., Campbell, C., Cavanagh, M., & Petocz, P. (2016). Characterising the perceived value of
mathematics educational apps in preservice teachers. Mathematics Education Research
Journal, 28(1), 199–221. doi: 10.1007/s13394-015-0160-0
Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning. New York: Routledge.
Hawkins, R. O., Collins, T., Hernan, C., & Flowers, E. (2017). Intervention in School and Clinic,
52(3), 141-147. doi: 10.1177/1053451216644827
Hirsh-Pasek, K., Zosh, J. M., Golinkoff, R., Gray, J. H., Robb, M. B., & Kaufman, J. (2015). Putting
education in “educational” apps: Lessons from the science of learning. Psychological Science in
the Public Interest, 16(1), 3–34. doi:10.1177/1529100615569721
Keengwe, J. (2013). iPad integration in an elementary classroom. Anderson, A. & Hur, J.W. (Eds.),
Pedagogical applications and social effects of mobile technology integration (pp. 42-54).
Hershey, PA: IGI Global
Kiili, K., Ketamo, H., Koivisto, H., & Finn, E. (2014) Studying the user experience of a tablet based
mathematics game. International Journal of Game-Based Learning, 4(1), 60-77.
doi:10.4018/ijgbl.2014010104
Milman, N., Carlson-Bancroft, A., & Boogart (2014). Examining differentiation and utilization of
iPads across content areas in an independent, preK–4th Grade elementary school. Computers in
the Schools, 31(3), 119-133. doi: 10.1080/07380569.2014.931776
Moyer-Packenham, P.S., Salkind, G., & Bolyard, J.J. (2008). Virtual manipulatives used by K-8
teachers for mathematics instruction: Considering mathematical, cognitive, and pedagogical
fidelity. Contemporary Issues in Technology and Teacher Education, 8(3), 202–218.
Murray, O. T., & Olcese, N. R. (2011). Teaching and learning with iPads, ready or not? TechTrends,
55(6), 42-48. doi: 10.1007/s11528-011-0540-6
Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2017). Designing and creating an educational app
rubric for preschool teachers. Education and Information Technology, 1-19 doi: 10.1007/s10639-
017-9579-0
Pitchford, N. (2014). Unlocking talent: Evaluation of a tablet-based Masamu intervention in a
Malawian primary school. Retrieved from https://onebillion.org.uk/ downloads/unlocking-talent-
final-report.pdf
Riconscente, M.M. (2013). Results from a controlled study of the iPad fractions game Motion Math.
Games and Culture, 8 (4), 186-214. doi: 10.1177/1555412013496894
National Council of Teachers of Mathematics (NCTM). (2000). Principles and standards for school
mathematics. Reston, VA: NCTM
Van Deursen, A.J.A.M., Ruijter, L.P., & Allouch, S.B. (2014). Tablet use in primary education:
Adoption hurdles and attitude determinants. Education and Information Technologies, 21(5), 971-
990. doi: 10.1007/s10639-014-9363-3
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd Ed.). Alexandria, VA: Association
for Supervision & Curriculum Development
Zhang, M., Trussell, R.P., Gallegos, B., & Asam, R.R..(2015). Using mathematics apps for improving
student learning: An exploratory study in an inclusive fourth grade classroom. Tech Trends, 59(2),
32-39. doi: 10.1007/s11528-015-0837-y
-760-
SITE 2018 - Washington, D.C., United States, March 26-30, 2018
