Understanding How Teachers Influence the Effectiveness of STEM-Based Mobile Apps

Abstract

Limited research has been conducted on teacher-related factors that influence student learning performance after using STEM-based mobile apps. The purpose of this study was to examine a wide range of teacher-based factors that might influence student learning performance including teacher attitudes, preparation and implementation strategies, and demographics. Twelve mathematics and science classes (n=838 students), grades 7 to 8, and 20 teachers participated in the study. Overall, student learning performance increased by 26% after using STEM-based apps. Teacher attitudes towards the design, engagement and learning value of apps were significantly correlated with learning performance. With respect to teacher preparation and implementation strategies, longer preparation time, customized (vs. pre-designed) support materials, not using a teacher-led approach, and using apps for review or homework resulted in significantly higher student gains in learning performance. Regarding teacher demographics, students who had female, older, or more experienced teachers achieved higher gains in learning performance. More research is needed, perhaps in the form interviews, focus groups and observational data to better understand the interaction between teacher-related factors and student performance.

Full text

Understanding How Teachers Influence the
Effectiveness of STEM-Based Mobile Apps
Robin Kay
Professor
University of Ontario Institute of Technology
Oshawa, Canada
robin.kay@uoit.ca
Abstract: Limited research has been conducted on teacher-related factors that influence student learning
performance after using STEM-based mobile apps. The purpose of this study was to examine a wide range
of teacher-based factors that might influence student learning performance including teacher attitudes,
preparation and implementation strategies, and demographics. Twelve mathematics and science classes
(n=838 students), grades 7 to 8, and 20 teachers participated in the study. Overall, student learning
performance increased by 26% after using STEM-based apps. Teacher attitudes towards the design,
engagement and learning value of apps were significantly correlated with learning performance. With
respect to teacher preparation and implementation strategies, longer preparation time, customized (vs. pre-
designed) support materials, not using a teacher-led approach, and using apps for review or homework
resulted in significantly higher student gains in learning performance. Regarding teacher demographics,
students who had female, older, or more experienced teachers achieved higher gains in learning
performance. More research is needed, perhaps in the form interviews, focus groups and observational
data to better understand the interaction between teacher-related factors and student performance.
Introduction
Constructive-based, educational mobile apps support exploration, investigation, constructing solutions, and
manipulating parameters instead of memorizing and retaining a series of facts (Kay & Knaack, 2008). Numerous
studies have reported that learning performance increased significantly when constructive STEM-based apps,
targeting mathematics and science concepts, are used by middle school students (Bulut et al., 2015; Chang et al.,
2013; Clark & Luckin, 2013; Freebody et al., 2007; Kay, 2011; Kay & Knaack, 2008; Lin et al., 2011; Moyer-
Packenham & Suh, 2012; Riconscente, 2013; Zhang et al., 2015). Keengwe et al. (2008) identified a number of
potential teacher factors that could have an impact on student learning outcomes including attitudes, lack of time,
planning, experience and confidence, and integration strategies.
To date, limited systematic research has been conducted on teacher-related factors that might affect student
learning performance when using STEM-based apps. One study looked at the impact of teacher attitudes on
learning with STEM-based apps. Kay, Knaack and Petrarca (2009) reported that teachers’ attitudes towards the
engagement value of an app (but not design or learning value) were significantly correlated with middle school
students’ learning performance. Another study examined teaching strategies and the use of apps (Kay, 2013). Kay
(2013) reported that middle school students were more positive about apps use and performed significantly better
when a teacher-led approach was used compared to individual use of apps. No research could be found regarding
the impact of teacher gender, age or experience on learning performance when STEM-based apps are used in middle
school classrooms. The purpose of this study, then, was to conduct a comprehensive analysis of teacher-based
variables that might influence learning performance when using constructive mobile apps in mathematics and
science. Three research questions were asked:
1. To what extent are teacher attitudes towards STEM-based apps related to middle school students’ learning
performance?
2. To what extent are lesson plan preparation, purpose and teaching strategies related to middle school
students’ learning performance with STEM-based apps?
3. To what extent are teacher gender, age and years of experience related to middle school students’ learning
performance with STEM-based apps?
Method
Participants
Students and teachers in this study lived in a suburban region (650,000 people) in Ontario, Canada. Eight-
hundred thirty-eight (325 males, 363 females, 150 other) between 10 and 13 years of age, participated. Students
-1182-
E-Learn 2018 - Las Vegas, NV, United States, October 15-18, 2018

were enrolled in either grade 7 (n=481) or 8 (n=357). Over three-quarters of the students (n=533, 77%) agreed or
strongly agreed that they were comfortable using computers (M=4.1 out of 5, SD=0.8). Almost two-thirds of the
students (n=445, 65%) agreed or strongly agreed that they were good at doing mathematics, however, one quarter
were neutral about their mathematical ability (M=3.8 out of 5, SD=0.9). Just of half of the students (n=384, 55%)
agreed or strongly agreed that they were good at doing science, with one third being neutral about their science
ability (M=3.6 out of 5, SD=0.9).
Twenty middle school teachers (10 male, 10 female) with 2 to 33 years of teaching experience ( M=11.1,
SD=7.6) participated in the study. Six teachers were born between 1946 to 1964 (Baby Boomers, Tapscott, 2009),
five teachers between 1965-1976 (Generation X, Tapscott, 2009) and seven teachers were born between 1977 and
1990 (Net Generation, Tapscott, 2009). All teachers agreed or strongly agreed that they were satisfied with their app
lessons. Ninety-five percent of the teachers were comfortable using apps, had enough time to complete their app
lessons, and found the supporting app materials helpful.
STEM Apps Used
Fifteen STEM-based apps were chosen from the Explore Learning Collection (www.explorelearning.com) and
focused on math (fractions, integer operations, slope, rotations, reflections, transitions, equations) and science
(conduction, convection, energy conservation, food chain, ecosystems, heat transfer, levers, pulleys). Links to the
actual apps used and support materials are available at tinyurl.com/ELEARN-STEM-AppList. All apps were
exploratory in nature and supported a constructive approach to learning. Each app came with a pre-designed,
student exploration sheet to guide learning.
Data Collection
Student learning performance was determined by pre- and post-tests developed by the instructor or provided by
the app. The content of each question was rated based on four Bloom’s taxonomy levels (remembering,
understanding, application, and analysis) (Krathwohl, 2002). Teacher attitudes were measured based on surveys
developed by Kay & Knaack (2008, 2009) and focused on design (n=5 items, r=0.73), engagement (n=4 items,
r=0.90) and learning (n=6 items, r=0.87). Internal reliability estimates were considered acceptable for measures
used in social sciences (Kline, 1999, Nunnally, 1978).
Procedure
All teachers participated and agreed they were properly trained in a half-day workshop focussing on the
selection, evaluation and use of mathematics apps. The extra time required to prepare for app lesson ranged from 0
to 60 minutes with an average of (M=13.5, SD=19.2). In a typical lesson, a pre-test was given, students used an app
with the student guide or were led by the teacher for 20 to 150 minutes (M=46.7, SD=25.9), then a post-test was
delivered.
Results
Student Learning Performance
Significant differences between pre- and post-test scores were observed for remembering (24% increase),
understanding (9% increase), application (21% increase) and analysis (27% increase) knowledge areas. Effect sizes
ranged from 0.20 to 0.65 reflecting a moderate to large differences (Cohen, 1988). See table 1.
Table 1. Learning Performance Before and After Using STEM-Based Apps
Learning Activity n % Pre-Test
Mean (SD)
% Post-Test
Mean (SD)
% Change
Mean(SD)
t(df) Cohen’s d
Remembering 380 39.3 (38.1) 63.3 (35.6) 24.0 (39.9) 11.7 (379) * 0.65
Understanding 203 44.8 (42.6) 53.4 (42.6) 8.6 (34.0) 3.6 (203) * 0.20
Application 144 54.9 (37.8) 75.9 (29.7) 21.0 (38.3) 6.6 (143) * 0.62
Analysis 177 41.6 (38.0) 69.0 (34.8) 27.4 (39.3) 9.3 (176) * 0.75
* p <.001
Teacher Attitudes and Learning Performance
Most teachers in this study rated the design of apps highly. The average score ranged from 4.0 to 4.5 out of 5
for the five design features rated. Ratings for the engagement value of apps were similar to the design value with
-1183-
E-Learn 2018 - Las Vegas, NV, United States, October 15-18, 2018

ratings ranging from 4.4 to 4.5. Teachers also rated the learning value of apps high with the average score ranging
from 4.1 to 4.5. Teacher attitudes towards the design (r=0.24, p <.001), engagement (r=0.20, p <.001), and learning
(r=0.20, p <.001) value of STEM-based apps were significantly correlated with student learning performance.
Teaching Strategies and Learning Performance
Time spent on extra lesson plan preparation was significantly and positively correlated with middle-school
students’ learning performance. Students who used customized teacher supports materials ( M=39.6%, SD=31.7)
showed significantly higher gains in performance than students who used the pre-designed app materials ( M=22.8%,
SD=23.1) (t(503)=4.8, p <.001, Cohen’s d = 0.61).
With respect to the purpose of a lesson, students who used STEM-based apps for homework ( M=31.0%,
SD=25.6) showed significantly higher gains in learning performance than students who did not use apps for this
purpose (M=23.1%, SD=31.0) (t(503)=2.5, p <.05, Cohen’s d = 0.28). Students who used STEM-based apps for
reviewing concepts (M=34.8%, SD=29.5) achieved significantly higher gains in learning performance than students
who did not use apps for this purpose (M=24.5%, SD=31.8) (t(476)=2.1, p <.05, Cohen’s d = 0.34). There was no
significant difference in learning performance gains between students who used STEM-based apps for exploring
new concepts and students who did not use apps for this purpose (t(476)=0.3, n.s.).
Students who experienced teacher-led lesson plans with STEM-based apps showed significantly lower gains in
learning performance (M=10.3%, SD=28.3) than students who did not experience this strategy (M=27.4%, SD=31.1)
(t(503) =5.0, p <.001, Cohen’s d = 0.58). There was no significant difference between students who followed a
pairs-based strategy with STEM-based apps and students who did not use this strategy (t(503)=0.4, n.s.).
Teacher Demographics and Learning Performance
Students who had male teachers (M=17.1%, SD=30.6) achieved significantly lower gains in learning
performance than students who had female teachers (M=26.8%, SD=31.2) (t(503)=2.9, p <.005, Cohen’s d = 0.31).
This difference may have been related to preparation time. Male teachers (M=9.7 minutes SD=11.3) spent
significantly less time preparing for STEM-based apps lesson than female teachers ( M=30.7 minutes, SD=27.1)
(t(836)=11.4, p <.001, Cohen’s d = 1.01).
An ANOVA indicated that there were significant differences in learning performance based on teachers’ age
(F(3, 485)= 17.4, p< .001). Students with older teachers (1946-1964) (M=31.9%, SD=31.3) achieved higher scores
than students with younger teachers (1946-1964) (M=13.7%, SD=31.0) (Cohen’s d=0.58).
Finally, teacher experience level was significantly correlated with gains in student learning performance ( r=0.16, p
<.001).
Discussion
Teacher Attitudes and Student Learning Performance
Teachers agreed that design, engagement and learning value of the STEM-based apps used in this study were of
high quality, a result that is consistent with previous research (Clarke & Bowe, 2006; Kay & Knaack, 2008; Kay,
2011; Kay et al., 2009; McCormick & Li, 2006). A new finding is that teacher attitudes toward the design,
engagement and learning value of the apps was significantly correlated with gains in learning performance. It is
possible that teacher commitment to the quality of an app influences student outcomes. However, the magnitude of
the correlation coefficients was relatively low indicating that other factors might be more important. More research
is needed to understand the nature of this effect and whether it is truly meaningful.
Lesson Preparations and Implementation Strategies and Student Learning Performance
Teacher preparation time was significantly related to student success. This result is particularly relevant given
that male teachers spent significantly less time than female teachers preparing for app-based lessons. Furthermore,
when teachers created customized guides for their students, learning gains increase significantly. These findings are
consistent with Clark & Luckin’s (2013) claim that careful planning is required for effective app use. It is
speculated that app-based lessons may require more time and effort than expected by some teachers.
Learning performance increased when apps were used for homework or review, but not for exploring new
concepts. This result is somewhat surprising given that the apps selected for this study were designed to help
students construct knowledge. It is conceivable that middle school students are too young to construct knowledge
-1184-
E-Learn 2018 - Las Vegas, NV, United States, October 15-18, 2018

effectively or need more time with apps to fully develop their understanding of more challenging concepts. It is also
possible that a teacher-led approach undermined the constructive value of STEM-based apps.
It is worth noting that lower student performance as a result of using a teacher-led strategy was not consistent
with the results reported by Kay (2013). It is unclear why the exact opposite result was reported in this study.
Potential confounding variables might include the purpose for using apps, preparation time, type of knowledge
acquired, and support materials used. A more detailed analysis is required involving observational data and
interviews or focus groups to fully understand the dynamics of how teaching purpose and strategy are affecting
student learning.
Teacher Demographics and Student Learning Performance
It was somewhat unexpected and concerning that teacher gender had a significant impact on student learning.
At first glance, it appears that female teachers spent more time preparing for lessons and that may have been the
main contributor to improving student performance. Interview data might help further explain this unusual, but
important finding.
Tapscott (2009) suggested that students born in a digital world are more comfortable with technology.
Consequently, one might expect younger teachers in this study to implement STEM-based apps more effectively
than older teachers. However, the opposite result was observed. Older, more experienced teachers were more
effective with respect to using apps to improve student performance. One explanation might be that the apps used in
this study were well designed and easy to use – all the teachers said they were comfortable using them.
Consequently, the potential challenge of using an app may have been rendered non-factor. Additionally, the result
might indicate that the effective integration of apps requires solid pedagogical knowledge and teaching experience to
increase student success. It would be helpful in future studies to examine how more experienced teachers think
when they incorporate mobile apps compared to their less experienced peers.
Conclusions and Future Research
This study examined the influence of teacher-related factors on middle school students’ learning performance
with STEM-based mobile apps. Teacher attitudes towards the design, engagement and learning value of STEM-
based apps were significantly correlated with gains in learning performance, however, the magnitude of the
correlations was relatively small. Teaching implementation and strategies appear to have a marked impact on
student learning performance. Preparation time, support materials, purpose and implementation strategy
significantly affect student learning. Specifically, increased preparation time, creating customized support materials,
and using apps for reviewing concepts appeared to work best for middle school students. Teacher demographics
also had a significant impact on the effectiveness of mobile apps. Older teachers with more experience appeared to
integrate apps more effectively than younger teachers with less experience. Furthermore, female teachers did a
more effective job at implementing apps. It is argued that qualitative research is needed to help explain why certain
teacher-related factors have a significant impact on student learning performance when STEM-based apps are used
in middle school classrooms.
.
References
Bulut, M. B., Akcakin, H. U., Kaya, G., & Akcakin, V. (2015). The effects of Geogebra on third grade primary
students’ academic achievement in fractions. Mathematics Education, 11(2), 347-355. doi:
10.12973/iser.2016.2109a
Chang, W.-L., Yuan, Y., Lee, C.-Y., Chen, M.-H., & Huang, W.-G. (2013). Using Magic Board as a teaching aid in
third grader learning of area concepts. Educational Technology & Society, 16(2), 163–173.
Clark, W., & Luckin, R. (2013). IPads in the Classroom. London Knowledge Lab,1, 1-31. Retrieved from
http://www.thepdfportal.com/ipads-in-the-classroom-report-lkl_61713.pdf
Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd edition). New York: Academic Press.
Freebody, P., Muspratt, S., & McRae, D. (2007). Evaluating the learning federation’s online curriculum content
initiative. 1-144. Retrieved from http://www.ndlrn.edu.au/verve/_resources/freebody_final_report_2007.pdf
Kay, R. H. (2013). Evaluating the instructional architecture of web-based learning tools (WBLTs): Direct instruction
vs. constructivism revisited. Journal of Interactive Online Learning, 24(1),33-51. doi:
https://doi.org/10.1080/10494820.2011.641675
-1185-
E-Learn 2018 - Las Vegas, NV, United States, October 15-18, 2018

Kay, R. H., & Knaack, L. (2008). Exploring the impact of learning objects in middle school mathematics and
science classrooms: A formative analysis. Canadian Journal of Learning and Technology, 34(1). Retrieved
from https://www.cjlt.ca/index.php/cjlt/article/view/26430/19612
Kay. R. H., & Knaack, L. (2009). Assessing learning, quality and engagement in learning objects: The learning
object evaluation scale for students (LOES-S). Education Technology Research and Development, 57(2), 147-
168. doi: 10.1007/s11423-008-9094-5
Kay, R. H., Knaack, L., & Petrarca, D. (2009). Exploring teacher perceptions of web-based learning
tools. Interdisciplinary Journal of E-Learning and Learning Objects, 5, 27-50.
Kay, R. H. (2011). Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT
Evaluation Scale. Computers in Human Behaviour, 27(5), 1849-1856. doi:10.1016/j.chb.2011.04.007
Kline, P. (1999). The handbook of psychological testing (2nd ed.). London: Routledge
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41(4): 212–218.
doi:10.1207/s15430421tip4104_2
Lin, C., Shao, Y., Wong, L., Li, Y., & Niramitranon, J. (2011). The impact of using synchronous collaborative
virtual tangram in children’s geometric. The Turkish Online Journal of Educational Technology, 10(2), 250-
257.
Moyer-Packenham, P. S., & Suh, J. M. (2012). Learning mathematics with technology: The influence of virtual
manipulatives on different achievement groups. Journal of Computers in Mathematics and Science Teaching,
31(1), 39–59.
Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill
Riconscente, M.M. (2013). Results from a controlled study of the iPad fractions game Motion Math. Games and
Culture, 8(4), 186-214. doi: 10.1177/1555412013496894
Tapscott. D. (2009). Grown up digital. McGraw Hill: New York.
Zhang, M., Trussell, R.P., Gallegos, B., & Asam, R.R. (2015). Using mathematics apps for improving student
learning: An exploratory study in an inclusive fourth grade classroom. Tech Trends, 59(2), 32-39. doi:
10.1007/s11528-015-0837-y
-1186-
E-Learn 2018 - Las Vegas, NV, United States, October 15-18, 2018