Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay on Sep 17, 2014
Content may be subject to copyright.
CANADIAN JOURNAL OF SCIENCE, MATHEMATICS
AND TECHNOLOGY EDUCATION, 12(4), 350–366, 2012
Copyright
C
OISE
ISSN: 1492-6156 print / 1942-4051 online
DOI: 10.1080/14926156.2012.732189
Examining Factors That Influence the Effectiveness
of Learning Objects in Mathematics Classrooms
Robin H. Kay
Faculty of Education, University of Ontario Institute of Technology, Oshawa, Ontario, Canada
Abstract:
Learning objects are interactive online tools that support the acquisition of specific
concepts. Limited research has been conducted on factors that affect the use of learning objects in
K–12 mathematics classrooms. The current study examines the influence of student characteristics
(gender, age, computer comfort level, subject comfort level, and mathematics grade), instructional
design (structured vs. open ended), and teaching strategy (teacher led vs. student based) on student
attitudes toward the use of learning objects and learning performance. Data in the form of surveys and
pre- and posttests were collected from 286 middle and secondary school students. Higher computer
and subject area comfort ratings were significantly correlated with more positive student attitudes
about learning objects. Older students in higher grades learned more than younger students in lower
grades after using learning objects. Learning performance was significantly higher for students who
used structured (vs. open-ended) learning objects and participated in teacher-led (vs. student-based)
lessons. It is speculated that younger students might need more scaffolding when using mathematics-
based learning objects.
R
´
esum
´
e: Les objets d’apprentissages sont des outils en ligne qui facilitent l’acquisition de cer-
tains concepts sp
´
ecifiques. Il y a peu de recherches sur les facteurs qui affectent l’utilisation des
objets d’apprentissage dans les cours de math
´
ematiques en cinqui
`
eme ann
´
ee de secondaire. La
pr
´
esente
´
etude se penche sur l’influence des caract
´
eristiques individuelles des
´
etudiants (sexe,
ˆ
age,
habilet
´
es informatiques, connaissance de la mati
`
ere et notes obtenues en math
´
ematiques), le type
de mat
´
eriel p
´
edagogique (structur
´
e ou ouvert) et les strat
´
egies d’enseignement (enseignement dirig
´
e
par les enseignant ou bas
´
e sur les apprenants) sur les attitudes
`
al’
´
egard de l’utilisation des objets
d’apprentissage et la performance. Les donn
´
ees, sous forme d’enqu
ˆ
etes et de pr
´
e-tests et post-tests,
ont
´
et
´
e recueillies
`
a partir des r
´
eponses de 286
´
etudiants de niveau
´
el
´
ementaire (deuxi
`
eme cycle)
et secondaire. Il y a une corr
´
elation significative entre d’une part les habilet
´
es informatiques ainsi
que le niveau de connaissance de la mati
`
ere, et d’autre part l’attitude positive devant les objets
d’apprentissage. Les
´
etudiants plus
ˆ
ag
´
es et ceux des niveaux sup
´
erieurs ont mieux appris gr
ˆ
ace
`
a
l’utilisation des objets d’apprentissage que les
´
el
`
eves les plus jeunes et ceux des niveaux inf
´
erieurs.
La performance d’apprentissage est significativement plus
´
elev
´
ee chez les
´
etudiants qui ont utilis
´
e
des objets d’apprentissage structur
´
es et chez ceux qui ont particip
´
e
`
a des cours dirig
´
es par les en-
seignants. Nous avanc¸ons l’hypoth
`
ese que les
´
el
`
eves les plus jeunes ont besoin d’un soutien plus
marqu
´
e lorsqu’ils se servent des objets d’apprentissage en math
´
ematiques.
Address correspondence to Dr. Robin H. Kay, Faculty of Education, University of Ontario Institute of Technology,
11 Simcoe Street North, Oshawa, ON L1H 7L7, Canada. E-mail: robin.kay@uoit.ca
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 351
OVERVIEW
Originally designed for higher education, learning objects are now being used more often in
middle and secondary school classrooms (e.g., Clarke & Bowe, 2006a, 2006b; Kay & Knaack,
2007b, 2008c, 2008d; Liu & Bera, 2005; Lopez-Morteo & Lopez, 2007; Nurmi & Jaakkola,
2006). Though some research has been done on the use of learning objects in mathematics (Kay
& Knaack, 2008d), a comprehensive examination of factors that might influence the effectiveness
of learning objects including student characteristics, design of learning objects, and teaching
strategy has yet to be conducted. The purpose of the current study was to examine key variables
that might contribute to the successful implementation of learning objects in middle and secondary
school mathematics classrooms.
Definition of Learning Objects
Learning objects are operationally defined in this study as interactive Web-based tools that
support the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive
processes of learners. This definition is an aggregate of previous attempts to define learning
objects (Agostinho, Bennett, Lockyer, & Harper, 2004; Butson, 2003; McGreal, 2004; Parrish,
2004; Wiley et al., 2004). Some examples used by teachers in this study include adding integers
with virtual colored tiles, visual presentation of how coordinates work on a two-dimensional
graph followed by a set of test questions, animations of how three-dimensional objects transform
to two-dimensional nets in order to examine surface area, and manipulation of parameters in a
parabolic equations. Links to all learning objects used in this study are provided in Appendix A
(Kay, 2011a).
Benefits of Using Learning Objects
Three key features of learning objects that benefit students are visual supports, motivation through
increased focus, and control over learning. Visual supports help make complex ideas more easily
understood by reducing working memory and cognitive load (Kay & Knaack, 2008c; Sedig &
Liang, 2006). Visualization is particularly important in mathematics where it is challenging for
many students to understand abstract concepts (Grouws, 2004; Kilpatrick, Martin, & Schifter,
2003; Sowder & Schappelle, 2002). Many learning objects also provide clear learning goals
and immediate feedback, characteristics that frequently lead to increased focus and motivation
(Barkley, 2010; Wlodkowski, 2008). Focus and specific feedback is especially helpful in mul-
tistep mathematics problems. Finally, learning objects permit students to control the pace of
learning, thereby providing easier digestion of new concepts (Bransford, Brown, & Cocking,
2000; Kay & Knaack, 2008c, 2008d; Willingham, 2009). The rate at which students under-
stand mathematics concepts varies considerably, so being able to control the pace of learning
is important (Grouws, 2004; Sowder & Schappelle, 2002). In summary, a carefully designed
learning object can provide visual scaffolding, decreased cognitive load, increased motivation
and focus, and control over the learning process, thereby resulting in more productive learning
experiences.
352 KAY
Overview—Learning Objects and Mathematics
A comprehensive review of articles on the use of learning objects in the past 10 years revealed
nine studies examining the use of mathematics-based learning objects in K–12 schools (Bower,
2005; Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2007c, 2008d; Kong & Kwok, 2005;
Lopez-Morteo & Lopez, 2007; Nurmi & Jaakkola, 2006a; Reimer & Moyer, 2005).
Seven studies indicated that student attitudes toward mathematics-based learning objects was
largely positive (Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2007c, 2008d; Lopez-Morteo
& Lopez, 2007; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). Features that students liked
about learning objects included ease of use, controlling the pace of learning, timely feedback,
using a wide range of multimedia tools, and support for learning (Clarke & Bowe, 2006a,
2006b; Kay & Knaack, 2007c, 2008d; Lopez-Morteo & Lopez, 2007; Nurmi & Jaakkola, 2006;
Reimer & Moyer, 2005). Lim, Lee, and Richards (2006) and Nurmi and Jaakkola (2006) added
that the acceptance of learning objects was partially dependent on the type of learning object
used. Students favored interactive, constructive learning objects over an “electronic” textbook
prototype. Kay and Knack (2007a) offered quantitative evidence that students were moderately
positive about using mathematics-based learning objects.
Four of the nine studies reviewed reported that elementary and middle school students who
used learning objects showed significant improvement on various learning performance measures
(Bower, 2005; Kong & Kwok, 2005; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). Nurmi
and Jaakkola (2006) noted that learning performance gains were dependent on the type of learning
object and how it was used. Students working with drill and practice learning objects were more
focused on competing with their peers than on learning. Students involved in a mixed learning
object/lab-based lesson performed significantly better than in other learning scenarios.
Student Characteristics and Learning Objects
Though research on student characteristics and the use of learning objects is relatively limited,
several recent studies in the area of mathematics and science (e.g., Kay, 2009a, 2011c; Kay &
Knaack, 2007b, 2008a, 2008c) suggested that there are at least five potential attributes that could
influence the effectiveness of learning objects, including gender, age, computer comfort level,
subject comfort level, and ability.
Three studies (Kay & Knaack, 2007b, 2008a, 2008c) reported no significant differences
between male and female secondary school students’ attitudes and learning performance when
learning objects were used. Kay and Knaack (2007b, 2008c) observed that older students (Grade
12) were more positive about learning objects and performed better than younger students (Grades
9 and 10). De Salas and Ellis (2006) added that second- and third-year university students were far
more open to using learning objects than first-year students. Several studies noted that computer
comfort was significantly and positively correlated with student attitudes toward the learning,
design, and engagement value of learning objects (Kay & Knaack, 2005, 2007b, 2008a, 2008c;
Kay, 2009a). Lim et al. (2006) added, in a case study, that students who were not comfortable
with computers used learning objects less. Finally, Kay and Knaack (2008c) recommended that
subject-area comfort level and aptitude be examined when looking at individual differences in
the use of learning objects.
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 353
Lear ning Objects and Instructional Architecture
As least two distinct categories of learning objects exist: structured and open ended. Structured
learning objects typically deliver short sequences of information and then test students’ knowledge
or allow limited practice with the concepts being learned. Clark (2008) referred to this type
of learning object as receptive or directive. Open-ended learning objects use a problem-based
format where students explore and test what-if scenarios to discover relationships and/or improve
understanding of specific concepts.
Substantial debate exists about the optimum level of instructional guidance required for suc-
cessful learning (Kirschner, Sweller, & Clark, 2006). Some researchers maintain that students
need to be provided with sufficient structure to learn effectively (e.g., Cronbach & Snow, 1977;
Mayer, 2004; Sweller, 1998), particularly when their knowledge and understanding is limited
within a given subject area (Kirschner et al., 2006). Kirschner et al. (2006) added that stu-
dents can become overwhelmed with learning new concepts and the cognitive demands of an
open-ended format.
Other researchers suggested that learning is best supported when students are given the es-
sential tools in an open-ended environment and required to construct understanding themselves
(e.g., Bruner, 1986; Steffe & Gale, 1995; Vannatta & Beyerbach, 2000; Vygotsky, 1978). This
minimal level of instructional guidance is referred to by a variety of names, including discovery,
problem-based, and inquiry learning (Kirschner et al., 2006). The more open-ended, construc-
tivist approach has grown in popularity over the past 10 years (Kirschner et al., 2006) and is
prominent in the National Council of Teachers of Mathematics’ Principles and Standards for
School Mathematics (2001).
Bransford et al. (2000), in their seminal work, How People Learn, proposed that both structured
and open-ended approaches are necessary for students to develop competence in an area of
learning. At times, structured delivery of information can work to help build a foundation of factual
knowledge organized within a conceptual framework. As students become more knowledgeable,
an open-ended approach is more viable because students can redirect their cognitive attention
from understanding basic concepts to controlling, monitoring, and assessing their own progress
(Bransford et al., 2000). To date, the influence of the underlying instructional architecture of
learning objects has not been examined.
Teaching Strategies
An increasing number of theorists argue that the effectiveness of any learning object is largely
dependent on the pedagogical choices of the instructor (e.g., Alonso, Lopez, Manrique, & Vines,
2005; Haughey & Muirhead, 2005; McCormick & Li, 2005). Strategies that have been suc-
cessfully used with learning objects include coaching or facilitating (e.g., Liu & Bera, 2005),
establishing context (e.g., Schoner, Buzza, Harrigan, & Strampel, 2005), instructing students to
evaluate their own actions (e.g., van Merrienboer & Ayres, 2005), and providing some sort of
instructional guide or scaffolding (e.g., Concannon, Flynn, & Campbell, 2005; Kay, Knaack, &
Muirhead, 2009; Lim et al., 2006; Mason, Pegler, &Weller, 2005).
One important issue is the amount of support offered by an instructor. Minimal teacher guidance
appears to work in higher education (e.g., Kong & Kwok, 2005; Reimer & Moyer, 2005) but
not in middle and high school classrooms (Kay & Knaack, 2007a; Nurmi & Jaakkola, 2006).
354 KAY
The cognitive demands of using a learning object independently may be too high for younger
students; consequently, a teacher-led approach may work better than a student-based approach.
Purpose
The purpose of this study was to examine the influence of student characteristics, instructional
architecture, and teaching strategy on the effectiveness of learning objects in the secondary school
mathematics classrooms.
METHOD
Overview
The following protocol was followed to maximize the quality of data collected:
1. A relatively large group of students was sampled.
2. Reliable, valid, and research-based survey tools were employed to collect data on student
attitudes toward learning objects (Kay & Knaack, 2009b).
3. Mathematics-based learning objects were preselected for teachers based on Kay and
Knaack’s (2008b) multicomponent approach for evaluating learning objects.
4. Predesigned lesson plans were created by trained teachers and derived from previous
research looking at effective strategies for using learning objects.
5. An enhanced measure of student performance was developed for each learning object
based on the revised Bloom’s taxonomy (Anderson & Krathwohl, 2001).
Sample
Students
The student sample consisted of 286 middle and secondary school students (132 males, 154
females) who were 11–17 years of age (M = 12.8, SD = 0.91). Most students reported average
grades of 60–69% (n = 43, 15%), 70–79% (n = 113, 40%), or 80–89% (n = 83; 29%). Just
over 60% (n = 182) agreed that they were good at the subject in which the learning object was
used. Only 40% (n = 119) agreed that they liked the subject taught with the learning object.
Three quarters of the students (n = 217) agreed or thought that they were good at working with
computers. The sample population was collected from 15 different middle and secondary school
classrooms located within two suburban regions with over 500,000 people each.
Teachers
Twelve middle school and three secondary school teachers participated in this study (5 males,
10 females). Specific grades taught were 7th (n = 6), 8h (n = 6), 9th (n = 2) and 10th (n = 1).
Teaching experience ranged from 0.5–23 years with a mean of 7.6 (SD = 7.0). Thirteen out of
15 teachers (87%) agreed that they were comfortable with using computers at school. Frequency
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 355
of typical classroom computer use varied widely, with two teachers never using computers, one
teacher using computers once a year, four teachers using computers each term, five teachers using
computers monthly, two teachers using computers weekly, and one teacher using computers on a
daily basis.
Learning Objects and Lesson Plans
Four teachers, not involved in the study, were recruited to select high-quality learning objects
and create lesson plans. Each of these teachers participated in a two-day workshop looking at
how to (a) select learning objects for the classroom and (b) develop effective lesson plans. The
criteria for choosing learning objects was based on Kay and Knaack’s (2008b) multicomponent
model for evaluating learning objects. The lesson plan design was derived from the results of
Kay et al.’s (2009) study on effective strategies for using learning objects. Key components of
a standard lesson plan included (a) a guiding set of questions; (b) a structured, well-organized
plan for using the learning objects; and (c) time to consolidate concepts learned. A typical lesson
was approximately 70 minutes in duration and included an introduction (10 minutes), guiding
questions and activities involving the use of an learning object (50 minutes), and consolidation
(10 minutes).
A database of 44 lesson plans and learning objects was created over a period of 2 months.
Nine unique learning objects were selected by teachers from the learning object database. See
Appendix A (Kay, 2011a) for a links to all learning objects, lesson plans, and pre- and posttests
used by mathematics teachers in this study.
Procedure
Mathematics teachers from two boards of education were informed of the research study by an
educational coordinator. Teachers who volunteered for the study participated in full-day training
workshop on using learning objects and implementing the predesigned lesson plans. If teachers
still wanted to be part of the study after the workshop, they were asked to use at least one
learning object in their mathematics classroom within the following 3 months. E-mail support
was available for the duration of the study. All students in a given teacher’s class participated in
the learning object lesson chosen by the teacher; however, survey and pre- and posttest data were
only collected from those students with signed parental permission forms.
Explanatory Variables
Three explanatory variables were examined in this study: student characteristics, learning object
instructional architecture, and teaching strategy. The five student characteristics were gender, age,
computer comfort level, subject comfort level, and average grade in subject area associated with
the learning object used. Computer comfort was assessed using a scale developed by Kay and
Knaack (2005), which showed good construct validity and reliability. The internal reliability for
the computer comfort scale used in the current study was 0.82. Subject comfort level was measured
using two questions asking students about their ability and attitude regarding the learning object
subject area. The internal reliability for subject comfort scale used in this study was 0.77. Finally,
356 KAY
students were asked to estimate their average grade in the subject area where the learning object
was used.
To assess instructional architecture, each learning object was categorized according to its main
pedagogical design: structured (n = 57) or open ended (n = 227). A structured learning object
presented information and then tested students for understanding. Information was presented in
a relatively passive manner. An open-ended learning object permitted students to change various
parameters, explore, and test what-if scenarios (Clark, 2008). See Appendix A (Kay, 2011a) for
examples of each learning object architecture type.
Finally, teaching strategy was determined by the lesson plan format used: teacher led (n =
59) vs. student based (n = 224). In a teacher-led format, the instructor displayed the learning
object at the front of the class using an LCD projector and guided the class with a deliberate
set of questions and activities. In a student-based lesson, students worked independently on a
guiding set of questions in a computer lab. See Appendix A (Kay, 2011a) for all teacher-led and
student-based lesson plans used in the study.
Response Variables
Student Attitudes Toward Learning Objects
When a learning object lesson was finished, students with signed permission forms filled
in the Learning Object Evaluation Scale for Students (Kay & Knaack, 2007, 2009b) to assess
their perceptions of how much they had learned (learning construct), the design of the learning
object (design construct), and how engaged they were while using the learning object (engagement
construct). These constructs were based on a comprehensive review of the literature on evaluating
learning objects (Kay & Knaack, 2007, 2009a). The scale showed good reliability, face validity,
construct validity, convergent validity, and predictive validity (Kay & Knaack, 2009b). Internal
reliability scale estimates in the current study were 0.94 (perceived learning), 0.82 (design of
learning object), and 0.92 (engagement). See Appendix B (Kay, 2011b) for a copy of the scale
used.
Student Performance
Students completed pre- and posttests based on the content of the specific learning object they
used in class. These tests were included with all predesigned lesson plans to match the learning
goals of the learning object (see Appendix A, Kay, 2011a). The percentage difference between
pre- and posttest scores was used to assess changes in student performance on four possible
knowledge categories from the revised Bloom’s taxonomy (Anderson & Krathwhol, 2001) and
included remembering, understanding, application, and analysis. The number of knowledge
categories assessed in any one class varied according the learning goals of each learning object
used.
Key Research Questions
The following five research questions were examined:
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 357
1. What are student attitudes (learning, design, engagement) toward mathematics-based
learning objects?
2. How do students perform as a result of using mathematics-based learning objects?
3. How are student characteristics (gender, age, teaching experience, computer comfort
level, and subject area comfort level) related to student attitudes about mathematics-
based learning objects and learning performance?
4. How is learning object instructional architecture (structured vs. open ended) related to
student attitudes about mathematics-based learning objects and learning performance?
5. How is teaching strategy (teacher-led vs. student-based lessons) related to student attitudes
about mathematics-based learning objects and learning performance?
RESULTS
Lesson Plan Evaluation
The lesson plans for the learning objects were designed by other teachers, so it is prudent to
evaluate the extent to which teachers accepted and followed these lesson plans. It is possible that
the influence of learning objects is partially dependent on teacher acceptance or rejection of the
predesigned lesson plans. Fourteen out of 15 teachers agreed or strongly agreed that the lesson
plans were easy to follow. Two thirds of the teachers believed that the lesson plans matched their
teaching style. Nearly three quarters of the teachers felt that the handouts were clear and over
85% believed that they were useful. Ninety-five percent of teachers felt that the lesson plans were
well designed and 80% believed that there was no need to make changes.
Student Attitudes Toward Learning Objects
Survey Data—Learning Construct
Students, on average, somewhat agreed that learning objects helped them learn (Items 8a–8e,
Appendix B, Kay, 2011b; M = 24.3, SD = 7.1), with a mean item rating of 4.9 out of 7. The
broad range of scores (5–35) indicates that there was considerable variability among students
with respect to their attitudes toward the learning impact of learning objects. Overall, a majority
of students agreed that learning objects helped their learning (49% agreed vs. 11% disagreed;
Table 1).
Survey Data—Design Construct
Students rated the design of learning objects (Items 7a–7d, Appendix B, Kay, 2011b) slightly
higher than the learning value (M = 21.1, SD = 4.5) with a mean item rating of 5.3 out of 7.
The range of student attitudes toward learning object design (4–28) demonstrated considerable
variance. Overall, most students agreed that the learning objects were well designed (66% agreed
vs. 4% disagreed; Table 1).
358 KAY
TABLE 1
Student Rating of Learning, Design, and Engagement for Mathematics-Based Learning Objects
Scale No. items Disagree
a
(%) Agree
b
(%) Mean (SD)
Learn 5 11.3 49.1 24.3 (7.1)
Design 4 4.2 65.8 21.1 (4.5)
Engagement 4 18.6 46.7 18.8 (6.2)
a
Percentage of students who disagreed that learning objects helped learning (including somewhat disagree, disagree,
strongly disagree).
b
Percentage of students who disagreed that learning objects helped learning (including somewhat
agree, agree, strongly agree).
Survey Data—Engagement Construct
Student ratings of learning object engagement (Items 9a–9d, Appendix B, Kay, 2011b) were
the lowest of all three attitude constructs (M = 18.8, SD = 6.2) with a mean item rating of 4.7 out
of 7. This means that students, on average, were neutral about or slightly agreed that the learning
object they used was engaging. High variability was observed among student engagement ratings
(4–28). Overall, almost half of the students felt that the learning objects were engaging (47%
agreed vs. 19% disagreed; Table 1).
Learning Performance
Five paired t-tests were performed to assess differences between pre- and posttest scores: four
knowledge categories and total test score. Though a multivariate analysis of variance (MANOVA)
is a common statistical procedure used with multiple dependent variables, not all of Bloom’s
knowledge categories were asked for each learning object. In other words, no learning object
targeted all four knowledge categories. The MANOVA analysis eliminated considerable data;
therefore, multiple t-tests were used to incorporate the maximum amount of information possible.
The alpha rate was not adjusted to reflect the fact that multiple tests were conducted because the
current study is considered formative in nature. All question categories showed significant gains
(see Table 2). Increases in scores ranging from 14 to 29% resulted in moderate to large effect
sizes based on Cohen’s d (Cohen, 1988, 1992). Note that the results for the understanding and
TABLE 2
Change in Learning Performance for Students Using Mathematics-Based Learning Objects
Question type Pretest mean (%) Posttest mean (%) % Change nt Effect size
Remembering 60.8 (43.5) 77.5 (38.5) 16.6 77 4.0
∗∗
0.41
Understanding 7.1 (18.9) 35.7 (37.8) 28.6 7 2.8
∗
0.96
Application 54.9 (31.9) 69.7 (29.8) 14.8 221 6.8
∗∗
0.48
Analysis 52.1 (43.7) 78.2 (39.7) 26.1 23 2.7
∗
0.63
Total score 55.1 (30.7) 70.6 (28.3) 15.4 228 7.5
∗∗
0.52
∗
p < .05.
∗∗
p < .001.
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 359
TABLE 3
Correlations Among Student Characteristics, Attitudes Toward Learning Objects, and Learning Performance
(
n
= 284)
Scale Gender Grade level Computer comfort Subject comfort Subject mark
Student attitudes
Learning 0.10 −0.07 0.29
∗
0.28
∗
−0.09
Design 0.03 0.03 0.27
∗
0.41
∗
0.11
Engagement 0.08 −0.04 0.28
∗
0.47
∗
0.04
Learning performance 0.00 0.26
∗
0.01 0.03 0.08
∗
p < .001.
analysis knowledge categories should be treated with caution because of small samples sizes and
marginal probability levels.
Student Characteristics and Learning Objects
Attitudes Toward Learning Objects
Subject and computer comfort level were significantly correlated with higher ratings of learning
( p < .001), design ( p < .001), and engagement ( p < .001). When students were more comfortable
with the subject area taught and with using computers, they had more positive attitudes toward
learning objects. Student gender, age, and average grade in mathematics were not significantly
correlated with student attitudes toward learning objects (Table 3).
Learning Performance
Student age was significantly correlated with learning performance ( p < .001). Older students
in higher grades performed better than younger students in lower grades when learning objects
were used. Student gender, subject area comfort level, average grade, and computer comfort level
were not significantly correlated with learning performance (Table 3).
Lear ning Object Instructional Architecture
Students who used structured learning objects rated learning value ( p < .05) significantly higher
than students who used open-ended learning objects. The effect size according to Cohen (1988,
1992) was considered moderate. No significant differences were observed between structured
and open-ended learning objects with respect to student ratings of design or engagement. Total
learning performance was significantly higher when structured as opposed to open-ended learning
objects were used ( p < .001). The effect size for this difference was considered large according
to Cohen (1988, 1992; see Table 4).
360 KAY
TABLE 4
Student Attitudes and Total Learning Performance as a Function of Learning Object Instructional Architecture
Structured (n = 57) Open-ended (n = 226)
Question type Mean (SD) Mean (SD) df t Effect size
Student attitudes
Learning 22.6 (7.4) 24.9 (6.9) 281 2.3
∗
0.32
Design 21.5 (4.0) 20.9 (4.6) 282 1.0
Engagement 18.4 (6.2) 19.0 (6.1) 283 0.6
Total learning performance 30.1 (34.4) 10.6 (28.2) 226 3.9
∗∗
0.62
∗
p < .05.
∗∗
p < .001.
Teaching Strategy and Learning Objects
Students rated learning object design significantly higher for teacher-led as opposed to student-
based lessons ( p < .005). The effect size for this difference is considered moderate according
to Cohen (1988, 1992). No significant differences were observed with respect to student ratings
of learning object learning and engagement constructs as a function of teaching strategy. Total
student performance was significantly higher for teacher-led versus student-based learning object
lessons ( p < .005). The effect size for this difference is considered moderate according to Cohen
(1988, 1992; see Table 5).
DISCUSSION
The purpose of this study was to examine the influence of student characteristics, instructional
architecture, and teaching strategy on student attitudes toward learning objects and learning
performance in middle and secondary school mathematics classrooms.
TABLE 5
Student Attitudes Toward Learning Objects and Total Learning Performance as a Function of Teaching
Strategy.
Teacher led Student based
Question type (n = 59) (n = 224) df t Effect size
Student attitudes
Learning 24.9 (6.4) 24.1 (7.3) 281 0.8
Design 22.5 (2.9) 20.7 (4.7) 282 2.9
∗
0.46
Engagement 19.3 (5.7) 18.7 (6.3) 283 0.7
Total learning performance 28.8 (32.1) 12.4 (30.0) 226 3.1
∗
0.53
∗
p < .005.
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 361
Student Attitudes and Learning Performance for Mathematics-Based Learning
Objects
Middle and secondary school students, on average, agreed that mathematics-based learning
objects were well designed, engaging, and helpful when learning. This result was also reported
in previous studies (Kay & Knaack, 2007a, 2009a; Lowe et al., 2010). The wide range of ratings
for learning, design, and engagement constructs suggests that some students do not benefit from
using learning objects. For example, a small group (11–19%) believed that learning objects did
not help them learn or were not engaging. More research is needed to determine the source of
resistance toward using learning objects.
Significant learning performance gains were seen in all four of Bloom’s knowledge categories
(Anderson & Krathwhol, 2001). Increases from 15 to 29% and effect sizes in the moderate to large
range (Cohen 1988, 1992) indicate that the change was sizeable, not just statistically significant.
The largest increases were observed in questions focusing on understanding (29%) and analysis;
however, the sample size was small, so more research is needed to confirm whether these gains
are consistent. The smallest gains were observed for remembering (17%) and application (15%)
knowledge areas. These two areas were the main learning targets in this study, perhaps a reflection
that learning objects for this age group tend to focus on basic as opposed to higher level concepts.
Because this is a first study examining the impact of learning objects on different knowledge
areas, the results should be treated with caution.
Student Characteristics
Student Attitudes Toward Learning Objects
Only two variables were significantly correlated with student attitudes toward learning objects:
computer comfort level and subject comfort level. Students who were more comfortable with
computers and mathematics had significantly more positive attitudes than their less confident
peers. This result was partially confirmed by previous research on computer comfort level (Kay
& Knaack, 2005, 2007b, 2008c; Lim et al., 2006); however, the result for the impact of subject
area confidence was new.
Gender, grade level, and self-reported ability in mathematics were not significantly correlated
with student attitudes toward learning objects. The results for gender were consistent with previous
studies on gender differences and learning objects (e.g., Kay & Knaack, 2007b, 2008c). A pattern
is beginning to emerge that learning objects are relatively gender neutral for mathematics-based
learning objects. On the other hand, the absence of a grade-level effect on student attitudes toward
learning objects does not match the results of previous research (e.g., De Salas & Ellis, 2006; Kay
& Knaack, 2007b, 2008c). One possible explanation for this difference may be that the students
in the current study were younger. These students born in the net generation (Tapscott, 2008)
have grown up on a steady diet of computer technology, perhaps negating the impact of grade
level on attitudes toward using learning objects (Montgomery, 2009; Palfrey & Gasser, 2008;
Tapscott, 2008). The absence of any relationship between self-rated ability in mathematics and
attitudes toward learning objects has not been examined prior to this study, so the results should
be considered preliminary.
362 KAY
Learning Performance
The only student characteristic significantly correlated with learning performance was age.
Older students in higher grades performed significantly better than younger students in lower
grades. This result was partially supported by previous research reporting a modest, positive
age effect on general learning performance after using learning objects (Kay & Knaack, 2007b,
2008c). One explanation for the impact of age on learning performance might involve the range
of skills required to use a learning object including reading instructions, writing down results, and
interpreting what-if scenarios. Older students may be able to cope with these cognitive demands,
whereas younger students may need more scaffolding.
Gender, computer comfort level, subject comfort level, and self-reported math ability were not
significantly correlated with learning performance. The absence of a gender effect is consistent
with past research (e.g., Kay & Knaack, 2007b, 2008c). This result provides further evidence
that learning objects are educational tools that serve male and female students equally well.
The conclusion is noteworthy given that small but persistent gender differences in the use of
technology have been observed over the past 20 years, usually in favor of males (e.g., American
Association of University Women, 2000; Sanders, 2006; Whitley, 1997).
It is interesting that computer and student comfort level had a significant impact on student
attitude but not learning. One implication from this finding may be that learning objects work
reasonable well regardless of how comfortable a student is with using computers or doing
mathematics.
It is somewhat surprising that mathematics grades were not correlated with learning perfor-
mance. One would expect that students with higher grades would perform better than students
with lower grades most of the time. One obvious explanation is that students were not able to
accurately assess their current grades. However, the range of grades selected by students was
highly variable and there was no reason to assume a systematic bias. It was assumed that students
had a general sense of how well they were doing. It is also possible that using learning objects
minimized the impact of average grade and helped level the academic playing field. Future re-
search should collect actual student grades in order to determine whether the current results are
robust.
Lear ning Object and Instructional Architecture
Students were equally positive about the design and engagement value for structured versus
open-ended learning objects. However, students using structured learning objects felt that they
learned significantly more and outperformed peers who used open-ended learning objects by
almost 20%. This result supports Kirschner et al.’s (2006) assertion that younger students may
not be able to handle the cognitive demands of self-guided discovery required in an open-ended
format, particularly when a firm foundation in mathematical concepts has not been established.
However, before making the assumption that structured learning objects are more appropriate
for younger students, several alternative explanations need to be considered. For example, it
is conceivable that younger students are simply more familiar with structured learning objects
and therefore respond to them more positively. Another interpretation might be that structured
learning objects are easier to complete in a single class than open-ended learning objects, which
may require more time to discover and construct solutions. A third possibility is that structured
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 363
learning objects may be more effective in addressing remembering and application knowledge
areas, the two main Bloom’s categories assessed in this study. It is reasonable to speculate that
open-ended learning objects might be more effective in targeting understanding and analysis
knowledge areas. More research is needed, perhaps in the form of qualitative observations of
interviews, in order to understand why middle and secondary school students respond differently
to a structured vs. open-ended learning object design. In addition, a larger sample is needed to
determine the impact of instructional architecture on specific knowledge areas.
Teaching Strategy
Student learning performance and, to a lesser extent, student attitudes were significantly higher
for teacher-led as opposed to student-based lesson plans. These findings are consistent with those
observed by several previous studies (Kay & Knaack, 2007a; Nurmi & Jaakkola, 2006). Middle
and secondary school students in Grades 7–10 appear to need more scaffolding and guidance
when using learning objects. This result is important given that the traditional approach to using
learning objects is to encourage students to work independently on their own computers (e.g.,
Kong & Kwok, 2005; Reimer & Moyer, 2005).
Again, these results need to be treated with caution for at least two reasons. First, the sample size
of students who used teacher-led learning objects was relatively small. Second, the instructional
architecture used within each teaching strategy was not strictly balanced (see Appendix A, Kay,
2011a), so it is possible that there was an interaction effect. For example, open-ended learning
objects may require more teacher guidance than structured learning objects. Future research
using equal numbers of structured and open-ended learning objects within each teaching strategy
would help investigate possible interactions between instructional architecture and method of
instruction.
Caveats and Future Research
Several steps were followed to ensure the quality of data collection and analysis in this study,
including controlling for the design of learning objects and lesson plans; using a wide range of
learning objects; employing reliable, valid data assessment tools; and assessing a wide range of
learning performance knowledge areas. Still, several limitations remain and need to be addressed
in future studies.
First, the sample size needs to be increased to more thoroughly assess instructional design,
teaching strategies, and all four of the revised Bloom’s knowledge categories. Second, a more
in-depth analysis involving qualitative data is needed to determine whether younger students
require more scaffolding when using learning objects and what this support might look like.
Third, the results of the current study are based on one-time use of learning objects. It is not clear
what the long-term impact of student characteristics, instructional design, and teaching strategy
on attitudes and learning performance would be. Finally, a wider range of teaching strategies
could be examined to determine the optimal guidance needed when using learning objects in
middle and secondary school mathematics classrooms.
364 KAY
Summary
This study looked at factors that influenced student attitudes toward mathematics-based learning
objects and learning performance. Overall, students thought that learning objects were well-
designed, engaging tools that helped them learn. Posttest scores increased significantly over
pretest scores after using learning objects, an average of 15%. However, individual student
characteristics, instructional design, and teaching strategy were intricately linked to the impact
of learning objects. A student’s computer and subject area comfort level (student characteristics)
were significantly correlated with student attitudes toward learning objects but not learning
performance. Older students in higher grades performed better with learning objects than younger
students in lower grades. Students who used structured learning objects performed significantly
better than students who worked with open-ended learning objects; however, this may be a
reflection of the type of knowledge areas assessed. Finally, students involved in a teacher-led
learning object lesson significantly outperformed students who participated in a student-based
lesson. The results suggest that younger students may need more guidance and scaffolding if
learning objects are to be successful in middle and secondary school mathematics classrooms.
REFERENCES
Agostinho, S., Bennett, S., Lockyear, L., & Harper, B. (2004). Developing a learning object metadata application profile
based on LOM suitable for the Australian higher education market. Australasian Journal of Educational Technology,
20(2), 191–208. Retrieved from http://www.ascilite.org.au/ajet/ajet20/agostinho.html
Alonso, F., Lopez, G., Manrique, D., & Vines, J. M. (2005). An instructional model for Web-based e-learning ed-
ucation with a blended learning process approach. British Journal of Educational Technology, 36(2), 217–235.
doi:10.1111/j.1467-8535.2005.00454.x
American Association of University Women. (2000). Tech-savvy: Educating girls in the new computer age. Washington,
DC: Author. Retrieved from http://www.aauw.org/member
center/publications/TechSavvy/TechSavvy.pdf
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of
Bloom’s taxonomy of educational objectives. New York, NY: Longman.
Barkley, E. F. (2010). Student engagement techniques. San Francisco, CA: John Wiley & Sons.
Bower, M. (2005). Online assessment feedback: Competitive, individualistic, or ...preferred form! Journal of Computers
in Mathematics and Science Teaching, 24(2), 121–147.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn. Washington, DC: National Academy
Press.
Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.
Butson, R. (2003). Learning objects: weapons of mass instruction. British Journal of Educational Technology, 34(5),
667–669. doi:10.1046/j.0007-1013.2003.00359.x
Clark, R. V. (2008). Building expertise: Cognitive methods for training and performance improvement. San Francisco,
CA: John Wiley & Sons.
Clarke, O., & Bowe, L. (2006a). The Learning Federation and the Victorian Department of Education and Train-
ing trial of online curriculum content with Indigenous students. Retrieved from http://www.sofweb.vic.edu.
au/edulibrary/public/teachlearn/ict/TLF
DETVIC indig trial mar06.pdf
Clarke, O., & Bowe, L. (2006b). The Learning Federation and the Victorian Department of Education and Train-
ing trial of online curriculum content with ESL students. Retrieved from http://www.thelearningfederation.
edu.au/verve/
resources/report esl final.pdf
Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd ed.). New York, NY: Academic Press.
Cohen, J. (1992). A power primer. Psychological Bulletin , 112(1), 155–159. doi:10.1037/0033-2909.112.1.155
Concannon, F., Flynn, A., & Campbell, M. (2005). What campus-based students think about the quality and benefits of
e-learning. British Journal of Educational Technology, 36(3), 501–512. doi:10.1111/j.1467-8535.2005.00482.x.
LEARNING OBJECTS IN MATHEMATICS CLASSROOMS 365
Cronbach, L. J., & Snow, R. E. (1977). Aptitudes and instructional methods: A handbook for research on interactions.
New York, NY: Irvington.
De Salas, K., & Ellis, L. (2006). The development and implementation of learning objects in a higher education. Interdisci-
plinary Journal of Knowledge and Learning Objects, 2, 1–22. Retrieved from http://www.ijello.org/Volume2/v2p001-
022deSalas.pdf.
Grouws, D. A. (2004). Handbook of research on mathematics teaching and learning. Reston, VA: National Council for
Teachers of Mathematics.
Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. Australasian Journal of Educational
Technology, 21(4), 470–490. Retrieved from http://www.ascilite.org.au/ajet/ajet21/haughey.html.
Kay, R. H. (2009). Understanding factors that influence of the effectiveness of learning objects in secondary school
classrooms. In L. T. W. Hin & R. Subramaniam (Eds.), Handbook of research on new media literacy at the K–12
level: Issues and challenges (pp. 419–435). Hershey, PA: Information Science Reference.
Kay, R. H. (2011a). Appendix A—List of learning objects used in mathematics study. Retrieved from
http://faculty.uoit.ca/kay/res/math/.
Kay, R. H. (2011b). Appendix B—Learning object survey for students. Retrieved from http://faculty.
uoit.ca/kay/res/math/ss
math.pdf.
Kay, R. H. (2011c). Exploring the influence of context on attitudes toward Web-Based Learning Tools
(WBLTs) and learning performance. Journal of E-Learning and Learning Objects, 7, 125–142. Retrieved from
http://www.ijello.org/Volume7/IJELLOv7p125-142Kay748.pdf
Kay, R. H., & Knaack, L. (2005). Developing learning objects for secondary school students: A multi-
component model. Interdisciplinary Journal of E-Learning and Learning Objects, 1, 229–254. Retrieved from
http://www.ijello.org/Volume1/v1p229–254Kay
Knaack.pdf
Kay, R. H., & Knaack, L. (2007a). Evaluating the learning in learning objects. Open Learning, 22(1), 5–28.
doi:10.1080/02680510601100135
Kay, R. H., & Knaack, L. (2007b). Evaluating the use of learning objects for secondary school science. Journal of
Computers in Mathematics and Science Teaching, 26(4), 261–289.
Kay, R. H., & Knaack, L. (2008a). A formative analysis of individual differences in the effectiveness of learning objects
in secondary school. Computers & Education, 51(3), 1304–1320.
Kay, R. H., & Knaack, L. (2008b). A multi-component model for assessing learning objects: The learning object
evaluation metric (LOEM). Australasian Journal of Educational Technology, 24(5), 574–591. Retrieved from
http://www.ascilite.org.au/ajet/ajet24/kay.pdf.
Kay, R. H., & Knaack, L. (2008c). An examination of the impact of learning objects in secondary school. Journal of
Computer Assisted Learning, 24(6), 447–461. doi:10.1111/j.1365-2729.2008.00278.x
Kay, R. H., & Knaack, L. (2008d). Investigating the use of learning objects in secondary school math-
ematics. Interdisciplinary Journal of E-Learning and Learning Objects, 4, 269–289. Retrieved from
http://ijello.org/Volume4/IJELLOv4p269–289Kay.pdf
Kay, R. H., & Knaack, L. (2009a). Analyzing the effectiveness of learning objects for secondary school science classrooms.
Journal of Educational Multimedia and Hypermedia, 18(1), 113–135.
Kay, R. H., & Knaack, L. (2009b). Assessing learning, design and engagement in learning objects: The learning
object evaluation scale for students (LOES-S). Education Technology Research and Development, 57(2), 147–168.
doi:10.1007/s11423-008-9094-5
Kay, R. H., Knaack, L., & Muirhead, B. (2009). A formative analysis of instructional strategies for using learning objects.
Journal of Interactive Learning Research, 20(3), 295–315.
Kilpatrick, J., Martin, W. G., & Schifter, D. (2003). A research companion to principles and standards for school
mathematics. Reston, VA: National Council for Teachers of Mathematics.
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis
of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational
Psychologist, 41(2), 75–86. doi:10.1207/s15326985ep4102
1
Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common fractions: A model
of affordances. Computers and Education, 45(2), 245–265. doi:10.1016/j.compedu.2004.12.002
Lim, C. P., Lee, S. L., & Richards, C. (2006). Developing interactive learning objects for a computing mathematics
models. International Journal on E-Learning, 5(2), 221–244.
366 KAY
Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational
Technology, Research and Development, 53(1), 5–21. doi:10.1007/BF02504854
Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning environment based on
recreational learning objects. Computers and Education, 48(4), 618–641. doi:10.1016/j.compedu.2005.04.014
Lowe, K., Lee, L., Schibeci, R., Cummings, R., Phillips, R., & Lake, D. (2010). Learning objects and engagement
of students in Australian and New Zealand schools. British Journal of Educational Technology, 41(2), 227–241.
doi:10.1111/j.1467-8535.2009.00964.x
Mason, R., Pegler, C., & Weller, M. (2005). A learning object success story. Journal of Asynchronous Learning Networks,
9(1), 97–105. Retrieved from http://oro.open.ac.uk/6624/1/v9n1
mason.pdf
Mayer, R. (2004). Should there be a three-strikes rule against pure discovery learning? The case for guided methods of
instruction. American Psychologist, 59(1), 14–19. doi:10.1037/0003-066X.59.1.1
McCormick, R., & Li, N. (2005). An evaluation of European learning objects in use. Learning, Media and Technology,
31(3), 213–231. doi:10.1080/17439880600893275
McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional Technology and
Distance Learning, 1(9). Retrieved from http://www.itdl.org/journal/sep
04/article02.htm
Montgomery, K. C. (2009). Generation digital. Cambridge, MA: MIT Press.
National Council of Teachers of Mathematics. (2011). Principles and standards for school mathematics. Retrieved from
http://www.nctm.org/uploadedFiles/Math
Standards/12752 exec pssm.pdf
Nurmi, S., & Jaakkola, T. (2006). Effectiveness of learning objects in various instructional settings. Learning, Media,
and Technology, 31(3), 233–247. doi:10.1080/17439880600893283
Palfrey, J., & Gasser, U. (2008). Born digital. New York, NY: Basic Books.
Parrish, P. E. (2004). The trouble with learning objects. Educational Technology Research & Development, 52(1), 49–67.
doi:10.1007/BF02504772
Reimer, K., & Moyer, P. S. (2005). Third-graders learning about fractions using virtual manipulatives: A classroom study.
Journal of Computers in Mathematics and Science Teaching, 24(1), 5–25.
Sanders, J. (2006). Gender and technology: A research review. In C. Skelton, B. Francis, & L. Smulyan (Eds.), Handbook
of gender and education (pp. 307–322). London, England: Sage.
Schoner, V., Buzza, D., Harrigan, K., & Strampel, K. (2005). Learning objects in use: “Lite” assessment for field studies.
Journal of Online Learning and Teaching, 1(1), 1–18.
Sedig, K., & Liang, H. (2006). Interactivity of visual mathematical representations: Factors affecting learning and
cognitive processes. Journal of Interactive Learning Research, 17(2), 179–212.
Sowder, J., & Schappelle, B. (2002). Lessons learned from research. Reston, VA: National Council for Teachers of
Mathematics.
Steffe, L., & Gale, J. (Eds.). (1995). Constructivism in education. Hillsdale, NJ: Lawrence Erlbaum.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.
doi:10.1207/s15516709cog1202
4.
Tapscott, D. (2008). Grown up digital: How the Net generation is changing your world. New York, NY: McGraw-Hill.
van Merrienboer, J. J. G., & Ayres, P. (2005). Research on cognitive load theory and its design implications for e-learning.
Education Technology Research and Development, 53(3), 1042–1629. doi:10.1007/BF02504793
Vannatta, R. A., & Beyerbach, B. (2000). Facilitating a constructivist vision of technology integration among education
faculty and preservice teachers. Journal of Research on Computing in Education, 33(2), 132–148.
Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.
Whitley, B. E., Jr. (1997). Gender differences in computer-related attitudes and behaviors: A meta-analysis. Computers
in Human Behavior, 13(1), 1–22. doi:10.1016/S0747-5632(96)00026-X
Wiley, D., Waters, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004). Overcoming the limitations of learning
objects. Journal of Educational Multimedia and Hypermedia, 13(4), 507–521.
Willingham, D. T. (2009). Why don’t students like school? San Francisco, CA: Jossey-Bass.
Wlodkowski, R. J. (2008). Enhancing adult motivation to learn: A comprehensive guide for teaching all adults.San
Francisco, CA: Jossey-Bass.
