Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay on Nov 27, 2017
Content may be subject to copyright.
An examination of the impact of learning objects
in secondary school
R.H. Kay & L. Knaack
Faculty of Education, University of Ontario Institute of Technology, 2000 Simcoe St. North, Oshawa, Ontario, Canada
Abstract Very few studies have systematically evaluated the effect of learning objects in secondary
school classrooms. The vast majority of studies have focussed on higher education. The current
study examined the impact of learning objects from the perspective of 850 students and 27
teachers (50 classrooms) of science, mathematics, or social science. The results suggest that
teachers typically spend 1 to 2 h finding and preparing for learning object based lesson plans
that focus on the review of previous concepts. Both teachers and students are positive about the
learning benefits, quality, and engagement value of learning objects, although teachers are
more positive than students. Student performance increased significantly – almost 30% – when
learning objects were used in conjunction with a variety of teaching strategies. It is reasonable
to conclude that learning objects are a viable teaching tool in a secondary school environment.
Keywords assess, evaluate, learning object, quality, secondary school.
Examining the impact of learning objects in
secondary school
Overview
The design, development, reuse and accessibility of
learning objects has been examined in some detail for
almost 10 years (Kay & Knaack 2007b). However,
research on the effectiveness and usefulness of learning
objects is limited (Sosteric & Hesemeier 2002; Kay &
Knaack 2005; Nurmi & Jaakkola 2005, 2006a,b). Until
recently, learning objects were solely used in higher
education (Haughey & Muirhead 2005; Kay & Knaack
2005, 2007b). The purpose of the current study is to
examine the impact of learning objects in secondary
school classrooms.
Literature review
Benefit of using learning objects
Learning objects, defined in this article as ‘interactive
Web-based tools that support learning by enhancing,
amplifying, and guiding the cognitive processes of
learners’ (Butson 2003; Polsani 2003; Agostinho
et al. 2004; McGreal 2004; Parrish 2004; Wiley
et al. 2004) offer a number of advantages for educators
and students including accessibility (Wiley 2000), ease
of use (e.g. Gadanidis et al. 2003; Sedig & Liang
2006), reusability (e.g. Rehak & Mason 2003;
Agostinho et al. 2004; Duval et al. 2004), interactivity
(e.g. Gadanidis et al. 2003; Sedig & Liang 2006) and
visual supports (e.g. Gadanidis et al. 2003; Sedig &
Liang 2006).
In spite of this list of potential benefits, little system-
atic research has been done examining the actual use
and impact of learning objects in the classroom (Kenny
et al. 1999; Van Zele et al. 2003; Bradley & Boyle
2004).
Accepted: 03 February 2008
Correspondence: Robin H. Kay, Faculty of Education, University of
Ontario Institute of Technology, 2000 Simcoe St. North, Oshawa,
Ontario, L1H 7L7, Canada. Email: Robin.Kay@uoit.ca
doi: 10.1111/j.1365-2729.2008.00278.x
Original article
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd Journal of Computer Assisted Learning (2008), 24, 447–461 447
Impact of learning objects in higher education
Eighteen articles were reviewed for this study, looking
at the use of learning objects in higher education. Five
studies examined faculty attitudes, 11 explored student
attitudes and seven assessed student performance. A
meta-analysis of these studies was not possible given
that 15 of the papers reviewed relied on informal quali-
tative evidence.
Faculty perspective
With respect to attitudes towards learning objects,
higher education faculty reported that there were at least
three obstacles. First, the faculty did not have sufficient
awareness and understanding of learning objects to
determine teaching advantages (Collis & Strijker 2003;
Shea et al. 2006). Second, it was anticipated that the
time required to integrate learning objects into an exist-
ing course or curriculum would be prohibitive (Collis &
Strijker 2003; Koppi et al. 2004). Finally, the time
needed to find good learning objects was deemed to
be extensive (Collis & Strijker 2003; Christiansen &
Anderson 2004).
Only two studies assessed teacher attitudes towards
actually using learning objects in a real classroom
(Bradley & Boyle 2004; De Salas & Ellis 2006). In the
first study, teachers were informally polled and reported
that learning objects complimented the course textbook
well (Bradley & Boyle 2004). In the second study, the
general perception was that learning objects helped stu-
dents be more engaged and better prepared for class (De
Salas & Ellis 2006).
Student perspective
Regarding perceptions of learning objects, under-
graduate or graduate students had positive attitudes
about learning objects in eight studies (Kenny et al.
1999; Bradley & Boyle 2004; Docherty et al.
2005; MacDonald et al. 2005; Mason et al. 2005;
Schoner et al. 2005; De Salas & Ellis 2006; Lim
et al. 2006), neutral attitudes in one study (Concannon
et al. 2005) and negative attitudes in one study (Van
Zele et al. 2003). Most papers offered informal or
qualitative evidence, while three studies reported that
approximately 50% to 60% of higher education stu-
dents liked using learning objects (Howard-Rose &
Harrigan 2003; Bradley & Boyle 2004; De Salas &
Ellis 2006).
Students offered positive comments about a wide
range of characteristics including animations (Bradley
& Boyle 2004), self-assessment (Lim et al. 2006),
attractiveness (Bradley & Boyle 2004), control over
learning (Lim et al. 2006), ease of use (Kenny et al.
1999; Schoner et al. 2005), feedback (Concannon et al.
2005; Lim et al. 2006), scaffolding or support (Lim
et al. 2006), interactivity (Concannon et al. 2005;
Lim et al. 2006), navigation (Concannon et al. 2005)
and self-efficacy (Docherty et al. 2005). Negative
comments focussed on problems with navigation
(Concannon et al. 2005), technology (Concannon et al.
2005) and workload (Van Zele et al. 2003).
Student performance
Regarding student performance, three studies offered
qualitative evidence that student learning performance
improved when learning objects were used (Windschitl
& Andre 1998; Kenny et al. 1999; Lim et al. 2006).
Two studies offered descriptive evidence suggesting
that learning objects enhanced student learning
(Bradley & Boyle 2004; MacDonald et al. 2005).
MacDonald et al. (2005) reported fewer students
having to make major changes on their assignments
and Bradley and Boyle (2004) observed pass rates
increasing by 12% to 23%. Three papers presented
formal statistical analyses with respect to student per-
formance and the use of learning objects (Windschitl &
Andre 1998; Rieber et al. 2004; Docherty et al. 2005).
Docherty et al. (2005) reported that nursing students
were significantly more positive about learning objects
than control groups, but the marks on the final course
were not significantly different. Rieber et al. (2004)
observed that students, provided with embedded
explanations and graphical representations, performed
significantly better than those students who did not
receive these supports. Finally, Windschitl and Andre
(1998) noted that simulation-based learning objects
showed significant gains in two out of six concepts
taught. The remaining four concepts showed no signifi-
cant difference.
Use of learning objects in secondary school
Only four published studies were found investigating
the use of learning objects with secondary school stu-
dents (Brush & Saye 2001; McCormick & Li 2005; Kay
& Knaack 2007b; Lopez-Morteo & Lopez 2007).
448 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Teacher perspective
Two studies looked at how teachers viewed learning
objects (McCormick & Li 2005; Kay & Knaack 2007b).
Kay and Knaack (2007b) reported preservice and expe-
rienced teachers strongly agreed that
• learning objects were a beneficial tool for students.
• they helped students with respect to understanding
concepts.
• they would be interested in using the learning objects
in their classrooms again.
McCormick and Li (2005) noted that 60% to 75% of
teachers felt learning objects were useful and enjoyed
using learning objects, although over 50% reported
technical problems local to their schools.
Student perspective
Brush and Saye (2001) observed that students tended to
look at superficial content in a learning object when left
to their own devices and that more active guidance and
structure was needed when using information-based
learning objects. Kay and Knaack (2007b) used a com-
prehensive assessment measure and reported that stu-
dents were moderately positive about learning objects.
In addition, overall usefulness, clear instructions, orga-
nized layout and good theme/motivation were particu-
larly important to students. Finally, Lopez-Morteo and
Lopez (2007) reported that students perceived interac-
tive, recreation-based, collaborative learning objects
positively.
Student performance
To date, no studies have been done looking at the impact
of learning objects on the performance of secondary
school students.
Methodological issues
This study looked at a total of 22 articles (18 in higher
education, four in secondary schools). Several good
quality studies have been performed. Ten studies incor-
porated mixed methods, which included qualitative,
quantitative and performance metrics. Six studies had
sample populations of over 100. Nonetheless, a number
of challenges remain with respect to improving the
investigation of learning objects, particularly in second-
ary schools.
First, only four studies have been done examining
the use of learning objects outside the higher education
populace. Learning objects were originally targeted
at tertiary education, and use in secondary schools
requires further analysis. Second, there is inadequate
data on how teachers react to and use learning objects
in a real world situation. Only four of the 22 studies
reviewed examined teacher perspectives on the actual
use of learning objects (Bradley & Boyle 2004;
McCormick & Li 2005; De Salas & Ellis 2006; Kay &
Knaack 2007b).
Third, even though a wide range of learning objects
exist, the majority of papers focus on a single learning
object (e.g. Kenny et al. 1999; Bradley & Boyle 2004;
Krauss & Ally 2005; MacDonald et al. 2005). It is diffi-
cult to determine whether the evaluation tools used in
one study generalize to the full range of learning objects
that are available.
Fourth, sample populations tested in many studies are
relatively small and poorly described (e.g. Van Zele
et al. 2003; Krauss & Ally 2005; MacDonald et al.
2005) making it challenging to extend any conclusions
to a larger population. This observation is more evident
in secondary school studies where only one paper
looked at more than 70 students (Kay & Knaack 2007b).
Fifth, while triangulation of data collection has been
achieved in a number of studies by using multiple data
collection tools (e.g. Howard-Rose & Harrigan 2003;
Van Zele et al. 2003; Docherty et al. 2005; Schoner
et al. 2005) only four studies have looked at both
student and teacher perspectives at the same time
(Bradley & Boyle 2004; McCormick & Li 2005; De
Salas & Ellis 2006; Kay & Knaack 2007b). No studies
have examined student attitude, teacher attitude and
student performance simultaneously.
Sixth, while most evaluation studies reported that
students benefited from using learning objects, the
evidence is based on loosely designed assessment
tools with no reliability or validity (e.g. Kenny et al.
1999; Howard-Rose & Harrigan 2003; Van Zele et al.
2003; Bradley & Boyle 2004; Krauss & Ally 2005;
Schoner et al. 2005; Lopez-Morteo & Lopez 2007).
Only two out of the 22 studies reviewed offered esti-
mates of reliability (Windschitl & Andre 1998; Kay &
Knaack 2007b) and one study provided data validity
(Kay & Knaack 2007b). As well, few evaluation
studies (e.g. Windschitl & Andre 1998; Kenny et al.
1999; Van Zele et al. 2003; Rieber et al. 2004;
Learning objects in secondary school 449
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Docherty et al. 2005; Kay & Knaack 2007b) used
formal statistics, particularly in the secondary school
domain (Kay & Knaack 2007b). The lack of reliable
and valid evaluation tools combined with an absence of
statistical rigour reduce confidence in the results pre-
sented to date.
Finally, a promising trend in learning object evalua-
tion research is the inclusion of performance measures
(e.g. Bradley & Boyle 2004; Docherty et al. 2005;
MacDonald et al. 2005). Until recently, there has been
little evidence to support the usefulness or pedagogical
impact of learning objects. The next step is to refine
current evaluation tools to determine which specific
qualities of learning objects influence performance.
In summary, previous methods used to evaluate
learning objects have offered extensive descriptive and
anecdotal evaluations of single learning objects, but are
limited with respect to sample size, representative popu-
lations, reliability and validity of data collection tools
and the use of formal statistics. Recent evaluation
efforts to incorporate learning performance should be
encouraged in order to advance knowledge of learning
object features that may influence learning.
Purpose
The purpose of this study was to examine the impact of
learning objects in secondary schools. With respect
to impact, teacher and student perceptions of quality,
engagement and learning were assessed, as well as
student learning performance.
Method
Overview
In order to address the key methodological challenges
noted in previous evaluation of learning objects, the fol-
lowing steps were taken:
1A large, diverse, sample was used.
2Reliability and valid surveys were used.
3Formal statistics were used where applicable.
4Both qualitative and quantitative data were collected.
5Both teacher and student perspectives were assessed.
6A measure of student performance was included.
7A wide range of learning objects in a variety of
subject areas was tested.
Sample
Teachers
The teacher sample consisted of 27 teachers (12 males,
15 females) and 50 classrooms (a number of teachers
used learning objects more than once). Teaching experi-
ence ranged from 1 to 33 years with a mean of 9.2
(sd =8.2). Subject areas taught were science (biology,
chemistry, general science, physics, n=15), math
(n=10) and social science (geography, history, n=2).
Eighty-five percent of the teachers rated their ability to
use computers as strong or very strong and their attitude
towards using computers as positive or very positive.
However, only six of the teachers used computers in
their classrooms more than once a month.
Students
The student sample consisted of 850 secondary school
students (444 males, 406 females) ranging in age from
10 to 22 years (M =16.5, sd =1.1). The population
base spanned three separate boards of education, 15 sec-
ondary schools and 27 different classrooms. The stu-
dents were selected through convenience sampling and
had to obtain signed parental permission to participate.
Learning objects
A majority of teachers selected learning objects from
a repository located at the LORDEC website (Learn-
ing Object Research Development and Evaluation
Collaboratory; http://www.education.uoit.ca/lordec/
collections.html), although several reported that they
also used Google. A total of 33 unique learning objects
were selected covering concepts in biology, chemistry,
general science, geography, mathematics and physics.
Procedure
Teachers from three boards of education volunteered to
use learning objects in their classrooms. Each teacher
received a half day of training in November 2006 on
how to choose, use and assess learning objects (see
http://www.education.uoit.ca/lordec/lo_use.html for
more details on the training provided). They were then
asked to use at least one learning object in their class-
rooms by April 2007 of the following year. E-mail
support was available throughout the duration of the
study. All students in a given teacher’s class used the
learning object that the teacher selected. However, only
450 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
those students with signed parental permission forms
were permitted to fill in an anonymous, online survey
about their use of the learning object. In addition, stu-
dents completed a pre- and post-test based on the
content of the learning object.
Data sources
Teacher use
Teachers were asked
1how long it took them to find and integrate learning
objects into their classroom;
2their purpose for using the learning object (e.g. moti-
vate students, teach a new concept, review, supple-
menting a lesson); and
3strategies they used to integrate learning objects (e.g.
demonstration, providing a set of guiding questions,
let student explore, discussion after learning object).
Teacher survey
Each teacher completed the Learning Object Evaluation
Scale for Teachers (LOES-T – see Appendix A) to deter-
mine their perception of how much their students
learned (learning construct), the quality of the learning
object (quality construct) and how much their students
were engaged with the learning object (engagement con-
struct). The constructs selected were based on a thor-
ough review of the literature (Kay & Knaack 2005,
2007b). The LOES-T showed fair to moderate reliability
and good construct validity (see Kay & Knaack 2007c).
Teacher comments
Finally, teachers were asked to comment on the overall
impact that the learning object had on learning (ques-
tion 9, Appendix A).
Student survey
After using a learning object, students completed
the Learning Object Evaluation Scale for Students
(LOES-S) in Appendix B to determine their perception
of how much they had learned (learning construct), the
quality of the learning object (quality construct) and
how much they were engaged with the learning object
(engagement construct). The constructs selected were
based on a thorough review of the literature (Kay &
Knaack 2005, 2007a,b,c). The scale showed good reli-
ability (0.78 to 0.89), face validity, construct validity,
convergent validity and predictive validity (see Kay &
Knaack 2007a).
Student comments
Students were asked to comment on what they liked and
disliked about the learning object (Appendix B – ques-
tions 13 and 14). These qualitative items were organized
according to the three main constructs identified in the
literature review (learning, quality and engagement) and
analysed using the coding scheme provided in Table 1.
This coding scheme (Kay & Knaack 2007a) was used to
categorize 1302 student comments. Each comment was
then rated on a 5-point Likert scale (-2=very negative,
-1=negative,0=neutral,1=positive,2=very posi-
tive). Two raters assessed all comments made by stu-
dents and achieved inter-rater reliability of 99% on the
categories and 100% on the ratings.
Note that the total impact of any one category was
determined by multiplying the mean rating by the total
number of students who made a comment. For example,
from Table 2, the impact of visual supports on learning
was calculated by multiplying the mean which was 0.91
by the number of students who commented about visual
supports (92) for a total of 84.0.
Student performance
Students completed a pre- and post-test based on the
content of the learning object used in class. The differ-
ence between pre- and post-test scores was used to
determine student performance.
Key questions and data analysis
In order to examine the impact of learning objects on
secondary school students, the following questions
were addressed in the data analysis:
• How do teachers use learning objects in their
classrooms? (descriptive analysis of teacher-use
questions)
• How do teachers rate learning, quality and engage-
ment of learning objects? (descriptive analysis of
teacher survey – LOES-T)
• What was the overall impact of learning objects?
(analysis of qualitative teacher comments)
• How do students rate learning, quality and engage-
ment of learning objects? (descriptive analysis of
student survey – LOES-S)
Learning objects in secondary school 451
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
• What do students like and dislike most about learning
objects? (qualitative analysis of student comments)
• How do teacher ratings of learning objects compare
with student ratings? (correlation among learning,
quality and engagement constructs)
• How do learning objects affect student performance
(t-test comparing pre and post scores)?
Results
Use of learning objects
Finding a learning object
Forty-two percent of the teachers reported that finding a
suitable learning object took them less than 30 min.
Thirty-six percent took 30 to 60 min to find an appropri-
ate learning object. The remaining 22% took over an
hour to finding the learning object they wanted to use in
their class.
Preparing a learning object lesson
With respect to preparation for using the learning object
in class, 6% of the teachers spent little or no time, 42%
spent less than 30 min, 28% spent 30 to 60 min and the
remaining 24% spent over an hour.
Using a learning object
On average, teachers used learning objects for 38.1 min
(sd =30.5). However there was considerable variability
(6 to 210 min). Students worked independently on their
Table 1. Coding scheme to categorize student comments about learning objects.
(a) Learning
Category label Criteria
Challenge Refers to the ease/difficulty of the concepts being covered. Basically whether the content level of the LO
matched the student’s cognitive level/understanding.
Code ‘it was easy’ in here, but not ‘it was easy to use’
Learn Student comments about a specific or general learning/teaching issue involved in using the LO
Visual The student mention as visual feature of the LO that helped/inhibited their learning
(b) Engagement
Category label Criteria
Compare Student compares LO to another method of learning
Engage Student refers to program as being OR not being fun/enjoyable/engaging/interesting
Technology The student mention a technological issue with respect to using the LO
(c) Quality
Category label Criteria
Animate Refers to quality of animations/moving pictures
Audio Refers to some audio/sound aspect of the learning object
Easy Refers to clarity of instructions or how easy/hard the LO was to use. It does not refer to how
easy/hard the concept was to learn.
Graphics Refers to static picture or look of the program (e.g. colours)
Help Refers specifically to help/hints/instructions/feedback provided by the LO
Interactive Student refers to some interactive part feature of the LO
Control Refers to student control of choice/pace in using the LO
Organization/design Refers to quality of organization/design or the LO
Text Refers to quality/amount of text in LO
Theme Refers to overall/general theme or CONTENT of LO
LO, learning object.
452 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
own computers in a majority of classrooms (90%), with
cooperative learning chosen only 10% of the time.
Reason for using a learning object
The top four reasons cited by teachers for using learning
objects was to review a previous concept (54%), to
provide another way to look at a concept (48%), to moti-
vate students about a topic (38%) and to introduce a
concept before a formal lesson (30%). Teachers rarely
used learning objects to teach a new concept (8%),
explore a new concept after a formal lesson (6%) or for
homework (4%).
Strategies for using learning objects
Teachers in this study typically provided a brief intro-
duction to a learning object (58%) or let the students
start exploring on their own (46%). Only 16% offered a
formal demonstration of the learning object before the
class used it. Forty percent of teachers prepared a formal
handout for students to guide the use of the learning
object. Thirty-eight percent of teachers chose to have a
class discussion about the learning object after it was
used by students.
Teacher rating of learning objects
Learning
The mean rating for the learning construct was 11.4
(sd =1.4) or 5.7 on a 7-point scale. This suggests that
most teachers agreed that learning objects had a positive
impact on student learning. Note that the range of learn-
ing construct scores was relatively narrow (8 to 14) pro-
viding additional support for the conclusion that a
majority of teachers felt learning objects offered learn-
ing benefits (Table 3).
Quality of learning object
The mean rating of learning object quality was 17.3
(sd =1.4) or 5.8 on a 7-point scale. Most teachers
agreed or strongly agreed the learning object was of
Table 2. Summary of student comments about learning objects.
Category Mean SD nTotal effect (mean ¥n)
Learning
Visual supports 0.91 0.41 92 84.0
Overall learning –0.07 1.11 191 –13.0
Challenge –0.48 1.04 152 –73.0
Quality
Easy 0.95 0.60 74 70.0
Animation 0.67 0.75 43 29.0
Graphics 0.24 1.08 97 23.0
Audio –1.00 0.00 5 –5.0
Theme –0.19 1.22 42 –8.0
Control –0.32 1.04 34 –11.0
Organization –0.25 1.11 55 –14.0
Help –0.53 1.00 68 –36.0
Text –1.03 0.59 38 –39.0
Engagement
Interactivity 0.79 0.71 73 58.0
Compare with other method 0.70 0.75 63 44.0
Engagement/motivation 0.21 1.16 107 22.0
Liking technology –0.10 1.21 59 –6.0
Table 3. Teacher rating of learning, quality, and engagement for learning objects.
Scale No. of items Possible range Actual range observed Mean (SD)
Learn 2 2 to 14 8 to 14 11.4 (1.4)
Quality 3 3 to 21 11 to 21 17.3 (2.6)
Engagement 3 2 to 21 9 to 21 17.1 (2.9)
Learning objects in secondary school 453
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
good quality. The range of learning object quality
scores was broader than the learning construct
scores (11 to 21) but never dipped into negative rating
(Table 3).
Engagement
Teachers also rated engagement with learning objects
high with a mean score of 17.1 (sd =1.4) or 5.7 on a
7-point scale. A majority of teachers, then, felt students
were engaged while using learning objects. The range
of learning object engagement scores was relatively
large compared to the learning and quality constructs (9
to 21) (Table 3).
Teacher comments about learning objects
Four themes emerged from the 68 comments that teach-
ers made about the overall impact of the learning object:
learning (63%), engagement (22%), time (13%) and
individual differences (3%). Details for each theme and
sample comments offered by teachers are presented in
Table 4.
Of the 43 comments that teachers made about learn-
ing, 12 focussed on overall impact. Eight teachers made
positive comments about how much was learned or how
students were on task. However, four teachers observed
that the learning object was not as successful as they had
hoped. Twelve teachers commented on the effectiveness
Table 4. Qualitative comments from teachers.
Category n% Sample comments
Learning
Overall positive 12 18% ‘It was a success, and I think it helped students prepare for an upcoming lab
on the topic.’
’Students had a much better background in the subject when it was introduced
[using learning objects]’
‘[The learning object] kept them on task and the class basically ran itself’
Overall negative 4 5% ‘They still had great difficulty distinguishing between vertical and horizontal
stretches and compressions.’
’The method of explanation and some of the wording was different than our . . .
textbook’
Visual supports 15 22% ‘I feel like the visual . . . interaction with the stages of mitosis really helps them to
understand the basic concepts of cell division’
’They had not been very good at solving problems like this earlier in the semester.
The graph that it drew to show the relationship helped them a lot’
‘For many students, using the balance scale was an excellent visual representation
of solving algebra problems.’
Review 12 18% ‘It helped review a topic that I had taught them last year in grade 11’
’This learning object was a great interactive review of concepts learned in grade 6. It
not only helped review concepts, but it also motivated students about probability
(the topic of our new unit)’
‘I am always reviewing balancing equations, and this learning object provided [the]
students with an interactive and immediate means to assess what they
remembered.’
Interactive 4 6% ‘The interactivity of the learning object helped the students view the relationships
between the sizes of the different objects.’
Engagement 15 22% ‘It helped to motivate them during the formal teaching after the learning object.’
‘Most students seemed to enjoy working with the learning object’
Time 9 13% ‘The learning object allowed me to take a much shorter time to teach to teach
the concept’
‘I had only had one class prior to using the learning object to introduce the concept
to the class because of problems at the school in terms of booking computer
time.’
Individual differences 2 3% ‘Some students seemed to work well, however, one student was finished in 5
minutes, whilst another was finished in 35 minutes’
’The learning object was a good one, but only my strongest students were able to
learn the concept using the learning object and accompanying worksheets.’
454 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
of learning objects to provide a good review and 15
teachers noted that visual supports provided good
opportunities for learning. Finally, four teachers
remarked that learning objects provided good interac-
tivity for the students.
Almost one quarter of the teachers felt that one of the
key impacts of learning objects was engagement. They
believed that students were more interested or motivated
when using this tool. Nine teachers mentioned that time
was an issue either in creating a good lesson plan with a
learning object, saving time, booking the right time to
use a learning object or not having enough time. Finally,
three teachers observed individual differences with
respect to the impact of learning objects. Some students
worked with learning objects quickly and efficiently
whereas others struggled and took more time.
Student rating of learning objects
Learning
Students rated learning objects lower than teachers with
respect to learning (M =17.01, sd =4.3) with a mean
item rating of 3.4 out 5 (or 4.8 out of 7). Students were
between ‘neutral’ and ‘agree’with respect to how much
they felt the learning objects contributed to their
learning. The range of scores was extensive (5 to 25),
indicating that there was considerable variability
(Table 5). The mean range was 13.5 (sd =3.9) out of a
possible 20-point spread.
Quality of learning objects
Students rated the quality of learning objects higher
than the learning construct, although the mean item
rating was still lower than that of the teachers. The mean
item rating of 3.7 out of 5 (5.2 out of 7) indicated that
most students agreed that the learning objects they used
were of good quality. The range of learning object
quality scores (4 to 20) showed considerable variability
(Table 5). The mean range was 9.2 (sd =3.3) out of a
possible 16-point spread.
Engagement
Ratings of learning object engagement were moderate
(M =10.2, sd =2.6) with a mean item rating of 3.4 out
or 5 (or 4.8 out of 7). In other words, as was the case
with the learning construct, students were somewhere
in between ‘neutral’ and ‘agree’ when assessing the
engagement value of the learning objects they used.
High variability among student engagement ratings is
supported by the wide range of scores reported (3 to 15)
and a mean range of 8.3 (sd =2.5) out of a possible 12
point spread.
Student comments about learning objects
Student comments are summarized in Table 2. With
respect to learning, the visual support that a learning
object offered towards learning was rated the highest,
whereas the pedagogical challenge of learning objects
was rated quite low. In other words, many students liked
the visual affordances of learning objects. However,
quite a few felt the learning object was not challenging
enough.
With respect to rating the quality of learning objects,
ease of use was the highest rated feature, followed by
animation and graphics. The quality of help and the
excessive amount of text was rated the lowest.
Finally, regarding engagement, ‘interactivity’ and
‘comparing learning objects to other methods of teach-
ing’ were rated highest. A number of the students liked
the interactive qualities of learning objects and felt they
were an improvement over other teaching strategies.
Teacher vs. student ratings
Teacher ratings of learning (r=0.51, P<0.001),
quality (r=0.52, P<0.001) and engagement (r=0.39,
P<0.001) were significantly correlated with student
ratings of the same constructs. Note that the teachers
and students did not appear to agree on the engagement
construct as much as they did on the learning and quality
constructs.
Table 5. Description of Student Learning Object Evaluation Scales.
Scale No. of items Possible range Actual range observed Mean (SD)
Learn 5 5 to 25 5 to 25 17.0 (4.3)
Quality 4 4 to 20 4 to 20 14.9 (3.3)
Engagement 3 3 to 15 3 to 15 10.2 (2.6)
Learning objects in secondary school 455
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Student performance
Differences between pre- and post-test scores were cal-
culated for classes where the learning object was not
used for review. This yielded a total of 194 students.
Student performance scores increased by an average of
29.3% from 40.5% to 69.9%. This change was signifi-
cant (t=-13.6, d.f. =193, P<0.001). The effect size
(based on Cohen’s d) of 1.10 is considered very large
according to Thalheimer and Cook (2002).
Discussion
The purpose of this study was to examine the impact of
learning objects in secondary school classrooms. Six
sources of data were examined: teacher use, teacher
ratings, teacher comments about overall impact, student
ratings, student comments about what they liked and did
not like and student performance. Each of these data
sources will be discussed in turn.
Teacher use
Previous research is relatively silent with respect to how
teachers use computers. This study provides new infor-
mation in this area. On the surface, it appears that
finding and preparing to use a learning object for a sec-
ondary school classroom does not take excessive time –
roughly an hour on average. However, a number of
teachers commented that time was an issue in using
learning objects. While an hour is not overwhelming in
terms of time, it may be more than the typical teacher
spends on a regular lesson plan.
It is interesting that most teachers used learning
objects to support concepts that they had already taught.
Only 30% used learning objects to help teach a new
topic before a formal lesson and less than 10% used
learning objects to teach a new concept on its own. One
could speculate that teachers were being cautious with
respect to introducing a new teaching strategy in their
classroom.
When using learning objects, most teachers offered
a brief introduction and let the students explore on
their own. About 40% of teachers prepared a formal
handout and/or had a class discussion about the learn-
ing object. This meant that the majority of teachers
did not provide scaffolding – students were left to
investigate and draw conclusions on their own. Perhaps
teachers felt that since the concept being covered was
a review, there was no need to provide additional
support. It might also mean that teachers felt that the
learning objects they selected should stand on their
own in terms of teaching.
Teacher ratings and comments (learning,
quality and engagement)
It is safe to say that teachers felt that the learning objects
they selected were good quality engaging tools that sup-
ported learning. Ratings were very high. On the one
hand, it is somewhat predictable that teachers would
rate learning objects high – after all they were the ones
who selected them. On the other hand, teachers rated
these learning objects after they watched them being
used by students in their classroom. The fact that teacher
ratings were relatively consistent with student ratings
partially confirms reliability and validity. Positive reac-
tion from teachers in this study is consistent with
previous research on secondary education teachers
and learning objects (McCormick & Li 2005; Kay &
Knaack 2007b).
Teacher comments
With respect to learning, teacher comments were con-
sistent with the survey results. Most teachers felt their
students were on task and that the learning objects
offered a good review of concepts. In addition, learning
objects were thought to be engaging. These comments
are reflective of previous findings (De Salas & Ellis
2006; Kay & Knaack 2007b). Asmaller group of teach-
ers reported that time was a concern – an issue that
worried higher education faculty (Collis & Strijker
2003; Christiansen & Anderson 2004). Finally, three
teachers reported that they noticed individual differ-
ences with respect to students’ interaction with learning
objects in terms of pace and ease of use. Clearly the
results from this small sample should be treated cau-
tiously, although it may be worthwhile to explore indi-
vidual differences in future studies.
Student ratings (learning, quality and engagement)
Students were moderately positive about the quality and
engagement of learning objects and closer to neutral
456 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
when assessing the learning value. These results some-
what contradict more positive findings reported for
higher education (Kenny et al. 1999; Bradley & Boyle
2004; Docherty et al. 2005; MacDonald et al. 2005;
Mason et al. 2005; Schoner et al. 2005; De Salas &
Ellis 2006; Lim et al. 2006). However, when quantita-
tive data is gathered (e.g. Howard-Rose & Harrigan
2003; Bradley & Boyle 2004; De Salas & Ellis 2006),
students’ satisfaction with learning objects hovers
around 50%–60%. In addition, the results of this study
match those reported by Kay and Knaack (2007b) who
observed that students were ‘somewhat positive’ when
rating learning objects.
It is worth noting that the range of scores is broad for
all three constructs. This means that for any given learn-
ing object, some students like it a lot and others dislike it
a lot, even when it is the same learning object and the
same teacher. As stated earlier, some teachers noted
individual differences in the use and acceptance of
learning objects. The age old problem of trying to
address diversity in needs and ability still exists when
learning objects are used.
Student comments about learning objects
Student comments offer some insights into why stu-
dents liked and did not like using learning objects. Stu-
dents liked learning objects that were easy to use and
had good interactivity, visual supports, animations and
graphics. They did not like learning objects that were
not challenging enough, nor did they like poor help fea-
tures and excessive amounts of text. These findings are
closely aligned to the results reported by Kay and
Knaack (2007b). It is also worth noting that even though
a good number of students may not have rated learning
objects very highly in terms of learning, quality and
engagement, quite a few students reported that using
learning objects was an improvement over other teach-
ing strategies.
Teacher vs. student ratings
Teacher and student impressions of learning objects
and their effectiveness were relatively consistent. It
appears however that teachers were more positive than
students on all three survey constructs and especially
when it came to rating engagement. This is an impor-
tant finding because few studies compare teacher–
student perceptions. Studies that glean feedback from
only one of these populations may be underestimating
or overestimating the impact of learning objects.
Clearly it is important to gather data from both teacher
and students to get a balanced perspective.
Student performance
Regardless of either teacher or student perceptions of
the impact of learning objects, it is clear that learning
performance increased markedly when learning objects
were used. Although no previous research has looked at
student performance in secondary schools, these results
are consistent with those reported in higher educa-
tion (Windschitl & Andre 1998; Rieber et al. 2004;
Docherty et al. 2005). They also suggest that teacher
analysis of learning in this study is more closely aligned
to actual performance than student analysis. Students’
modest ratings of learning did not the match significant
jumps observed in performance.
The improvement in student performance does not
mean that learning objects were uniquely responsible
for these gains. Anumber of teachers used these tools in
combination with a more formal lesson or class discus-
sion, so the influence of learning objects is partially
confounded by additional teaching techniques. It is
reasonable, and perhaps ideal, that learning objects are
integrated within a full classroom lesson that involves
multiple teaching strategies.
Implications for education
There are several implications for secondary school
educators who want to use learning objects in their
classrooms. First, it will take about an hour to find and
prepare a learning object lesson plan, but it may take as
long as 2 h. Second, both teacher and students are posi-
tive about the use of learning objects in the classroom,
although the impact of learning objects may vary con-
siderably within the same classroom. Accommodations
will have to be made for students with different ability
and interest levels. Third, when learning objects are
integrated into a lesson plan, student performance
increases significantly. From the behaviour of most
teachers in this study, combining the use of learning
objects with a formal lesson using a brief introduction,
supporting handout and/or class discussion may work
well.
Learning objects in secondary school 457
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Caveats and future research
While considerable effort was made to ensure the reli-
ability and validity of the results reported in this study,
several limitations remain which offer opportunities for
future research efforts. First, while the evidence sug-
gests that learning objects had a positive impact in the
classroom in terms of attitude and performance, without
a control group it is difficult to determine the relative
impact of the learning object technology itself. It is con-
ceivable, for example, that other factors such as the
structure afforded by the use of learning objects may be
the critical component in the learning process, and not
the technology. Second, even though the population was
large and balanced in terms of gender, the subject speci-
ality of most teachers was science and mathematics. Dif-
ferent results may be observed with other subject areas.
Third, while the overall impact of learning object-based
lessons on student performance was large, the effect of
specific instructional strategies was not examined. In
other words, we do not know which teaching strategies
work best with learning objects. Finally, the study was
designed to look at the overall impact of learning objects
– the impact of specific kinds of learning objects was not
looked at. It is possible that certain categories of learning
objects may have decidedly different effectson learning.
Conclusions
This study adds significantly to the current knowledge
regarding the use of learning objects. First, it looks
at the secondary school population, a sector that has not
been examined in much detail. Second, it looks at how
teachers prepare for and use learning objects. Most sec-
ondary teachers take anywhere from one to 2 h to
produce a lesson that is often focussed on review and to
a lesser extent introducing a new concept. Previous
learning object research is almost silent with respect
to instructional strategies used with learning objects.
Third, three forms of data collection were combined
to analyse the impact of learning objects: teacher atti-
tude, student attitude and student performance. This
approach, rarely followed in learning object research,
offered a reliable and valid method of evaluation.
All three data sources confirmed that learning objects
have a positive effect in secondary school classrooms.
Finally, student performance increased by an average of
almost 30% when learning objects were used in con-
junction with other teaching strategies. This finding
supports the premise that learning objects can be an
effective teaching aid in secondary schools classrooms.
Acknowledgements
Research funded by the Canadian Council of Learning
and the Waterloo Region District School Board.
Appendix A – Learning object evaluation scale – teachers.
Strongly
disagree 1
Disagree
2
Slightly
disagree 3
Neutral
4
Slightly
agree 5
Agree
6
Strongly
agree 7
Learning
1. The graphics and animations from the
learning object helped students learn.
1234567
2. The students were able to learn from
the learning object.
1234567
Quality
3. The learning object was easy for
students to use.
1234567
4. The learning object was easy to learn. 1 2 3 4 5 6 7
458 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Appendix A Continued
Strongly
disagree 1
Disagree
2
Slightly
disagree 3
Neutral
4
Slightly
agree 5
Agree
6
Strongly
agree 7
5. The students found the learning
object instructions clear
1234567
Engagement
6. The students liked interacting with
the learning object.
1234567
7. The students were on task while
using the learning object.
1234567
8. Students were motivated while using
the learning object.
1234567
Overall Impact on Learning
9. What was the overall impact of the learning object on your lesson?
Appendix B – Learning object evaluation survey – students.
Strongly
disagree 1
Disagree
2
Neutral
3
Agree
4
Strongly
agree 5
Learning
1. Working with the learning object helped me learn. 1 2 3 4 5
2. The feedback from the learning object helped me learn. 1 2 3 4 5
3. The graphics and animations from the learning object helped
me learn.
12345
4. The learning object helped teach me a new concept. 1 2 3 4 5
5. Overall, the learning object helped me learn. 1 2 3 4 5
Quality
6. The help features in the learning object were useful. 1 2 3 4 5
7. The instructions in the learning object were easy to follow. 1 2 3 4 5
8. The learning object was easy to use. 1 2 3 4 5
9. The learning object was well organized. 1 2 3 4 5
Engagement 1 2 3 4 5
10. I liked the overall theme of the learning object. 1 2 3 4 5
11. I found the learning object motivating. 1 2 3 4 5
12. I would like to use the learning object again. 1 2 3 4 5
13. What, if anything, did you LIKE about the learning object?
14. What, if anything, did you NOT LIKE about the learning object?
References
Agostinho S., Bennett S., Lockyear L. & Harper B. (2004)
Developing a learning object metadata application profile
based on LOM suitable for the Australian higher education
market. Australasian Journal of Educational Technology
20, 191–208.
Bradley C. & Boyle T. (2004) The design, development, and
use of multimedia learning objects. Journal of Educational
Multimedia and Hypermedia 13, 371–389.
Brush T. & Saye J. (2001) The use of embedded scaffolds with
hypermedia-supported student-centered learning. Journal
of Educational Multimedia and Hypermedia 10, 333–
356.
Learning objects in secondary school 459
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Butson R. (2003) Learning objects: weapons of mass
instruction. British Journal of Educational Technology 34,
667–669.
Christiansen J. & Anderson T. (2004) Feasibility of course
development based on learning objects: research analysis of
three case studies. International Journal of Instructional
Technology and Distance Learning 1. Available at: http://
www.itdl.org/Journal/Mar_04/article02.htm (last accessed
30 July 2005).
Collis B. & Strijker A. (2003) Re-usable learning objects in
context. International Journal on E-Learning 2, 5–16.
Concannon F., Flynn A. & Campbell M. (2005) What campus-
based students think about the quality and benefits of
e-learning. British Journal of Educational Technology 36,
501–512.
De Salas K. & Ellis L. (2006) The development and imple-
mentation of learning objects in a higher education. Inter-
disciplinary Journal of E-Learning and Learning Objects
2, 1–22.
Docherty C., Hoy D., Topp H. & Trinder K. (2005) e-Learning
techniques supporting problem-based learning in clinical
simulation. International Journal of Medical Informatics
74, 527–533.
Duval E., Hodgins W., Rehak D. & Robson R. (2004) Learning
objects symposium special issue guest editorial. Journal of
Educational Multimedia and Hypermedia 13, 331–342.
Gadanidis G., Gadanidis J. & Schindler K. (2003) Factors
mediating the use of online applets in the lesson planning of
pre-service mathematics teachers. Journal of Computers in
Mathematics and Science Teaching 22, 323–344.
Haughey M. & Muirhead B. (2005) Evaluating learning
objects for schools. E-Journal of Instructional Sciences
and Technology 8. Available at: http://www.usq.edu.au/
electpub/e-jist/docs/vol8_no1/fullpapers/
eval_learnobjects_school.htm (last accessed 1 June 2007).
Howard-Rose D. & Harrigan K. CLOE learning impact studies
lite: evaluating learning objects in nine Ontario university
courses. Available at: http://cloe.on.ca/documents/
merlotconference10.doc (last accessed 3 July 2007).
Kay R. & Knaack L. (2005) Developing learning objects for
secondary school students: a multi-component model.
Interdisciplinary Journal of E-Learning and Learning
Objects 2005, 229–254.
Kay R.H. & Knaack L. (2007a) Assessing learning, quality
and engagement in learning objects: the Learning Object
Evaluation Scale for Students (LOES-S). Available at:
http://faculty.uoit.ca/kay/papers/LOES_Student_2007.pdf
(last accessed 3 March 2008).
Kay R.H. & Knaack L. (2007b) A systematic evaluation of
learning objects for secondary school students. Journal of
Educational Technology Systems 35, 411–448.
Kay R.H. & Knaack L. (2007c) Teacher evaluation of learning
objects in middle and secondary school classrooms.
Available at: http://faculty.uoit.ca/kay/papers/LOES_
Teacher_2007.pdf (last accessed 3 March 2008).
Kenny R.F., Andrews B.W., Vignola M.V., Schilz M.A. &
Covert J. (1999) Towards guidelines for the design of inter-
active multimedia instruction: fostering the reflective
decision-making of preservice teachers. Journal of Tech-
nology and Teacher Education 7, 13–31.
Koppi T., Bogle L. & Lavitt N. (2004) Institutional use of
learning objects: lessons learned and future directions.
Journal of Educational Multimedia and Hypermedia 13,
449–463.
Krauss F. & Ally M. (2005) Astudy of the design and evalua-
tion of a learning object and implications for content
development. Interdisciplinary Journal of E-Learning and
Learning Objects,1, 1–22.
Lim C.P., Lee S.L. & Richards C. (2006) Developing
interactive learning objects for a computing mathematics
models. International Journal on E-Learning 5, 221–244.
Lopez-Morteo G. & Lopez G. (2007) Computer support for
learning mathematics: a learning environment based on
recreational learning objects. Computers and Education 48,
618–641.
McCormick R. & Li N. (2005) An evaluation of European
learning objects in use. Learning Media and Technology 31,
213–231.
MacDonald C.J., Stodel E., Thompson T.L., Muirhead B.,
Hinton C., Carson B. & Banit E. (2005) Addressing the
eLearning contradiction: a collaborative approach for
developing a conceptual framework learning object. Inter-
disciplinary Journal of E-Learning and Learning Objects
1, 79–98.
McGreal R. (2004) Learning objects: a practical definition.
International Journal of Instructional Technology and Dis-
tance Learning 1. Available at: http://www.itdl.org/Journal/
Sep_04/article02.htm (last accessed 5 August 2005).
Mason R., Pegler C. & Weller M. (2005) A learning object
success story. Journal of Asynchronous Learning Networks
9. Available at: http://www.sloan-c.org/publications/jaln/
v9n1/v9n1_mason.asp (last accessed 1 June 2007).
Nurmi S. & Jaakkola T. (2005) Problems underlying the learn-
ing object approach. International Journal of Instructional
Technology and Distance Learning 2. Available at: http://
www.itdl.org/Journal/Nov_05/article07.htm (last accessed
9 April 2007).
Nurmi S. & Jaakkola T. (2006a) Effectiveness of learning
objects in various instructional settings. Learning Media
and Technology 31, 233–247.
Nurmi S. & Jaakkola T. (2006b) Promises and pitfall of learn-
ing objects. Learning, Media and Technology 31, 269–285.
460 R.H. Kay & L. Knaack
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
Parrish P.E. (2004) The trouble with learning objects. Educa-
tional Technology Research & Development 52, 49–67.
Polsani P.R. (2003) Use and abuse of reusable learning
objects. Journal of Digital Information 3. Available at:
http://journals.tdl.org/jodi/article/view/jodi-105/88 (last
accessed 1 January 2008).
Rehak D. & Mason R. (2003) Chapter 3: keeping the learning
in learning objects. Journal of Interactive Media in Educa-
tion 2003. Available at: http://www-jime.open.ac.uk/2003/
1/reuse-05.html (last accessed 1 July 2005).
Rieber L.P., Tzeng S. & Tribble K. (2004) Discovery learning,
representation, and explanation within a computer-based
simulation: finding the right mix. Learning and Instruction
14, 307–323.
Schoner V., Buzza D., Harrigan K. & Strampel K. (2005)
Learning objects in use: ‘lite’ assessment for field studies.
Journal of Online Learning and Teaching 1, 1–18.
Sedig K. & Liang H. (2006) Interactivity of visual mathemati-
cal representations: factors affecting learning and cognitive
processes. Journal of Interactive Learning Research 17,
179–212.
Shea P., McCall S. & Ozdogru A. (2006) Adoption of the mul-
timedia educational resource for learning and online teach-
ing (MERLOT) among higher education faculty: evidence
from the State University of New York learning network.
Journal of Online Learning and Teaching 2, 136–156.
Sosteric M. & Hesemeier S. (2002) When is a learning object
not an object: a first step towards a theory of learning
objects. International Review of Research in Open and Dis-
tance Learning 3, 1–16.
Thalheimer W. & Cook S. (2002) How to calculate effect sizes
from published research articles: a simplified methodology.
Available at: http://work-learning.com/effect_sizes.htm
(last accessed 10 November 2004).
Van Zele E., Vandaele P., Botteldooren D. & Lenaerts J.
(2003) Implementation and evaluation of a course concept
based on reusable learning objects. Journal of Educational
Computing and Research 28, 355–372.
Wiley D., Waters S., Dawson D., Lambert B., Barclay M. &
Wade D. (2004) Overcoming the limitations of learning
objects. Journal of Educational Multimedia and Hyperme-
dia 13, 507–521.
Wiley D.A. (2000) Connecting learning objects to instruc-
tional design theory: a definition, a metaphor, and a
taxonomy. In The Instructional Use of Learning Objects:
Online Version (ed. D.A. Wiley). Available at: http://
reusability.org/read/chapters/wiley.doc (last accessed 1
July 2005).
Windschitl M. & Andre T. (1998) Using computer simulations
to enhance conceptual change: the roles of constructivist
instruction and student epistemological beliefs. Journal of
Research in Science Teaching 35, 145–160.
Learning objects in secondary school 461
© 2008 The Authors. Journal compilation © 2008 Blackwell Publishing Ltd
