Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay
Content may be subject to copyright.
A Formative Analysis of Instructional Strategies
for Using Learning Objects
ROBIN KAY, LIESEL KNAACK, AND BILL MUIRHEAD
University of Ontario Institute of Technology, Canada
robin.kay@uoit.ca
liesel.knaack@uoit.ca
bill.muirhead@uoit.ca
To date, limited research has been done examining and eval-
uating the instructional wrap for using learning objects effec-
tively. The current study examined instructional strategies
used by 15 teachers to integrate learning objects into 30 sec-
ondary school classrooms (510 students). Four key areas
were examined: preparation time, purpose for using a learn-
ing object, integration strategies, and time spent using a learn-
ing object. A small, but significant, correlation was observed
between preparation time and student attitudes toward learn-
ing objects. When the purpose of using a learning object was
to introduce a concept before a formal lesson, motivate stu-
dents, or teach a new concept, student attitudes and perfor-
mance were significantly higher. On the other hand, choosing
to use a learning object after a formal lesson or to review a
concept resulted in significantly lower student attitudes and
performance. Regarding integration strategies, providing a
guiding set of questions was associated with more positive
student attitudes and increased performance, whereas allow-
ing students to explore on their own (without direction) and
class discussion after use led to significantly lower student
attitudes and performance. Finally, time spent using learning
objects was inversely correlated with student attitudes and
performance. It is reasonable to conclude that decisions about
instructional wrap had a significant impact on the effective-
ness of learning objects in a secondary school environment.
Learning objects are operationally defined in this study as interactive
web-based tools that support the learning of specific concepts by enhancing,
amplifying, and/or guiding the cognitive processes of learners (Bennett &
Jl. of Interactive Learning Research (2009) 20(3), 295-315
McGee, 2005; Bradley & Boyle, 2004; Caws, Friesen, & Beaudoin, 2006;
Cochrane, 2005; McGreal, 2004; Kay & Knaack, 2007a; Wiley, Waters,
Dawson, Lambert, Barclay, & Wade, 2004). One of the first serious discus-
sions about learning objects occurred in the form of an online edited book
entitled The Instructional Use of Learning Objects (Wiley, 2000). Since that
seminal work appeared, numerous papers have been written about learning
objects focusing primarily on four key areas: finding an acceptable defini-
tion (e.g., McGreal, 2004, Wiley, 2000), design and development (Ally,
2004; Bradley & Boyle, 2004; Kay & Knaack, 2005), storage and retriev-
able (Agostinho, Bennett, & Lockyear, 2004; Carey, Swallow, & Oldfield,
2002), and reuse (Collis & Strijker, 2004; Koppi, Bogle, & Bogle, 2005).
Within the last three years, the scope of research has expanded to three more
issues: learning, evaluation, and use (Kay & Knaack, 2007d).
Systematic investigation of instructional strategies used by educators to
integrate learning objects into the curriculum has been noticeably absent
(Bennett & McGee, 2005; Kay & Knaack, 2007e). One possible reason for
this omission is the original paradigm for defining and designing learning
objects. A majority of articles, particularly in the domain of higher educa-
tion, view learning objects as independent, stand-alone, web-based tools
(e.g., Kong & Kwok, 2005; Oliver & McLoughlin, 1999; Poldoja, Leinonen,
Valjataga, Ellonen, & Priha, 2006). In addition, a number of higher educa-
tion institutions supported the creation of self-directed objects in an effort to
reduce demands on instructor time and save money (Weller, 2004).
However, recent research on the use of learning objects reveals that not
all learning objects have a positive impact on learning (Kay & Knaack,
2007c). Nurmi & Jaakola (2006b) suggest that learning objects can facilitate
learning through active knowledge construction, although they can also lead
to reductionist teaching practices that stress content delivery and knowledge
transmission. In fact, several researchers have suggested that teachers need
training to maximize the effectiveness of learning objects (Bratina, Hayes,
& Blumsack, 2002; COHERE Group, 2002; Gadanidis, Gadanidis &
Schindler, 2003; Haughey & Muirhead, 2005). In addition, a growing num-
ber of theorists have argued that the ultimate effectiveness of any learning
object is largely dependent on the pedagogical choices of the instructor
(Alonso, Lopez, Manrique, & Vines, 2005; Bratina et al., 2002; Haughey &
Muirhead, 2005; Koppi et al., 2004; McCormick & Li, 2005; Moyer, 2002;
Thorpe, Kubiak, & Thorpe, 2003). It appears that while learning objects may
have been conceived as autonomous learning tools, how educators use them
may be critical to their overall effectiveness.
While a number of papers have focused on the more traditional, self-
guided use of learning objects (Docherty, Hoy, Topp, & Trinder, 2005; Kong
& Kwok, 2005; Nurmi & Jaakkola, 2006a; Reimer & Moyer, 2005), sever-
al studies have looked at the effectiveness of specific strategies used to sup-
296 Kay, Knaack, and Muirhead
A Formative Analysis of Instructional Strategies for Using Learning Objects 297
port learning objects including coaching or facilitating (Liu & Bera, 2005),
establishing context (Schoner, Buzza, Harrigan, & Strampel, 2006), instruct-
ing students to evaluate their own actions (van Marrienboer & Ayres, 2005),
and providing some sort of instructional guide or scaffolding (Brush & Saye,
2001; Concannon, Flynn, & Campbell, 2005; Lim, Lee, & Richards, 2006;
Mason, Pegler, &Weller, 2005; Mayer, 2004).
When learning objects were used with minimal interaction from an instruc-
tor, moderate success was experienced for higher education students (Docher-
ty et al., 2005; Kong & Kwok, 2005; Reimer & Moyer, 2005). However, stu-
dents in grades seven to ten did not fair as well (Kay & Knaack, 2007a; Nurmi
& Jaakkola, 2006a). It is possible that demands of self-regulation may be too
high for younger students. With respect to context, Schoner et al. (2006) noted
linking course objectives and learning outcomes to specific learning objects
improved educational value. When students were instructed to evaluate their
actions, van Marrienboer & Ayres (2005) observed learning objects yielded
more positive results. Finally, considerable evidence suggests that learning
objects are more effective when scaffolding in the form of worksheets and
guiding questions are provided (Brush & Saye, 2001; Concannon et al., 2005;
Lim et al., 2006; Mason et al., 2005; Mayer, 2004).
In summary, the agenda for the majority of articles written to date has
been to look at the design and developmental process of stand-alone objects
that are readily accessed and reused. Only a handful of studies have exam-
ined the impact of individual teaching strategies used with learning objects
and no studies have compared strategies. However, a substantial number of
theorists (Alonso et al., 2005; Bratina et al., 2002; Haughey & Muirhead,
2005; Koppi et al., 2004; McCormick & Li, 2005; Moyer, 2002; Thorpe,
Kubiak, & Thorpe, 2003) believe that how a teacher chooses to use a learn-
ing object is critical for successful implementation. The purpose of the cur-
rent study was to examine and evaluate instructional strategies that teachers
use when integrating learning objects into secondary school classrooms.
METHOD
Overview
This study reviewed a total of 17 research studies (6 in elementary
schools, 2 in secondary schools, 9 in higher education) looking at the impact
of specific strategies on the use of learning objects. Many of these studies
used mixed methods that included qualitative, quantitative, and performance
metrics (n=11); however, a number of challenges remain with respect to
improving the investigation of learning objects.
First, while a wide range of learning objects exist, the majority of papers
focused on a single learning object. It is difficult to determine whether the
results in one study generalize to the full range of learning objects that are
available. Second, sample populations tested were relatively small (M = 65.6)
and poorly described, making it challenging to extend any conclusions to a
larger population. Third, while most evaluation studies reported that students
benefited from using learning objects, the evidence is based on loosely
designed assessment tools with no reliability or validity. Only three out of the
17 studies reviewed offered estimates of reliability (Kay & Knaack, 2007b;
Kong & Kwok, 2005; Liu & Bera, 2005), and only one study provided valid-
ity data (Kay & Knaack, 2007b). As well, few evaluation studies (e.g.,
Docherty et al., 2005; Kenny, Andrews, Vignola, Schilz, & Covert, 1999;
Kay & Knaack, 2007b; Rieber, Tzeng, & Tribble, 2004; Windschitl & Andre,
1998; Van Zele, Vandaele, Botteldooren, & Lenaerts, 2003) use formal sta-
tistics, particularly in the secondary school domain (Kay & Knaack, 2007b).
In order to address key methodological challenges, the following steps
were taken:
1. a large, diverse, sample was used
2. reliable and valid surveys were used where possible
3. formal statistics were used when appropriate
4. multiple assessments of impact were used including student per-
ceptions of learning, quality, and engagement with respect to learn-
ing objects
5. a measure of student performance was included
6. a wide range of learning objects in a variety of subject areas was tested
Sample
Teachers
The teacher sample consisted of 15 teachers (7 males, 8 females) and 30
classrooms (a number of teachers used learning objects more than once).
Teaching experience ranged from 2 to 33 years with a mean of 7.8 (SD =
8.1). Subject areas taught were science (biology, chemistry, general science,
physics) and math. A majority of the teachers rated their ability to use com-
puters as strong or very strong (n=14) and their attitude toward using com-
puters as positive or very positive (n=14). In spite of the high ability and
positive attitude, only three of the teachers used computers in their class-
rooms more than once a month.
Students
The student sample consisted of 510 secondary school students (248
males, 262 females), 10 to 22 years of age (M = 16.5, SD = 1.1). The popu-
lation base spanned three boards of education, 10 secondary schools, and 30
different classrooms. The students were selected through convenience sam-
pling and had to obtain signed parental permission to participate.
298 Kay, Knaack, and Muirhead
Learning Objects
A majority of teachers selected learning objects from a repository located at
the LORDEC website (http://www.education.uoit.ca/lordec/collections.html),
although several reported that they also used Google.A total of 16 unique learn-
ing objects were selected covering concepts in biology, chemistry, general sci-
ence, mathematics, and physics (see Appendix A for a complete list of learning
objects used).
Procedure
Teachers from three boards of education were asked to volunteer to use
learning objects in their classrooms. Each teacher received a half-day of train-
ing in November on how to choose, use, and assess learning objects (see
http://www.education.uoit.ca/lordec/lo_use.html for more details on the train-
ing provided). They were then asked to use at least one learning object in their
classrooms by April of the following year. Email support was available
throughout the duration of the study. All students in a given teacher’s class
used the learning object that the teacher selected; only those students with
signed parental permission forms were permitted to fill in an anonymous,
online survey about their use of the learning object. In addition, students com-
pleted a pre- and post-test based on the content of the learning object.
Data Sources
Independent Variables
Four categories of independent variables were used to assess teacher use
of learning objects: (1) preparation, (2) purpose, (3) integration, and (4) time
spent using the learning object. Preparation referred to the time taken to find
an appropriate learning object and to plan its integration into a lesson plan.
Purpose included assessing whether learning objects were used to (a) intro-
duce a lesson, (b) motivate students, (c) teach a new concept, (d) review a
previous concept, and/or (e) extend a concept. Integration included the fol-
lowing strategies for using the learning objects in a classroom: (a) indepen-
dent use of computers, (b) introducing the learning object, (c) supports pro-
vided for learning object use, and (d) consolidation of a learning object les-
son. Time referred to the number of minutes that the learning object compo-
nent of the lesson took. The coding for each of the independent variables is
provided in Appendix A.
It is important to note that teachers could select multiple purposes and
integration strategies from the online survey they filled in after using the
learning object in their classroom. Sixty-five percent of the teachers (n=19)
chose more than one purpose and integration strategy. An attempt was made
to find purposes and integration strategies that grouped together using corre-
lation and factor analysis; however, only one consistent pattern was observed.
Teachers tended to choose both “introducing a new concept” and “exploring
A Formative Analysis of Instructional Strategies for Using Learning Objects 299
a new concept” together when selecting integration techniques. Therefore, it
was decided to analyse individual purpose and integration items.
Dependent Variables
Four dependent variables were chosen for this study: learning, quality,
engagement, and student performance. Learning referred to a student’s self-
assessment of how much a learning object helped them to learn. Quality was
determined by student perceptions of the quality of the learning object.
Engagement referred to student ratings of how engaging or motivating a
learning object was. Student performance was determined by calculating the
percent difference between pre-test and post-test created by each teacher
based on content of the learning object used in class.
Student self-assessment of learning, quality, and engagement were col-
lected using the Learning Object Evaluation Scale for Students (LOES-S).
These constructs were selected based on a detailed review of the learning
object literature over the past 10 years (Kay & Knaack, 2007b). According
to Kay & Knaack (2007), the LOES-S displayed good reliability, construct
validity, convergent validity, and predictive validity. Scale items are pre-
sented in Appendix B.
Key Questions & Data Analysis
In order to evaluate teacher use of learning objects with secondary school
students, the following questions were addressed in the data analysis:
1. What is the relationship between preparation time for using a learn-
ing object and the four dependent variables (learning, quality,
engagement, student performance)?
2. How is the intended purpose for using a learning object related to
the four dependent variables (learning, quality, engagement, student
performance)?
3. How are strategies used to integrate learning objects related to the
four dependent variables (learning, quality, engagement, student
performance)?
4. Is time spent using a learning object significantly related to the
four dependent variables (learning, quality, engagement, student
performance)?
RESULTS
Preparation for a Learning Object Lesson
Thirty-one percent (n=9) of the teachers reported that finding a suitable
learning object took them less than 30 minutes. Forty-eight percent (n=14)
300 Kay, Knaack, and Muirhead
took 30 to 60 minutes to find an appropriate learning object. The remaining
21% (n=6) took over an hour to find the learning object they wanted to use
in their class.
With respect to preparation for using the learning object in class, seven
percent (n=2) of the teachers spent little or no time, 48% (n=14) spent less
than 30 minutes, 35% (n=10) spent 30 to 60 minutes, and the remaining 13%
(n=3) spent over an hour.
The time spent finding a learning object was not significantly correlated
with student perceptions of learning (r= .03, n.s.), quality (r= .00, n.s.), and
engagement (r= .08, n.s.), nor was it correlated with increased student per-
formance (r= .00, n.s.). However, time spent on integrating a learning object
into a lesson showed small but significant correlations with student percep-
tions of learning (r= 0.12, p < .05), quality (r= 0.13, p < .01), and engage-
ment (r= 0.15, p < .005), but not with student performance (r= -.08, n.s.).
Purpose of Using Learning Object
The most frequent reasons that teachers chose to use learning objects
were to review a previous concept (n=16, 55%), motivate students (n=14,
48%), to provide another way of looking at a concept (n=9, 31%), and to
introduce or explore a new concept before a lesson (n=7, 24%). Teachers
rarely chose to use learning objects to explore a new concept after a lesson
(n=2, 7%), teach a new concept (n=1, 3%), or to extend a concept (n=1, 3%).
Introducing a New Lesson
When a teacher decided to use a learning object to “introduce a new topic,
then teach a formal lesson,” student perceptions of learning (t = -2.17, df
=469, p < .05) and learning object quality (t = -2.87, df =459, p < .005) were
significantly higher than when they did not choose this purpose (see Table 1).
In addition, student performance was significantly higher (p < .001; see Table
2). Assessment of student engagement showed no differences.
If a learning object was used to “explore a new concept before a formal
lesson,” student perceptions of learning (t = -2.25, df =469, p < .05) were
significantly higher (Table 1), as was the increase in student performance (p
< .001; Table 2). Student assessment of learning object quality and engage-
ment showed no differences (see Table 1 below).
Motivating Students
Differences in student perceptions of learning and engagement were not sig-
nificantly different between teachers who chose to use learning objects as a
motivational tool and those who chose not to use it this way (see Table 1). How-
ever, learning object quality was rated higher (t = 2.33, df =459, p < .05; Table
1) and student performance increased significantly (p < .005; Table 2) when
one of the main goals for using a learning object was to motivate students.
A Formative Analysis of Instructional Strategies for Using Learning Objects 301
302 Kay, Knaack, and Muirhead
Table 1
Mean Learning, Quality, and Engagement Scores as a Function of
Purpose Chosen by Teachers
Learning
6
Quality
7
Engagement
8
Purpose Yes No Yes No Yes No
M (SD) M (SD) M (SD) M (SD) M (SD) M (SD)
Introduce first 18.1(3.9) 17.1(4.3)
4
15.9(3.9) 15.0(4.3)
2
10.2(3.9) 10.3(4.3)
5
Explore first 18.0(4.1) 17.1(4.3)
4
15.4(2.7) 15.2(3.4)
5
10.2(2.3) 10.3(2.6)
5
Motivate students 17.4(4.4) 17.3(4.1)
5
14.8(3.5) 15.5(2.9)
4
10.1(2.7) 10.4(2.3)
5
Teach new concept 20.6(3.0) 17.2(4.2)
2
17.5(1.7) 15.2(3.2)
3
11.9(1.9) 10.2(2.5)
4
Another way/method 16.2(4.2) 17.9(4.1)
1
14.3(3.1) 15.7(3.1)
1
9.8(2.7) 10.5(2.3)
3
Review concepts 17.3(4.0) 17.3(4.5)
5
15.4(3.1) 14.9(3.1)
4
10.6(2.3) 9.9(2.6)
1
Explore after 14.3(4.7) 17.6(4.1)
1
12.1(4.2) 15.5(2.9)
1
8.5(2.6) 10.5(2.4)
1
Extend concept 20.0(3.0) 17.2(4.2)
3
17.5(1.6) 15.1(3.2)
2
12.0(2.0) 10.2(2.5)
2
1
p < .001
2
p < .005
3
p < .01
4
p < .05
5
not significant
6
Possible range is 5 to 25
7
Possible range is 4 to 20
8
Possible range is 3 to 15
Table 2
Student Performance as a Function of Purpose Chosen by Teachers
Student Performance
Percent Change
Purpose Yes No % df t
M (SD) M (SD) Diff
Introduce first, then lesson 34.7% (28.0%) 15.6% (25.5%) 19.1% 421 -6.68
1
Explore first, then lesson 30.5% (26.0%) 16.3% (27.1%) 14.2% 421 -5.07
1
Motivate students 26.0% (28.7%) 17.2% (26.0%) 8.8% 421 -3.27
2
Teach new concept 50.8% (24.2%) 19.9% (27.1%) 8.8% 421 -3.27
1
Another way/method 8.3% (20.9%) 26.0% (28.3%) -17.7% 421 6.26
1
Review concepts 13.5% (22.7%) 29.4% (30.1%) -15.9% 421 6.16
1
Explore after lesson 5.5% (29.6%) 22.1% (27.0%) -30.9% 421 -3.90
1
Extend a concept 11.3% (15.0%) 21.2% (27.9%) -9.9% 421 1.49
3
1
p < .001
2
p < .005
3
not significant
Teach a New Concept
When a learning object was used to “teach a new concept,” student per-
ceptions of learning (t = -2.87, df =469, p < .005), learning object quality (t
= -2.73, df =459, p < .01), and engagement (t = -2.59, df =497, p < .05) were
higher, and student performance increased significantly (t = -3.90, df =421,
p < .001; Table 2).
Review a Previous Concept
When a teacher chose to use a learning object to “provide another way of
looking at a concept,” student perceptions of learning (t = 3.89, df =469, p <
.001), learning object quality (t = 4.42, df =459, p < .001), and engagement
(t = 2.82, df =497, p < .01) decreased significantly (Table 1), and student per-
formance was lower (t = 6.26, df =497, p < .001; Table 2).
If a teacher was using a learning object to “review a previous concept,”
student perceptions of learning and learning object quality were unaffected,
engagement (t = -3.3, df =497, p < .001) increased significantly (Table 1),
and student performance was significantly lower (t = 6.12, df =421, p < .001;
Table 2).
Choosing to use a learning object to “explore a concept after a formal les-
son” resulted in lower scores for student perceptions of learning (t = 4.67, df
=469, p < .001), learning object quality (t = 6.60, df =459, p < .001), engage-
ment (t = 4.96, df =497, p < .001) (Table 1), and student performance (t =
3.31, df =421, p < .005; Table 2).
Extending a Concept
When a teacher wanted to use a learning object to “extend a concept,”
student perceptions of learning (p < .01), learning object quality (p < .01),
and engagement (p < .005) decreased significantly (Table 1), while student
performance was unaffected.
Integration of Learning Object
Almost all teachers (n=28, 97%) chose to have students work indepen-
dently on their own computers. With respect to introducing the learning
object, 62% (n=18) provided a brief introduction and seven percent (n=2)
formally demonstrated the learning object. In terms of supports provided,
35% of the teachers (n=10) created a set of guiding questions, while 28%
(n=8) provided a worksheet. Thirty-eight percent (n=11) of the teachers
chose to discuss the learning object after it had been used.
Independent Use of Learning Object
Choosing to have students work independently on computers as opposed
to in pairs or larger groups was not significantly related to student percep-
A Formative Analysis of Instructional Strategies for Using Learning Objects 303
tions of learning, learning object quality, and engagement (see Table 3), nor
was it related to student performance (see Table 4).
304 Kay, Knaack, and Muirhead
Table 3
Mean Learning, Quality, and Engagement Scores as a Function of
Integration Strategys
Learning
5
Quality
6
Engagement
7
Purpose Yes No Yes No Yes No
M (SD) M (SD) M (SD) M (SD) M (SD) M (SD)
Independent Use 17.3(4.3) 17.7(3.1)
4
15.1(3.2) 16.4(1.9)
4
10.3(2.5) 10.0(2.3)
4
Demonstration 17.8(3.6) 17.3(4.3)
4
15.1(2.8) 15.3(3.2)
4
10.2(1.8) 10.3(2.6)
4
Brief Introduction 17.2(4.3) 17.5(4.0)
4
15.0(3.3) 15.5(3.0)
4
10.2(2.5) 10.5(2.5)
4
Let students explore 16.6(4.2) 17.8(4.1)
2
14.8(3.6) 15.5(2.9)
3
10.1(2.6) 10.4(2.4)
4
Worksheet 18.1(3.7) 17.1(4.4)
3
15.8(2.5) 15.0(3.4)
3
10.6(2.5) 10.2(2.5)
4
Guiding Questions 18.2(3.9) 16.9(4.3)
2
15.9(2.6) 14.9(3.4)
2
10.5(2.5) 10.2(2.5)
4
Discuss After 16.7(4.2) 17.8(4.2)
2
14.4(3.3) 15.8(2.9)
1
10.0(2.5) 10.5(2.5)
4
1
p < .001
2
p < .005
3
p < .05
4
not significant
5
Possible range is 5 to 25
6
Possible range is 4 to 20
7
Possible range is 3 to 15
Table 4
Student Performance as a Function of Integration Strategy
Student Performance
Percent Change
Strategy Yes No % df t
M (SD) M (SD) Diff
Independent Use 21.2% (27.8%) 10.5% (15.7%) 10.7% 421 -1.58
2
Demonstration 18.4% (24.4%) 21.1% (27.9%) -2.7% 421 0.63
2
Brief Introduction 22.4% (28.8%) 17.9% (25.0%) 4.4% 421 -1.60
2
Let students explore 13.6% (25.1%) 24.9% (28.0%) -11.2% 421 4.10
1
Worksheet 20.9% (23.7%) 20.7% (29.0%) 0.2% 421 -0.07
2
Guiding Questions 29.1% (26.3%) 15.9% (27.1%) 13.3% 421 -4.92
1
Discuss After 14.0% (26.0%) 25.2% (27.6%) -11.2% 421 4.18
1
1
p < .001
2
not significant
Introduction of Learning Object
Demonstrating a learning object or providing a brief introduction was not
significantly related to the four dependent variables used in this study (learn-
ing, quality, engagement, student performance) (as shown in Tables 3 & 4).
Simply letting students explore on their own was negatively related to stu-
dent perceptions of learning (t = 2.88, df =469, p < .005) and quality (t =
2.29, df =459, p < .05), but student engagement scores were unaffected
(Table 3). Student performance dropped significantly if students were left to
explore on their own (Table 4).
Supports Provided
When worksheets were provided, students rated learning (t = -2.29, df =
469, p < .05) and learning object quality (t = -2.27, df =459, p < .05) high-
er, but not engagement (Table 3). Student performance was unaffected
(Table 4). If a teacher created a set of guiding questions, students rated learn-
ing (t = -3.23, df = 469, p < .005) and learning object quality (t = -3.33 df
=459, p < .005) higher, but not engagement (Table 3). Student performance
increased significantly (p < .001; Table 4).
Consolidation
When teachers chose to discuss the learning object after students worked
with it, students rated learning (t = -2.71, df = 469, p < .005) and learning
object quality (t = -4.65 df =459, p < .001) lower, but not engagement (Table
3). Student performance decreased significantly (p < .001; Table 4).
Time Spent Using the Learning Object
The mean amount of time spent on the learning object component of the
lesson was 34.8 minutes (SD = 19.8), with a range of 6 to 75 minutes. Time
spent using the learning object was negatively correlated with perceived
learning object quality (r= -0.12, p < .01) and student performance (r= -0.12,
p < .05). It should be noted that these correlations are quite small.
DISCUSSION
The purpose of this study was to explore and evaluate strategies for using
learning objects in secondary school classrooms based on (a) preparation for
a learning object lesson, (b) purpose of using learning object, (c) integration
of learning object, and (d) time spent using the learning object. Each of these
areas will be discussed in turn.
Preparation for a Learning Object Lesson
The time spent to find a suitable learning object ranged from less than 30
minutes to over an hour, with almost 80% of teachers taking less than hour.
A Formative Analysis of Instructional Strategies for Using Learning Objects 305
However, time directed toward searching for learning objects was not related
to students’ attitudes or performance. While one might predict that searching
for the best learning object should take more time, several other search scenar-
ios may have occurred. It is possible that teachers who took longer to search for
learning objects were simply unable to find one that fit their needs. They may
have settled for a lower quality learning object simply because they ran out of
time. Conversely, teachers who found learning objects quickly might have been
impressed by a high-quality learning object early on in the search process.
The majority of teachers spent less than 60 minutes preparing to use
learning objects. Unlike search time, preparation time was significantly, but
minimally, related to student perceptions of learning, quality, and engage-
ment. However, preparation was not related to student performance. The
expectation would be that increased time spent preparing for the use of
learning objects would lead to increased success in the classroom; prepara-
tion quality may be more important that absolute preparation time. In other
words, absolute time spent on preparation may not be as important as learn-
ing goals and strategies selected by the teacher.
Purpose of Using Learning Object
Five reasons for using learning objects were evaluated in this study. When a
learning object was used to introduce or explore a concept before a formal les-
son, student perceptions were more positive and performance increased signif-
icantly. This result is partially confounded by the fact that a formal lesson was
used in conjunction with a learning object. It is impossible to determine the rel-
ative contribution of the learning object to final performance. However, the
order in which the learning object is introduced is important. When a learning
object was used to explore a concept after a formal lesson, student perceptions,
and student performance was significantly lower. It is conceivable that when a
learning object is used before a formal lesson, the teacher can build on the expe-
riences of the class and repair any misconceptions, if required. Conversely, if a
learning object is used after a formal lesson, students are left to their own
devices to make connections and resolve any misconceptions.
Using a learning object to motivate students resulted in favorable student
feedback on learning object quality and increased student performance.
However, student perceptions of learning and engagement were unaffected.
One would expect that if a teacher selected a learning object for motivation-
al reasons, students would be more enthusiastic. This was not the case. There
may be disconnect between what teachers and students think is motivating.
Furthermore, teachers were allowed to select more than one reason for using
a learning object. Other reasons or strategies for using learning objects may
have influenced student assessment of engagement value.
While only one teacher chose to use a learning object to teach a new con-
cept without a formal lesson, the impact was positive with respect to student
306 Kay, Knaack, and Muirhead
attitudes and performance. It is risky to extrapolate from a sample size of one
classroom; however, this may be a strategy that works well. It is speculated that
the learning object for this kind of lesson would have to be chosen carefully.
Over 50% of the teachers in this study chose to use learning objects for
review purposes, a choice that resulted in lower student attitudes and per-
formance scores. One explanation for this result is that students already
knew the material, so differences in pre- and post-test scores were minimal.
Furthermore, significantly lower perceptions of learning object quality,
engagement, and learning value may reflect student frustration at spending
time using technology to review information they already know. Regardless
of the possible interpretation of this result, it appears that using learning
objects for review purposes with secondary school high school students may
not be an effective approach.
Finally, using a learning object to extend a concept, an approach that was
used by only one teacher, resulted in negative student attitudes but did not
affect student performance. Because of the limited sample size, this result
should be treated with caution and needs to be examined in more detail.
Integration Strategies
Four areas of integration were evaluated in this study. First, the decision
to have students work independently on computers and not in pairs was
made by 97% of the teachers. While there was no difference between stu-
dent attitude and performance between independent and cooperative use of
computers, this result is compromised by disparate sample sizes. In this
study, having students work at their own computer appeared to have a neu-
tral influence on attitude and learning outcomes.
Second, providing a brief or extended introduction appeared to be neces-
sary but not sufficient for improving student attitudes and performance.
While the type of introduction (brief vs. extended) was unrelated to student
perceptions and learning outcomes, post-test scores were significantly lower
if students were simply allowed to explore on their own. Paradoxically, stu-
dents preferred the “explore on your own” approach. In this situation, stu-
dents’ attitudes were not the best predictor of student performance. Some
type of introduction and guidance is probably a good starting strategy when
using learning objects. This result is consistent with previous research on
providing sufficient context (Schoner et al., 2006).
Third, regarding the provision of instruction supports, the results of this
study are consistent with previous studies in suggesting that worksheets or
guiding questions are essential for the successful use of learning objects
(Brush & Saye, 2001; Concannon et al., 2005; Lim et al., 2006; Mason et al.,
2005; Mayer, 2004). However, the precise nature of supports appears to be
important. When simple worksheets were used, student performance was
unaffected, but when guiding questions were offered, student performance
A Formative Analysis of Instructional Strategies for Using Learning Objects 307
increased significantly. Guiding questions may have offered a clearer path-
way to the intended goals of the instructor.
Finally, and somewhat surprisingly, consolidation or class discussion
after the use of a learning object appears to have a negative effect on student
attitude and learning performance. This finding is opposite to what one
would expect. One explanation might be that class discussion was used
when the use of learning objects did not go smoothly, when there were prob-
lems, and perhaps when students experienced confusion. A more detailed
description of the discussion is required to fully understand this result.
Time Spent Using the Learning Object
While there was considerable variability in the time spent using learning
objects, a significant and negative correlation was observed with respect to
perceived learning object quality and student performance. The longer stu-
dents spent on learning objects, the lower the quality and performance
scores. While the magnitude of this correlation was very small, the results
may suggest that allowing students to use learning objects without time con-
straints may be counter productive.
Implications for Education
It is always wise to be cautious with respect to providing educational advice,
especially when doing a formative analysis of a topic that has not been sys-
tematically evaluated before. However, there are several preliminary sugges-
tions that can be made based on the results of this study. First, while prepara-
tion time for the use of learning objects is related to student attitude and per-
formance, it is the specific choices made during this preparation that determine
successful learning object implementation. Effective choices made by teachers
in this study included using learning objects to motivate students and introduce
or explore a concept before a formal lesson, as well as providing guiding ques-
tions. Less effective choices involved using learning objects to review concepts
and letting students explore on their own without direction. Overall, instructor
decisions have a marked impact on the effectiveness of learning objects in a
secondary school environment.
Caveats and Future Research
In this study, careful attention was directed toward collecting good qual-
ity data by sampling a large, relatively diverse population, establishing the
reliability and validity of measures, and using multiple data sources to estab-
lish triangulation. Nonetheless, several limitations exist which provide
opportunities for future researchers.
First, variability in the kinds of learning objects selected has not been
accounted for in this study. Different learning objects may have an impact
308 Kay, Knaack, and Muirhead
on the strategies selected by teachers. For example, question and answer
learning objects may promote a different learning environment than tool-
based learning objects. In addition, specific characteristics of learning
objects may have an impact on student perceptions and learning perfor-
mance, regardless of the instructional strategies chosen. Engaging learning
objects with high-quality multimedia may be perceived as more useful than
text-based learning objects with limited interactivity. To date, little system-
atic research has been done examining the qualities of learning objects that
promote more effective learning.
Second, student ability was not examined and may have an impact on the
success of any learning tool, let alone a learning object. A number of
researchers have reported that high ability students may use learning objects
differently than low ability students (Akpinar & Bal, 2006; Deaudelin, Dus-
sault, & Brodeur, 2003; Haughey & Muirhead, 2005; van Marrienboer &
Ayres, 2005). It is important, then, to investigate these differences with
respect to instructional wrap.
Third, while providing guiding questions proved to be a successful strat-
egy, the actual quality of questions was not examined. It is possible that cer-
tain kinds of questions are more effective than others in supporting the use
of learning objects (Brush & Saye, 2001).
Fourth, more qualitative research is needed to help interpret some of the
more puzzling findings reported in this study. It is unclear, for example, why
consolidation when using learning objects resulted in lower student perfor-
mance. A qualitative discussion of what is actually said in the class discus-
sion would help make sense of this counterintuitive result.
Finally, the type of knowledge gains associated with instructional strate-
gies need to be looked at in more detail. The results from this study suggest
that certain strategies lead to significant gains in learning performance, but
nothing is said about the qualitative nature of knowledge for these gains. For
example, Reimer & Moyer (2005) observed increases in conceptual knowl-
edge with learning objects, but not in procedural knowledge.
References
Agostinho, S., Bennett, S., Lockyear, L., & Harper, B. (2004). Developing a learning object metadata
application profile based on LOM suitable for the Australian higher education market. Aus-
tralasian Journal of Educational Technology, 20(2), 191-208.
Akpinar, Y., & Bal, V. (2006). Student tools supported by collaboratively authored tasks: The case
of work learning unit. Journal of Interactive Learning Research, 17(2), 101-119.
Ally, M. (2004). Designing effective learning objects. In R. McGreal (Ed), Online education using
learning objects (pp. 87-97). London: Routledge Falmer.
Alonso, F., Lopez, G., Manrique, D., & Vines, J.M. (2005). An instructional model for web-based
e-learning education with a blended learning process approach. British Journal of Educa-
tional Technology, 36(2), 217-235.
A Formative Analysis of Instructional Strategies for Using Learning Objects 309
310 Kay, Knaack, and Muirhead
Bennett, K., & McGee, P. (2005). Transformative power of the learning object debate. Open
Learning, 20(1), 15-30.
Bradley, C., & Boyle, T. (2004). The design, development, and use of multimedia learning objects.
Journal of Educational Multimedia and Hypermedia, 13(4), 371-389.
Bratina, T.A., Hayes, D., & Blumsack, S.L. (2002). Preparing teachers to use learning objects. The
Technology Source Archives. Retrieved January 10, 2007 from http://technologysource.org/
article/preparing_teachers_to_use_learning_objects/
Brush, T., & Saye, J. (2001). The use of embedded scaffolds with hypermedia-supported
student-centered learning. Journal of Educational Multimedia and Hypermedia, 10(4), 333-356.
Carey, T., Swallow, J., & Oldfield, W. (2002). Educational rationale metadata for learning objects.
Canadian Journal of Learning and Technology, 28(3). Retrieved July 1, 2005 from
http://www.cjlt.ca/content/vol28.3/carey_etal.html
Caws, C., Friesen, N., & Beaudoin, M. (2006). A new learning object repository for language
learning: Methods and possible outcomes. Interdisciplinary Journal of Knowledge and Learn-
ing Objects, 2006(2), 112-124.
Cochrane, T. (2005). Interactive QuickTime: Developing and evaluating multimedia learning
objects to enhance both face-to-face and distance e-learning environments. Interdisciplinary
Journal of Knowledge and Learning Objects, 1. Retrieved August 3, 2005 from
http://ijklo.org/Volume1/v1p033-054Cochrane.pdf
COHERE Group (2002). The learning object economy: Implications for developing faculty expertise.
Canadian Journal of Learning and Technology, 28(3). Retrieved July 1, 2005 from
http://www.cjlt.ca/content/vol28.3/cohere.html
Collis, B. & Strijker, A. (2004). Technology and human issues in reusing learning objects. Journal
of Interactive Media in Education, 2004(1), 1-32.
Concannon, F., Flynn, A., & Campbell, M. (2005). What campus-based students think about the
quality and benefits of e-learning. British Journal of Educational Technology, 36(3), 501-512.
Deaudelin, C., Dussault, M., & Brodeur, M. (2003). Human-computer interaction: A review of the
research on its affective and social aspects. Canadian Journal of Learning and Technology,
29(1), 89-110. Retreived June 1, 2000 from http://www.cjlt.ca/content/vol29.1/
05_deaudelin_etal.html
Docherty, C., Hoy, D.,Topp, H., & Trinder, K. (2005). E-Learning techniques supporting problem based
learning in clinical simulation. International Journal of Medical Informatics, 74(7-8), 527-533.
Gadanidis, G., Gadanidis, J., & Schindler, K. (2003). Factors mediating the use of online applets
in the lesson planning of pre-service mathematics teachers. Journal of Computers in Math-
ematics and Science Teaching, 22(4), 323-344.
Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of
Instructional Sciences and Technology, 8(1). Retrieved June 1, 2007 from http://www.usq.edu.au/
electpub/e-jist/docs/vol8_no1/fullpapers/ eval_learnobjects_school.htm
Kay, R. H., & Knaack, L. (2005). Developing learning objects for secondary school students:A multi-
component model. Interdisciplinary Journal of Knowledge and Learning Objects, 1, 229-254.
Kay, R. H., & Knaack, L. (2007a). A formative analysis of individual differences in the effectiveness
of learning objects in secondary school. Computers & Education. Manuscript submitted for
publication.
Kay, R. H., & Knaack, L. (2007b). Assessing learning, quality and engagement in learning objects:
the learning object evaluation scale for students (LOES-S). Educational Technology Research
& Development. Manuscript submitted for publication.
Kay, R. H., & Knaack, L. (2007c). A systematic evaluation of learning objects for secondary school
students. Journal of Educational Technology Systems, 35(4), 411-448.
Kay, R. H., & Knaack, L. (2007d). Evaluating the learning in learning objects. Open Learning,
22(1), 5-28.
Kay, R. H., & Knaack, L. (2007e). Investigating the use of learning objects in secondary school
mathematics. Journal of Computers in Mathematics and Science Teaching. Manuscript sub-
mitted for publication.
Kenny, R. F., Andrews, B. W., Vignola, M. V., Schilz, M. A., & Covert, J. (1999). Towards guidelines
for the design of interactive multimedia instruction: Fostering the reflective decision-making
of preservice teachers. Journal of Technology and Teacher Education, 7(1), 13-31.
Kong, S.C., & Kwok, L.F. (2005). A cognitive tool for teaching the addition/subtraction of common
fractions: A model of affordances. Computers and Education, 45(2), 245-265.
Koppi, T., Bogle, L., & Lavitt, N. (2004). Institutional use of learning objects: lessons learned and
future directions. Journal of Educational Multimedia and Hypermedia, 13(4), 449-463.
Lim, C.P., Lee, S.L., & Richards, C. (2006). Developing interactive learning objects for a computing
mathematics models. International Journal on E-Learning, 5(2), 221-244.
Liu, M. & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning
environment. Educational Technology, Research and Development, 53(1), 5-21.
Mason, R., Pegler, C., & Weller, M. (2005). A learning object success story. Journal of Asynchronous
Learning Networks, 9(1). Retreived June 1, 2007 from http://www.sloan-c.org/publications/
jaln/v9n1/v9n1_mason.asp
Mayer, R. E. (2004). Should there be a three-strikes rule against pure discovery learning? The
case for guided methods of instruction. American Psychologist, 59(1), 14-19.
McCormick, R., & Li, N. (2005). An evaluation of European learning objects in use. Learning,
Media and Technology, 31(3), 213-231.
McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional
Technology and Distance Learning, 1(9). Retrieved August 5, 2005 from http://www.itdl.org/
Journal/Sep_04/article02.htm
Moyer, P. S. (2001). Are we having fun yet? How teachers use manipulatives to teach mathematics.
Educational Studies in Mathematics, 47(2), 175-197.
Nurmi, S., & Jaakkola, T. (2006a). Effectiveness of learning objects in various instructional settings.
Learning, Media, and Technology, 31(3), 233-247.
Nurmi, S., & Jaakkola, T. (2006b). Promises and pitfall of learning objects. Learning, Media, and
Technology, 31(3), 269-285.
Oliver, R., & McLoughlin, C. (1999). Curriculum and learning-resources issues arising from the
use of web-based course support systems. International Journal of Educational Telecommu-
nications, 5(4), 419-435.
Poldoja, H., Leinonen, T., Valjataga, T., Ellonen, A., & Priha, M. (2006). Progressive inquiry learning
object templates (PILOT). International Journal on E-Learning, 5(1), 103-111.
Reimer, K., & Moyer, P.S. (2005). Third-graders learning about fractions using virtual manipulatives:
A classroom study. Journal of Computers in Mathematics and Science Teaching, 24(1), 5-25.
A Formative Analysis of Instructional Strategies for Using Learning Objects 311
Rieber, L. P., Tzeng, S., & Tribble, K. (2004). Discovery learning, representation, and explanation
within a computer-based simulation: Finding the right mix. Learning and Instruction, 14(3),
307-323.
Schoner,V., Buzza, D., Harrigan, K., & Strampel, K. (2005). Learning objects in use:‘Lite’ assessment
for field studies. Journal of Online Learning and Teaching, 1(1), 1-18.
Thorpe, M., Kubiak, C., & Thorpe, K. (2003). Designing for reuse and versioning. In A. Littlejohn
(Ed.), Reusing online resources (pp.116-120). London, UK: Kogan Page.
Van Merrienboer, J.J.G. (2005). Research on cognitive load theory and its design implications for
e-learning. Education Theory, Research and Development, 53(3), 1042-1629.
Van Zele, E., Vandaele, P., Botteldooren, D., & Lenaerts, J. (2003). Implementation and evaluation
of a course concept based on reusable learning objects. Journal of Educational Computing
and Research, 28(4), 355-372.
Weller, M. (2004). Learning objects and the e-leaning cost dilemma. Open Learning, 19(3),293-302.
Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a
metaphor, and a taxonomy. In D. A. Wiley (Ed.), The Instructional Use of Learning Objects:
Online Version. Retrieved July, 1, 2005, from http://reusability.org/read/chapters/wiley.doc
Wiley, D., Waters, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004). Overcoming the
limitations of learning objects. Journal of Educational Multimedia and Hypermedia, 13(4),
507-521.
Windschitl, M., & Andre, T. (1998). Using Computer Simulations to Enhance Conceptual Change:
The Roles of Constructivist Instruction and Student Epistemological Beliefs. Journal of
Research in Science Teaching, 35(2), 145-160.
312 Kay, Knaack, and Muirhead
APPENDIX A
List of Independent Variables
Variable Coding
Preparation
About how long did it take you to find this learning object? 1 – less than 30 minutes
2 – 31 to 60 minutes
3 – 61 to 90 minutes
4 – 91 to 120 minutes
5 – More than 2 hours
Aside from finding the learning object, how much extra time did it take to
integrate the learning objects into your lesson? 1 – less than 30 minutes
2 – 31 to 60 minutes
3 – 61 to 90 minutes
4 – 91 to 120 minutes
5 – More than 2 hours
Purpose - What was the main purpose of using your learning object?
Check all that apply.
Introduce a lesson
Introduce a new topic, then teach a formal lesson 0 = No, 1 = Yes
Explore a new concept before a formal lesson 0 = No, 1 = Yes
Motivate students
To motivate students about a topic 0 = No, 1 = Yes
Teach a New Concept
Teach a new concept on its own 0 = No, 1 = Yes
Review a previous concept
To provide another way of looking at a concept 0 = No, 1 = Yes
Review a previous concept 0 = No, 1 = Yes
Explore a new concept after a formal lesson 0 = No, 1 = Yes
Integration - How did you integrate the learning object into your lesson?
Check all that apply.
Independent Use of Computers
Students used the learning object on their own computer
in class 0 = No, 1 = Yes
Introduce Learning Object
Did a demonstration of the learning object prior to use 0 = No, 1 = Yes
Provided a brief introduction to the learning object, but
did not demonstrate how to use it 0 = No, 1 = Yes
Let the students start exploring the learning object
on heir own 0 = No, 1 = Yes
Supports Provided
Provided a worksheet to support the use of the learning
object while they used it 0 = No, 1 = Yes
Provided a set of guiding questions 0 = No, 1 = Yes
Consolidation
Discussed the learning object after it had been used 0 = No, 1 = Yes
Time - How many minutes did the learning object component of the lesson take?Open ended
A Formative Analysis of Instructional Strategies for Using Learning Objects 313
APPENDIX B
Learning Object Evaluation Survey - Students
Strongly Disagree Neutral Agree Strongly
Disagree Agree
1 2 345
Learning
1. Working with the learning object helped me learn. 1 2 345
2. The feedback from the learning object helped me learn. 1 2 345
3. The graphics and animations from the learning
object helped me learn. 1 2 345
4. The learning object helped teach me a new concept. 1 2 345
5. Overall, the learning object helped me learn. 1 2 345
Quality
6. The help features in the learning object were useful. 1 2 345
7. The instructions in the learning object were easy
to follow. 1 2 345
8. The learning object was easy to use. 1 2 345
9. The learning object was well organized. 1 2 345
Engagement 1 2 345
10. I liked the overall theme of the learning object. 1 2 345
11. I found the learning object motivating. 1 2 345
12. I would like to use the learning object again. 1 2 345
314 Kay, Knaack, and Muirhead
A Formative Analysis of Instructional Strategies for Using Learning Objects 315
Collection Name of Learning Object Web Address Status
NLVM Algebra Balance Scales http://nlvm.usu.edu/en/nav/frames_asid_201_g_4_t_2.html?open=instructions Open
TLF Alpha, Beta, Gamma of Radiation http://www.thelearningfederation.edu.au/tlf2/ Closed
Learn Alberta Ammeters and Voltmeters http://www.learnalberta.ca/ Closed
UOIT Capillary Fluid Exchange http://education.uoit.ca/EN/main/151820/151827/research_teach_locollection.php Open
FunBased Classic Chembalancer http://funbasedlearning.com/chemistry/chemBalancer/ Open
Independent Congruent Triangles http://argyll.epsb.ca/jreed/math9/strand3/3203.htm Open
PHET Energy Skate Park http://phet.colorado.edu/simulations/energyconservation/energyconservation.jnlp Open
DNA Int Gel electrophoresis http://www.dnai.org/b/index.html Open
Shodor Maze Game http://www.shodor.org/interactivate/ Open
Independent Metals in Aqueous Solutions http://www.chem.iastate.edu/group/Greenbowe/sections/projectfolder/animationsindex.htm Open
Learn Alberta Multiplying and Dividing Cells http://www.learnalberta.ca/ Closed
WISC Online Periodic Table http://www.wisc-online.com/objects/index_tj.asp?objid=SCI202 Open
TLF Reading Between the Lines http://education.uoit.ca/lordec/lo/L80/LV5536/ Open
PBS Structure of Metals http://www.pbs.org/wgbh/nova/wtc/metal.html Open
UOIT Transformation of Parabola http://education.uoit.ca/EN/main/151820/151827/research_teach_locollection.php Open
UW Madison Wild Weather http://cimss.ssec.wisc.edu/satmet/modules/wild_weather/index.html Open
APPENDIX C
List of Learning Objects Used in the Study
