Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay
Content may be subject to copyright.
Interdisciplinary Journal of E-Learning a nd Learning Objects Volume 4, 2008
For mer l y the I nter disci plinar y Jo urnal of Knowl edge and L earni ng Obj ects
Editor: Susan Nash
Investigating the Use of Learning Objects for
Secondary School Mathematics
Robin Kay and Liesel Knaack
University of Ontario Institute of Technology, Canada
robin.kay@uoit.ca; liesel.knaack@uoit.ca
Abstract
Research on the impact of learning objects in secondary school mathematics classes is limited.
To date, only two investigations have been reported in this area. The current study presents a
comprehensive analysis of the use and effect of learning objects in secondary school mathematics
classrooms. T eacher and st udent att itudes, as well as student performance, were used t o assess
the impact of learning objects for 298 students and 11 teachers in 20 different classrooms.
T eacher att itudes were posit ive with r espect to qualit y, en gagement, and learning v alue of learn-
in g o bject s. Student att itudes, on t he other hand, varied markedly. Overall, studen t perfor m ance
increased significantly after using learning objects, although gains observed were small and high-
ly variable. Student performance was significantly related to the intended purpose of a lesson and
teaching strategy selected. Planning time, using learning objects for review or to introduce new
concepts, and supplying guided handouts improved student performance. It is reasonable to con-
clude t hat mathematics-based learning objects are viable teaching tools when used with t he ap-
propriat e goals an d st r at e gies.
Keywords: mathematics, use, evaluate, secondary school, learning object
Introduction
Learning objects are operat ionally defined in t his st udy as interactive web-based t ools that sup-
port the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive
processes of learners. While the design, development, reuse, and accessibilit y of learning objects
has been examined in some detail for almost 10 years (Kay & Knaack, 2007a, 2007b), research
on the effectiveness and usefulness of learning objects is limited, particularly in the area of sec-
ondary school mathematics (Kay & Knaack, 2007c; Lopez-Morteo & Lopez, 2007).
It is argued that learning objects help address a number of obstacles secondary school teachers
face with respect to using technology including not having enough time, difficulties in learning
new soft ware, creating effective integration strategies, and accessibility (e.g. Agostinho, Bennett,
Locky er, & Ha rp er, 2004; Duval, Hodgin s, Reh a k, & Ro bso n, 2004; Gada n idis, Ga dan idis, &
Schindler, 2003; Kay& Knaack, 2007c; Rehak & Mason, 2003). In addition, comprehensive,
theoretically-based, reliable, and valid
evaluation tools for assessing learning
objects are scarce (Kay& Knaack, in
press). The purpose of this study was to
examine the impact of learning objects
in secondary school mathematics class-
rooms by usin g a com pre h ensive set of
measures to assess teaching strategies,
t eache r att it udes, st ude nt att it ude s, and
learning performance.
Material published as part of this publication, either on-line or
in print, is copyrighted by the Informing Science Institute.
Permission to make digital or paper copy of part or all of these
works for personal or classroom use is granted without fee
provided that the copies are not made or distributed for profit
or commercial advantag e AND that copies 1) bear this noti ce
in full and 2) give the full citation on the first page. It is per-
mis si ble to abstract t hese works so long as credit i s given . To
co py i n all ot h er cases o r t o repu blis h or t o pos t on a serv er or
to redistribute to lists requires specific permission and payment
of a fee. Contact Publisher@InformingScience.org to request
red is t ribu ti on permis si on .
Inv estigating the Use of Learning Obj ects
270
Literature Review
Overview
Unt il recently, learning objects were solely used in higher educat ion; consequently the majority of
research has taken place in this domain (Haughey & Muirhead, 2005; Kay & Knaack, in press).
Increased use of learning objects in the K-12 domain (e.g., Brush & Saye, 2001; Clarke & Bowe,
2006a, 2006b; Kay & Knaack, 2005; Liu & Bera, 2005; Lopez-Morteo & Lopez, 2007; Nurmi &
Jaakkola, 2006) demands that the focus of investigation shift, at least in part, to the needs of mid-
dle and secon dary school students. The current literature review will focus on three key areas:
the potential of learning objects as an effective educational tool in secondary school classrooms,
the proposed pedagogical benefits of learning objects, and research on the use of learning objects
in K-12 mathematics classrooms to dat e.
Potential for Using Learning Objects in Secondary School
Without doubt, efforts to increase the use of technology in K-12 classrooms have been substantial
in the past 10 years (Compton & Harwood, 2003; McRobbie, Ginns, & Stein, 2000; Plante &
Beattie, 2004; US Department of Education, National Center for Education Statistics, 2002). In
spite of this push, a number of researchers have argued that technology has had a minor and even
negative impact on student learning (e.g., Cuban, 2001; Roberston, 2003; Russell, Bebell, O'D-
wyer, & O' Con no r, 20 03 ; Waxman, Connell, & Gray, 2002). P art of the pr o blem stems from a
considerable list of obstacles that have prevent ed successful implementation.
Educators face a number of challenges when attempting t o use technology in their classrooms
in cludin g t he amo unt of t im e required ( E ifler, Green e, & Carroll, 2001; We pn er, Ziom ek, & T ao,
2003), having to work with limited t echnological skills (Eifler et al., 2001; Strudler, Archambault ,
Bendixen, Anderson, & Weiss, 2003; Thompson, Schmidt, & Davis, 2003), fear of technology
(Bullock, 2004; Doering, Hughes, & Huffman., 2003), difficulty in understanding how to inte-
grate technology into teaching (Cuban, 2001), and insufficient access (Bartlett, 2002; Brush et al.,
2003; Russell et al., 2003).
Learning objects offer several promising solutions to the challenges that everyday teachers face
wit h r e sp ect to using t echnology. Fir st and forem ost , learn ing object s ar e easy to use. T each ers,
even those who have limited computer-based skills, do not need to devote considerable blocks of
time toward understanding how to use t hese straightforward tools (Gadanidis, et al., 2003; Kay &
Knaack, 2007c). Second, good learning objects have well defined objectives and a clear narrow
focus making it easier to develop effective lesson plans and int egration st rategies (Kay & Knaack,
2007c). Third, learning objects are readily accessible over the Internet. Given that over 90% of
all public schools in North America and Europe now have access to the Internet (and therefore
learning objects) with most having high-speed broadband connections (Compton & Harwood,
2003; McRobbie et al., 2000; Plante & Beattie, 2004; US Department of Education, National
Center for Education St atist ics, 2002), teachers need not worry about software accessibility. Fi-
nally , reusability permits learning objects to be useful for a large audience, particularly when the
objects are placed in well organized, searchable databases (e.g., Agostinho et al., 2004; Duval,
Hodgins, Rehak & Mason, 2004).
In summary, the accessibility and straight forward design of reusable, concept-focussed learning
objects help address a number of obstacles teacher face with respect to using technology includ-
ing time, difficult ies in learning new software, int egration strategies, and accessibility.
Kay & Knaack
271
Pedagogical Benefits of Learning Ob jects
A number of researchers have argued that learning objects, if carefully selected, have a consider-
able potential to aid student learning (Akpinar & Bal, 2006; Liu & Bera, 2005; Nurmi & Jaak-
kola, 2006; Reimer & Moyer, 2005). It is hypothesized that effective learning objects (a) require
students to construct and manipulate information (Akpinar & Bal, 2006; Baser, 2005; Nurmi &
Jaakkola, 2006), (b) provide rich feedback and interactive illustrations (Akpinar & Bal, 2006), (c)
help student s un der st and abst r act ideas wit h concrete r epresentat ions (Akpinar & Bal, 200 6 ; Rei-
mer & Moyer, 2005) and (d) support key student weaknesses like limited working memory, diffi-
culty in retrieving long term memory, and ineffective cognitive strategies (Liu & Bera, 2005).
In addition, it is emphasized that instructional strategies supporting the use of learning objects are
crit ical for success, regardless of the quality of the learning object selected (Akpinar & Bal, 2006;
Clarke & Bowe, 2006a; Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005). A judicious mix of
instruction, exploration, pract ice, and reflections is ideal (Nurmi & Jaakkola, 2006). To dat e, the
above list of hypotheses and speculations about the effective use of learning objects remain large-
ly untested, particularly in the secondary school classroom.
Impact of Learning Objects in Mathematics
An extensive review of learning objects articles in the past 10 years revealed 10 studies looking at
the use of mathematics-based learning objects (Bower, 2005; Clarke & Bowe, 2006a, 2006b;
Gadanidis et al., 2003; Kay & Knaack, 2007c; Kong & Kwok, 2005; Lim, Lee, & Richards, 2006;
Nurmi & Jaakkola, 2006; Reimer & Moyer, 2005. It is worth noting that, aside from Gadanidis et
al. (2003), these studies go back no further than 2005, indicating that research in this domain is
are relatively new.
Context
While the 10 st udies reviewed for this paper focussed on mathematics, the context of use varied
widely with respect to grade level, time spent using learning objects, number of objects evaluated,
and implementation strategies. In terms of grade level, four studies looked at elementary school
st uden t s (Bo wer, 20 05 ; Clar ke & Bo we, 2 006a, 20 06 b; Reim e r & Moyer, 200 5) , three st udie s
investigated middle school students (Bower, 2005; Kong & Kwok, 2005; Nurmi & Jaakkola,
2006), two studies examined high school students (Kay & Knaack, 2007c; Lopez-Morteo & Lo-
pez , 200 7) , and t wo studies f o cusse d o n h igh er educ at ion st udent s ( Gada n idis et al., 2003; L im et
al., 2006). Time spent using learning objects, varied from 40 to 60 minutes (Akinpar & Bal,
2006; Kay & Knaack, 2007c; Nurmi & Jaakkola, 2006) to several weeks (Clarke & Bowe, 2006a,
2006b; Liu & Bera, 2005; Reimer & Moyer, 2005). Regarding the number of learning objects
evaluated, half of the papers focussed on a single learning object, and half looked at multiple
learning objects (Clarke & Bowe, 2006a, 2006b; Kay & Knaack, 2007c; Nurmi & Jaakkola,
2006; Reimer & Moyer, 2005). Finally, with implementation strategies, some teachers played a
facilit at ing role, allowing students to explore t he learnin g object on their own (Kay & Knnack,
2007c; Kong & Kwok, 2005; Nurmi & Jaakola, 2006), while other teachers report ed the use of
multiple teaching techniques such as large group discussion, guiding worksheets, collaborative
learning, and writing reflective comments (Clarke & Bowe, 2006a; 2006b; Lim et al., 2006).
Teacher perspective
Only four studies looked at teacher attitudes toward t he use of learning objects in the mathematics
classroom (Clarke & Bowe, 2006a; 2006b; Gadanidis et al., 2003; Kay & Knaack, 2007c). Based
largely on qualitative dat a collect ion methods, teachers valued several characteristics of learning
object s: the immediate feedback provided, the ability t o replay and redo t asks for bot h enjoyment
Inv estigating the Use of Learning Obj ects
272
and mast ery, and the motivational impact. Kay and Knaack (2007c) added quantitative evidence
suggesting that pre-service teachers felt learning objects benefited student learning.
Student perspective
Seven st udies lo oked a t st ude nt att it ude s to war d lea r n ing o bject s. Mathematic s st udent s report ed
liking learning objects because they (a) were fun and enjoyable, (b) were easy to control with re-
spect to the pace of learning, (c) provided timely feedback, (d) were easy t o use, (e) consisted of a
number multimedia tools, and (f) helped with learning (Clarke & Bowe, 2006a, 2006b; Kay &
Knaack, 2007c; Lim et al., 2006; Lopez-Morteo & Lopez, 2007; Nurmi & Jaakkola, 2006; Rei-
mer & Moyer, 2005). Lim et al. (2006) and Nurmi and Jaakkola, (2006) added that the accep-
tance of learning objects was somewhat dependent on the kind of learning object used. Students
in these studies favoured interactive, constructive learning objects over the “electronic” textbook
protot ype. Kay and Kn ack (2007 c) offere d quant itative evidence that student s were mo der at e ly,
but not exceedingly, positive about using mathematics-based learning objects.
Student performance
Fo ur st udies lo ok ed at learning per fo rm ance aft er st udent s used m at h-based lea rn ing obje ct s, but
none were done at the secondary school level (Bower, 2005; Kong & Kwok, 2005; Nurmi & Ja-
akkola, 2006; Reimer & Moyer, 2005). In all four studies, elementary and middle school students
who used learning objects showed significant improvement on various performance measures.
Sever a l co nt ext ual de t ails, th o ugh, are worth not ing.
Nurmi and Jaakkola (2006) noted that learning performance was dependent on the type of learn-
ing object and how it was used. Students working wit h drill and practice learning object s were
more focussed o n c om p et ing wit h th eir peer s th a n on lea rn ing. Student s involved in a mixed
learning object / lab-based lesson, performed significantly better than in other learning scenarios.
In addition, Bower (2005) observed that students’ performance improved when they received
feedback comparing their personal performance to their peers.
Purpose
The purpose of this study was to examine the use of learning objects in the secondary school ma-
thematics classroom from the perspective of both teachers and students.
Method
Overview
This paper reviewed 10 articles looking at the use of learning objects in mathematics classrooms,
although just two papers focussed on secondary schools. While half of the papers used mixed
data collection methods, a number of issues need to be addressed to improve the investigation of
learning objects.
First , eight o ut of t en st udie s used a qualit at iv e approach to data collect ion th at relie d heavily on
descript ive data and anecdotal reports to assess the merit s of learning object s. T he reliability and
validity of these informal qualitative observations are questionable, as is relying on only one data
source.
Second, limited information is provided on teacher attitudes toward learning objects. Only four
studies examined teacher perspectives’ on the actual use of learning objects (Clarke & Bowe,
2006a, 2006b; Kay & Knaack, 2007c), however three of those studies reported “general impres-
sions” of relatively small samples.
Kay & Knaack
273
Thir d, sample pop ulat ions tested in K-12 studies are small and wea k ly described mak in g it chal-
lenging to extend any conclusions to a larger population. Only two studies (Bower, 2005; Kay &
Knaack, 2007c) examined more than 100 students and average sample size was 70 st udents.
Fourth, triangulat ion of data collection is somewhat limited with only one st udy using more t han
t wo dat a collect ion p ro cedure s ( Reimer & Moy er, 2005). In addit ion , no resear chers look ed at
st uden t at t itude, teacher att itude, and st udent p erforman c e sim ult aneo usly.
Finally , while mo st evaluat ion st udies r eport e d that student s benefited fr om usin g learn ing ob-
ject s, the evidence is based on assessment tools with no validit y or reliability. Only two studies
offered reliability est imates (Kay & Knaack, 2007c; Kong & Kwok, 2005) and one study, validity
data (Kay & Knaack, in press, 2007c). T he lack of reliability and validity of evaluation tools
compromises the accuracy of results presented to dat e.
In summary, previous methods used to evaluate learning objects are limited with respect to col-
lecting multiple data sources, evaluating t eacher perspectives, acquiring an adequate sample size,
and providing est imat es of reliability, and validity.
In order to address the key methodological challenges, the following steps were taken:
1. a large, diverse, sample was used;
2. reliability and valid surveys were used ;
3. formal st atistics were used where applicable;
4. bo t h qualit ativ e and quan t it at ive data were co llected;
5. bot h teacher and student perspectives were assessed;
6. a m easure of st uden t perfor m an ce was included; and
7. a number of learning objects were tested.
Sample
Teachers
The teacher sample consisted of 11 teachers (5 males, 6 females) and 20 classrooms (a number of
teachers used learning objects more than once). Teaching experience ranged from 2 to 27 years
with a mean of 9.3 (SD = 9.8). A majority of the teachers rated their abilit y to use comput ers as
strong or very strong (n=9) and their at tit ude t oward using computers as positive or very positive
(n=10). In spite of the high comp uter ability and positive att itudes, only t wo of the t eachers used
computers in their classrooms more than once a month.
Students
The student sample consisted of 298 secondary school students (161 males, 137 females), 11 to
22 years of age (M = 16.3, SD = 1. 2). Th e pop ulat ion base sp ann ed three separate bo ar ds of edu-
cat ion , 9 se condary schools, an d 20 different classrooms. The st udent s wer e select ed th ro ugh
co nv en ience samplin g and had t o obt a in signed parental p erm ission t o participate.
Learning objects
Teachers were permitt ed to select any learning object they deemed appropriate for their curricu-
lum. As a starting point, they were introduced to a wide range of learning objects located at the
LORDEC website (http ://www.education.uoit.ca/lordec/collections.ht ml). Sev enty per cent of the
teachers select ed learning objects from the LORDEC repository - the remaining teachers reported
that they used Google. A t otal of 13 unique learning objects were selected (see
http://facult y.uoit.ca/kay/papers/Math_LO.html for links to of all the learning objects used). Key
topics covered included algebra, coordinate planes, congruent triangles, equation of a line, integer
Inv estigating the Use of Learning Obj ects
274
addition and multiplication, slope of a line, proportional relationships, parabolic transformations,
in v e stiga t ing triangle cent res, exponent ial funct ions, and th e bino mial distribut ion .
Procedure
Teachers from three boards of education were emailed by an educational coordinator and in-
formed o f the lear n ing o bject st udy . Part icipat ion was volunt ary and a subject could wit hdraw
from the study at any time. Each teacher received a half day of training in November on how to
choose, use, and assess learning objects (see http://www.educat ion.uoit.ca/lordec/lo_use.html for
more details on the training provided). They were then asked to use at least one learning object in
their classrooms by April of t he following year. Email support was available throughout the du-
ration of t he st udy. All students in a given teacher’s class used the learning object that the t eacher
selected. However, only those students with signed parental permission forms were permitted to
fill in an anonymous, online survey about their use of t he learning object . In addit ion, students
co mp let ed a pre- and po st-t est based o n th e cont ent of t h e learnin g object.
All teachers in this study used learning object s in a lab setting. In order t o simulate real secon-
dary school environment as much as possible, teachers were given full control over the learning
object they selected, the purpose for using the learning object, how long the learning object was
used, teaching strategies for using the learning objects, and the design of t he pre- and post-tests.
Data Sources
Teacher use
Teachers were asked (a) how much preparation was involved when using learning objects (e.g.,
how long it took them to find and integrate learning objects into their classroom), (b) their pur-
pose for using the learning object (e.g., motivate students, teach a new concept, review, supple-
menting a lesson), (c) strategies they used to integrate learning objects (e.g., demonstration, pro-
viding a set of guiding questions, let student explore, discussion after learning object) and (d)
how long the learning object was used in their classroom.
Teacher survey
After using a learning object, each teacher completed t he Learning Object Evaluation Scale for
Teachers (LOES-T ) to det ermine their perception of (a) how much their students learned (learn-
ing construct), (b) the quality of the learning object (qualit y construct), and (c) how much t heir
students were engaged with the learning object (engagement construct). Data from the LOES-T
showed low to moderate reliability (0.63 for learning construct , 0.69 for learn ing object quality
construct, 0.84 for engagement construct), and good construct validity using a principal compo-
nents factor analysis. See Kay and Knaack (2007d) for a det ailed of the t eacher-based learning
object scale.
Teacher comments
Finally, teachers were asked to comment on t he overall impact that the learning object had on
learning (see Q9, Appendix A).
Student survey
After using a learning object, students completed the Learning Object Evaluation Scale for Stu-
dent s (LOES-S) in Appendix B to determine their perception of (a) how much they learned
(learning construct), the quality of the learning object (quality const ruct), how much they were
engaged with the learning object (engagement const ruct). The const ructs selected were based on
Kay & Knaack
275
a thorough review of the literature (Kay & Knaack, 2007a, 2007b). The scale showed good reli-
ability (0.78 to 0.89), face validit y, construct validity, convergent validity, and predict ive validity
(see Kay & Knaack, in press).
Student comments
Students were asked to comment on what they liked and disliked about the learning object (Ap-
pendix B – questions 13 and 14). These qualitative items were analysed according to the coding
schem e pro v ided in Table 1. T h is codin g sch eme (Kay & Knaack, 2007) was use d t o catego r ize
447 st udent comment s. Each comment was th en rat ed on a five-point Lik ert scale (-2 = v ery neg-
Ta bl e 1 . Co di n
g
Scheme to Cate
g
orize Student Comments about Learnin
g
Ob
j
ects
Learning C ate gories Criteria
Learn Student comments about a specific or general learning/t eaching
issue involved in using the LO
Visual The student mention as visual feat ure of the LO that
helped/inhibited their learning
En gagement Categorie s
Challenge Refers to the ease/difficulty of the concepts being covered. Basi-
cally wheth er t h e cont e nt level of the L O matched the st ude nt ’s
cognitive level/understanding.
Code “ it was easy” in h ere, but not “ it was easy to use”
Compare Student compares LO to another method of learning
Engage Studen t refer s to program as being OR not being
fun/en joyable/engaging/interesting
Interactive Student refers to some int eractive part feature of the LO
Technology The student mention a t echnological issue wit h respect to using the
LO
Quali ty C ate gories
Animate Refers to quality of animations /moving pictures
Audio Refers to some audio/sound aspect of the learning object
Easy Refers to clarity of instructions or how easy/hard the LO was to
use. It does not refer to how easy/hard the concept was to learn.
Graph ics Refe r s to static picture or look of the pro gr am (e.g. , colour s)
Help Refers specifically to help/h ints/instruct ions/feedback provided by
the LO
Control Refers to student control of choice/pace in using the LO
Organization/Design Refers to quality of organization/design or the LO
T ext Refers to quality/amount of text in LO
Theme Refers to ov erall/general theme or CONT ENT of LO
Inv estigating the Use of Learning Obj ects
276
ative, -1 = negative, 0 = neutral, 1 = positive, 2 = very positive). Two raters assessed all com-
ment s made by st udent s based on category an d rat ing v alue . Comm ent s whe re cat ego r ies or rat-
ings were not exact ly the same were shared and reviewed a second time by each rater. Using this
app roach, an inter-rater reliability of 99% was attained for categories an d 100% for the rating val-
ues.
Note that t he total impact of any one category was determined by multiplying the mean rating by
the total number of students who made a comment. For example, from Table 4, the impact of
visual supports on learning was calculated by multiplying the mean which was 1.00 by the num-
ber of student who commented about visual supports (24) for a total of 24.0.
Student performance
Students completed a pre-test and post-test created by each teacher based on the content of the
learning object used in class. Questions for pre- and post -test were identical in form, but differed
in the raw numbers used. T he type of questions asked varied according to t he goal of the specific
learning objects. Some tests focussed primarily on factual knowledge while others assess higher
order thinking f ocussing o n “what-if ” scen ario s. T h e mea sur e was used t o det e rmine st udent pe r -
formance. Because of the wide range of learning objects used, it was not possible to assess the
validit y of this test dat a.
Ke y Questions & Data Analysi s
In order to examine the impact of learning objects on secondary school students, the following
quest ions wer e addressed in the data analy sis:
1. How do teachers use learning objects in their classrooms? (descriptive analysis of teacher
use que st ion s) ;
2. H o w do t eacher s rate learn ing, qualit y , and en gagement of learning o bject s? (descript ive
analysis of teacher survey – LOES-T);
3. What was the overall impact of learning objects according to teachers? (analysis of quali-
tative teacher comments);
4. H o w do st uden t s rat e learning, qua lit y, and en ga gement of learning o bject s? (descript ive
analysis of student survey – LOES-S);
5. W hat do st udent s like and dislike most a bo ut learnin g o bjects? (qualit at ive an aly sis of
st uden t c omment s);
6. How do teacher ratings of learning objects compare with student ratings? (correlation
am on g lear n ing, qualit y , and engagement co n struct s) and
7. How do learning objects affect student performance? (t-test comparing pre and post
scores); and
8. How does teacher use of learning objects affect performance? (correlations among prepa-
ration, purpose, teaching strat egies, t ime used, and st udent performance scores)
Kay & Knaack
277
Results
Use of Learning Objects
Finding a learning object
Fifty percent (n=10) of the teachers reported that finding a suitable learning object took them less
than 30 minutes. Thirty percent (n=6) took 30 to 60 minutes to find an appropriate learning ob-
ject. The remaining 20% (n=4) took over an hour t o finding the learning object they want ed to
use in their class.
Preparing a learning object lesson
With respect to preparation for using the learning object in class, 5% of the teachers (n=1) spent
little or no t ime, 4 0% (n=8) spent less than 3 0 minutes, 25% (n=5) spent 30 to 60 minutes, and
the remaining 30% (n=6) spent over an hour.
Using a learning object
On average, t eachers used learning objects for 37.1 minutes (S.D. 17.0); however there was con-
siderable variability (15 to 70 minutes). Students worked on their own on comput ers in a major-
ity of classrooms (85%, n=17), with cooperative learning chosen only 15% of the time.
Reason for using a learning object
The top four reasons cited by teachers for using learning objects were to provide another way of
looking at a concept (75%, n=15), review a previous concept (45%, n=9), explore a new concept
before a lesson (40%, n=8), and motivate students about a topic (35%, n=7). Teachers rarely or
never used learning objects to teach a new concept, explore a new concept after a formal lesson,
or fo r hom e wo r k (4 %, n =2) .
Strategies for using learning objects
T eachers in this study typically provided a brief introduct ion to a learning object (55%, n=11) or
let the students start exploring on their own (45%, n=9). Only 20% (n=4) offered a formal dem-
onstration of the learning object before the class used it. Forty-five percent of teachers (n=9) pre-
pared a formal handout or guiding questions (30%, n=6). Forty percent of teachers (n=8) chose
to have a class discussion about the learning object after it was used by students.
Teacher Rating of Learning Objects
Learning
The mean rat ing for impact learning was 10.8 (SD = 1 .3) o r 5.4 on a 7-point scale. This sugge sts
that most teachers agreed somewhat that learning objects had a positive impact on student learn-
ing. Note that the range of learning construct scores was relatively narrow (8 to 13) (Table 2).
Quality of Learning Object
The mean rating of learning object quality was 16.0 (SD = 2. 8 ) or 5.3 on a 7-point scale. Mo st
teachers somewhat agreed or agreed that learning objects were of good quality. The range of
learning object quality scores was broader than that of the learning construct (11 to 20), but never
dipped into negative rating (Table 2).
Inv estigating the Use of Learning Obj ects
278
Engagement
T eachers rat ed en gagement of learnin g objec t s the highest o f all constructs wit h a mean score of
16.9 (SD = 2. 8) or 5.6 on a 7-po int scale. A majority of t eacher s, then, f e lt student s were en gaged
while using learning objects. The range of learning object engagement scores was relatively large
compared to the learning and quality constructs (9 to 21) (Table 2).
Tabl e 2. Tea cher Rating of Learning, Quality, and Engagement for Learning Objects
Scale No.
Items
Po ssi ble
Range
Actual Range
Observed
Me an (S .D)
Learn 2 2 to 14 8 to 13 10.8 (1.3)
Quality 3 3 to 21 11 to 20 16.0 (2.8)
Engagement 3 2 to 21 9 to 21 16.9 (2.9)
Teacher comments about learning objects
Four themes emerged from t he 24 comments that teachers made about the overall impact of t he
learning objects: overall learning, visual supports for learning, engagement, and time.
With respect to overall learning, three teachers reported that the learning object was not as suc-
cessful h as they h ad ho pe d:
“ A lth o ugh the learn in g o bject did n ot im pr ov e as many studen t s' init ial un derstan din g of
equat io n solv ing as I had hoped, it was useful. ”
“ [Aft e r the usin g the learn ing obje ct ] they st ill had great difficulty dist in guish ing bet wee n
vert ic al and ho r izont al stretches and co mp r essions.
However, six teachers commented that visual supports in the learning objects helped their stu-
dent s learn bett er:
“I felt that this particular learning object helped them gain a solid visual of what happens
to a parabola as you change the numbers of the vertex form of a parabola.”
“ For m any studen t s, usin g the balan ce scale was an ex cellent visual representat ion of
solvin g algebr a proble m s. ”
Regar ding engagem ent, f ive teach ers not ed that th eir st ude nt s seemed to enjoy usin g the learn ing
objects:
“ St ude nt s enjoy e d using the learn ing obje ct an d sp en ding t im e o n the co mp ut er in math
class.”
“I think students seemed to enjoy using it, especially the quiz.”
Finally, four teachers mentioned that time was an issue, either in creating a good lesson plan with
a learning object, saving time, booking the right time to use learning objects, or not having
en ough tim e.
“The learning object allowed me to take a much shorter time to teach t o t each the concept
after the students had been introduced to it in t he learning object.
“I had only had one class prior to using the learning object to introduce the concept to the
class because of problems at the school in terms of booking computer time.”
Kay & Knaack
279
Student Rating of Learning Objects
Learning
Students rated learning object s lower than teachers with respect to learning (M=1 5.65 , SD = 4.6)
with a mean item rating of 3.1 out of 5 (or 4.4 out of 7). Students were relatively neutral with
respect to how much t hey felt the learning objects contributed to their learning. The range of
scores was extensive (5 t o 25) indicating that there was considerable variability with respect to
whether students felt learning objects helped them learn (Table 3).
Quality of learning objects
Students rated t he quality of learning objects higher t han their learning value, although the mean
item rating was still lower than t hat of the t eachers. The mean item rating was 3.4 out of 5 (4.8 or
out 7) indicated that most students somewhat agreed that the learning object s they used were of
good quality. The range of learning object quality scores (4 t o 20) was highly variable (Table 3).
Engagement
Ratings of learning object engagement were moderate (M=9.34, S.D. = 2.8) with a mean item rat -
ing of 3.1 out or 5 (or 4.4 out of 7). In other words, as was the case with the learning construct,
students were neutral about engagement value of the learning objects they used. High variability
among student engagement ratings is supported by the wide range of scores reported (3 to 15).
Table 3. Description of Student Learning Object Evaluation Scales (LOES-S)
Scale No.
Items
Possible Range Actual Range
Observed
Mean (S.D)
Learn 5 5 to 25 5 to 25 15.4 (4.3)
Quality 4 4 to 20 4 to 20 13.5 (3.4)
Engagement 3 3 to 15 3 to 15 9.1 (2.6)
Student comments about learning objects
Student comments are summarized in Table 4. With respect to learning, the visual support that a
learning object offered toward learning was rated the highest, whereas overall learning and the
peda go gical ch allen ge of the learning object wer e rat ed quit e low. I n ot h er words, many st udent s
liked the visual affordances of learning objects, however quite a few felt the learning object did
not support learning or was not challenging enough.
With respect to rating the quality of learning object s, ease of use was the highest rated feature.
On the other hand, th e quality of help and having ex cessive amounts of text were rated t he lowest.
Finally , regardin g engagem e nt, int eract ivity, and comp ariso n with other m ethods of teachin g
were rat ed the highest . A number of t h e student s liked t he int eractive qualities of learning objects
and felt they were an improvement over other teaching strategies.
Inv estigating the Use of Learning Obj ects
280
Table 4. Summary of Student Comments about Learning Objects
Category Me an S.D n To t a l Effe c t
Me an * n
Learning
Visual Supports 1.00 0.00 24 24.0
Challenge -0.46 1.02 52 -24.0
Overall Learning -0.37 1.11 75 -28.0
Quality
Easy 0.88 0.80 24 21.0
Control 0.71 0.76 7 5.0
Animation 1.00 0.00 2 2.0
Graphics -0.14 1.25 29 -4.0
Theme -0.43 1.22 14 -6.0
Organization -0.81 0.98 16 -13.0
Help -0.86 0.75 28 -24.0
Text -1.19 0.40 21 -25.0
Engage m ent
Interactivity 0.90 0.44 21 19.0
Compare with other method 0.67 0.76 24 16.0
Liking Technology 0.50 0.91 12 6.0
Engage m ent / Mot ivation -0.04 1 .18 50 -2.0
Student Performance
Overall, student performance scores increased by an average of 7.6% from 56.4% to 64.0%. This
change was significant (t = -3.85, df = 123, p < .001). T he effect size (based on Cohen’s d) of
0.29 is considered a small effect according to Thalheimer & Cook (2002). Note that the average
student performance score by class varied considerably ranging from a 4.2% decrease to a 27.1%
increase.
Teacher Use and Student Performance
Four areas of teacher use were examined wit h respect to student performance: preparation, pur-
pose, strategies for teaching, and the amount of time a learning object was used. With respect to
preparation, time spent finding a learning object was inversely correlated with student perform-
ance (r = -0.34, p < .001). In other words, the more time a teacher spent looking for a learning
object, the less successful the learning object was in terms of student performance. Planning time
was strongly correlated with student performance (r = 0.69 , p < .001).
Regar ding t h e main purpose or o bject ive for usin g a learn ing object, intro ducin g a new co n cept ,
then following up with a formal lesson (r = 0.68, p < .001) an d r evie win g p r ev iously learn e d con-
cepts (r = 0.36, p < .001 ) wer e signif icantly correlat ed wit h st udent perfo rma nce. Howev er, us-
Kay & Knaack
281
ing a learning object to motivate students and give them another way of looking at a concept was
nega t ively co rrelated wit h student performan ce (r = -0.31, p < .001).
In terms of teaching strategies used with learning object s, posit ive correlations with student per-
formance were seen when a worksheet and guiding questions were provided (r = 0.57, p < .001)
and when students were encouraged to explore on their own (r = 0.29, p < .005). On the other
hand, negative correlations were observed when a brief demonstration of the learning object was
given (r = -0.41, p < .001) or when the learning object was discussed after the student had used it
(r = -0.43, p < .001).
Finally, the amount of time that students used a learning object in class was negatively correlated
with student performance (r = -0.28, p < .005).
Discussion
The purpose of this study was to investigate the impact of learning objects in secondary school
mathematics classrooms. Evidence gathered to evaluate effectiveness included a descript ion of
t eache r use, t each er att itudes, t each er com m ent s, studen t att itudes, st udent comm ent s, and st udent
performance. Each of these sources of evidence will be disc ussed.
Teacher Use
Preparation
Previous research has not been done looking at how teachers find learning objects and prepare to
use them in the science classroom. This st udy provides new information in this area. Searching
for and planning to use a learning object does not take an inordinate amount of time – on average,
a half hour for each activity. It is interesting to note that planning time was strongly correlated
with increases in student performance. While a few teachers commented t hat time was a concern,
it is clear that this time investment is worthwhile.
P aradoxically, tim e spent sear chin g for learn ing obje ct s was inversely relat ed to st udent perform-
ance. It is possible that teachers who took longer to search for learning objects were simply un-
able to find one that fit their needs. They may have settled for a lower quality learning object
simply because they ran out of time. Conversely, teachers who found learning objects quickly
may have been impressed by a high quality learning object early on in t he search process.
Purpose for using learning object
Three quarters of the teachers used learning objects to provide another way of looking at a con-
cept; however this approach resulted in significantly lower student performance. It is unclear
why students faltered with this approach. Perhaps the “other” way of learning a concept was
viewed as unnecessary and maybe confusing. Almost half the teachers used learning objects for
rev iew, a goal that cor r elated sign ificant ly wit h ga ins in st uden t perfor m an ce. St uden t s appear to
have reacted more positively when a deliberat e review was planned, as opposed to a less focussed
lesson that simply looked at a current concept in an alternative way. Students may also have been
more motivated and focussed during a review because a formal test was imminent.
Four out of ten t eachers chose to use learning object s to int roduce a new concept before a formal
lesson, an objective t hat was strongly correlated with improved student performance. T his result
is partially confounded by the fact that a formal lesson was used in conjunction wit h a learning
object. It is impossible to determine the relative contribution of the learning object to final per-
formance. That said, using this tool to lead into more formal instruction works very well.
Inv estigating the Use of Learning Obj ects
282
Teaching strategies
The num be r on e strategy for usin g lear n ing obje ct s, na m ely pr ov idin g a br ief int r o duc t ion, was
nega t ively co rrelated wit h student performan ce. At fir st glan ce, this fin ding seems count er intui-
tive. It would be expected t hat st udent s, even in the worst case scenario, would react in a neutral
manner to a brief introduction, simply because it is “brief”. Then again, learning objects that tru-
ly require and introduct ion may be of lower quality and harder to use.
An ot her str at egy that was used r e lat ive ly o ft en, but th at result ed in lower st udent performa n ce,
was the pract ice of having a class discussion after students used the learning object. Again, this
finding is opposite to what one would expect. One explanation might be that class discussion was
used whe n the use of learn ing obje ct s did n ot go smo othly , when t here were pro blem s, an d per-
haps when confusion was experienced by students. A more detailed description of the discussion
is required to fully understand this result.
A strategy t hat was significantly correlat ed with learning was the inclusion of a worksheet with
guiding questions. This kind of support may have focussed students to concentrate on the key
concepts at hand, thereby resulting in increased understanding of tested items.
Finally, st udents clearly prefer to explore learning objects on their own, a strategy that resulted in
increased student performance. Learning objects t hat permit st udent s to simply “jump in” may be
better designed in terms of usabilit y, part icularly if t hey are supplemented with guiding work-
sheet s.
Teacher Ratings (Learning, Quality, and Engagement)
There is no question t hat the vast majority of teachers believed that the learning object s they se-
lect ed were good quality, engaging tools that supported learning. Ratings of these construct s
were fairly high. On the one hand, it may be obvious that teachers would give high ratings, since
they selected the learning objects in t he first place. On the other hand, teachers rated these learn-
in g o bject s aft er th ey wat ched th em bein g use d by st udent s in their classr oo m . In addit ion , they
had no personal stake in approving these tools, and three teachers felt open enough to acknowl-
edge that learning objects had not fully met their expectations. Positive react ion from teachers in
t h is study is consist ent wit h result s report e d in p revio us research (Clark e & Bowe, 2 00 6 a ; 2006b;
Gadanidis et al., 2003; Kay & Knaack, 2007d).
Teacher Comments
With respect to learning, teacher comments supported both survey and learning performance re-
sults, however a few teachers were disappointed in the final impact. A number of teachers felt
that the learning objects offered helpful visual supports and that the tool was engaging for stu-
dents. In addit ion, learning objects were t hought to be engaging. T hese comments are consistent
with observations made by previous researchers (Clarke & Bowe, 2006a; 2006b; Gadanidis et al.,
2003; Kay & Knaack, 2007d).
Student Ratings (Learning, Quality, and Engagement)
On a verage , studen t s were neutra l wit h r esp ect to th e learning, qua lit y, an d en gagement imp a ct of
learning objects. These results are somewhat consistent with t he modest enthusiasm expressed in
the one other study that formally evaluated the use of math-based learning object s by secondary
school students (Kay & Knaack, 2007d). It is critical to note, though, that the range of scores was
very broad for all three constructs. For any given learning object, even when it was used by the
same class, some students liked it a lot and others disliked it intensely. Learning objects clearly
do not suit every student’s learning style. The challenge remaining is to identify the source and
per ha p s cause of in div idual dif f erences.
Kay & Knaack
283
Finally, teachers rated learning, quality, and engagement much higher t han st udents. What ma-
thematics teachers think is an effective, high quality, motivating t eaching tool, may be perceived
as neutral or barely acceptable by students. Teachers might be wise to test learning objects with a
few representative students, before exposure to the whole class.
Student Comments about Learning Objects
Student comments offer insight into what st udent s like and do not like about learning object s.
Not surprisingly, students liked easy to use, interactive learning objects that provided good visual
supports. They reacted negatively, though, to poor quality help and having to read excessive
am o unt s of text . Th ese react ion s are sim ilar to th o se made by st udent s in prev ious st udies
(Clarke & Bowe, 2006a; Clarke & Bowe, 2006b; Kay & Knaack, 2007d). Finally, it is interesting
to note that even though students were relatively neut ral about the learning objects they used, a
noticeable number of student s report ed that using learning object s was an improvement over oth-
er teaching methods.
Student Performance
This is a first study on the impact of learning objects in secondary school mathematics classes
with respect to learning performance. The fact that learning performance increased is not surpris-
in g given t he wea lt h of pr evious research t h at has repo rt ed sim ilar result s at other gr a de lev els
(Akpinar & Bal, 2006; Baser, 2005; Liu & Bera, 2005; Nurmi & Jaakkola, 2006; Rieber, Tzeng,
& Tribble, 2004; Schoner, Buzza, Harrigan, & Strampel, 2005; Windschitl & Andre, 1998).
However, the magnitude of the overall effect (7.6 %) was fairly small. It is import ant, t hough, to
be cognisant of t wo other results. First, th e variability in student performance among classrooms
was considerable – from a drop of 4% to and increase of 27%. So the more critical question is,
“Why are learning objects successful in some classes, but ineffective in others?” The results of
t h is study suggest t hat t h e answer , at least in part, is increa se d plann in g tim e, usin g lear n ing o b-
jects for review or to introduce a new concept, and supplying guided handouts to focus students
Implications for Education
This st udy of f ers sev eral practical obse rv at ions an d sugge st io ns for usin g lear n in g object s in sec -
ondary school mathematics classrooms. First, it is important to take sufficient time to prepare for
lessons involving learning objects. Second, lessons should have a clear focus like introducing a
new concept or reviewing previous concepts. Simply using a learning object as another way of
looking at a concept may confuse students and have a detrimental effect. Third, a supporting
han dout wit h guided que st ions sh o uld improve st udent perfor m an ce. Fourt h, the effect of learn -
ing objects may vary greatly within the same classroom. Accommodations could have to be
made f or studen t s wit h diffe r ent ability and int erest le v els. Finally, it might be wise t o p re-t est
learning objects on a few students. St udents and teachers differ considerably on what they see as
high quality, engaging learning objects that promote learning. On average, students appear to
have a more accurat e sense of these qualities, at least when it comes t o predicting learning per-
formance.
Caveats and Future Research
In th is st udy , careful att ent ion was direct ed t o ward co lle ct ing goo d qualit y data by sampling a
large, relatively diverse populat ion, establishing t he reliability and validity of measures used, an d
using multiple data sources to establish t riangulation. Nonetheless, several limitations exist
which provide opportunities for future researchers. First, the overall results saw inconsistencies
between teacher attitudes, student attit udes, and student performance. More extensive qualitative
Inv estigating the Use of Learning Obj ects
284
dat a in the for m of fo cus gr o up s and/or interviews migh t shed furth er light one why these dis-
crepancies exist and what role learning objects have to play.
Second, more qualitative data is needed to understand how teacher goals and strategies influence
student performance. A number of strong, significant correlations were observed, but it was chal-
lenging to interpret what teachers were doing. For, example brief introduct ions to learning ob-
ject s were negat ively correlated wit h st udent perfo r mance. It is difficult to comp r ehend t his un-
expected finding without knowing the intended message of these introductory segments.
Finally, the impact of specific kinds of learning objects was not looked at. It is possible that cer-
tain categories of learning objects may have decidedly different impacts on learning. The wide
variability in student performance among classes may be p artially explained by teachers’ goals
and strategies, but the type of learning object used may also have an impact. Developing and
evaluating a classification system for learning objects is an important next step in learning object
research.
References
Agostinho, S., Bennett, S., Lockyear, L., & Harper, B. (2004). Developing a learning object metadata ap-
plication profile based on LOM suitable for the Australian higher edu cation market. Australasian
Journal of Educational Technology, 20(2), 191-208.
Akpinar, Y. & Bal, V. (2006). Student tools supported by collaboratively authored tasks: The case of work
learning unit. Journal of Interactive Learning Research, 17(2), 101-119.
Bartlett, A. (2002). Preparing preservice teachers to implement performance assessment and technology
through electronic portfolios. Action in Teacher Education, 24(1), 90-97.
Baser, M. (2005). Promoting conceptual change through active learning using open source software for
physics simulations. Australasian Journal of Educational Technology, 22(3), 336-354.
Bower, M. (2005). Online assessment feedback: Competitive, individualistic, or…preferred form! Journal
of Computers in Mathematics and Science Teaching, 24(2), 121-147.
Brush, T., Glazewski, K., Rut owski, K., Berg, K., Stromfors, C ., Van-Nest, M., Stock, L., & Sutton, J.
(2003). Integrating technology in a field-based teach er training program : T he PT 3@AS U project . Edu-
cational Technology, Research and Development, 51(1), 57-73.
Brush, T. & Saye, J. (2001). T he use of embedded scaffolds with hypermedia-supported student-centered
learning. Journal of Educational Multimedia and Hypermedia, 10(4), 333-356.
Bullock, D. (2004). Moving from theory to practice: An examination of the factors that preservice teachers
encounter as the attempt to gain experience teaching with technology during field placement exp e ri -
ences. Journal of Technology and Teacher Education. 12 (2), 211-237.
Clarke, O., & Bowe, L. (2006a). The learning federation and the Victorian department of education and
training trial of online curriculum content with Indigenous students. 1-14. Retrieved from
htt p:/ /www. thelearnin gfederati on. edu. au/verve/_ resou rces/ tl f_det vic_i ndig_ trial_ mar06. pd f
Clarke, O., & Bowe, L. (2006b). The learning federation and the Victorian department of education and
training trial of online curriculum content with ESL students. 1-16. Retrieved from
htt p:/ /www. thelearnin gfederati on. edu. au/verve/_ resou rces/ report _esl_ final. pd f
Compton, V., & Harwood, C. (2003). Enhancing technological practice: An assessment framework for
technology education in New Zealand. International Journal of Technology and Design Education,
13(1), 1-26.
Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard Uni-
versity Press.
Kay & Knaack
285
Doering, A., Hughes, J., & Huffman, D. (2003). Preservice teachers: Are we thinking with technology?
Journal of Research on Technology in Education, 35(3), 342-361.
Duval, E., Hodgins, W., Rehak, D., & Robson, R. (2004). Learning objects symposium special issue guest
editori al. Journal of Educational Multimedia and Hypermedia, 13(4), 331-342.
Eifler, K. Greene, T., & Carroll, J. (2001). Walking the talk is tough: From a single technology course to
infusion. The Educational Forum. 65(4), 366-375.
Gadanidis, G., Gadanidis, J., & Schindler, K. (2003). Factors mediating the use of online applets in the les-
son planning of pre-service mathematics teachers. Journal of Computers in Mathematics and Science
Teaching, 22(4), 323-344.
Haughey, M. , & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of Instructional
Sciences and Technology, 8(1). Retrieved June 1, 2007 from ht tp://www. usq.edu.au/electpub/e-
jist/docs/vol8_no1/fullpapers/eval_learnobjects_school.htm
Kay, R . H., & Knaack, L. (in press). As sess ing learning, quality and engagement in learning objects: The
learning object evaluation scale for students (LOES-S). ETR& D.
Kay, R . H., & Knaack, L. (2005). Developing learning objects for secondary school students: A multi-
component model. Interdisciplinary Journal of Knowledge and Learning Objects, 1, 229-254. Re-
trieved from http://ijello.org/Volume1/v1p229-254Kay_Knaack.pdf
Kay, R . H., & Knaack, L. (2007a). A systematic evaluation of learning objects for secondary school stu-
dents. Journal of Educational Technology Systems, 35(4), 411-448.
Kay, R . H., & Knaack, L. (2007b). Evaluating the learning in learning objects. Open Learning, 22(1), 5-28.
Kay, R . H., & Knaack, L. (2007c). Evaluating the use of learning objects for secondary school science.
Journal of Computers in Mathematics and Science Teaching, 26(4), 261-289.
Kay, R . H., & Knaack, L. (2007d). T eacher evaluat ion o f learning object s in middl e and secondary school
classrooms. Manuscript submitted for publication. Retrieved Feb, 2008 from
http://faculty.uoit.ca/kay/papers/LOES_Teach er _2007.pdf.
Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common frac-
tions: A model of affordances. Computers and Education, 45(2), 245-265.
Lim, C. P., Lee, S. L. & Richards, C. (2006). Developing interactive learning objects fo r a computing ma-
thematics models. International Journal on E-Learning, 5(2), 221-244.
Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environ-
ment. Educational Technology, Research and Development, 53(1), 5-21.
Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning environ-
ment based on recreational learning objects. Computers and Education, 48(4), 618-641.
McRobbie, C. J., Ginns, I. S., & Stein, S. J. (2000). Preservice primary teachers' thinking about technology
and technology education. International Journal of Technology and Design Education, 10, 81-101.
Nurmi, S., & Jaakkola, T. (2006). Effectiveness of learning objects in various instructional settings. Learn-
ing, Media, and Technology, 31(3), 233-247.
Plante, J., & Beattie, D. (2004). Education, skills and learning – Research papers Connectivity and ICT
integration in Canadian elementary and secondary schools: First results from the Info rmation and
Communications Technologies in Schools Survey, 2003-2004. Statistics Canada. Retrieved Aug 29,
2004 from http://www. schoolnet.ca/home/documents/Report_EN.pdf
Rehak, D., & Mason, R. (2003). Chapter 3: Keeping the learning in learning objects. Journal of Interactive
Media in Education, 1. Retrieved July 1, 2005 from http://www-jime.open.ac.uk/2003/1/reuse-05.html
Reimer, K., & Moyer, P. S. (2005). Third-graders learning about fractions using virtual manipulatives: A
classroom study. Journal of Computers in Mathematics and Science Teaching, 24(1), 5-25.
Inv estigating the Use of Learning Obj ects
286
Rieber, L. P., Tzeng, S., & T ribble, K. (2004). Discovery learning, representation, and explanation within a
computer-based simulation: Finding the right mix. Learning and Instruction, 14(3), 307-323.
Robertson, H. (2003). Toward a theory of negativity: T eacher educ ation and information and communica-
tions technology. Journal of Teacher Education, 54(4), 280-296.
Russell, M., Bebell, D., O’Dwyer, L., & O’Connor, K. (2003). Examining teacher technology use: Implica-
ti ons for prese rvice and i nservice teach er prep aration. Journal of Teacher Education, 54(4), 297-310.
Schoner, V., Buzza, D. , Harrigan, K. & St ram pel , K. . (2005). Learning objects in use: ‘Lite’ assessment for
fi eld studies. Journal of Online Learning and Teaching, 1(1), 1-18.
Strudler, N., Archambault, L., Bendixen, L., Anderson, D., & Weiss, R. (2003). Project THREAD: Tech-
nology helping restructure educational access and delivery. Educational Technology Research and De-
velopment, 51(1), 39-54.
Thalheimer, W., & Cook, S. (2002). How to calculate effect sizes from published research articles: A sim-
plified methodology. Retrieved July 14, 2007 from http://work-learning.com/effect_sizes.htm
Thompson, A. D., Schmidt, D. A., & Davis, N. E. (2003). Technology collaboratives for simultaneous re-
newal in t eacher education. Educational Technology Research and Develo pm en t, 51(1), 73-89.
U.S. Department of Education, National Center for Education Statistics (2002). Int ernet acces s in U. S.
public schools and classrooms: 1994-2002. Retrieved August 30, 2004 from
http://nces.ed.gov/programs/digest/d02/tables/dt419.asp
Waxman, H. C., Connell, M. L., & Gray, J. (2002). A quantitative synthesis of recent r esearch on t he ef -
fects of teaching and learning with technology on student outcomes. Naperville, IL: North Central Re-
gional Laboratory.
Wepner, S. B., Ziomek, N., & T ao L. (2003). T hree teacher educ ators ’ perspectives about the shifting re-
sponsibilities of infusing technology into the curriculum. Action in Teacher Education, 24(4), 53-63.
Windschitl, M., & Andre, T. (1998). Using computer simulations to enhance conceptual change: T he roles
of constructivist instruction and student epistemological beliefs. Journal of Research in Science Teach-
ing, 35(2), 145-160.
Kay & Knaack
287
Appendix A - Learning Object Survey – Teachers
Strongly
Disagree
1
Disagree
2
Slightly
Disagree
3
Neutral
4
Slightly
Agree
5
Agree
6
Strongly
Agree
7
Learning
1. The graphics and an-
imations from the
learning object helped
students learn.
1 2 3 4 5 6 7
2. The students were
able to learn from the
learning object.
1 2 3 4 5 6 7
Quality
3. The learning object
was easy for students
to use.
1 2 3 4 5 6 7
4. The learning object
was easy to learn.
1 2 3 4 5 6 7
5. The students found
the learning object in-
structions clear
1 2 3 4 5 6 7
Engage men t
6. The students liked
interacti ng with the
learning object.
1 2 3 4 5 6 7
7. The students were on
task while using the
learning object.
1 2 3 4 5 6 7
8. Students were moti-
vated while using the
learning object.
1 2 3 4 5 6 7
Ove ral l I mpact on Le arni ng
9. What was the overall impact of the learning object on your lesson?
Inv estigating the Use of Learning Obj ects
288
Appendix B - Learning Object Survey – Students
Strongly
Disagree
1
Disagree
2
Neutral
3
Agree
4
Strongly
Agree
5
Learning
1. Working with the learning object helped me
learn.
1 2 3 4 5
2. The feedback from the learning object helped
me learn.
1 2 3 4 5
3. The graphics and animations from the learn-
ing object helped me learn.
1 2 3 4 5
4. The learning object helped teach me a new
concept.
1 2 3 4 5
5. Overall, the learning object helped me learn.
1 2 3 4 5
Quality
6. The help features in the learning object were
useful.
1 2 3 4 5
7. The instructions in the learning object were
ea sy to follow.
1 2 3 4 5
8. The learning object was easy to use. 1 2 3 4 5
9. The learning object was well organized. 1 2 3 4 5
Engage me nt 1 2 3 4 5
10. I liked the overall theme of the learning ob-
ject.
1 2 3 4 5
11. I found the learning object motivating. 1 2 3 4 5
12. I would like to use the learning object
again.
1 2 3 4 5
13. What, if anything, did you LIKE about the learning object?
14. What, if anything, did you NOT LIKE about the learning object?
Kay & Knaack
289
Biographies
Robin Kay, Ph.D. is an Associate Professor in the Facult y of Educa-
tion at the University of Ontario Institute of Technology. He has pub-
lished over 40 articles in the area of computers in education, present ed
numerous papers at 10 international conferences, refereed three promi-
nent comput er educa t ion journals, and ta ugh t com put e r s, mat h em at ics,
an d t echn o lo gy fo r ov er 18 year s. Current project s in c lude r esearch on
lapt op use in tea cher educat ion, discussio n bo ard use, electronic
evaluation of teacher education programs, and factors that influence
how students learn with technology.
Liesel Knaack, Ph.D. is an Associate Professor in the Faculty of Edu-
cation at the University of Ontario Institute of Technology. Her re-
search interests are in the areas of design, development and evaluation
of learning objects, effect ive integrat ion of computers in the curricula,
in st r uct ional design of digit al learn ing environ m ent s and t he pr o cess of
ch a n ge in imp lem ent in g tech no logy use at inst itut ions o f h igher e duca-
t ion. Current pro ject s include lapt op use in preserv ice educat ion pro-
gram s, on line discussion s a n d digit al performan c e app r a isals.
