Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay
Content may be subject to copyright.
Interdisciplinary Journal of Knowledge and Learning Objects Volume 1, 2005
Editor: Alex Koohang
Developing Learning Objects for Secondary
School Students: A Multi-Component Model
Robin Kay and Liesel Knaack
University of Ontario, Institute of Technology,
Oshawa, Ontario, Canada
robin.kay@uoit.ca liesel.knaack@uoit.ca
Abstract
Previous research on the development of learning objects have three notable characteristics: a
focus on either technical or learning features, but not both, a target audience consisting exclu-
sively of higher education students, and the absence of formal evaluation. This study provides a
detailed description and formal evaluation of a multi-component model used to develop five
learning objects for secondary school students. Overall, two thirds of the students reported that
the learning objects were beneficial, citing a motivating theme, interactivity, and visual qualities
as the most important features. However, almost 60% of all students were critical about the learn-
ing object quality. Clarity of instructions, help functions and organization/layout presented the
most problems. While the development model used in the study emphasized both technical and
learning components, the latter was far more important to teachers and students. Key steps that
appeared to be beneficial in the development process included a collaborative team approach,
understanding the learner, a focus on clear instruction and organization, and using a comprehen-
sive, theoretically supported evaluation metric to examine the quality and benefits of learning
objects.
Keywords: Development, evaluation, secondary school, learning object
Introduction
Over the past five years, the educational potential of learning objects has been examined in some
detail, however, relatively little research has been done looking at developmental process. A re-
view of 58 articles (Kay & Knaack, 2006) revealed only five papers that documented the process
of developing a learning object (Bradley & Boyle, 2004; Cochrane, 2005; Krauss & Ally, 2005;
MacDonald et al., 2005; Metros, 2005). Three patterns emerged from these studies. First, only
higher education students have been looked at. Second, technical features are emphasized ahead
of learning features. Third, there is a clear absence of systematic formal evaluation. The purpose
of this study was to document and formally evaluate a pedagogically guided model of developing
learning objects for secondary school
students.
In order to fully understand the model
used in this study, a clear definition of
learning objects will be established,
followed by an analysis of previous
research on the development of learn-
ing objects, and finally, an examina-
tion of the design principles used in
past studies. The primary goals of this
paper are to (a) examine a new popu-
Material published as part of this journal, either on-line or in
print, is copyrighted by the publisher of the Interdisciplinary
Journal of Knowledge and Learning Objects. Permission to make
digital or paper copy of part or all of these works for personal or
classroom use is granted without fee provided that the copies are
not made or distributed for profit or commercial advantage AND
that copies 1) bear this notice in full and 2) give the full citation
on the first page. It is permissible to abstract these works so long
as credit is given. To copy in all other cases or to republish or to
post on a server or to redistribute to lists requires specific permis-
sion and payment of a fee. Contact Publisher@ijklo.org to re-
quest redistribution permission.
Developing Learning Objects for Secondary School Students
230
lation for using learning objects, (b) describe a multi-component development model based on a
composite of key features examined previously, and (c) formally evaluate the key qualities of
learning objects that appear to have an impact on learning.
Literature Review
Definition of Learning Object
In order to critically examine the design and development of learning objects, a clear definition is
necessary. Considerable effort has been directed toward this goal (Agostinho, Bennett, Lockyer &
Harper, 2004; Butson, 2003; Friesen, 2001; Gibbons, Nelson & Richards, 2002; Littlejohn, 2003;
McGreal, 2004; Metros, 2005; Muzio, Heins & Mundell, 2002; Parrish, 2004; Polsani, 2003;
Wiley, 2000; Wiley, et al., 2004), however consensus has yet to be reached.
Two definition pathways have been pursued by learning object theorists. The original “technol-
ogy-focussed” pathway was founded on an object-oriented programming model. A learning ob-
ject was seen as a discrete, reusable, context-free learning chunk (e.g., Baruque & Melo, 2004;
Downes, 2003; Hamel & Ryan-Jones, 2002; Parrish, 2004; Siqueira, Melo, & Braz, 2004). In the-
ory, this kind of definition leads to a cost-efficient, well-organized, searchable repository of learn-
ing objects that can be used for a variety of purposes. The object-oriented definition has domi-
nated the design and development of learning objects over the past five years (Butson, 2003; Gib-
bons et al, 2000; Wiley, et al. 2004).
A second “learning-focussed” pathway to defining learning objects emerged as a reaction to an
overemphasis of technological characteristics. Butson (2003) claimed that learning objects, as
defined by object-oriented theorists, were actually hindering instruction by reducing learning to
small chunks and taking the meaning and holistic nature of discovery out of education. He advo-
cated a more open-ended, context-based, non-prescriptive design. A number of other researchers
and designers advocate a more learning–focused framework for the design of learning objects
(Baruque & Melo, 2004; Bradley & Boyle, 2004; Cochrane, 2005; Wiley et al., 2004).
Both technical and learning-based definitions offer important qualities that can contribute to the
success of learning objects. With respect to a technologically guided definition, key features in-
clude accessibility, ease of use, and reusability. Making learning objects readily accessible over
the web helps address the difficulties teachers experience in acquiring the latest versions of edu-
cational software. Well over 90% of all public schools in North America and Europe now have
access to the Internet (and therefore learning objects) with most having high-speed broadband
connections (Compton & Harwood, 2003; McRobbie, Ginns, & Stein, 2000; Plante & Beattie,
2004; US Department of Education, National Center for Education Statistics, 2002). Therefore, it
makes sense to design learning objects to be delivered through the Internet. In addition, limiting
the focus and size of learning objects increases ease of use and makes them much more attractive
to busy educators who have little time to learn more complex, advanced software packages
(Gadanidis, Gadanidis, & Schindler, 2003). Finally, reusability permits learning objects to be use-
ful for a large audience, particularly when the objects are placed in well organized, searchable
databases (e.g., Agostinho et al., 2004; Duval, Hodgins, Rehak & Robson, 2004; Rehak & Ma-
son, 2003).
With respect to enhancing learning, many learning objects are interactive tools that support explo-
ration, investigation, constructing solutions, and manipulating parameters instead of memorizing
and retaining a series of facts. The success of this constructivist based model is well documented
(e.g., Albanese & Mitchell, 1993; Bruner, 1983, 1986; Carroll, 1990; Caroll & Mack, 1984;
Collins, Brown, & Newman, 1989; Vygotsky, 1978). In addition, a number of learning objects
have a graphical component that helps make abstract concepts more concrete (Gadanidis et al.,
Kay & Knaack
231
2003). Furthermore, certain learning objects allow students to explore higher level concepts by
reducing cognitive load. They act as perceptual and cognitive supports, permitting students to
examine more complex and interesting relationships (Sedig & Liang, 2006). Finally, learning ob-
jects are adaptive, allowing users to have a certain degree of control over their learning environ-
ments, particularly “when” they are learning and for “how long”.
In this study, key components of technical and learning-based definitions are used. Learning ob-
jects are as defined as “reusable, interactive web-based tools that support the learning of specific
concepts by enhancing, amplifying, and guiding the cognitive processes of learners”.
Design of Learning Objects
Somewhat predictably, the design of learning objects has mirrored the two definitions presented
above. More technically inclined designers have emphasized accessibility (Downes, 2003),
adaptability (Siqueira, et al., 2004), the effect use of metadata (Baruque & Melo, 2004; Boyle,
2003; Hamel & Ryan-Jones, 2002; Littlejohn, 2003), reusability (Cochrane, 2005; Downes, 2003;
Hamel & Ryan-Jones, 2002; Littlejohn, 2003; Muzio et al., 2002; Paquette & Rosca, 2002;
Siqueira, et al., 2004; Wiley et al., 2004) and standardization (Downes, 2003; Hamel & Ryan-
Jones, 2002; Laurillard, 2002; Littlejohn, 2003; Polsani, 2003).
Pedagogically-focused designers of learning objects have emphasized principles of instructional
design (Baruque & Melo, 2004; Krauss & Ally, 2005; MacDonald et al., 2005; Muzio et al.,
2002), interactivity (Bradley & Boyle, 2004; Cochrane, 2005; Sedig & Liang, 2006), clear in-
structions (Cochrane, 2005), formative assessment (Bradley & Boyle, 2004; Cochrane, 2005) and
solid learning theory (Baruque & Melo, 2004; Bradely & Boyle, 2004; Cochrane, 2005; Wiley et
al., 2004).
Two observations are worth noting in the literature on the design of learning objects. First, there
is relatively little research on the design principles for learning objects (Cochrane, 2005; Wil-
liams, 2000). Recommendations for specific design characteristics are made but are rarely evalu-
ated (Downes, 2003; Krauss & Ally, 2005). Second, designers tend to bifurcate with respect to
guiding frameworks emphasizing either a technical (Boyle, 2003; Bradley & Boyle, 2004; Hamel
& Ryan-Jones, 2002; Laurillard, 2002; Littlejohn, 2003; Paquette & Rosca, 2002; Petrinjak &
Graham, 2004; Polsani, 2003; Siqueira et al., 2004) or a learning-focussed (Baruque & Melo,
2004; Cochrane, 2005; Krauss & Ally, 2005; Muzio et al., 2002; Sedig & Liang, 2006; Wiley et
al., 2004) model, but not both.
In this study, an attempt has been made to merge the design goals of both technical and learning-
based theorists. However, when there was a theoretical conflict between technical and learning
philosophies, pedagogy principles took precedence.
Development of Learning Objects
Developing high quality learning objects is a daunting task involving collaboration among subject
specialists, programmers, multimedia designers, and evaluators/researchers (Bradley & Boyle,
2004; Cochrane, 2005; Krauss & Ally, 2005; Metros, 2005; MacDonald et al., 2005; Polsani,
2003). Subject specialists are necessary to provide the instructional goals and perceptive of where
the learning objects fits in the larger picture. Programmers work with the subject specialists and
multimedia designers to translate ideas and written plans into a digital, interactive, motivating,
and easy to use format. Finally researchers informally and formally assess the learning object,
providing constructive feedback for future modifications.
A review of 58 articles (Kay & Knaack, 2006) revealed only five papers that documented the
process of developing a learning object (Bradley & Boyle, 2004; Cochrane, 2005; Krauss & Ally,
2005; Metros, 2005; MacDonald et al., 2005). All five studies developed learning objects for
Developing Learning Objects for Secondary School Students
232
higher education students with a collaborative team which always included subject matter and
technology specialists. In addition, all studies collected feedback from various users while the
learning object was being designed. However, only three studies did a formal descriptive evalua-
tion of the final learning object product (Cochrane, 2005; Krauss & Ally, 2005; MacDonald et al.,
2005). No studies did a comprehensive analysis of the impact of specific learning object qualities.
In other words, while a number of design features, both technical and learning-based, have been
incorporated by developers of learning objects, the impact of these features has not been formally
tested.
This study followed the collaborative model of previous researchers, but added a formal, system-
atic evaluation metric to explore the relative contributions of specific design features.
Purpose
The purpose of this study was to examine the quality and perceived benefit of learning objects
developed for secondary school students using a multi-component model that incorporated both
technical and learning based features.
Method
Overview
The learning object development project was based on a partnership between the Faculty of Edu-
cation and the University of Ontario Institute of Technology (UOIT) and the Centre for Learning
and Teaching Through Technology (LT3) at the University of Waterloo (UW). The principle
framework for development was derived from the Cooperative Learning Object Exchange
(CLOE), a collaboration among Ontario universities and colleges focussed on the development,
sharing and reuse of learning objects (see McGreal et al., 2004 for detailed description). This
CLOE model (http://tlc.uwaterloo.ca/projects/cloe/CaseStory/) has been replicated many times
and has proven effective in the design of learning objects for the CLOE repository
(http://www.cloe.on.ca). A modified version of a five-day CLOE workshop designed for higher
education was used to guide the development of learning objects for secondary school students.
Team Members
The key team members in this study were:
• An LT3 trainer to assist in the delivery of the CLOE workshop
• Preservice teacher candidates to assist in the organization, management, and development
of learning objects
• Experienced teachers to provide subject matter expertise
• A UOIT Flash programmer and multimedia designer
• A member of the UOIT Faculty of Education to guide evaluation and research
In order to account for the additional workload required for the preservice teacher candidates, the
entire project was counted as a course (Independent Study) on their transcripts.
Kay & Knaack
233
Development Process
Overview
The design, development and evaluation of the five learning objects took place over eight months.
A timeline of the development process is presented in Table 1. A detailed description of each step
follows.
Table 1: Key Steps and Timeline for Developing the Learning Objects
Step Time Description
Selecting Team Members March 2004 Experienced teachers were contacted to
participate in learning object study
August 2004 Preservice Teachers were given opportunity
to participate in the study
Mock Prototyping Sept 2004 (2 hours) Subject team introduced to learning objects
by creating paper-based prototype
Role Assignment Sept 2004 (1 hour) Each team completed profiles to determine
role of each member
Project Planning Nov 2004 (half day) Subject teams brainstormed topics, selected
topic for learning object, a listed goals
Understanding Learners Nov 2004 (half day) Formal presentation on key qualities of
learning objects and analysis of learners
completed
Prototyping and Usability Nov 2004 (1.5 days) Subject teams produced detailed paper and
pencil prototype of their learning objects
Electronic Prototype Dec 2004 One team member creates PowerPoint
prototype of learning object
Programming Learning
Object
Jan 2005 One team member programs Flash version
of learning object with multimedia expert
Team Formative Evaluation Feb 2005 (half day) Subject teams evaluate each other’s Flash
versions of learning objects
Pilot Test Feb 2005 (1 day) Learning objects tested on 40 volunteer
students
External Formative
Evaluation
Feb 2005 (half day) CLOE expert provides feedback on learning
objects
Revision Plan Feb 2005 (half day) Subject teams digest student and expert
feedback and make plan for further
revisions
Final Revisions March 2005 Flash multimedia programmer modifies all
5 learning objects based on revision plans
Implementation March – Apr 2005 Five learning objects are implemented and
evaluated by students
Teacher Evaluation April 2005 Subject teams brought together to evaluate
implementation of learning objects future
revisions
Developing Learning Objects for Secondary School Students
234
Selection of learning objects participants (March)
Six months prior to the start of the project, experienced secondary school teachers were recruited
from local school boards in the Toronto area via email and phone. Nine certified, secondary
school teachers (10 to 25 years experience), representing the five key subject areas (biology,
physics, chemistry, computer science and mathematics) offered at UOIT’s Faculty of Education,
volunteered to be part of the project.
A month prior to the start of the project, teacher candidates were given the opportunity to be part
of the research study. The researcher interviewed applicants and selected 21 teacher candidates.
The criterion for selection was based on a student’s ability to handle the extra workload involved.
These candidates were training to teach in one of the five subject areas which would form the
learning object groups. Team sizes ranged from five to eight members.
Mock prototyping (2 hours – September workshop)
The subject matter teams were given various case scenarios and required to create a paper-based,
mock design of what the first few screens of a learning object might look like. After one hour,
several members of each team systematically circulated around the room to view the prototypes.
Each team left behind a few members to demonstrate their mock learning object and obtain feed-
back. This first step in the CLOE model helps team members to (a) experience the first stages of
learning object design, (b) develop an understanding of each other’s strengths and potential con-
tributions to the group, and (c) see the benefits of the feedback process.
Role assignment (1 hour – September workshop)
Each subject team was asked to complete a group profile sheet to identify the various strengths of
team members. This profile sheet showed what skills were present in the group in terms of sub-
ject matter expertise, instructional design abilities, Flash experience, programming, and leader-
ship experience. The team then assigned roles to each member based on the profile feedback
(e.g., subject matter expert, flash programmer, graphical designer, group co-ordinator, content
writer)
Project planning (half day –November workshop)
The subject groups brainstormed the possible topics for their learning object. The experienced
teachers guided the discussion toward topics students struggled with at the secondary school
level. After choosing a topic, team members created a summary of the goals and objectives. The
entire group was warned of ‘scope creep’ where ambitious ideas “creep” toward being too com-
plex to complete within the eight month time frame allotted. Next, teams were asked to describe
how their subject areas was currently taught and where the learning object fit in with regards to
the course of study and teaching strategies. The experienced teachers lead and guided this part of
the discussion. Finally, the team had to clearly outline the specific tasks, duties and milestones to
be reached over the next five months. A calendar was provided so team members could start as-
signing deadline dates and tasks for group members. Teams were to have a completed a paper-
prototype by mid-December, a digital prototype (using PowerPoint) by mid-January, and a com-
plete Flash version of the learning object to be pilot-tested in February.
Understanding learners (half day – November workshop)
The LT3 faculty member and researcher gave a formal presentation on the key characteristics that
make a learning object beneficial. The researcher then showcased two exemplary learning objects
to provide teams with a concrete idea of typical layouts, organization, graphics, and key func-
tions.
Kay & Knaack
235
Subject teams were then asked to complete a task sheet to consider how their secondary school
students would learn the topic they had proposed for the learning object. This was step was done
so that teams could identify sound learning strategies to use within the learning object.
The final component for “understanding the learners” was a presentation and work period to pre-
pare a persona. A persona is a hypothetical archetype of actual users including age, grade, gender,
interests, reason for using learning object, etc. This persona exercise was to help the team mem-
bers hone in on what motivated the intended audience.
Prototyping and usability testing (1.5 days – November workshop)
First, the subjects teams were asked to produce a “low fidelity” paper prototype of their learning
object. This task involved a rather quick articulation of the components of their learning object on
paper. Debriefing of the process and sharing of their work with other subject teams also occurred.
Next, the subject teams were asked to create detailed, screen-by-screen paper prototypes (medium
fidelity) of their learning objects. Each screen had to include complete text, description of but-
tons, navigational elements and graphics. If a button indicated movement to another page, another
piece of paper was produced with that screen’s content.
Paper prototyping works particularly well if regular feedback is worked into the process (Nielson,
1994). Every two hours subject teams were asked to circulate around the room, and give feedback
on other group’s learning object designs. The feedback was digested by the designers and actively
incorporated by teams into a next version of the prototype. It was not uncommon for 10-20 ver-
sions of the paper-prototype to emerge over the span of this one and a half day workshop.
The primary product at the end of the “Prototyping and Usability Testing” session was to have a
medium fidelity paper prototype complete with descriptions of each button, page, and activity
within the prototype. This final paper prototype was given to the team member who was assigned
to work on the electronic prototype.
Electronic prototyping (one month - December)
Microsoft PowerPoint was used to transform the paper prototype into a semi-working electronic
version of the learning object. Over the course of a month, one team member took on the respon-
sibility of completing the electronic prototype which included interactivity with buttons, sounds
and graphics. Throughout this process, feedback was solicited from other team members. Edits
and modifications were made through an online discussion board where various versions of the
prototype were posted and comments from team members were discussed. All electronic proto-
types were done by early to mid-January and shared with the researcher and other designers.
Programming (one month - January)
A Flash programmer/ multimedia designer sat down with each group and observed their elec-
tronic prototype. He discussed what challenges they would have and different strategies for get-
ting started in Flash. He also worked with some groups on file structures and programming ba-
sics. Additionally, key Flash tutorials and websites were made available to students for getting
started in working with Flash. Most of the Flash work was done independently by the teacher
candidates.
After consultation with the Flash programmer, one or more team members (depending on as-
signed team roles) worked on transforming the PowerPoint digital prototype into a Macromedia
Flash learning object. This stage was the most demanding as no one in the subject teams had ever
used Flash. The Flash developer was made available to them from January to March when they
were on campus. Some students took advantage of individual learning sessions to acquire the ex-
Developing Learning Objects for Secondary School Students
236
pertise in manipulating Flash components and nuances. All Flash learning objects were turned
into the Flash programmer a week before the pilot study in late February. The programmer fixed
bugs and critical interface problems.
Formative evaluation of Macromedia Flash learning object (half day-
February)
The subject teams then met again to present the first Flash versions of their learning objects. Each
team had approximately 15 minutes to go through their learning object, describe interactivity
components, and highlight sections that each member had done. The entire learning object group
provided feedback during and after each presentation.
Pilot testing (1 day - February)
Forty secondary school students from a single high school volunteered to pilot test the learning
objects. Prior to arriving at the school, the subject teams were coached on how to approach the
testing process. This involved writing a common script, understanding the evaluation tool pro-
duced by the researcher, and practising how they would set up and have students work with the
learning object. Secondary school students from specific subject areas engaged with the learning
objects for approximately 45 minutes and then completed a 15 minute survey. Subject teams de-
briefed afterwards to consolidate the feedback given.
External formative evaluation (half day - February)
The following day, the CLOE Director from the University of Waterloo visited and provided one-
to-one guidance and feedback for improving the learning objects. The CLOE Director had exper-
tise in review and assessment of learning objects and provided on-the-spot advice to each team
during their pre-arranged meeting.
Revision discussions (half day - February)
Subject teams returned to the afternoon session with a strong sense of modifications needed. Both
the secondary school students and the CLOE expert had given them a number of key areas for
improvement. The team spent the rest of session making plans for revisions, rewording compo-
nents, and reorganizing the screen layout. Helpful guides or cartoon characters were devised, help
menus arose, and assessment ideas were incorporated. The researcher then met with teacher can-
didates from each subject team to discuss and review plans for revising learning objects.
Final revisions for learning object (one month- March)
The Flash developer was assigned to revise all five learning objects. He incorporated changes
from the expert and student testing. By the beginning of April, all learning objects were uploaded
to the UOIT website (see http://education.uoit.ca/learningobjects).
Implementation (5 weeks – March to April)
Consent forms and letters of invitation were mailed out to the 30 participants (both teachers and
teacher candidates) in the study. Consent forms had to be completed by both students and parents
before the learning object was administered.
Teachers and teacher candidates were instructed to use the learning object as authentically as pos-
sible. Since the evaluation period was confined to specific five week period in the middle of the
term, sometimes the subject matter of the learning object had been taught, so it was used as a re-
view tool. Sometimes the subject matter had yet to be taught and therefore the learning object was
used as an introduction to the topic.
Kay & Knaack
237
Students were taken to a computer lab with Internet access, given some preliminary introduction
to the learning object, and then asked to use it. Teachers provided input and assistance to indi-
vidual students as needed. After one period of using the learning object (approximately 70 min-
utes), students were asked to fill out a survey (see Appendix A).
Debriefing (half day - late April)
Teachers and teacher candidates filled out a survey (Appendix B). They also gathered in their
teams and completed charts on the process of implementing the learning object. In addition, they
also completed a list of items they felt were necessary for further revision and editing of their
learning objects.
Guiding Design Principles
Technical
Two key technical features were addressed in the development of the learning objects in this
study – reusability and accessibility. The guidance and perspective of experienced subject matter
teachers was used to create meaningful objects that could exist in a larger context. Polsani (2003)
identifies this “conceptualization” as critical to developing reusable products. In addition, a spe-
cific concept was chosen for each of the five learning objects created, allowing for multiple uses
at different grade levels. With respect to accessibility, a Flash format was used create the final
web-based, versions of the learning objects. This is a format that is readily and quickly accessible
to all schools in Ontario and most Internet users. The Sharable Courseware Object Reference
Model (SCORM) standards were not strictly adhered to because of (a) the reported inflexibility of
this model to allow for a variety of pedagogies (Downes, 2003) and (b) the potential complexities
of the standards slowing down other aspects of the development process.
Learning
The learning objects were designed at the grassroots level by preservice and experienced teachers.
Wiley (2000) maintained that learning objects need to be sufficiently challenging, so experienced
teachers were asked to brainstorm about and select areas where their students had the most diffi-
culty. Second, the learning objects were designed to be content-rich, however they focussed on a
relatively specific topic areas that could be shared by different grades. Reusability, while impor-
tant, took a back seat to developing meaningful and motivating problems. This approach is sup-
ported by a number of learning theorists (Brown, Collins & Duguid, 1989; Lampert, 1986;
Larkin, 1989; Lave & Wenger, 1991; Sternberg, 1989). Third, the learning objects were both in-
teractive and constructivist in nature. Students interacted with the computer, but not simply by
clicking “next, next, next.” They had to construct solutions to genuine problems. Finally, the “oc-
topus” or resource model proposed by Wiley et al., (2004) was used. The learning objects were
designed to support and reinforce understanding of specific concepts. They were not designed as
stand alone modules that could teach concepts.
Description of the Learning Objects
All learning objects can be accessed at: http://education.uoit.ca/learningobjects. A brief descrip-
tion is provided below.
Mathematics
This learning object (Deep Space Line) was designed to help grade 9 students explore the formula
and calculations for the slope of a line. Students used their knowledge of slope to navigate a
Developing Learning Objects for Secondary School Students
238
spacecraft through four missions. As the missions progressed from level one to level four, less
scaffolding was provided to solve the mathematical challenges.
Physics
This learning object (Relative Velocity) helped grade 11 and 12 students explore the concept of
relative velocity. Students completed two case study questions, and then actively manipulated the
speed and direction of a boat, along with the river speed, to see how these variables affect relative
velocity.
Biology
This learning object (Goovy Genetics) was designed to help grade 11 students investigate the ba-
sics of Mendel’s genetics relating the genotype (genetic trait) with the phenotype (physical traits)
including monohybrid and dihybrid crosses. Students had a visual instruction to complete Punnett
squares. Each activity finished with an assessment.
Chemistry
This grade12-oriented learning object (Le Chatelier's Principle) demonstrated the three stresses
(concentration, temperature & pressure change) that can be imposed to a system at chemical equi-
librium. Students explored how equilibrium shifts related to Le Chatelier’s Principle. Students
assessed their learning in a simulated laboratory environment by imposing changes to equilibrated
systems and predicting the correct outcome.
Computer Science
This learning object (Logic Flows) was designed to teach grade 10 or 11 students the six basic
logic operations (gates) AND, OR, NOT, XOR (exclusive OR), NOR (NOT-OR) and NAND
(NOT-AND) through a visual metaphor of water flowing through pipes. Students selected the
least number of inputs (water taps) needed to get a result in the single output (water holding tank)
to learn the logical function of each operation.
Sample
Students
The sample consisted of a 221 secondary school students (104 males, 116 females, 1 missing
data), 13 to 17 years of age, in grades 9 (n=85), 11 (n=67), and 12 (n=69) from twelve different
high schools and three boards of education. The students were obtained through convenience
sampling.
Teachers
A total of 30 teachers (9 experienced, 21 preservice) participated in the development of the learn-
ing objects. The breakdown by subject area was eight for Biology (two experienced, six preser-
vice), five for Chemistry (two experienced, three preservice), five for Computer Science (one ex-
perienced, four preservice), five for Physics (one experienced, four preservice), and seven for
Math (three experienced, four preservice).
Learning objects
Five learning objects in five different subject areas were evaluated by secondary school students.
Seventy-eight students used the Mathematics learning object (grade 9), 40 used the Physics learn-
ing object (grades 11 and 12), 37 used the Chemistry learning object (grade 12), 34 used the Bi-
Kay & Knaack
239
ology learning object (grades 9 and 11), and 32 used the Computer Science learning object
(grades 11 and 12).
Data Sources
Students – perceived benefit of learning object
The data for this study was gathered using four items based on a 7-point Likert scale, and two
open ended questions (see Appendix A). Items one to four examined perceived student benefit
and had an internal reliability rating of .87. Item 5 (Appendix A) was an open ended question ask-
ing students whether the learning object was beneficial. Two-hundred and twenty five comments
were made and categorized according to nine post-hoc categories (Table 2). Each comment was
then rated on a five-point Likert scale (-2 = very negative, -1 = negative, 0 = neutral, 1 = positive,
2 = very positive).
Table 2: Coding Scheme for Assessing Learning Object Benefits (Item 5 – Appendix A)
Reason Category Criteria Sample Student Comments
1. Timing When the learning object was
introduced in the curriculum
“I think I would have benefited more if
I used this program while studying the
unit.”
“It didn't benefit me because that par-
ticular unit was over. It would have
helped better when I was first learning
the concepts. “
2. Review of Basics /
Reinforcement
Refers to reviewing, reinforcing
concept, practice.
“going over it more times is always
good for memory”
“it did help me to review the concept
and gave me practise in finding the
equation of a line.”
3. Interactive / Hands
On / Learner Con-
trol
Refers to interactive nature of the
process
“I believe I did, cause I got to do my
own pace … I prefer more hands on
things (like experiments).”
“Yes, it helped because it was interac-
tive.”
4. Good for visual
learners
Refers to some visual aspect of
the process
“I was able to picture how logic gates
function better through using the learn-
ing object.”
“I found it interesting. I need to see it”
5. Computer Based Refers more generally to liking to
work with computers
“I think that digital learning kind of
made the game confusing.”
“I think I somewhat did because I find
working on the computer is easier then
working on paper. “
6. Fun / Interesting Refers to process being fun, in-
teresting, motivating
“I think I learned the concepts better
because it made them more interesting.”
“I think I did. The learning object
grasped my attention better than a
teacher talking non-stop.”
7. Learning Related Refers to some aspect of the
learning process
“I don't think I learned the concept bet-
ter.”
“It did help me teach the concept better”
Developing Learning Objects for Secondary School Students
240
8. Clarity Refers to the clarity of the pro-
gram and/or the quality of in-
struction
“I think it was very confusing and hard
to understand.”
“Yes, this helped me. It made it much
clearer and was very educational.”
9. Not good at sub-
ject
Refers to personal difficulties in
subject areas
“No, to be honest it bothered me. In
general I don't enjoy math and this did
not help.”
10. Compare to other
method
Compared to other teaching
method / strategy
“Yes, because it… is better than having
the teacher tell you what to do.”
“Would rather learn from a book.”
11. No reason given “I didn't benefit from any of it.”
“Yes.”
Criterion related validity for perceived benefit score was assessed by correlating the survey score
(Items 1 to 4) with the qualitative ratings (Item 5). The correlation was significant (.64; p <.001).
Students – quality of learning object
Item 6 (Appendix A) asked students what they liked and did not like about the learning object. A
total of 757 comments were written down by 221 students. Student comments were coded based
on well-established principles of instructional design. Thirteen categories are presented with ex-
amples and references in Table 3. In addition, all comments were rated on a five-point Likert
scale (-2 = very negative, -1 = negative, 0 = neutral, 1 = positive, 2 = very positive).
Two raters assessed the first 100 comments made by students and achieved inter-rater reliability
of .78. They then met, discussed all discrepancies and attained 100% agreement. Next the raters
assessed the remaining 657 comments with an inter-rated reliability of .66. All discrepancies were
reviewed and 100% agreement was reached again.
Table 3: Coding Scheme for Assessing Learning Object Quality (Item 6 – Appendix A)
Category & References Criteria Sample Student Comments
1. Organization / Layout
(Calvi, 1997; Koehler & Lehrer,
1998; Lorch, 1989; Madhumita &
Kumar, 1995)
Refers to the location
or overall layout of
items on the screen
“Sometimes we didn't know where/what
to click.”
“I found that they were missing the next
button.”
“Easy to see layout”
“[Use a] full screen as opposed to small
box.”
2. Learner Control over Interface
(Akpinar & Hartley, 1996; Bagui,
1998; Druin et al. 1999; Hanna,
Risden, Czerwinski, & Alexan-
der, 1999; Kennedy &
McNaught, 1997)
Refers the control of
the user over specific
features of the learning
object including pace
of learning
“[I liked] that it was step by step and I
could go at my own pace.”
“I liked being able to increase and de-
crease volume, temperature and pres-
sure on my own. It made it easier to
learn and understand.”
“It was too brief and it went too fast.”
3. Animation
(Gadanidis et al., 2003; Oren,
1990; Stoney & Wild, 1998;
Sedig & Liang, 2006)
Refers specifically to
animation features of
the program
“You don't need all the animation. It's
good to give something good to look at,
but sometimes it can hinder progress.”
“I liked” the fun animations”
“Like how it was linked with little mov-
ies … demonstrating techniques.”
“I liked the moving spaceship.
Kay & Knaack
241
4. Graphics
(Gadanidis et al., 2003; Oren,
1990; Stoney & Wild, 1998;
Sedig & Liang, 2006)
Refers to graphics (non
animated of the pro-
gram), colours, size of
text
“The pictures were immature for the
age group.”
“I would correct several mistakes in the
graphics”
“The graphics and captions that ex-
plained the steps were helpful.”
“Change the colours to be brighter.”
5. Audio
(Gadanidis et al., 2003; Oren,
1990; Stoney & Wild, 1998;
Sedig & Liang, 2006)
Refers to audio fea-
tures
“Needed a voice to tell you what to do”
“Needs sound effects”
“Unable to hear the character (no sound
card on computers).”
6. Clear Instructions
(Acovelli & Gamble, 1997;
Jones, Farquhar, & Surry 1995;
Kennedy & McNaught, 1997 ;
MacDonald et al., 2005)
Refers to clarity of
instructions before
feedback or help is
given to the user
“Some of the instruction were confus-
ing”
“I … found it helpful running it through
first and showing you how to do it.”
“[I needed] … more explana-
tions/Clearer instructions.
7. Help Features
(Acovelli & Gamble, 1997; Jones
et al., 1995; Kennedy &
McNaught, 1997; MacDonald et
al., 2005)
Refers to help features
of the program
“The glossary was helpful.”
“Help function was really good”
“Wasn't very good in helping you when
you were having trouble…I got more
help from the teacher than it.”
8. Interactivity
(Akpinar & Hartley, 1996; Bagui,
1998; Druin et al. 1999; Hanna et
al., 1999; Kennedy & McNaught,
1997)
Refers to general inter-
active nature of the
program
“Using the computer helped me more
for genetics because it was interactive.”
“I like that it is on the computer and you
were able to type the answers.”
“I liked the interacting problems”
9. Incorrect Content / Errors Refers to incorrect
content
“There were a few errors on the sight.”
“In the dihybrid cross section, it showed
some blond girls who should have been
brunette.”
10. Difficulty / Challenge Levels
(Hanna et al., 1999; Klawe, 1999;
Savery & Duffy, 1995)
Was the program chal-
lenging? Too easy?
Just the right difficulty
level?
“Make it a bit more basic.”
“For someone who didn't know what
they were doing, the first few didn't
teach you anything but to drag and
drop.”
“I didn't like how the last mission was
too hard.”
11. Useful / Informative
(Sedig & Liang, 2006)
Refers to how useful
or informative the
learning object was
“I like how it helped me learn”
“I found the simulations to be very use-
ful”
“[The object] has excellent review ma-
terial and interesting activities.”
“I don't think I learned anything from it
though.”
12. Assessment
(Atkins, 1993; Kramarski &
Zeichner, 2001; Sedighian, 1998;
Wiest, 2001; Zammit, 2000)
Refers to summative
feedback/ evaluation
given after a major
task (as opposed to a
single action) is com-
pleted
No specific comments offered by stu-
dents
Developing Learning Objects for Secondary School Students
242
13. Theme / Motivation
(Akpibar & Hartley, 1996; Harp
& Mayer, 1998)
Refers to overall theme
and /or motivating
aspects of the learning
object
“Very boring. Confusing. Frustrating.”
“Better than paper or lecture - game is
good!”
“I liked it because I enjoy using com-
puters, and I learn better on them.”
Teachers – perceived benefit of learning object
Because teachers, who helped create the learning objects, also administered it to their classes for
testing, there may be a bias towards evaluating the learning objects more positively. Therefore,
teacher feedback was used to corroborate what students reported. A survey consisting of five
Likert and two open-ended questions was given to experienced and preservice teachers after stu-
dents had used the learning objects. The focus of these questions was to assess the benefits of
learning objects to students. (Appendix B). The number of teachers who filled in this survey is
relatively small (n=26), so internal reliability estimates are not reported.
Teachers – quality of learning object
Data from a half-day focus group was collected to evaluate the quality of the learning object and
suggestions for improvements.
Results
Perceived Benefit of Learning Object - Students
Based on the average perceived benefit rating from the survey (items 1 to 4 - Appendix A), it ap-
pears the students felt the learning object was more beneficial than not (M= 4.8, SD= 1.5; scale
ranged from 1 to 7). Fourteen percent of all students (n=30) disagreed (average score of 3 or less)
that the learning object was of benefit whereas 55% (n=122) agreed (average score of 5 or more)
that it was useful.
The qualitative comments (Q9 – Appendix A) supported the survey results. Sixty-six percent (n=
146) of the students felt that the learning objects were beneficial.
A more detailed examination indicated that the motivational, interactive, and visual qualities were
most important to students who benefited from the learning object. Whether they learned some-
thing new was also cited frequently and rated highly. Presenting the learning object after the topic
had already been learned and poor instructions were the top two reasons given by students who
did not benefit from the learning object (Table 4).
Table 4: Mean Ratings for Reasons Given for Benefits of Learning Objects (Q9)
Reason n Mean Std. Deviation
Fun / Interesting 17 1.35 0.74
Visual Learners 33 1.24 0.84
Interactive 30 1.17 1.34
Learning Related 37 0.81 1.13
Good Review 60 0.80 1.04
Computer Based 5 0.20 1.40
Compare to Another Method 24 0.00 1.18
Timing 21 -0.29 1.19
Clarity 33 -0.55 0.00
Not good at subject 3 -1.35 0.38
Kay & Knaack
243
Perceived Benefit of Learning Object – Teacher
Overall, experienced and preservice teachers strongly agreed that the learning object was a bene-
ficial learning strategy for students (Item 1 - M= 6.5, S.D. = 0.6) and were interested in using the
learning object in their classrooms again (Item 3 - M= 6.6, S.D. = 0.6). The teachers moderately
agreed that the learning object helped students with respect to understanding concepts (M= 5.4,
S.D. = 1.2) and that students would want to use the learning object again (M= 5.5, S.D. = 1.7).
Teachers agreed that the learning objects would have been more successful if they had been im-
plemented at the right time in the curriculum (M= 6.0, S.D. = 1.4). Recall, that the learning ob-
jects were used during the field experience placements and may not have been introduced at a
pedagogically appropriate time. Finally, there were non significant differences between experi-
enced and preservice teachers with respect to the perceived benefits (Items 1 to 4) of the learning
objects (Hotelling’s T2 – n.s.).
In the open-ended questions (Appendix B – Items 6 and 7), teachers reported that that students
liked the visual or graphics qualities so the learning objects best (n= 15; 58%) followed by the
interactivity (n=12; 46%) and motivating theme (n= 7; 27%). With respect to dislikes, teachers
noted that some students had difficulty following the instructions (n=5; 19%), found the theme
boring (n=5; 19%) or were distracted by some of the computer bugs found (n=4; 15%).
Quality of Learning Object – Students
Overview
Students were relatively negative with respect to their comments about learning object quality
(Item 6 – Appendix A). Fifty-seven percent of all comments were either very negative (n=42,
6%) or negative (n=392, 52%) whereas only 42% of the students made positive (n=258, 34%) or
very positive (n=57, 8%) statements about learning object quality.
Categories
An analysis of categories evaluating learning object quality (see Table 3 for description) identi-
fied animation, interactivity, and usefulness as the highest rated areas and audio, correct informa-
tion, difficulty, clarity of instructions, and help functions as the lowest rated. Table 5 provides
means and standard deviation for all categories assessing the quality of learning objects.
Table 5: Mean Ratings for Categories Evaluating Learning Object Quality
Category n Mean Std. Deviation
Animations 27 0.81 0.74
Interactivity 47 0.66 0.84
Useful 39 0.51 1.34
Assessment 9 0.44 1.13
Graphics 84 0.25 1.04
Theme/ Motivation 125 0.12 1.40
Organization 34 -0.06 1.18
Learner Control 75 -0.12 1.19
Help Functions 42 -0.43 1.02
Clear Instructions 138 -0.61 0.95
Difficulty 107 -0.67 0.81
Information Correct 17 -1.00 0.00
Audio 13 -1.15 0.38
Developing Learning Objects for Secondary School Students
244
A one-way ANOVA comparing categories of learning object quality was significant (p < .001).
Audio, correct information, and difficulty were rated significantly lower than animations, interac-
tivity, and usefulness (Scheffé post hoc analysis, p <.05).
Categories – likes only
One might assume that categories with mean ratings close to zero are not particularly important
with respect to evaluation. However, it is possible that a mean of zero could indicate an even split
between students who liked and disliked a specific category. Therefore, it is worth looking at
what students liked about the learning objects, without dislikes, to identify polar “hot spots.” A
comparison of means for positive comments confirmed that usefulness (M=1.33) was still impor-
tant, but that theme and motivation (M=1.35), learner control (M=1.35), and organization of the
layout (M=1.20) also received high ratings. These areas had mean ratings that were close to zero
when negative comments were included (see Table 5). This indicates than students had relatively
polar attitudes about these categories.
Categories – dislikes only
A comparison of means for negative comments indicated that usefulness (M=-1.33) remained im-
portant, however theme and motivation (M=-1.32) was also perceived as particularly negative.
Students appeared to either like or dislike the theme or motivating qualities of the learning object.
Correlation between learning object quality and perceived benefit
Theme and motivation (r=.45; p < .01), the organization of the layout (r=.33; p < .01), clear in-
structions (r=.33; p < .01), and usefulness (r=.33; p < .01) were significantly correlated with the
perceived benefit score measured by the survey (items 1 to 4 – Appendix A).
Quality of Learning Object – Teachers
With respect to the positive qualities of learning objects, two key themes emerged in the focus
groups for all five learning objects: graphics and interactivity. These were to two most salient
qualities that students liked best. Regarding areas for improvement, feedback varied according to
the specific learning object used. The Biology group reported that students wanted better audio
and more challenges. The Chemistry group noted that teacher support was necessary for the
learning objects and that some instructions were unclear. The Computer Science group noted that
students liked the learning object but wanted more difficult circuits. The Math group felt the suc-
cess of the learning object was tied closely to when the concept was taught and in what format
(group vs. individual). Finally, the Physics group noted a number of bugs slowed students down,
as well as some obscure instructions.
The focus groups also reported a series of programming changes that would help improve the
consistency and quality of the learning objects (see Table 6).
Kay & Knaack
245
Table 6: Proposed Programming Changes for Learning by Subject Area
Learning
Object
Proposed Changes
Biology • Integrate dihybrid activity.
• In dihybrid Punnett square #1:
o Blue squares updated to question marks.
o Prompt for next button removed.
• In dihybrid analysis #1:
o Click and drag too repetitive.
• Eye colour difficult to see in dihybrid section.
• Fix credits
• Monohybrid – “What is monohybrid” tab:
o Remove word “dominant” from heterozygous.
• Monohybrid Punnett #1:
o “The off spring…” – Should be on one line.
• Dihybrid Punnett #1:
o Words in combination box on one line.
Chemistry • Give example of catalyst/noble gas in year 3.
• “Correct” change text colour to green or blue.
• Update credits.
• Link to activities in each year.
• Simulation.
• Remove title from certificate frame.
• Size of screen – too small
Math • Make help more obvious (Have a bubble? Bubbles for areas of the screen (con-
sole and intro to the screen).
• Press “Enter” on the keyboard instead of “next” on screen (when prompted for
text).
• Mission 2 – Students didn’t know that they needed to do the calculations on their
own using pencil and paper. Instructions need to be more explicit.
• “Instruction” font size and colour are too small and too dark.
• Options to go back to other missions, and when they get to the end, more clarity
as to what or where they will goÆmore missions or choices.
• Variety of scenarios (Missions).
• Display the equation of the line drawn from “planet” to “planet”.
Developing Learning Objects for Secondary School Students
246
Computer
Science
• Buttons are active and hidden – e.g., upper left and main
choice buttons hidden behind instructions and help screen
• Level 2 - should there be a short pipe under the second-last
OR gate?
• Level 2 - no 'unsuccessful' message
• Level 3 - no 'unsuccessful' message
• Level 5 - incorrect message if choose right-most two taps
(message also appears twice)
• Level 5 - no 'unsuccessful' message
• Level 6 - incorrect message if choose either of the
right-most taps
• Level 6 - no 'unsuccessful' message
• On level 6, above NAND, there is no end to the pipes
• General - change pipe colour (to silver-grey?) and the 'on'
tap colour to green (to match text instructions and feedback from users)
• About screen - bump up the version number (v1.4?, v1.5?)
• Teacher Info / Expectations à 'Ontario Curriculum Unit
Planner' should be changed to 'the content and intentions of the
published Ontario curriculum'
• Teacher Info / Prerequisite- same wording as above
• The big grey box at the left with all the mouse-over help -
put this on the right and make it a water tank, and then feed the flow
from this - it would make it a useful part of the metaphor, rather than
a big help box taking up a large part of the screen
Physics • Program locked up at times.
• Instructions are not obvious.
• Screen Resolution problems.
• Labels inconsistent.
• No defined Learning Objects for Case 3.
• Boat can disappear!?
• Better used as a demonstration tool or as a problem solving simulation.
• Cannot teach the concept in isolation from a teacher.
Discussion
Evaluation of Learning Object
Perceived benefit
The results from this study suggest that learning objects designed and developed using a multi-
component model are viable learning tools for secondary students. Two-thirds of all students felt
that learning objects were beneficial, particularly when they had a motivating theme, with visual
supports, and interactivity. Independent evidence from experienced and preservice teachers was
consistent with student reports. These results are also consistent with previous research on in-
structional design (e.g., Akpinar & Hartley, 1996; Bagui, 1998; Druin et al. 1999; Gadanidis et
al., 2003; Hanna, Risden, Czerwinski, & Alexander, 1999; Harp & Mayer, 1998; Kennedy &
McNaught, 1997; Oren, 1990; Sedig & Liang, 2006; Stoney & Wild, 1998). Overall, the learning
features incorporated in the development process were well received by students and teachers.
Kay & Knaack
247
The technical features considered in the design of these learning objects were ignored by students
and teachers. Reusability and accessibility were not mentioned once in the qualitative feedback
given, however, the implementation of the learning objects minimized the impact of these fea-
tures. In other words, each learning object was used by a single grade only and was readily acces-
sible over the web. Reusability, accessibility and metadata may have been more of an issue had
teachers born the responsibility of selecting objects from a large database. It is unlikely, though,
that students will ever be preoccupied with technical design features typically addressed by learn-
ing objected theorists (Boyle, 2003; Bradley & Boyle, 2004; Hamel & Ryan-Jones, 2002; Lauril-
lard, 2002; Littlejohn, 2003; Paquette & Rosca, 2002; Petrinjak & Graham, 2004; Polsani, 2003;
Siqueira et al., 2004) simply because teachers select the learning objects to be used in the class-
rooms. Once a learning object is selected, challenges based on reusability, accessibility, and
metadata are essentially eliminated.
Quality of learning object
The results from this paper suggest that four of the thirteen learning quality categories (Table 3)
are particularly important in terms of learning object quality and benefit: usefulness, clear instruc-
tions, organization/layout, and theme/motivation. If the learning object provides clear instruc-
tions, is well organized, motivating, and perceived as being useful, secondary students are more
likely feel they have benefited from the experience. These results match the qualitative feedback
reported by Cochrane (2005) and MacDonald et al. (2005) for higher education students.
Teacher feedback confirmed the impressions reported by students, however, the detail and quality
of their suggestions was relatively vague. It is difficult for a teacher to observe the nuances of
learning for all students in a limited instructional time. Nonetheless, the teachers in this study
were able to get a general sense of the learning objects quality that was consistent with student
perceptions.
On the other hand, student feedback on programming bugs was typically vague and unfocussed.
Most of the time, students appeared to be trying to learn. The teachers, though, who were the de-
signers of these learning objects, were effective in gathering and providing detailed suggestions
on programming changes that needed to me made.
In summary, both student and teacher feedback were essential for examining the quality of learn-
ing objects. The student responses regarding learning features provided clear guidelines on what
works well: usefulness, clear instructions, organization/layout, and theme/motivation. Teacher
feedback provided clear suggestions for improving the consistency and clear presentation of the
learning object.
Multi-Component Model
There are five key conclusions that can be made regarding the multi-component development
model used to create learning objects for this study. First, as stated earlier, the learning features
were more important to students and teachers than technical features. This result was not pre-
dicted by previous literature, where a heavy bias has been placed on more technical issues such as
metadata, reusability, and accessibility (e.g., Boyle, 2003; Bradley & Boyle, 2004; Hamel &
Ryan-Jones, 2002; Laurillard, 2002; Littlejohn, 2003; Paquette & Rosca, 2002; Petrinjak & Gra-
ham, 2004; Polsani, 2003; Siqueira et al., 2004).
Second, the team approach worked reasonably well in producing five learning objects in a rela-
tively short time period. The experienced teacher proved valuable in selecting topics that were
relevant to student needs. It is doubtful that preservice candidates would have been able to accu-
rately anticipate important topics and motivating themes without the guidance of a veteran educa-
tor. It should be noted that the use of a Flash/multimedia programmer was also critical – the ob-
Developing Learning Objects for Secondary School Students
248
jects could not have been completed without the support of this individual, especially near the
final deadline. One modification supported by previous studies (Bradley & Boyle, 2004; Coch-
rane, 2005; Krauss & Ally, 2005; Metros, 2005; MacDonald et al., 2005; Polsani, 2003) would be
to have a full time programmer. Preservice teachers were good at forming story lines and Power-
Point simulations of the learning objects, however, learning Flash and its complexities to success-
fully program a learning object is an onerous task for a preservice teacher candidate. Addition-
ally, last minute programming pressures resulted in a number of “bugs” that might have been
avoided with the assistance of a dedicated programmer.
Third, the half day devoted to understanding learners and focussing on effective learning strate-
gies was critical for the success of the learning objects given that the perceived benefit by stu-
dents was predicated on interactivity, effective visual representations, and a motivating theme.
This emphasis on learning is supported by a handful of previous studies (e.g., Bradley & Boyle,
2004; Cochrane, 2005; Gadanidis et al., 2003; Sedig & Liang, 2006). That said, 45% of the stu-
dents were either neutral or negative with respect to the benefits of the learning object. Given that
the primary focus of students and teachers was pedagogy, it might be wise to dedicate even more
time to the “understanding the learner”.
Fourth, four of the five and half days devoted to team development of the learning object focus-
sed on organization, prototyping, and an extensive feedback cycle to ensure that the learning ob-
jects were easy to use and understand. This time was well spent given that students rated clear
instructions and organization/layout as two of the most important qualities of the learning object.
These findings are consistent with instructional design research (e.g., Baruque & Melo, 2004;
Krauss & Ally, 2005; MacDonald et al., 2005; Muzio et al., 2003), but have only been formally
tested with learning objects in one study (Cochrane, 2005). Nonetheless, even though over 70%
of formal design and development time was spent on clear instructions and organization/layout,
these were the two biggest obstacles noted by students in the study. More pilot testing may have
improved the instructional guidance offered by learning objects.
Finally, it is clear that a detailed, theoretically-based evaluation tool coupled with teacher focus
groups was a critical component in the design and development of learning objects. This compo-
nent has been noticeably absent in learning object research (Downes, 2003; Krauss & Ally,
2005). Without the comprehensive analysis of feedback from teachers and students, it would have
been challenging to identify key features important to the success of the learning objects.
Caveats
This study was first attempt to systematically develop learning objects for secondary school stu-
dents. While the study produced useful information for educators and researchers, there are at
least three key areas that could be improved in future research. First, a set of pre and post-test
content questions is important to assess whether any learning actually occurred. Second, a more
systematic survey requiring students to rate all benefit and quality categories (Tables 2 and 3)
would help to provide more comprehensive assessment data. Finally, details about how each
learning object is used are necessary to open up a meaningful dialogue on the kinds of instruc-
tional wrap that might affect use.
Future Development of Learning Objects
Based on the results from this study and previous research, there are a number of key suggestions
for the future development of learning objects including:
• A dedicated programmer and designer coupled with extensive pilot testing on a wide
range of subjects is recommended to ensure clarity of instructions and help, as well as
ease of use
Kay & Knaack
249
• Considerable time should be devoted to understanding what motivates the learner, as well
as the key elements of the learning object that enhance learning
• A comprehensive, theoretically-based evaluation tool is needed to identify critical com-
ponents that effect the success of learning objects
• Future research should look at actual learning outcomes and the nature of instructional
wrap that embraces the learning object
Summary
The purpose of this study was to examine the quality and perceived benefit of learning objects
developed for secondary school students using a multi-component model incorporating both
technical and learning based features. Overall, two thirds of the students stated they benefited
from using the learning object. Students benefited more if they were comfortable with computers,
the learning object had a well organized layout, instructions are clear, and the theme was fun or
motivating. Students appreciated the motivating, hands-on, and visual qualities of the learning
objects most. While the developmental model used in this study emphasized both technical and
learning base features, the latter proved to be more important to teachers and students. Key steps
that were beneficial in the development process included a collaborative team approach, under-
standing the learner, a focus on clear instruction and organization, and using comprehensive,
theoretically supported evaluation metric examining the quality and benefits of learning objects.
References
Acovelli, M., & Gamble, M. (1997). A coaching agent for learners using multimedia simulations. Educa-
tional Technology, 37(2), 44-49.
Agostinho, S., Bennett, S., Lockyear, L., & Harper, B. (2004). Developing a learning object metadata ap-
plication profile based on LOM suitable for the Australian higher education market. Australasian
Journal of Educational Technology, 20(2), 191-208.
Akpinar, Y., & Hartley, J. R. (1996). Designing interactive learning environments. Journal of Computer
Assisted Learning, 12(1), 33-46.
Albanese, M. A., & Mitchell, S. A. (1993). Problem-based learning: A review of the literature on its out-
comes and implementation issues. Academic Medicine, 68, 52-81.
Atkins, M. J. (1993). Theories of learning and multimedia applications: An overview. Research Papers in
Education, 8(2), 251-271.
Bagui, S. (1998). Reasons for increased learning using multimedia. Journal of Educational Multimedia and
Hypermedia, 7(1), 3-18.
Baruque, L. B., & Melo, R. N. (2004). Learning theory and instructional design using learning objects.
Journal of Educational Multimedia and Hypermedia, 13(4), 343-370.
Boyle, T. (2003). Design principles for authoring dynamic, reusable learning objects. Australian Journal of
Educational Technology, 19(1), 46-58.
Bradley, C., & Boyle, T. (2004). The design, development, and use of multimedia learning objects. Journal
of Educational Multimedia and Hypermedia, 13(4), 371-389.
Brown, J. S., Collins, A., & Duiguid, P. (1989). Situated cognition and the culture of learning. Educational
Researcher, 18(1), 32-42.
Bruner, J. (1983). Child's talk. Learning to use language, Toronto, Canada: George J. McLeod Ltd.
Bruner, J. (1986). Actual minds, possible worlds. Cambridge, MA: Harvard University Press.
Butson, R. (2003). Learning objects: Weapons of mass instruction. British Journal of Educational Technol-
ogy, 34(5), 667-669.
Developing Learning Objects for Secondary School Students
250
Calvi, L. (1997). Navigation and disorientation: A case study. Journal of Educational Multimedia and Hy-
permedia, 6(3/4), 305-320.
Carroll, J. B. (1990). The Nurnberg funnel. Cambridge, MA: MIT Press.
Carroll, J. M., & Mack, R. L. (1984). Learning to use a word processor: By doing, by thinking, and by know-
ing. In J. C. Thomas and M. Schneider (Eds.), Human factors in computer systems. Norwood, NJ: Ablex.
Cochrane, T. (2005). Interactive QuickTime: Developing and evaluating multimedia learning objects to
enhance both face-to-face and distance e-learning environments. Interdisciplinary Journal of Knowl-
edge and Learning Objects, 1. Retrieved August 3, 2005 from http://ijklo.org/Volume1/v1p033-
054Cochrane.pdf
Collins, A. Brown, J. S., Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writ-
ing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction (pp. 453-494). Hillsdale,
NJ: Erlbaum Associates.
Compton, V., & Harwood, C. (2003). Enhancing technological practice: An assessment framework for
technology education in New Zealand. International Journal of Technology and Design Education,
13(1), 1-26.
Downes, S. (2003). Design and reusability of learning objects in an academic context: A new economy of
education? USDLA Journal: A Refereed Journal of the United States Distance Learning Association,
17(1). Retrieved October 17, 2005 from
http://www.usdla.org/html/journal/JAN03_Issue/article01.html
Druin, A., Bederson, B., Boltman, A., Miura, A., Knotts-Callahan, D., & Platt, M. (1999). The design of
children's technology. San Francisco: Morgan Kaufmann Publishers, Inc.
Duval, E., Hodgins, W., Rehak, D., & Robson, R. (2004). Learning objects symposium special issue guest
editorial. Journal of Educational Multimedia and Hypermedia, 13(4), 331-342.
Friesen, N. (2001). What are educational objects? Interactive Learning Environments, 9(3).
Gadanidis, G., Gadanidis, J. & Schindler, K. (2003). Factors mediating the use of online applets in the les-
son planning of pre-service mathematics teachers. Journal of Computers in Mathematics and Science
Teaching, 22(4), 323-344.
Gibbons, A. S., Nelson, J. & Richards, R. (2000). The nature and origin of instructional objects. In D. A.
Wiley (Ed.), The instructional use of learning objects: Online version. Retrieved July, 1 2005 from
http://reusability.org/read/chapters/gibbons.doc
Hamel, C. J. and Ryan-Jones, D. (2002). Designing instruction with learning objects. International Journal
of Educational Technology, 3(1). Retrieved June 1, 2005 from:
http://www.ao.uiuc.edu/ijet/v3n1/hamel/
Hanna, L., Risden, K., Czerwinski, M., & Alexander, K. J. (1999). The Role of usability in designing chil-
dren's computer products. In A. Druin (Ed.), The design of children's technology. San Francisco: Mor-
gan Kaufmann Publishers.
Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cognitive interest
in science learning. Journal of Educational Psychology, 90(3), 414-434.
Jones, M. G., Farquhar, J. D., & Surry, D. W. (1995). Using metacognitive theories to design user inter-
faces for computer-based learning. Educational Technology, 35(4), 12-22.
Kay, R. H., & Knaack, L. (2006). A systematic evaluation of learning objects for secondary school stu-
dents. Manuscript submitted for publication.
Kennedy, D. M., & McNaught, C. (1997). Design elements for interactive multimedia. Australian Journal
of Educational Technology, 13(1), 1-22.
Klawe, M. M. (1999). Computer games, education and interfaces: The E-GEMS Project. Retrieved January
15, 2000 from http://www.graphicsinterface.org/proceedings/1999/203/
Koehler, M. J., & Lehrer, R. (1998). Designing a hypermedia tool for learning about children's mathemati-
cal cognition. Journal of Educational Computing Research, 18(2), 123-145.
Kay & Knaack
251
Kramarski, B., & Zeichner, O. (2001). Using technology to enhance mathematical reasoning: Effects of
feedback and self-regulation learning. Education Media International, 38(2/3).
Krauss, F., & Ally, M. (2005). A study of the design and evaluation of a learning object and implications
for content development. Interdisciplinary Journal of Knowledge and Learning Objects, 1, 1-22. Re-
trieved August 4, 2005 from http://ijklo.org/Volume1/v1p001-022Krauss.pdf
Lampert, M. (1986). Teaching multiplication. Journal of Mathematical Behaviour, 5, 241-280.
Larkin, J. H. (1989). What kind of knowledge transfers? In L. B. Resnick (Ed.), Knowing, learning, and in-
struction (pp. 283-305). Hillsdale, NJ: Erlbaum Associates.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cam-
bridge University Press.
Laurillard, D. (2002). Design tools for e-learning. In A. Williamson, C. Gunn, A. Young and T. Clear
(Eds.). Winds of change in the sea of learning. Proceedings of the 19th Annual Conference of the Aus-
tralasian Society for Computers in Tertiary Education (ASCILITE), December 8-11, UNITEC, Auck-
land, New Zealand. Retrieved January 26, 2004, from:
http://www.unitec.ac.nz/ascilite/proceedings/papers/key_laurillard.pdf
Littlejohn, A. (2003). Issues in reusing online resources. Journal of Interactive Media in Education, 1, Spe-
cial Issue on Reusing Online Resources. Retrieved July 1, 2005 from http://www-
jime.open.ac.uk/2003/1/
Lorch, R. F. (1989). Text-signalling devices and their effects on reading and memory processes. Educa-
tional Psychology Review, 1(3), 209-234.
MacDonald, C. J., Stodel, E., Thompson, T. L., Muirhead, B., Hinton, C., Carson, B., et al. (2005). Ad-
dressing the eLearning contradiction: A collaborative approach for developing a conceptual framework
learning object. Interdisciplinary Journal of Knowledge and Learning Objects, 1, 79-98. Retrieved
August 2, 2005 from http://ijklo.org/Volume1/v1p079-098McDonald.pdf
Madhumita & Kumar, K. L. (1995). Twenty-one guidelines for effective instructional design. Educational
Technology, 35(3), 58-61.
McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional Tech-
nology and Distance Learning, 1(9). Retrieved August 5, 2005 from
http://www.itdl.org/Journal/Sep_04/article02.htm
McGreal, R., Anderson, T., Babin, G., Downes, S., Friesen, N., Harrigan, K., et al. (2004). EduSource:
Canada's learning object repository network. International Journal of Instructional Technology and
Distance Learning, 1(3). Retrieved July 24, 2005 from
http://www.itdl.org/Journal/Mar_04/article01.htm
McRobbie, C. J., Ginns, I. S., & Stein, S. J. (2000). Preservice primary teachers' thinking about technology
and technology education. International Journal of Technology and Design Education, 10, 81-101.
Metros, S. E. (2005). Visualizing knowledge in new educational environments: A course on learning ob-
jects. Open Learning, 20(1), 93-102.
Muzio, J. A., Heins, T., & Mundell, R. (2002). Experiences with reusable e-learning objects from theory to
practice. Internet and Higher Education, 2002 (5), 21-34.
Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods.
New York, NY: John Wiley & Sons.
Oren, T. (1990). Cognitive load in hypermedia: Designing for the exploratory learner. In S. Ambron & K.
Hooper (Eds.), Learning with interactive multimedia (pp. 126-136). Washington: Microsoft Press.
Paquette, G., & Rosca, I. (2002). Organic aggregation of knowledge object in educational systems. Cana-
dian Journal of Learning and Technology, 28 (3). Retrieved July 1, 2005 from
http://www.cjlt.ca/content/vol28.3/paquette_rosca.html
Parrish, P. E. (2004). The trouble with learning objects. Educational Technology Research & Development,
52(1), 49-67.
Developing Learning Objects for Secondary School Students
252
Petrinjak, A., & Graham, R. (2004). Creating learning objects from pre-authored course materials: Seman-
tic structure of learning objects – design and technology. Canadian Journal of Learning and Technol-
ogy, 30 (3). Retrieved July 1, 2005 from http://www.cjlt.ca/content/vol30.3/petrinjak.html
Plante, J., & Beattie, D. (2004). Connectivity and ICT integration in Canadian elementary and secondary
schools: First results from the Information and Communications Technologies in Schools Survey,
2003-2004. Statistics Canada. Retrieved Aug 29, 2004 from
http://www.schoolnet.ca/home/documents/Report_EN.pdf
Polsani, P. R. (2003). Use and abuse of reusable learning objects. Journal of Digital Information, 3(4), Re-
trieved July 1, 2005 from http://jodi.ecs.soton.ac.uk/Articles/v03/i04/Polsani/
Rehak, D. & Mason, R. (2003). Chapter 3: Keeping the learning in learning objects. Journal of Interactive
Media in Education, 2003 (1). Retrieved July 1, 2005 from http://www-jime.open.ac.uk/2003/1/reuse-
05.html
Savery, J. R., & Duffy, T. M. (1995). Problem-based learning: An instructional model and its constructivist
framework. Educational Technology, 35(5), 31-34.
Sedig, K & Liang, H (2006). Interactivity of visual mathematical representations: Factors affecting learning
and cognitive processes. Manuscript submitted for publication.
Sedighian, K. (1998). Interface style, flow, and reflective cognition: Issues in designing interactive multi-
media mathematics learning environments for children. Unpublished Doctor of Philosophy disserta-
tion, University of British Columbia, Vancouver.
Siqueira, S. W. M., Melo, R. N., & Braz, M. H. L. B. (2004). Increasing the semantics of learning objects.
International Journal of Computer Processing of Oriental Languages, 17(1), 27-39.
Sternberg, R. J. (1989). Domain-generality versus domain-specificity: The life and impending death of a false
dichotomy. Merrill-Palmer Quarterly, 35(1), 115-130.
Stoney, S., & Wild, M. (1998). Motivation and interface design: Maximizing learning opportunities. Jour-
nal of Computer Assisted Learning, 14(1), 40-50.
U.S. Department of Education, National Center for Education Statistics (2002). Internet Access in U.S.
Public Schools and Classrooms: 1994-2002. Retrieved August 30, 2004 from
http://nces.ed.gov/programs/digest/d02/tables/dt419.asp
Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.
Wiest, L. R. (2001). The role of computers in mathematics teaching and learning. Computers in the
Schools, 17(1/2), 41-55.
Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor,
and a taxonomy. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. Re-
trieved July, 1, 2005, from http://reusability.org/read/chapters/wiley.doc
Wiley, D., Wayers, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004). Overcoming the limita-
tions of learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 507-521.
Williams, D. D. (2000). Evaluation of learning objects and instruction using learning objects. In D. A.
Wiley (Ed.), The instructional use of learning objects: Online version. Retrieved July 1, 2005 from
http://reusability.org/read/chapters/williams.doc
Zammit, K. (2000). Computer icons: A picture says a thousand words. Or does it? Journal of Educational
Computing Research, 23(2), 217-231.
Kay & Knaack
253
Appendix A - Learning Object Survey – Students
Strongly
Disagree
1
Dis-
agree
2
Slightly
Disagree
3
Neutral
4
Slightly
Agree
5
Agree
6
Strongly
Agree
7
1. The learning object has some
benefit in terms of providing me
with another learning strat-
egy/another tool.
1
2
3
4
5
6
7
2. I feel the learning object did
benefit my understanding of
the subject matter’s con-
cept/principle.
1
2
3
4
5
6
7
3. I did not benefit from using the
learning object.
1
2
3
4
5
6
7
4. I am interested in using the
learning object again.
1
2
3
4
5
6
7
5. Do you think you benefited from using this particular learning object? Do you think you learned the con-
cept better? Do you think it helped you review a concept you just learned? Why? Why not.
6. You used a digital learning object on the computer. Tell me about this experience when you used
the object.
a) What did you like? (found helpful, liked working with, what worked well for you)
b) What didn’t you like? (found confusing, or didn’t like, or didn’t understand)
Appendix B - Learning Object Survey – Teachers
Strongly
Disagree
1
Dis-
agree
2
Slightly
Disagree
3
Neutral
4
Slightly
Agree
5
Agree
6
Strongly
Agree
7
1. The learning object has benefit in
terms of providing students with
another learning strategy in my
classroom.
1
2
3
4
5
6
7
2. The learning object did benefit my
students in terms of their under-
standing of the con-
cept/principle explored in the
learning object.
1
2
3
4
5
6
7
3. I would be interested in using the
learning object again in my
class.
1
2
3
4
5
6
7
4. There would have been more
success with the learning object
had it been implemented during
the proper time within the unit.
1
2
3
4
5
6
7
5. Students were interested in using
the learning object again.
1
2
3
4
5
6
7
6. What did students like? What worked? What did they learn?
7. What didn’t students like? What didn’t work out well? What didn’t they learn?
Developing Learning Objects for Secondary School Students
254
Biographies
Robin Kay, Ph.D. is an Assistant Professor in the Faculty of Education
at the University of Ontario Institute of Technology. He has published
over 20 articles in the area of computers in education, presented nu-
merous papers at 10 international conferences, refereed three promi-
nent computer education journals, and taught computers, mathematics,
and technology for over 15 years. Current projects include research on
laptop use in teacher education, discussion board use, electronic
evaluation of teacher education programs, and factors that influence
how students learn with technology.
Liesel Knaack, Ph.D. is an Assistant Professor in the Faculty of Edu-
cation at the University of Ontario, Institute of Technology. Her re-
search interests are in the areas of design, development and evaluation
of learning objects, effective integration of computers in the curricula,
instructional design of digital learning environments and the process of
change in implementing technology use at institutions of higher educa-
tion. Current projects include laptop use in preservice education pro-
grams, online discussions and digital performance appraisals.
