Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay
Content may be subject to copyright.
Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay
Content may be subject to copyright.
Jl. of Computers in Mathematics and Science Teaching (2010) 29(1), 87-104
Evaluating the Use of Learning Objects for
Improving Calculus Readiness
ROBIN KAY
ILONA KLETSKIN
University of Ontario Institute of Technology, Canada
robin.kay@uoit.ca
ilona.kletskin@uoit.ca
Pre-calculus concepts such as working with functions and
solving equations are essential for students to explore limits,
rates of change, and integrals. Yet many students have a weak
understanding of these key concepts which impedes perfor-
mance in their first year university Calculus course. A series
of online learning objects was developed to provide students
with appropriate resources for self-study in pre-calculus con-
cepts. Each learning object consisted of text-based summary
sheets, interactive video-clips demonstrating sample solutions
to typical problems, and a set of online mastery practice ques-
tions. The results indicated that a majority of students used
the learning objects frequently, rated them as useful or very
useful, and reported significant knowledge gains in pre-calcu-
lus concepts.
To effectively tackle university level mathematics, a strong pre-calculus
background is essential. Yet many international studies over the past few
decades have found a decline in the algebraic ability of first year university
students and have raised concerns about numeracy (Barbeau et al., 1977;
Engineering Council, 2000; Jourdan, Cretchley, & Passmore, 2007; London
Mathematical Society, 1995; Mustoe & Lawson, 2002).
Learning objects, defined in this paper as “interactive web-based tools
that support learning by enhancing, amplifying, and guiding the cogni-
tive processes of learners” (Agostinho, Bennett, Lockyer & Harper, 2004;
Butson, 2003; McGreal , 2004; Parrish, 2004; Polsani, 2003; Wiley, et al.
2004) present a viable solution for addressing weakness in student pre-cal-
88
Kay and Kletskin
culus knowledge. Key advantages for using learning objects include acces-
sibility (Wiley, 2000), ease of use (e.g., Gadanidis, Gadanidis, & Schindler,
2003; Sedig & Liang, 2006), reusability (e.g., Agostinho et al., 2004; Du-
val, Hodgins, Rehak & Robson, 2004; Rehak & Mason, 2003), interactivity
(e.g., Gadanidis, Gadanidis, & Schindler, 2003; Sedig & Liang, 2006) and
visual supports (e.g., Gadanidis, Gadanidis, & Schindler, 2003; Sedig & Li-
ang, 2006).
The purpose of this paper was to evaluate a set of learning objects de-
signed to improve Calculus readiness for first year university students.
LITERATURE REVIEW
Addressing Gaps in Student Understanding
In order to begin their study of Calculus, students must possess a strong
foundation in working with functions and the ability to perform algebraic
manipulations with ease. This ability, however, is often lacking in first year
and impedes students’ efforts in university mathematics. Over a decade ago,
a report by the London Mathematical Society (1995) found that post-sec-
ondary mathematics, science, and engineering departments appeared unani-
mous in perceiving a qualitative change in the mathematical preparedness
of first year students, even among some high-attaining students. One of the
key problems noted was that students in mathematics courses were ham-
pered by a serious lack of essential skills in being able to do algebraic cal-
culations and manipulations fluently and accurately.
Several international studies have noted that students have substantial
difficulty with the kinds of concepts necessary to be successful in university
mathematics. Jourdan, Cretchley, & Passmore (2007) reported that students
experienced significant challenges in algebra, functions, and trigonometry.
Lawson (2003) compared diagnostic scores from 1991 to 2001 and found
a significant drop in competency in most areas of pre-calculus mathematics
tested. A recent Canadian report claimed that today’s students have less fa-
cility with basic algebra, trigonometry, and exponentials and logarithms and
that universities have observed significant declines in student performance
(Ontario Ministry of Education, 2006).
The Role of Learning Objects
Learning objects offer a number of key advantages that can benefit
both instructors and students including accessibility, ease of use, reusabil-
Evaluating the Use of Learning Objects for Improving Calculus Readiness
89
ity, graphical supports, and adaptivity. Learning objects are readily acces-
sible over the Internet and users need not worry about excessive costs or not
having the latest version (Wiley, 2000). With almost all universities having
access to high-speed broadband connections, access to learning objects is
not an obstacle. In addition, because of their limited size and focus, learning
objects are relatively easy to learn and use, making them much more attrac-
tive to busy educators who have little time to learn more complex, advanced
software packages (Gadanidis et al., 2003). Reusability allows learning
objects to be useful for a large audience, particularly when the objects are
placed in well organized, searchable databases (e.g., Agostinho et al., 2004;
Duval, Hodgins, Rehak & Robson, 2004; Rehak & Mason, 2003).
With respect to learning, a number of learning objects are interactive
tools that support exploration, investigation, constructing solutions, and ma-
nipulating parameters instead of memorizing and retaining a series of facts.
In addition, a number of learning objects have a graphical component that
helps present abstract concepts in a more concrete manner (Gadanidis et al.,
2003). Furthermore, some learning objects permit students to explore high-
er level concepts by reducing cognitive load. They provide perceptual and
cognitive supports, permitting students to examine more complex and in-
teresting relationships (Sedig & Liang, 2006). Finally, learning objects are
adaptive, allowing users to have a certain degree of control over their learn-
ing environments, particularly when they are learning and for how long.
Student perspective. In a review of 10 studies examining the use of
learning objects in higher education, undergraduate or graduate students had
positive attitudes about learning objects in eight studies (Bradley & Boyle,
2004; de Salas & Ellis, 2006; Docherty, Hoy, Topp & Trinder, 2005; Kenny,
Andrews, Vignola, Schilz, & Covert, 1999; Lim, Lee, & Richards, 2006;
MacDonald et al., 2005; Mason, Pegler, &Weller, 2005; Schoner, Buzza,
Harrigan, & Strampel 2005), neutral attitudes in one study (Concannon, Fly-
nn, & Campbell, 2005) and negative attitudes in only one study (Van Zele,
Vandaele, Botteldooren, & Lenaerts, 2003). Most papers presented infor-
mal or qualitative evidence, although three studies quantified their results
by noting that 50% to 60% of higher education students liked using learning
objects (Bradley & Boyle, 2004; de Salas & Ellis, 2006; Howard-Rose &
Harrigan, 2003).
Students made positive comments about a wide range of features in-
cluding animations (Bradley & Boyle, 2004), self-assessment (Lim et al.,
2006), attractiveness (Bradley & Boyle, 2004), control over learning (Lim
et al., 2006), ease of use (Kenny et al., 1999; Schoner et al., 2005), feedback
(Concannon et al., 2005; Lim et al., 2006), scaffolding or support (Lim et
al., 2006), interactivity (Concannon et al., 2005; Lim et al., 2006), naviga-
90
Kay and Kletskin
tion (Concannon et al., 2005) and self-efficacy (Docherty et al., 2005). Neg-
ative comments targeted problems with navigation (Concannon et al., 2005),
technology (Concannon et al., 2005) and workload (Van Zele et al., 2003).
Student performance. Regarding student performance, three studies
reported qualitative data suggesting that student learning performance im-
proved when learning objects were used (Kenny et al., 1999; Lim et al.,
2006; Windschitl & Andre, 1998). Two studies provided descriptive evi-
dence suggesting that learning objects enhanced student learning (Bradley &
Boyle, 2004; MacDonald et al., 2005). Bradley & Boyle (2004) noted pass
rates increased by 12 to 23%. Docherty et al. (2005) observed that nursing
students were significantly more positive about learning objects than control
groups, but the marks on the final course were not significantly different.
Rieber, Tzeng and Tribble (2004) cited that students, provided with embed-
ded explanations and graphical representations, performed significantly bet-
ter than those students who did not receive this scaffolding. Finally, Wind-
schitl & Andre (1998) noted that simulation-based learning objects resulted
in significant gains in two out of six concepts taught. The remaining four
concepts showed no significant difference.
The purpose of this study was to develop and evaluate learning objects
designed to improve student understanding of pre-calculus concepts.
METHOD
Sample
Students. The student sample consisted of 289 engineering or science
students (189 males, 100 females) enrolled in a first year Calculus course.
This sample size represents a survey response rate of about 60%. Students
reported high school calculus grades of 60-69 (12%, n=35), 70-79 (37%,
n=107), 80-89 (34%, n=98) and 90+ (13.5%, n=39).
Learning Object Design
As stated earlier, each learning object in this study consisted of three
components: a text-based summary of a topic, a set of 2-8 minute video
clips focusing on representative problems, and a set of online questions as-
sessing student understanding of the topic. Five key topics were covered in-
cluding operations with functions, solving equations, linear functions, expo-
nential and logarithmic functions, and trigonometric functions. A detailed
description of each component of the learning objects is provided below.
Evaluating the Use of Learning Objects for Improving Calculus Readiness
91
The text-based summaries, typically no longer than a page, summa-
rized the basic mathematical concepts within a specific topic. They con-
sisted of definitions, theorems, diagrams, and simple examples. The main
goal of these summaries was to present clear, text-based instructions with
well thought out explanations and simple illustrative examples. These sum-
maries were designed to help students decide whether they needed further
help on specific concepts. If a student believed that more instruction was
needed, he or she could choose to view the mini video clips (described be-
low). Each topic included from 2 to 6 summary sheets. The intent was to
focus on smaller chunks of information within a topic so that students would
not be overwhelmed.
Within each concept covered by a text-based summary, a set of short
video clips (also known as mini-clips) were available that showed students
how to solve specific problems in real time. In addition, students were asked
to solve a similar problem of their own based on the information provided in
the mini-clip. The benefits of using worked examples are well document-
ed in research on effective teaching strategies (Atikinson, Deryy, Renkl, &
Wortham, 2000; Clark & Mayer, 2008; Crippen & Earl, 2004; Renkl, 2005;
Zhu & Simon, 1987). Each mini-clip had video controls enabling a student
to easily pause, rewind, and fast forward a clip.
Students also had the option of testing their knowledge on a specific
topic using an online assessment system called Maple T.A. The system al-
lows for a variety of question types and supports the incorporation of algo-
rithmic questions, graphs, and free-form responses. Furthermore, Maple
T.A. provides a “scaffolded” approach to learning. The student is taken
through a series of difficulty levels; if a student is struggling at a particular
difficulty level, the student is directed to an easier level to build confidence
and a stronger mathematical foundation before moving on. Feedback is
immediate with full solutions provided, and a summary of the students’
progress at each difficulty level is shown throughout the test. Students can
also access their results at a later time, and can choose to retake the test
as many times as they like, being presented with new questions on each at-
tempt. Test-enhanced learning or the use of practice tests has been shown to
be particularly effective (Angus & Watson, 2009; Kotcherlakota & Brooks,
2008; Roediger & Karpicke, 2006).
Procedure
The learning objects were made available to students three weeks pri-
or to a pre-calculus diagnostic test implemented during the second week
92
Kay and Kletskin
of school. After students received their results, they were asked to fill in
a 10-15 minute survey asking about their use of and attitudes toward each
of the three learning object components: written documentation, mini-clips,
and online-testing (Maple T.A.). The learning objects used by students are
located at http://faculty.uoit.ca/kay/precalc/index2.html and the survey used
to evaluate the impact of the pre-calculus learning objects is available in Ap-
pendix A (Kay & Kletskin, 2009). Participation was voluntary and anony-
mous and in no way impacted a student’s grade.
Data Sources
Background information and learning object use. Students were asked
their gender, program of study, and their Calculus grade in high-school.
Use of learning objects. The number of site visits to each component of
the learning object (written documentation, mini-clips, and online mastery
testing) was recorded by a custom designed tracking tool. Students were
also asked to estimate when and how many times they visited the pre-calcu-
lus learning object. Finally, students were asked how much time they spent
using each of the three learning object components.
Written documentation. Students were first asked if they used the on-
line written documentation and their reasons for this decision. Students who
used the written documentation were asked which topics they reviewed, and
to rate overall usefulness.
Mini-clips. Students were asked if they used the online video clips and
their reasons for this decision. Students who used the mini-clips were asked
which topics they reviewed and to rate overall usefulness. They were then
asked to rate nine features about the mini-clips listed in table 1. The internal
reliability of all scale items (expect for how many times a single clip was
looked at) was 0.84. However, all items were also analysed individually in
order to gain further insights into this new format of learning object.
Online mastery testing (Maple T.A.). Students were asked if they used
online mastery testing and their reasons for this decision. Students who
used the mastery tests were asked which topics they reviewed, and to rate
overall usefulness. They were then asked to rate seven qualities of the on-
line mastery testing tool (see table 1). The internal reliability of all scale
items was 0.87. However, all items were also analysed indepedently in or-
der to gather detailed information on this part of the learning object.
Evaluating the Use of Learning Objects for Improving Calculus Readiness
93
Table 1
Description of Pre-Calculus Learning Object Survey
Background Information
1. Gender
2. Program of Study
3. Calculus Grade in High School
Use of Learning Object
1. Online tracking of website use (written documentation, mini clips, online mastery testing)
2. When did you visit the pre-calculus learning object ( 1 week before classes, first week of
classes, just before the test, after the test)
3. How many times did you use the pre-calculus learning objects?
4. How much time (in minutes) was spent using each of the three components of the learning
objects (written documentation, mini clips, online mastery testing).
Written Documentation
1. Did you use the written documentation? Why or why not?
2. Which topics were reviewed?
3. How useful was the written documentation (Not at all, Somewhat, Useful, Very Useful)
Mini-Clips
1. Did you use the mini-clips? Why or why not?
2. Which topics were reviewed?
3. How useful were the mini clips (Not at all, Somewhat, Useful, Very Useful)
4. Using a 5 point Likert scale (Strongly Disagree to Strong Agree)
a. The clips were easy to follow
b. The math problems were well explained
c. I liked doing the “student problem”
d. The writing in the clips was easy to read
e. Good tips were provided
f. The clips were helpful because I could do them on my own time
g. The clips helped me understand math concepts better
h. I liked using the clips better that a textbook
Online Mastery Testing (Maple T.A.)
1. Did you use Maple T.A.? Why or why not?
2. Which topics were reviewed?
3. How useful was Maple T.A. (Not at all, Somewhat, Useful, Very Useful)
4. Using a 5 point Likert scale (Strongly Disagree to Strong Agree)
a. The Maple TA questions were clear
b. The syntax required was easy to use
c. There was a sufficient variety of questions.
d. The solution provided from Maple TA was helpful.
e. The practice tests were easy to use.
f. I liked being able to try Maple TA questions on my own time
g. I liked the varying levels of question difficulty.
Performance
1. Using a 5 point Likert Scale (Very Weak to Very Strong), five areas of pre and post
pre-calculus knowledge were self-evaluated
a. Functions
b. Solving Equations
c. Linear Functions
d. Exponential and Logarithmic Functions
e. Trigonometric Functions
94
Kay and Kletskin
Performance. Students were asked to assess their pre-calculus knowl-
edge before and after using the learning objects (working with functions,
solving equations, linear functions, exponential and logarithmic functions,
trigonometric functions). They were also asked their pre-calculus diagnos-
tic test score.
Research Questions
The key research questions addressed regarding the use of the pre-cal-
culus learning objects were as follows:
1) When and how often was the pre-calculus learning object used?
2) Were there significant differences among the three learning object
components (written documentation, mini-clips, online mastery
testing) with respect to use?
3) How useful were each of the three learning object components?
4) How did students rate the overall quality of mini-clips?
5) How did students rate the overall quality of the online mastery test-
ing tool (Maple T.A.)?
6) Did student understanding of pre-calculus knowledge improve as a
result of using the learning objects?
RESULTS
Use of pre-calculus learning objects. During the three week period that
the learning object was available to students, 8988 hits and 969 unique visits
were recorded by the tracking program. The average time spent for each
visit was 307.5 seconds (SD = 445.8) and the mean number of items (writ-
ten documentation, mini clips, or Maple T.A. files) looked at each visit was
6.3 (SD = 9.0). Thirty percent (n=80) of the students used the learning ob-
ject one week before classes started , 50 % (n=133) used them during the
first week of classes, 29% (n=77) used them just before the Calculus readi-
ness diagnostic test, and 30% (n=81) used them after the diagnostic test.
The mini clips were selected 4681 times (52%), the written summary
files 4030 times (45%), and the Maple T.A. assessment tool 277 times (3%).
A one-way ANOVA comparing the time spent on each component of the
learning objects was significant (F= 83.8, p <.001). A Scheffe’s post hoc
analysis revealed that students spent the most time using the Maple T.A. fea-
ture (M =523.5 seconds, SD = 604.3), followed by the mini-clips (M =357.7
seconds, SD = 443.7), and then the written documentation (M =237.7 sec-
onds, SD = 426.1).
Evaluating the Use of Learning Objects for Improving Calculus Readiness
95
The pattern of learning object component use was consistent with the
survey results where students reported using mini-clips most often (68%,
n=195), followed by the summary text files (54%, n=154), and the online
testing with Maple T.A. (38%, n=107). With respect to the total time spent
on each component, students reported spending an average of 70.6 (SD =
71.1) minutes using mini-clips, 59.3 (SD = 64.5) minutes reading the writ-
ten summaries, and 44.9 (SD = 43.8) minutes participating in the Maple TA
online testing.
It should be noted that a total of 36 out of 288 students (13%) of the
students who filled in the survey did not use any of the learning object
tools. Reasons given by students for not using written documentation in-
cluded they did not know it was available (n=29 comments), they had no
time (n=28 comments), they preferred the mini clips (n=22 comments) or
they did not need any help (n=8 comments). Reasons offered for not using
the mini-clips included not knowing it was available (n=15 comments), not
needing any help (n=12 comments), having no time (n=14), technological
problems (n=6 comments) or preferring written documentation (n=4 com-
ments). Finally, reasons for not using the online mastery tool were as fol-
lows: not knowing it was available (n=46), not knowing how to use the tool
(n=25 comments), having no time (n=22 comments), not needing any help
(n=8 comments), preference for another method (n=7 comments).
Usefulness of learning object components. Overall, students rated
the mini clip (87%), written summary (79%) and Maple T.A. online test-
ing (69%) components of the learning objects as useful or very useful. A
one-way ANOVA comparing the usefulness among learning object compo-
nents was significant (F = 13.2, p < .001). The Scheffe’s post hoc analysis
showed that mini-clips (M =3.38, SD = 0.72) were rated significantly more
useful than written documentation (M =3.12, SD = 0.75) or the online mas-
tery testing (M =2.93, SD = 0.82).
With respect to the mini-clips, students commented that the videos were
clear, easy to follow, well explained, provide useful visual supports, and
helped improve understanding. They also enjoyed being able to proceed at
their own pace. Student comments about the written documentation indi-
cated that this feature was easy to use and follow, improved understanding,
and provided a helpful review of concepts. Regarding the online mastery
tool, students noted that it helped them understand and review concepts, and
provided helpful practice to re-enforce concepts.
Evaluation of mini-clips. Students rated the mini-clips highly (Likert
scale from 1 to 5) noting that they were easy to follow, helpful because they
could be viewed on the student’s own time, and well explained. Students
also preferred the mini-clips to working through a textbook. A summary of
the mini-clip ratings is provided in table 2.
96
Kay and Kletskin
Table 2
Summary of Rating for Mini-Clips
Item Mean Standard
Deviation
a) The clips were easy to follow 4.46 0.68
b) The math problems were well explained in the clips. 4.31 0.74
c) I liked doing the “student problem” in the clips. 3.87 1.02
d) The writing in the clips was easy to read. 4.30 0.75
e) Good tips were provided in the clips to help me understand
the problem.
4.10 0.76
f) The clips were helpful because I could do them on my own
time.
4.40 0.74
g) The clips helped me understand math concepts better. 4.16 0.77
h) I liked using the clips better that using a textbook to work through
examples.
4.17 0.99
Evaluation of online mastery testing (Maple T.A.). When rating the
online mastery testing tool, students rated the following qualities relatively
high: varying level of question difficulty, being able to use the tool on their
own time, clarity of questions, and the variety of questions. Ease of use and
the syntax required to use this component were rated somewhat lower than
the other features. A summary of the online mastery tool ratings is provided
in table 3.
Table 3
Summary of Rating for Online Mastery Testing Tool (Maple T.A.)
Item Mean Standard
Deviation
a) I liked the varying level of question difficulty offered
by Maple T.A.
3.86 0.83
b) I liked being able to try Maple T.A. questions on my
own time.
3.84 0.91
c) The Maple T.A. questions were clear.
3.82 0.74
d) There was a sufficient variety of Maple T.A. questions
available for me to practice on.
3.76 0.80
e) The solution provided from Maple T.A. was helpful.
3.66 0.87
f) I found Maple T.A. practice tests easy to use.
3.63 0.94
g) The syntax required to give answers to Maple T.A.
questions was easy to use.
3.32 1.14
Evaluating the Use of Learning Objects for Improving Calculus Readiness
97
Impact of pre-calculus learning object on understanding. While a for-
mal pre-post test analysis was not conducted, students self-rated five areas of
pre-calculus knowledge before and after using the learning objects. Paired
t-tests revealed significant gains in all five pre-calculus knowledge areas as-
sessed. The effect sizes (based on Cohen’s d) ranged from 0.14 to 0.44 and
are considered to be small to moderate (Thalheimer & Cook, 2002).
Table 4
Pre vs. Post Ratings of Pre-Calculus Knowledge (n=252)
Topic Pre-LO
Mean (SD)
Post-LO
Mean (SD)
Significance
t value
Cohen’s d
Operations with Functions 3.36 (0.90) 3.63 (0.82) - 5.58 ** 0.31
Solving Equations 3.61 (0.91) 3.74 (0.83) - 2.44 * 0.14
Linear Functions
3.49 (0.97) 3.72 (0.84) - 4.26 ** 0.25
Exp. and Log Functions 2.89 (0.96) 3.27 (0.84) - 6.85 ** 0.42
Trigonometric Functions 2.96 (1.01) 3.38 (0.89) - 7.56 ** 0.44
* p < .05
** p < .001
Correlations between use of written summaries or mini clips and self-
reported changes in pre- calculus knowledge were positive and significant
for all five pre-calculus concepts. Correlations between the use of the on-
line mastery tool and change in pre-post test scores were not significant for
any of the five pre-calculus concepts assessed (Table 5).
Table 5
Correlation between Use of Learning Objects Components and
Change in Pre-Calculus Knowledge (n=288)
Topic Written Summaries Mini Clips Online Testing
Operations with Functions 0.21 ** 0.27 ** 0.08
Solving Equations 0.20 ** 0.14 * 0.04
Linear Functions 0.17 ** 0.22 ** 0.09
Exp. and Log Functions 0.23 ** 0.27 ** - 0.01
Trigonometric Functions 0.20 ** 0.25 ** 0.06
* p < .05
** p < .01
A regression analysis assessing the relative contributions of the three
learning tools indicated that the impact of mini-clips and written documen-
98
Kay and Kletskin
tation were both significant predictors of the total change in pre-calculus
knowledge. This means that the impact of these two tools appears to be
cumulative. In other words, the contribution of either mini-clips or written
documentation in isolation is less than the combined contribution of both
tools.
Students who used the written summaries noted that they provided a
good overview of concepts (30 out of 102 comments), helped with under-
standing and review (56 out of 102 comments), and were useful (16 out of
102 comments). Students who viewed the mini clips reported that the step-
by-step explanations were helpful (40 out of 149 comments), the clips were
helpful for understanding and reviewing concepts (51 out of 149 comments),
the visual representations helped learning (30 out of 149 comments), and
that the tool was useful (21 out of 149 comments). Finally, students who
chose to engage in online mastery testing thought Maple T.A. was helpful
for understanding and reviewing concepts (30 out of 68 comments), believed
they were required to learn this tool for the course (22 out of 68 comments),
felt Maple T.A. was useful (9 out of 68 comments) or simply wanted to ex-
plore the program (7 out of 68 comments).
DISCUSSION
Overall Use of Pre-Calculus Learning Objects
One can be reasonably confident that the students in this study were ac-
tive participants in using the learning object tools. Even though they were
introduced to the clips before class officially started and the concepts cov-
ered were not being directly evaluated for marks, the learning object web
site received almost 9000 hits in a three week period. In addition, only 13%
of those students surveyed did not use any of the tools. It appears that the
use of learning objects is an attractive option for students who want to up-
grade their mathematics skills.
Differences in Use Among Learning Object Components
With respect to the three tools offered, students preferred using interac-
tive mini-clips most followed by the written documentation. This appears
to be a reasonable choice for students who are reviewing concepts they may
have forgotten. It seems unlikely that a student would jump directly to an
online mastery test, unless he/she felt confident in his/her knowledge.
Evaluating the Use of Learning Objects for Improving Calculus Readiness
99
If one sets aside the frequency of use, it is interesting to note that stu-
dents spent the most time using the online mastery tool, followed by the
mini clips and then the written documentation. The online mastery tool is
the most complicated to learn and would require extra time to answer a wide
range of review questions. The mini-clips would take extra time too as the
students had to view clips and complete the student problems. Written doc-
umentation could be downloaded quickly and read offline. More research
is needed, perhaps in the form of interview or focus groups to confirm the
reasons why students selected specific tools.
Usefulness of Learning Object Components
Overall, the vast majority of students rated all three learning tools as
useful or very useful. The consensus, based on student comments, was that
the tools provided a useful review and helped to improve understanding. A
comparison among the three learning object tools indicated that almost 90%
of the students thought mini-clips were useful and rated them significantly
higher than written documentation or the online mastery tool. This result
is consistent with previous research reporting that students respond very
positively to the visual supports and interactivity of learning objects (e.g.,
Bradley & Boyle, 2004; Concannon et al., 2005; Lim et al., 2006). Both the
written documentation and online mastery tools were predominantly based
on the display of static text. It is also possible that students appreciated the
auditory component of the mini-clips, a communication mode that was not
available in either the written documentation or online mastery tools.
Evaluation of mini-clips
Students rated all aspects of the mini clips highly, averaging four or bet-
ter on a five-point Likert scale. Top rated features were that the clips were
easy to follow and provided clear explanations. Student also appreciated the
fact that they could use them on their own time. The qualities that students
appreciated in the mini-clips were similar to those reported in previous re-
search on learning objects (e.g., Docherty et al., 2005; Kenny et al., 1999;
Lim et al., 2006; Schoner et al., 2005). Overall, the mini-clips were a highly
rated tool in the pre-calculus learning object.
100
Kay and Kletskin
Evaluation of Online Mastery Testing (Maple T.A.)
Ratings of the online mastery tool were, on average, a half-point to one
point lower than those for the mini-clips. Students liked the varying level
of question difficulty offered and working on their own time. Obstacles,
based on the survey data and student comments, appeared to be the syntax
involved in using this tool. Some students found the tool difficult to use.
Overall, comparatively fewer students used online mastery testing, however
those students who used this tool thought it was effective in terms of testing
their knowledge and practicing questions. More research is needed on how
to encourage students to use this tool.
Impact of Pre-Calculus Learning Object on Understanding
There is considerable indirect evidence that students experienced sig-
nificant gains in the knowledge areas targeted by the learning object. Dif-
ferences in pre and post learning object scores show significant gains for
all five categories of knowledge assessed. The effect size was considered
small to moderate. It is important to note that this data came from student’s
self assessment of their own skills. A more rigorous analysis would have
students do a pre- and post diagnostic test. Nonetheless, the data indicates
that, at the very least, students believe their knowledge of pre-calculus con-
cepts increases.
It is important to note that while the use of written documentation
and mini clips were significantly and positively correlated with change in
pre- and post- scores, the use of online mastery testing was not significant-
ly related to perceived changes in pre-calculus knowledge. One possible
explanation is that the written documentation and mini-clips are designed
to teach whereas the online mastery tool focuses more on assessment. Of
course, students could learn by attempting the online mastery questions and
responding to the feedback given, but perhaps this more indirect form of in-
struction was not as effective. Interview data would be helpful in sorting
out why online mastery tools did not appear to have a short-term impact on
improving pre-calculus knowledge.
Finally, the impact of mini-clips and written documentation appears
to be complementary. In other words, students did better if they used both
tools as opposed to only one. However, adding the use of the online mastery
tool to the use of mini clips and the written documentation did not signifi-
cantly improve pre-calculus knowledge in this study.
Evaluating the Use of Learning Objects for Improving Calculus Readiness
101
Caveats and Future Research
This study is a first attempt at using learning objects to address the def-
icits in pre-calculus that students have when entering a first year calculus
course. A large sample combined with triangulation of data collection tools
was used to assess the impact of learning objects. Nonetheless, several ca-
veats remain that are worth addressing for future research. First, students
were not interviewed in this study. This type of data would have been par-
ticularly useful in providing information on why online mastery tools were
used less frequently and appeared to have a minimal impact. Second, a for-
mal pre- and post- test on precalculus concepts would provide stronger data
on supporting the effectiveness of the learning objects used in this study.
Third, since this study was a formative analysis, the reliability and validity
of the survey instruments were not provided. Fourth, students had limited
time to use the learning objects prior to the start of classes. It would be in-
teresting to investigate whether earlier access to the learning objects would
result in increased use of the online mastery tool, and perhaps a larger im-
pact on student pre- and post- learning object scores.
Summary
Previous research suggests that students have a weak understanding of
the kinds of mathematical concepts required to be successful in university
Calculus courses. A set of learning objects was designed to help students
augment their pre-calculus knowledge. Three tools were used: written doc-
umentation, interactive mini-clips, and online mastery testing. Students ac-
cessed and used these tools of almost 9000 times in a three week period.
Most students rated all three tools as useful for review and improving un-
derstanding. Students reported significant pre-calculus knowledge gains as
a result of using mini-clips and written documentation, but not the online
mastery tool. Overall, it appears there is evidence to support the effective-
ness of learning objects in improving the pre-calculus knowledge of first-
year university students.
References
Agostinho, S., Bennett, S., Lockyer, L., & Harper, B. (2004). Developing a
learning object metadata application profile based on LOM suitable for the
Australian higher education context. Australasian Journal of Educational
Technology, 20(2), 191-208.
102
Kay and Kletskin
Angus, S. D., & Watson, J. (2009). Does regular online testing enhance student
learning in the numerical science? Robust evidence from a large data set.
British Journal of Educational Technology, 40(2), 255-272.
Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from
examples: Instructional principles from the worked examples research. Re-
view of Educational Research, 70(2), 181-214.
Barbeau, M., Bruce, I., Clake, J., Morgan, J., Patry, R. & Porter, M. (1977). The
role of the university with respect to enrolments and career opportunities,
admission policies, continuing education and community colleges. Ottawa:
Association of Universities and Colleges of Canada.
Bradley, C., & Boyle, T. (2004). The design, development, and use of multime-
dia learning objects. Journal of Educational Multimedia and Hypermedia,
13 (4), 371-389.
Butson, R. (2003). Learning objects: weapons of mass instruction. British Jour-
nal of Educational Technology, 34 (5), 667-669.
Clark, R. C., & Mayer, R. E. (2008). E-Learning and the Science of Instruction.
San Francisco: Pfeiffer.
Concannon, F., Flynn, A. & Campbell, M. (2005). What campus-based students
think about the quality and benefits of e-learning. British Journal of Educa-
tional Technology, 36 (3), 501-512.
Crippen, K. J., & Earl, B. L. (2004). Considering the effectiveness of web-based
worked example in introductory chemistry. Journal of Computers in Math-
ematics and Science Teaching, 23(2), 151-167.
De Salas, K. & Ellis, L. (2006). The development and implementation of learn-
ing objects in a higher education. Interdisciplinary Journal of Knowledge
and Learning Objects, 2006 (2), 1-22.
Docherty, C., Hoy, D., Topp, H., Trinder, K. (2005). E-Learning techniques
supporting problem based learning in clinical simulation. International
Journal of Medical Informatics, 74 (7-8), 527-533.
Duval, E., Hodgins, W., Rehak, D., & Robson, R. (2004). Learning objects sym-
posium special issue guest editorial. Journal of Educational Multimedia and
Hypermedia, 13 (4), 331-342.
Gadanidis, G., Gadanidis, J. & Schindler, K. (2003). Factors mediating the use
of online applets in the lesson planning of pre-service mathematics teach-
ers. Journal of Computers in Mathematics and Science Teaching, 22(4),
323-344.
Howard-Rose, D. & Harrigan, K. CLOE learning impact studies lite: Evaluat-
ing learning objects in nine Ontario university courses. Retrieved July 3,
2007 from
http://cloe.on.ca/documents/merlotconference10.doc
Engineering Council. (2000). Measuring the mathematics problem. London:
Engineering Council.
Evaluating the Use of Learning Objects for Improving Calculus Readiness
103
Jourdan, N., Cretchley, P., & Passmore, T. (2007). Secondary-tertiary transi-
tion: what mathematics skills can and should we expect this decade? In J.
Watson & K. Beswick (Eds.), Mathematics: Essential Research, Essential
Practice (Vol. 2, pp. 463 - 472). Adelaide, Australia: Mathematics Educa-
tion Research Group of Australasia Inc.
Kay, R. H., & Kletskin, I. (2009). Appendix A – Pre-Calculus Survey retrieved
Oct 14
th
, 2009 at
http://faculty.uoit.ca/kay/res/pcs/PreCalcSurvey.pdf
Kenny, R. F., Andrews, B. W., Vignola, M. V., Schilz, M. A., & Covert, J. (1999).
Towards guidelines for the design of interactive multimedia instruction:
Fostering the reflective decision-making of preservice teachers. Journal of
Technology and Teacher Education, 7(1), 13-31.
Kotcherlakota, S., & Brooks, D. W. (2008). A test of strategies for enhanced
learning of AP descriptive chemistry. Journal of Science Education and
Technology, 17(4), 297-304.
Lawson, D.A. (2003). Changes in student entry competencies 1991 - 2001.
Teaching Mathematics and Its Applications, 22 (4), 171-175.
Lim, C.P., Lee, S.L. & Richards, C. (2006). Developing interactive learning
objects for a computing mathematics models. International Journal on E-
Learning, 5 (2), 221-244.
London Mathematical Society. (1995). Tackling the Mathematics Problem.
London: LMS.
MacDonald, C. J., Stodel, E., Thompson, T. L., Muirhead, B., Hinton, C., Car-
son, B., et al. (2005). Addressing the eLearning contradiction: A collabora-
tive approach for developing a conceptual framework learning object. In-
terdisciplinary Journal of Knowledge and Learning Objects, 1. Retrieved
August 2, 2005 from
http://ijklo.org/Volume1/v1p079-098McDonald.pdf
.
Mason, R., Pegler, C., &Weller, M. (2005). A learning object success story.
Journal of Asynchronous Learning Networks, 9(1). Retrieved June 1, 2007
from
http://www.sloan-c.org/publications/jaln/v9n1/v9n1_mason.asp
.
McGreal, R. (2004). Learning objects: A practical definition. International Jour-
nal of Instructional Technology and Distance Learning, 1(9). Retrieved Au-
gust 5, 2005 from
http://www.itdl.org/Journal/Sep_04/article02.htm
.
Mustoe, L. & Lawson, D. (eds). (2002). Mathematics for the European engi-
neer: A curriculum for the twenty-first century. A report by the SEFI Math-
ematics Working Group, SEFI HQ, Brussels, Belgium. Retrieved Aug 23,
2009 from
http://learn.lboro.ac.uk/mwg/core/latest/sefimarch2002.pdf
Ontario Ministry of Education. (2006). Report of the minister’s task force on
senior high school mathematics. Retrieved Aug 23, 2009 from
http://www.
edu.gov.on.ca/eng/document/reports/seniormath/
Parrish, P. E. (2004). The trouble with learning objects. Educational Technology
Research & Development, 52 (1), 49-67.
Polsani, P. R. (2003). Use and abuse of reusable learning objects. Journal of
Digital Information, 3(4), Retrieved July 1, 2007 from
http://journals.tdl.org/
jodi/article/view/jodi-105/88
104
Kay and Kletskin
Renkl, A. (2005). The worked-out examples principle in multimedia learning.
In R. E. Mayer (Ed.) The Cambridge Handbook of Multimedia Learning.
New York: Cambridge University Press.
Rehak, D. & Mason, R. (2003). Chapter 3: Keeping the Learning in Learning
Objects. Journal of Interactive Media in Education, 2003 (1). Retrieved
July 1, 2005 from
http://www-jime.open.ac.uk/2003/1/reuse-05.html
.
Rieber, L. P., Tzeng, S., & Tribble, K. (2004). Discovery learning, representa-
tion, and explanation within a computer-based simulation: Finding the right
mix. Learning and Instruction, 14(3), 307-323.
Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning – Taking
memory tests improves long term retention, Psychological Science, 17 (3),
249-255.
Schoner, V., Buzza, D., Harrigan, K. & Strampel, K. (2005). Learning objects
in use: ‘Lite’ assessment for field studies. Journal of Online Learning and
Teaching, 1 (1), 1-18.
Sedig, K & Liang, H (2006). Interactivity of visual mathematical representa-
tions: Factors affecting learning and cognitive processes. Journal of Inter-
active Learning Research. 17(2), 179-212.
Thalheimer, W., & Cook, S. (2002). How to calculate effect sizes from published
research articles: A simplified methodology. Retrieved July 14, 2007 from
http://work-learning.com/effect_sizes.htm
.
Van Zele, E., Vandaele, P., Botteldooren, D., & Lenaerts, J. (2003). Implementa-
tion and evaluation of a course concept based on reusable learning objects.
Journal of Educational Computing and Research, 28(4), 355-372.
Wiley, D. A. (2000). Connecting learning objects to instructional design theory:
A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The In-
structional Use of Learning Objects: Online Version. Retrieved July, 1,
2005, from
http://reusability.org/read/chapters/wiley.doc
.
Wiley, D., Waters, S., Dawson, D., Lambert, B., Barclay, M., & Wade, D. (2004).
Overcoming the limitations of learning objects. Journal of Educational
Multimedia and Hypermedia, 13 (4), 507-521.
Windschitl, M., & Andre, T. (1998). Using Computer Simulations to Enhance
Conceptual Change: The Roles of Constructivist Instruction and Student
Epistemological Beliefs. Journal of Research in Science Teaching, 35(2),
145-160.
Zhu, X., & Simon, H. A. (1987). Learning mathematics from examples and by
doing. Cognition and Instruction, 4, 137-166.
