Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay on Sep 27, 2024
Content may be subject to copyright.
Research and Practice in Technology Enhanced Learning
Vol. 7, No. 2 (2012) 89－104
Ó Asia-Pacific Society for Computers in Education
89
EXPLORING INDIVIDUAL DIFFERENCES IN THE IMPACT OF WEB-BASED
LEARNING TOOLS (WBLTS)
ROBIN KAY
University of Ontario Institute for Technology
Oshawa, Ontario, Canada
robin.kay@uoit.ca
http://faculty.uoit.ca/kay/home
The purpose of this study was to explore individual differences in middle and secondary school
student attitudes and learning performance regarding Web-Based Learning Tools (WBLTs). The
student characteristics assessed were gender, age, computer comfort level, subject comfort level, and
average grade. Attitudes toward WBLTs were measured using a reliable, valid survey designed to
gather data on student perceptions of learning, design, and engagement. Learning performance was
assessed by comparing pre- and post-test scores on four knowledge categories (remembering,
understanding, application, analysis) based on the revised Bloom’s taxonomy. Female students had
significantly more positive attitudes toward WBLTs. Students who were more comfortable with
using computers and the subject area addressed by a WBLT had significantly more positive attitudes
toward WBLTs. Average grade was unrelated to student attitudes toward WBLTs. Student age was
the only student characteristic that was significantly associated with learning performance. When
older students use WBLTs (different from those used by younger students), learning performance is
significantly greater than younger students. It is speculated that WBLTs may be better suited toward
older students who have better self-regulation skills.
Keywords: Evaluate; attitudes; performance; individual differences; secondary school; middle school;
learning objects; web-based learning tool.
1. Introduction
Web-Based Learning Tools (WBLTs), also known as learning objects, are operationally
defined in this study as interactive web-based tools that support the learning of specific
concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners
(Kay & Knaack, 2008b, 2009). Recent evidence suggests that the impact of WBLTs on
student attitudes and learning performance is positive in middle and secondary school
classrooms. Students report that WBLTs are engaging (e.g. Kay, 2009; Kay & Knaack,
2005, 2007a, 2007b), enjoyable (Clarke & Bowe, 2006a, 2006b; Kay, 2009; Reimer &
Moyer, 2005) and easy to control with respect to the pace of learning (Clarke & Bowe,
2006b; Docherty, Hoy, Topp, & Trinder, 2005; Kay, 2009; Reimer & Moyer, 2005).
They also note that WBLTs provide timely feedback (Brown & Voltz, 2005; Reimer &
Moyer, 2005), include a wide range of motivating multimedia (Clarke & Bowe, 2006b;
Kay & Knaack, 2007a, 2007b, 2009), and help them learn (Bradley & Boyle, 2004; de
Salas & Ellis, 2006; Kay, 2009; Kay & Knaack, 2007a, 2007b, 2009; Lim, Lee, &
90 R. H. Kay
Richards, 2006; MacDonald et al., 2005; Schoner, Buzza, Harrigan, & Strampel, 2005).
In addition, considerable evidence suggests that student learning performance improves
when WBLTs are used (Akpinar & Bal, 2006; Bower, 2005; Docherty et al., 2005; Kay
& Knaack, 2007a, 2007b, 2009; Kong & Kwok, 2005, Liu & Bera, 2005; Nurmi &
Jaakkola, 2006; Reimer & Moyer, 2005; Rieber, Tzeng, & Tribble, 2004; Windschitl &
Andre, 1998).
However, with the exception of Kay and Knaack (2008a), limited research has been
conducted on student characteristics and the use WBLTs in the classroom. Examining the
impact of these characteristics is important for at least two reasons. First, the data could
be used to alter the conditions and strategies for using WBLTs, thereby helping
instructors to adapt to the individual needs of students. Second, analysis of student
characteristics might provide a more comprehensive model for understanding the WBLT
learning environment. Blanket statements about the effectiveness of WBLTs in middle
and secondary school classrooms are a starting point, however a potentially more
productive question for both educators and researchers is, “Which student characteristics
influence student attitudes toward WBLTs and learning performance?”
2. Individual Differences and WBLTs - Students
A previous content analysis of 183 peer-reviewed articles conducted by Kay and Knaack
(2009) revealed only seven papers mention individual differences and WBLTs, and no
studies directly examine the impact of student characteristics. Five areas of potential
differences in the use and impact of WBLTs included gender, age, computer comfort
level, subject comfort level and ability. This is not meant to be a comprehensive list of
student characteristics, but rather a reasonable starting point based on previous research.
2.1. Gender
Extensive research has looked at gender differences in computer-related behavior (see
American Association of University Women (AAUW), 2000; Barker & Aspray, 2006;
Kay, 2008; Sanders, 2006; Whitley, 1997 for detailed reviews of the literature). Overall,
there is a persistent pattern of small, but statistically significant differences in computer
attitude, ability, and use that typically favors males, however considerable variability
exists. Therefore, it is reasonable to examine gender differences in any new computer-
based technology to determine the impact of potential gender biases.
Limited research has been conducted on gender differences and the use of WBLTs -
only two peer-reviewed studies could be found (Kay & Knaack, 2007b, 2008a). Kay and
Knaack (2007b, 2008a) reported no significant differences between male and female
secondary school students’ attitudes and learning performance when WBLTs were used.
More data is needed though, particularly for middle school students, to either conform or
deny a gender effect with WBLTs.
Exploring Individual Differences in the Impact 91
2.2. Age
A number of researchers have explored differences in computer attitude and ability
among various age groups. In several studies, older students (15-16 years old) viewed
computers as tools for getting work done (e.g. word processing, programming, use of the
Internet, and email), whereas younger students (11-12 years old) saw computers more as
a source of entertainment (e.g. play games and use graphics software) (Colley, 2003;
Colley & Comber, 2001; Comber, Colley, Hargreaves, & Dorn, 1997). On the other hand,
Harris and Granfgenett (1996) and Kubek, Miller-Albrecht, and Murphy (1999) observed
that age had a negligible effect on computer attitudes.
With respect to WBLTs, Kay and Knaack (2007b, 2008a) reported that older students
(Grade 12) were more positive about WBLTs and performed better than younger students
(Grade 9 and 10). De Salas & Ellis (2006) added that second and third year university
students were far more open to using WBLTs than first year students. Since WBLTs were
originally designed for higher education students (Haughley & Muirhead, 2005), it is
conceivable that they might not work as well for younger students, who may be less
prepared to engage in the rigors of self-guided discovery. More research is needed to
assess the influence of age on using WBLTs, particularly at the middle and secondary
school levels.
2.3. Computer comfort
Computer comfort, also referred as self-efficacy and confidence, has been studied in
some depth with respect to general computer-related behavior (Barbeite & Weiss, 2004;
Christensen & Knezek, 2000; Durndell & Haag, 2002; Liu, Hsieh, Cho, & Schallert,
2006; Torkzadeh, Pflughoeft, & Hall, 1999). A greater comfort level with computers is
typically associated with a higher level of computer ability and/or use.
Regarding WBLTs, attitudes toward learning value, design, and engagement were
positively and significantly correlated with computer comfort level (Kay & Knaack, 2005,
2007b, 2008a). Lim et al. (2006) added in a case study, that students who were not
comfortable with computers used WBLTs less. These studies, though, focused on older
students, so the results may not apply to their younger counterparts. Because many
WBLTs are easy to use (e.g. Haughey & Muirhead, 2005; MacDonald et al., 2005; Nesbit
& Belfer, 2004) and most of today’s net generation feel comfortable with computers (e.g.
Montgomery, 2009; Palfrey & Gasser, 2008; Tapscott, 2008), it is speculated that
computer comfort level should have a minor impact on the perceptions and use of
WBLTs with younger students.
2.4. Subject comfort level and ability
Two variables that have not been examined with respect to WBLTs are student comfort
level and ability in the subject area that a particular WBLT covers. When students are not
comfortable with a subject or do not feel competent, motivation is typically low (Cross &
Steadman, 1996; Wlodkowski, 2008) and they may not be particularly receptive to
92 R. H. Kay
WBLTs when they are introduced. A negative reaction to WBLTs, then, might reflect a
feeling of incompetence in the subject area being addressed. Furthermore, the additional
cognitive load of having to use a new learning tool combined with weak subject-area
knowledge may result in an overall negative learning experience (Chandler & Sweller,
1991). Kay and Knaack (2008a) recommended that subject area comfort level and
aptitude be examined when looking at individual differences and WBLTs.
2.5. Summary and purpose
In summary, five potential influences, gleaned from previous research on computer-
related behavior, use of WBLTs, and motivational learning theory include gender, age,
computer comfort level, subject area comfort level, and subject aptitude. The purpose of
this study was to examine the influence of five student-based characteristics (gender, age,
computer comfort level, subject comfort level, and average grade) on attitudes toward
WBLTs and learning performance.
3. Method
3.1. Overview
After conducting an extensive review of WBLT research, Kay and Knaack (2009) noted
at least three areas of concern: (a) focusing on a narrow range of WBLTs (e.g. Bradley &
Boyle, 2004; Krauss & Ally, 2005; MacDonald et al., 2005), (b) small and poorly
described sample populations (e.g. Cochrane, 2005; Krauss & Ally, 2005; MacDonald et
al., 2005; Van Zele, Vandaele, Botteldooren, & Lenaerts, 2003), and (c) a noticeable
absence of reliability and validity in data collection tools (e.g. Howard-Rose & Harrigan,
2003; Lopez-Morteo & Lopez, 2007; Schoner et al., 2005; Vacik, Wolfslehner, Spork, &
Kortschak, 2006; Vargo, Nesbit, Belfer, & Archambault, 2003).
Three additional problems not mentioned by Kay and Knaack (2009) included
random selection and use of WBLTs with no theoretical structure or guidance, a wide
range of loosely described teaching strategies employed, and the application of general,
non-standardized measures of learning performance.
To address these methodological challenges, a wide range of WBLTs were used,
reliable and valid measurement tools were included, a database of pre-selected WBLTs
was created, pre-designed lesson plans were developed by experienced teachers, and an
enhanced measure of learning performance was custom designed for each WBLT.
3.2. Sample
3.2.1. Students
This sample was comprised of 834 middle (n=444) and secondary (n=390) school
students (392 males, 441 females, 1 missing data), 11 to 17 years of age (M = 13.3, SD =
0.97). Most students were enrolled in grades seven (n=229), eight (n=215), and nine
Exploring Individual Differences in the Impact 93
(n=340). Over three quarters (n=628) of the students reported that their average mark was
70% or more in the subject area where the WBLT was used. In addition, over 75% of the
students agreed that they were good at working with computers. The sample population
was selected from 25 middle and 20 secondary school classes located in a sub-urban
region of nearly 600,000 people.
3.2.2. Teachers
This sample included 28 teachers (8 males, 20 females) who taught mathematics (n=15)
or science (n=13) in grades seven (n=9), eight (n=9), nine (n=7) or ten (n=2). Class size
ranged from 9 to 28 with a mean of 18 students (SD= 5.4). Teaching experience varied
from 0.5 to 23 years with a mean of 7.1 (SD= 6.7). Twenty-three out of 28 teachers
agreed that they (a) were good at working with computers and (b) liked working with
computers at school.
3.2.3. WBLT selection and lesson plan design
Four teachers (not involved in the study) were hired and trained for two days on how to
choose WBLTs for the classroom and develop effective lesson plans. WBLTs were
selected based on Kay and Knaack’s (2008b) multi-component model for assessing
WBLTs. Lesson plans were designed from previous research on effective teaching
strategies for using WBLTs (Kay, Knaack, & Muirhead, 2009). Key features of each
lesson plan included a guiding set of questions, a structured well-organized format for
using the WBLTs, and time to consolidate concepts learned. All lessons were designed to
be approximately 70 minutes in length with 10 minutes for introduction, 50 minutes for
WBLT use, and 10 minutes for consolidation. In order to minimize the impact of
extraneous teaching and learning variables, pre-tests were given immediately before the
lesson and post-tests were given right after.
Over a period of 2 months, a database of 122 lesson plans and WBLTs was created
(78 for mathematics and 44 for science). A total of 22 unique WBLTs were chosen from
the WBLT database and used by classroom teachers in this study. A wide variety of
WBLTs were used involving experimentation, virtual manipulatives, task-based
applications, and formal presentation of concepts followed by a question and answer
assessment. See Kay (2011) for links to all WBLTs, detailed descriptions of the lesson
plans, and pre/post tests used in this study.
3.3. Procedure
Teachers from two boards of education were emailed by an educational coordinator and
invited to participate in WBLT study. Participation was voluntary and teachers could
withdraw at any time. Each participant received a full day of training on using and
implementing the pre-designed WBLT lesson plans. They were then asked to use at least
one WBLT in their classroom. Email support was available for the duration of the study.
All students in a given teacher’s class used the WBLT that the teacher selected, however,
94 R. H. Kay
only those students with signed parental permission forms were permitted to fill in an
anonymous, online survey (Appendix A). Students also completed pre- and post-tests
based on the content of the WBLT.
3.4. Data sources
3.4.1. Explanatory variables
Five student-based explanatory variables were examined in this study: gender, age,
computer comfort level, subject comfort level, and average grade in subject area
associated with the WBLT used. Computer comfort was assessed using a scale developed
by Kay and Knaack (2005) which showed good construct validity and reliability. The
internal reliability for the computer comfort scale was 0.82. Subject comfort level was
assessed using two questions asking students about their ability and attitude regarding the
WBLT subject area. The internal reliability for the subject comfort scale was 0.77.
Finally, students were asked to estimate their average grade in the subject area where the
WBLT was used.
3.4.2. Response variables
Two categories of response variables were used in this study: student attitudes toward
WBLTs and learning performance. Student attitudes were assessed using the WBLT
Evaluation Scale for Students. This scale consisted of 13, seven-point Likert scale items
asking students about their perceptions of how much they had learned (learning construct
- 5 items), the design of the WBLT (design construct - 4 items) and how much they were
engaged when using the WBLT (engagement construct - 4 items). According to Kay and
Knaack (2009), the scale displayed good internal reliability, construct validity,
convergent validity, and predictive validity (see Appendix A for the scale items).
To assess learning performance, students were asked to complete a pre- and post-test
based on the content of the WBLT used in class. These tests were included with all pre-
designed lesson plans to match the learning goals of the WBLT. All tests consisted of two
to six questions worth a total of five to eight marks. The type of questions varied
according to the learning goals of the WBLT and included open-ended, short-answer,
multiple choice, fill in the blank, and application questions. All pre- and post-tests with
scoring rubrics are provided in Kay (2011).
The difference between pre- and post-test scores was used to determine changes in
student performance on four possible knowledge areas: remembering, understanding,
application, and analysis. These categories were derived from the revised Bloom’s
Taxonomy (Anderson & Krathwhol, 2001). The number of Bloom’s knowledge
categories assessed varied according to the learning goals and type of the specific WBLT
used.
Exploring Individual Differences in the Impact 95
3.5. Research questions
In order to examine individual differences in the impact of WBLTs on middle and
secondary school students, the following questions were addressed in the data analysis:
(1) Are student gender, age, computer comfort level, subject area comfort level, or
average grade significantly related to student perceptions of learning, quality, or
engagement for WBLTs?
(2) Are student gender, age, computer comfort level, subject area comfort level, or
average grade significantly related to learning performance?
3.6. Data analysis
3.6.1. Gender differences
To assess whether gender had an impact on student perceptions of WBLTs, a MANOVA
was run for gender and the three constructs assessing student perceptions of WBLTs. Age,
computer comfort level, subject area comfort level, and average grade in subject area
were entered as covariates to ensure that any differences observed were due to gender.
To evaluate the relationship between gender and learning performance, independent t-
tests were run on the four learning performance measures. While a MANOVA is
generally considered a better test when multiple response variables are assessed, most
WBLTs focused on only one or two knowledge areas, consequently the sample size was
not large enough to evaluate all four knowledge areas simultaneously using a MANOVA.
3.6.2. Age, computer comfort level, subject comfort level, and grades
To examine the relationship between the remaining four student characteristics (age,
computer comfort level, subject area comfort level, grades), student attitudes and learning
performance, simple correlation coefficients were used. This approach was followed
instead of a multiple regression because there was no theoretical background to support a
predictive model (Fields, 2005).
4. Results
4.1. Gender differences
4.1.1. Perceptions of WBLTs
The MANOVA run for student gender and the three constructs assessing student
perceptions of WBLTs revealed that Hotelling’s T was significant (p < .001), therefore
independent comparisons of WBLT quality constructs were analyzed. Female student
attitudes toward the learning value (p < .001), design (p < .001), and engagement (p < .05)
of WBLTs were significantly higher than those of male students. The effect size for these
differences based on Cohen’s d are considered small (Cohen, 1988, 1992) (Table 1).
96 R. H. Kay
Table 1. Student perceptions and learning performance as a function of student gender.
Female Male Test Effect Size
Cohen’s D
M(SD) M (SD)
Perceptions of:
Learning 26.0 (6.3) 24.2 (7.1) F = 17.7 ** 0.27
Design 21.8 (4.7) 20.8 (5.0) F=13.1 ** 0.21
Engagement 20.0 (5.8) 19.2 (6.1) F = 6.0 * 0.13
Learning Perf
(% Change)
Remembering 30.9 (44.3) 25.8 (42.9) t= 1.2 ns ---
Understanding 31.5 (43.7) 41.5 (44.4) t= 1.8 ns ---
Application 17.0 (33.8) 15.7 (27.9) t= 0.4 ns ---
Analysis 39.9 (46.3) 33.5 (50.4) t= 0.6 ns ---
*p < .05 *** p < .001
4.1.2. Learning performance
The independent t-tests revealed no significant gender differences in percent change of
learning performance scores for the remembering, understanding, application, or analysis
knowledge area.
4.2. Age differences
Correlations among age and student perceptions of learning were either very small or not
significant (Table 2). In other words, a student’s age was not related to his/her attitudes
toward WBLTs. On the other hand, age was significantly correlated with percent change
in remembering, understanding, application and analysis knowledge areas (Table 2).
Older students in higher grades (using different WBLTs) performed better than younger
students in lower grades.
Table 2. Age, computer comfort, subject-area comfort, average grade correlated with
student perceptions and learning performance.
Age Computer
Comfort Subject-Area
Comfort Average
Grade
Perceptions of
Learning (n=822) 0.08 * 0.29 *** 0.35 *** 0.03
Design (n=827) 0.05 0.27 *** 0.44 *** 0.12 **
Engagement (n=825) 0.06 0.29 *** 0.45 *** 0.06
Learning Performance
Remembering (n=421) 0.34 *** - 0.09 - 0.08 - 0.01
Understanding (n=254) 0.21 ** - 0.04 - 0.02 - 0.02
Application (n=422) 0.24 *** 0.05 0.11 * 0.08
Analysis (n=87) 0.36 *** 0.12 0.19 0.02
*p < .05 ** p < .01 *** p < .001
Exploring Individual Differences in the Impact 97
4.3. Computer comfort level
Correlations among computer comfort and student attitudes toward learning, design, and
engagement of WBLTs were positive and significant (Table 2). Students who were more
comfortable with computers, rated the quality of WBLTs higher in learning, design, and
engagement. Computer comfort, though, was not significantly correlated with changes in
learning performance for any of the four knowledge areas assessed (Table 2).
4.4. Subject area comfort level
Comfort level in the subject area addressed by a WBLT was significantly correlated with
student attitudes toward learning, design, and engagement of WBLTs (Table 2). Students
who were more comfortable with the subject area covered by a WBLT rated the learning,
design, and engagement quality higher. Subject area comfort level was significantly and
positively correlated with the application knowledge area, although the magnitude of the
correlation was small (Table 2). Subject area comfort level was not significantly
correlated with remaining three knowledge areas (remembering, understanding, and
analysis).
4.5. Average grade
The self-reported average grade in the subject area where the WBLT was used was not
significantly correlated with student attitudes about the learning or engagement value of
WBLTs (Table 2). A small but significant positive correlation was observed between
self-reported average grade and ratings of WBLT design. Overall, a student’s average
grade in the subject area covered by the WBLT did not appear to influence student
attitudes toward WBLTs. In addition, average grade was not significantly correlated with
changes in learning performance for any of the four knowledge areas assessed (Table 2).
5. Discussion
This study looked at the impact of student characteristics on middle and secondary
students’ attitudes toward WBLTs and learning performance. Five characteristics were
examined including gender, age, computer comfort level, subject area comfort level, and
self-reported average grade. The influence of each of these variables will be discussed in
turn.
5.1. Gender
Past research on general computer behavior suggests that there would be small, but
significant gender differences in favor of males for perceptions and learning performance
associated with the use of WBLTs (AAUW, 2000; Barker & Aspray, 2006; Kay, 2008;
Sanders, 2006; Whitley, 1997). However, the only two previous studies focusing on
gender differences and WBLTs (Kay & Knaack, 2007b, 2008a) reported that male and
female students had similar attitudes toward WBLTs and performed equally well on pre-
98 R. H. Kay
and post-tests. In the current study, a significant, but small effect was observed with
respect to student attitudes in favor of female students. No significant gender differences
were reported with respect to learning performance.
There are at least two possible explanations for the marginal impact of gender when
using WBLTs. First, a number of studies suggest that WBLTs are very easy to use (e.g.
Kay & Knaack, 2008b, 2009), therefore factors such as anxiety, confidence or ability are
unlikely to influence attitudes or undermine learning performance. The impact of gender
on attitudes toward WBLTs and learning performance may be minimized because the
tools are so easy to use.
A second reason for the limited impact of student gender in this study might reflect a
recent trend citing fewer gender differences in computer-related behaviors for younger
students (Kay, 2008). The ubiquity of computer use for today’s new generation of
students also supports the notion of gender neutrality in computer-related behaviors (e.g.
Montgomery, 2009; Palfrey & Gasser, 2008; Tapscott, 2008).
5.2. Age
Student age appeared to have a negligible impact on student attitudes toward WBLTs.
This result confirms previous research on the impact of age on computer attitudes (e.g.
Harris & Granfgenett, 1996; Kubek et al., 1999) and perceptions of WBLTs (Kay &
Knaack, 2007b, 2008a). In other words, older and younger students view WBLTs the
same with respect to learning, engagement and overall design. Since most of the students
in this study are members of the “net generation” described by Tapscott (2008), it is
speculated that WBLTs are viewed as just another interactive interface in a long list of
Internet tools used on a daily basis.
Age was significantly correlated with all four measures of learning performance.
Older students in higher grades performed significantly better than younger students in
lower grades in remembering, understanding, application, and analysis knowledge areas.
This result was partially supported by research suggesting that older students may be
more serious about using computers for learning rather than entertainment (e.g. Colley,
2003; Colley & Comber, 2003; Comber et al., 1997). Kay and Knaack (2007b, 2008a)
reported a modest, positive age effect on general learning performance, however, the
results in the current study suggest a more robust effect, perhaps because the age range
extended down to middle school.
One explanation for the impact of age on learning performance might involve student
expectations as well as the range of cognitive skills required to use a WBLT including
reading instructions, writing down results, interpreting and digesting “what-if” scenarios,
and working independently. Younger students expecting to be entertained, might be
surprised and even overwhelmed by how much effort is required to learn with WBLTs.
5.3. Computer comfort level
Students who were more comfortable with computers rated WBLTs higher for learning,
engagement, and design. In other words, they appeared to have more positive attitudes
Exploring Individual Differences in the Impact 99
toward WBLTs than their less able peers. However, computer comfort level was not
significantly correlated with any of the four learning performance measures. This result is
congruent with previous research (e.g. Kay & Knaack, 2005, 2007b, 2008a) and confirms
while that students who are not as comfortable with computers may not enjoy using
WBLTs as much as their more confident peers, learning performance is largely
unaffected.
5.4. Subject area comfort level
The results for the impact of subject area comfort level mirrored those of computer
comfort level. Students who were more at ease with the subject area covered by a WBLT,
rated the WBLT higher, but did not perform better than their less comfortable peers on
three of the four learning performances measures. While motivation learning theory
(Cross & Steadman, 1996; Wlodkowski, 2008) suggests that subject comfort level would
have a positive impact on learning, it seems that with WBLTs, the influence is less
pronounced and is reflected by student attitudes toward WBLTs, not performance.
Further research is needed to determine the rigor of these findings.
5.5. Average grade
Overall, the average grade of a student in the WBLT subject area was unrelated to
attitudes toward WBLTs and the four measures of learning performance. As stated earlier,
because most of the students in this study are exposed to a steady diet of web activity (e.g.
Montgomery, 2009; Palfrey & Gasser, 2008; Tapscott, 2008), it is reasonable to assume
relatively uniform acceptance WBLTs regardless of average grade. This assumption is
supported by the fact that there is no significant correlation between average grade and
computer comfort level.
It is surprising that average grade was not correlated with learning performance. One
would predict, in general, that students with higher grades would perform better than
students with lower grades. It appears that WBLTs, as they were used in this study,
minimized the impact of average grade. Using WBLTs may be a teaching approach that
helps level the academic playing field.
6. Implications for Education
This study is one of the first comprehensive efforts to explore the influence of student
characteristics on student attitudes toward WBLTs and learning performance, so it would
be bold to offer strong recommendations to educators. That said, several tentative
suggestions are worth considering. First, it appears educators need not be overly
concerned about gender biases in the use of WBLTs, particularly with respect to student
learning performance. This is the second study that has confirmed the minimal impact of
gender on WBLT use. Second, WBLTs may be better suited to older students (e.g.
secondary vs. middle school), perhaps because of the extensive self-supporting cognitive
skills required to use these tools effectively. If WBLTs are used with younger students, it
100 R. H. Kay
may be prudent to offer more scaffolding and guidance to ensure gains in learning
performance. Third, students who are less comfortable with computers or the subject
areas addressed by a WBLT may be more resistant to using them as teaching tools. While
this resistance does not appear to translate into reduced learning performance, students
with these limitations may benefit from additional support. Finally, WBLTs may be a
useful tool to reach a wider range of student ability levels, as average grade did not
appear to influence student attitudes toward WBLTs or learning performance.
7. Future Research
Careful attention was paid to method in this study. Well tested, reliable, valid, and
comprehensive measures were used to assess a wide range of systematically selected
WBLTs in a large, diverse sample of middle and secondary school students. Nonetheless,
because the investigation of individual differences in the use of WBLT is relatively new,
more research is needed to replicate the findings and to address unanswered questions.
Perhaps the most important new direction to pursue is collecting detailed qualitative data
in the form of interviews, focus groups, or open-ended questions to help explain why
certain differences exist and whether these differences are idiosyncratic or robust. Finally,
this study examined the impact of individual differences in personal characteristics like
age, gender, and comfort level with computers and subject area. Other variables such as
socio-economic status or cognitive style might be important. In addition, the effect of
variations context, such as teaching strategies, subject area, and technological support
need to be explored too. In other words, the next important question to ask is, “Under
what environmental conditions are WBLTs less or more useful?”
References
Akpinar, Y., & Bal, V. (2006). Student tools supported by collaboratively authored tasks: The case
of work learning unit. Journal of Interactive Learning Research,17(2), 101–119. Retrieved
from http://www.editlib.org.uproxy.library.dc-uoit.ca/p/6167
American Association of University Women (AAUW). (2000). Tech-Savvy: Educating girls in the
new computer age. Washington, DC: American Association of University Women
Foundation. Retrieved from
http://www.aauw.org/member_center/publications/TechSavvy/TechSavvy.pdf
Anderson, L. W., & Krathwohl, D. R. (Eds.) (2001). A taxonomy for learning, teaching, and
assessing: A revision of bloom’s taxonomy of educational objectives. New York: Longman.
Barbeite, F. G., & Weiss, E. M. (2004). Computer self-efficacy and anxiety scales for an Internet
sample: Testing measurement equivalence of existing measures and development of new
scales. Computers in Human Behavior,20(1), 1–15. doi:10.1016/S0747-5632(03)00049-9
Barker, L. J., & Aspray, W. (2006). The state of research on girls and IT. In J. M. Cohoon & W.
Aspray (Eds.), Women and information technology (pp. 3–54). Cambridge, MA: The MIT
Press.
Bower, M. (2005). Online assessment feedback: Competitive, individualistic, or…preferred form!
Journal of Computers in Mathematics and Science Teaching,24(2),121–147. Retrieved from
http://www.editlib.org.uproxy.library.dc-uoit.ca/p/5681
Exploring Individual Differences in the Impact 101
Bradley, C., & Boyle, T. (2004). The design, development, and use of multimedia learning objects.
Journal of Educational Multimedia and Hypermedia,13(4), 371–389. Retrieved from
http://www.editlib.org.uproxy.library.dc-uoit.ca/p/18905
Brown, A. R., & Voltz, B. D. (2005). Elements of effective e-learning design.The International Review
of Research in Open and Distance Learning,6(1). Retrieved from
http://www.irrodl.org/index.php/irrodl/article/view/217/300
Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and
Instruction,8(4), 293–332. doi: 10.1207/s1532690xci0804_2
Christensen, R. W., & Knezek, G. A. (2000). Internal consistency reliabilities for 14 computer
attitude scales. Journal of Technology and Teacher Education,8(4), 327–336. Retrieved from
http://www.editlib.org.uproxy.library.dc-uoit.ca/p/8041
Clarke, O., & Bowe, L. (2006a). The learning federation and the Victorian department of education
and training trial of online curriculum content with Indigenous students. 1–14. Retrieved
from
http://www.sofweb.vic.edu.au/edulibrary/public/teachlearn/ict/TLF_DETVIC_indig_trial_ma
r06.pdf
Clarke, O., & Bowe, L. (2006b). The learning federation and the Victorian department of
education and training trial of online curriculum content with ESL students. 1–16. Retrieved
from http://www.thelearningfederation.edu.au/verve/_resources/report_esl_final.pdf
Cochrane, T. (2005). Interactive QuickTime: Developing and evaluating multimedia learning
objects to enhance both face-to-face and distance e-learning environments. Interdisciplinary
Journal of Knowledge and Learning Objects, 1. Retrieved from
http://ijklo.org/Volume1/v1p033-054Cochrane.pdf
Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd edition). New York:
Academic Press.
Cohen, J. (1992). A power primer. Psychological Bulletin,112(1), 155–159. doi: 10.1037/0033-
2909.112.1.155
Colley, A. (2003). Gender differences in adolescents’ perceptions of the best and worst aspects of
computing at school. Computers in Human Behaviour,19(6), 673–682. doi:10.1016/S0747-
5632(03)00022-0
Colley, A., & Comber, C. (2001). Age and gender differences in computer use and attitudes among
secondary school students: What has changed? Educational Research,45(2), 155–165. doi:
10.1080/0013188032000103235
Comber, C., Colley, A., Hargreaves, D. J., & Dorn, L. (1997). The effects of age, gender and
computer experience upon computer attitudes. Educational Research Volume,39(2), 123–133.
Cross, K. P., & Steadman, M. (1996). Classroom research: Implementing the scholarship of
teaching. San Francisco: Jossey-Bass.
De Salas, K., & Ellis, L. (2006). The development and implementation of learning objects in a
higher education. Interdisciplinary Journal of E-Learning and Learning Objects,2006(2), 1–
22. Retrieved from http://www.ijello.org/Volume2/v2p001-022deSalas.pdf
Docherty, C., Hoy, D., Topp, H., & Trinder, K. (2005). E-Learning techniques supporting problem
based learning in clinical simulation. International Journal of Medical Informatics,74(7-8),
527–533. doi:10.1016/j.ijmedinf.2005.03.00
Durndell, A., & Haag, Z. (2002). Computer self efficacy, computer anxiety, attitude towards the
internet and reported experience with the internet, by gender, in an east European sample.
Computers in Human Behaviour,18(5), 521–535. doi:10.1016/S0747-5632(02)00006-7
102 R. H. Kay
Fields, A. (2005). Discovering statistics using SPSS. Thousand Oaks, CA: Sage Publications Ltd.
Harris, J. B., & Grandgenett, N. (1996). Correlates among teachers’ anxieties, demographics, and
tele-computing activity. Journal of Research on Computing in Education,28(3), 300–317.
Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. Australasian
Journal of Educational Technology,21(4), 470–490. Retrieved from
http://www.ascilite.org.au/ajet/ajet21/haughey.html
Howard-Rose, D., & Harrigan, K. (2003). CLOE learning impact studies lite: Evaluating learning
objects in nine Ontario university courses.Retrieved from
http://cloe.on.ca/documents/merlotconference10.doc
Kay, R. H. (2008). Exploring gender differences in computer-related behaviour: Past, present, and
future. In T .T. Kidd & I. Chen (Eds.),Social information technology: Connecting society and
cultural issues (pp. 12–30). Hershey, PA: Information Science Reference.
Kay, R. H. (2009). Understanding factors that influence of the effectiveness of learning objects in
secondary school classrooms. In L. T. W. Hin & R. Subramaniam (Eds.), Handbook of
research on new media literacy at the K-12 level: Issues and challenges (pp.419–435).
Hershey, PA: Information Science Reference.
Kay, R. H. (2011). List of WBLTs used in individual differences study. Retrieved from
http://faculty.uoit.ca/kay/res/ind_diff/ind_diff.html
Kay, R. H., & Knaack, L. (2005). Developing learning objects for secondary school students: A
multi-component model. Interdisciplinary Journal of E-Learning and Learning Objects, 1,
229–254. Retrieved from http://www.ijello.org/Volume1/v1p229-254Kay_Knaack.pdf
Kay, R. H., & Knaack, L. (2007a). Evaluating the learning in learning objects. Open Learning,
22(1), 5–28. doi: 10.1080/02680510601100135
Kay, R. H., & Knaack, L. (2007b). Evaluating the use of learning objects for secondary school
science. Journal of Computers in Mathematics and Science Teaching,26(4), 261–289.
Retrieved from http://www.editlib.org.uproxy.library.dc-uoit.ca/p/23577
Kay, R. H., & Knaack, L. (2008a). A formative analysis of individual differences in the
effectiveness of learning objects in secondary school. Computers and Education,51(3),
1304–1320. doi: 10.1016/j.compedu.2008.01.001
Kay, R. H., & Knaack, L. (2008b). A multi-component model for assessing learning objects: The
learning object evaluation metric (LOEM). Australasian Journal of Educational Technology,
24(5), 574–591. Retrieved from http://www.ascilite.org.au/ajet/ajet24/kay.pdf
Kay. R. H., & Knaack, L. (2009). Assessing learning, quality and engagement in learning objects:
The learning object evaluation scale for students (LOES-S). Education Technology Research
and Development, 57(2), 147–168. doi: 10.1007/s11423-008-9094-5
Kay. R. H., Knaack, L., & Muirhead, B. (2009). A formative analysis of instructional strategies for
using learning objects. Journal of Interactive Learning Research,20(3), 295–315. Retrieved
from http://www.editlib.org.uproxy.library.dc-uoit.ca/p/26233
Kubek, J. E., Miller-Albrecht, S. A., & Murphy, M. D. (1999). Finding information on the World
Wide Web: Exploring older adults’ exploration. Educational Gerontology,25(2), 167–183.
doi: 10.1080/036012799267945
Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of
common fractions: A model of affordances. Computers and Education,45(2), 245–265.
doi:10.1016/j.compedu.2004.12.002
Exploring Individual Differences in the Impact 103
Krauss, F., & Ally, M. (2005). A study of the design and evaluation of a learning object and
implications for content development. Interdisciplinary Journal of Knowledge and Learning
Objects, 1. Retrieved from http://ijklo.org/Volume1/v1p001-022Krauss.pdf
Lim, C. P., Lee, S. L., & Richards, C. (2006). Developing interactive learning objects for a
computing mathematics models. International Journal on E-Learning,5(2),221–244.
Retrieved from http://www.editlib.org.uproxy.library.dc-uoit.ca/p/5543
Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning
environment. Educational Technology, Research and Development,53(1), 5–21. doi:
10.1007/BF02504854
Liu, M., Hsieh, P., Cho, Y., & Schallert, D. (2006). Middle school students’ self-efficacy, attitudes,
and achievement in a computer-enhanced problem-based learning environment. Journal of
Interactive Learning Research,17(3), 225–242. Retrieved from
http://www.editlib.org.uproxy.library.dc-uoit.ca/p/18928
Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning
environment based on recreational learning objects. Computers and Education,48(4), 618–
641. doi: 10.1016/j.compedu.2005.04.014
MacDonald, C. J., Stodel, E., Thompson, T. L., Muirhead, B., Hinton, C., Carson, B., et al. (2005).
Addressing the eLearning contradiction: A collaborative approach for developing a
conceptual framework learning object. Interdisciplinary Journal of Knowledge and Learning
Objects, 1, 79–98. Retrieved from http://ijklo.org/Volume1/v1p079-098McDonald.pdf
Montgomery, K. C. (2009). Generation digital. MA: MIT Press.
Nesbit, J., & Belfer, K. (2004). Collaborative evaluation of learning objects. In R. McGreal (Ed.),
Online education using learning objects (pp. 138–153). New York: Routledge Falmer.
Nurmi, S., & Jaakkola, T. (2006). Effectiveness of learning objects in various instructional settings.
Learning, Media, and Technology,31(3), 233–247. doi: 10.1080/17439880600893283
Palfrey, J., & Gasser, U. (2008). Born digital. NY: Basic Books.
Reimer, K., & Moyer, P. S. (2005). Third-graders learning about fractions using virtual
manipulatives: A classroom study. Journal of Computers in Mathematics and Science
Teaching,24(1), 5–25. Retrieved from http://www.editlib.org.uproxy.library.dc-
uoit.ca/p/18889
Rieber, L. P., Tzeng, S., & Tribble, K. (2004). Discovery learning, representation, and explanation
within a computer-based simulation: Finding the right mix. Learning and Instruction,14(3),
307–323. doi:10.1016/j.learninstruc.2004.06.008
Sanders, J. (2006). Gender and technology: What the research tells us. In C. Skelton, B. Francis &
L. Smulyan (Eds.), Handbook of gender and education (pp. 307–324). London: Sage.
Schoner, V., Buzza, D., Harrigan, K., & Strampel, K. (2005). Learning objects in use: ‘Lite’
assessment for field studies. Journal of Online Learning and Teaching,1(1), 1–18. Retrieved
from http://jolt.merlot.org/documents/vol1_no1_schoner_001.pdf
Tapscott, D. (2008). Grown up digital: How the net generation is changing your world. NY:
McGraw-Hill.
Torkzadeh, R., Pflughoeft, K., & Hall, L. (1999). Computer self-efficacy, training effectiveness and
user attitudes: An empirical study. Behaviour and Information Technology,18(4), 299–309.
doi: 10.1080/014492999119039
Vacik, H., Wolfslehner, B., Spork, J., & Kortschak, E. (2006). The use of COCOON in teaching
silviculture. Computers and Education, 47(3), 245–259. doi:10.1016/j.compedu.2004.10.009
104 R. H. Kay
Van Zele, E., Vandaele, P., Botteldooren, D., & Lenaerts, J. (2003). Implementation and evaluation
of a course concept based on reusable learning objects. Journal of Educational Computing
Research,28(4), 355–372. doi:10.2190/PYTN-HTG8-UMBD-3T6C
Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation:
Computer mediated collaboration and inter-rater reliability. International Journal of
Computers and Applications,25(3), 1–8.
Whitley, B. E., Jr. (1997). Gender differences in computer-related attitudes and behaviors: A meta-
analysis. Computers in Human Behavior,13(1), 1–22. doi:10.1016/S0747-5632(96)00026-X
Windschitl, M., & Andre, T. (1998). Using computer simulations to enhance conceptual change:
The roles of constructivist instruction and student epistemological beliefs. Journal of
Research in Science Teaching, 35(2), 145–160. doi: 10.1002/(SICI)1098-
2736(199802)35:2<145::AID-TEA5>3.0.CO;2-S
Wlodkowski, R. J. (2008). Enhancing adult motivation to learn: A comprehensive guide for teach
all adults. San Francisco: Jossey-Bass.
Appendix A. WBLT Evaluation Scale
Learning
1. Working with the learning object helped me learn.
2. The feedback from the learning object helped me learn.
3. The graphics and animations from the learning object helped me learn.
4. The learning object helped teach me a new concept.
5. Overall, the learning object helped me learn.
Design
6. The help features in the learning object were useful.
7. The instructions in the learning object were easy to follow.
8. The learning object was easy to use.
9. The learning object was well organized.
Engagement
10. I liked the overall theme of the learning object.
11. I found the learning object engaging.
12. The learning object made learning fun.
13. I would like to use the learning object again.
All scale items used the following 7-point Likert scale:
1 = Strongly Disagree, 2 = Disagree, 3 = Somewhat Disagree, 4 = Neutral, 5 = Somewhat Agree, 6 = Agree,
7 = Strongly Agree
