Developing a Comprehensive Metric for Assessing Discussion Board Effectiveness

Abstract

The use of online discussion boards has grown extensively in the past 5 years, yet some researchers argue that our understanding of how to use this tool in an effective and meaningful way is minimal at best. Part of the problem in acquiring more cohesive and useful information rests in the absence of a comprehensive, theory-driven metric to assess quality and effectiveness. Based on an extensive review of the research, the following variables were used to assess traditional discussion board use: thread, location of message within thread, author (student vs. educator), subject line clarity, time of posting, response time from previous message, number of times message was read, number of words, primary purpose, message quality, difficulty level of topic, knowledge level, processing level and use of external resources. These variables proved to be effective in assessing 12 key areas of discussion board use. It is argued that this kind of metric is essential if we wish to advance our understanding of online discussion boards for both educators and researchers.

Full text

© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency. Published by
Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
British Journal of Educational Technology Vol 37 No 5 2006
761–783
doi:10.1111/j.1467-8535.2006.00560.x
Blackwell Publishing Ltd.Oxford, UKBJETBritish Journal of Educational Technology0007-1013British Educational Communications and Technology Agency, 20052005375761783Articles
Developing a comprehensive metric
British Journal of Educational Technology
Developing a comprehensive metric for assessing discussion
board effectiveness
Robin H. Kay
Assistant Professor, Faculty of Education, University of Ontario Institute of Technology, Oshawa,
Ontario, Canada. Email: Robin.Kay@uoit.ca
Abstract
The use of online discussion boards has grown extensively in the past 5 years,
yet some researchers argue that our understanding of how to use this tool in
an effective and meaningful way is minimal at best. Part of the problem in
acquiring more cohesive and useful information rests in the absence of a
comprehensive, theory-driven metric to assess quality and effectiveness. Based
on an extensive review of the research, the following variables were used to
assess traditional discussion board use: thread, location of message within
thread, author (student vs. educator), subject line clarity, time of posting,
response time from previous message, number of times message was read,
number of words, primary purpose, message quality, difficulty level of topic,
knowledge level, processing level and use of external resources. These variables
proved to be effective in assessing 12 key areas of discussion board use. It is
argued that this kind of metric is essential if we wish to advance our
understanding of online discussion boards for both educators and researchers.
Overview
The use of online discussion boards has grown extensively in the past 5 years (Cooper,
2001). While this tool is viewed as revolutionary by some researchers (Hara, Bonk &
Angeli, 1998; Li, 2003), others argue that our understanding of how to use online
discussion in an effective and meaningful way is limited (Blignaut & Trollip, 2003).
Acquiring more cohesive and useful information on the use of discussion boards is
partially dependent on developing a consistent, comprehensive, theory-driven metric to
assess quality and effectiveness. Considerable research has been carried out on the use
of online discussion (eg, Aviv, Erlich, Ravid & Geva, 2003; Benigno & Trentin, 2000;
Berge & Muilenburg, 2000; Blignaut & Trollip, 2003; Burstall, 2000; Hara
et al
, 1998;
Henri, 1992; Im & Lee, 2003–04; Love, 2002; Mazzolini & Maddison, 2003; Meyer,

762
British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
2003; Roblyer & Wiencke, 2003; Rourke & Anderson, 2002; Wickstrom, 2003); how-
ever, methodological differences in assessment make it difficult to combine the results
into a cohesive base of knowledge that can guide practice and education. In addition,
many studies have looked at only one or two specific aspects of online discussion in
detail. Several researchers attempted more complete and detailed analyses (eg, Aviv
et al
, 2003; Benigno & Trentin, 2000; Edelstein & Edwards, 2002; Hara
et al
; Im & Lee,
2003–04; Meyer, 2003; Roblyer & Wiencke, 2003; Zhu, 1998), although the scope was
still somewhat limited with respect to the full range of factors that could influence
successful performance.
It is argued that a more comprehensive, theory-driven assessment tool is needed to
advance our understanding of how to best use discussion boards in education. The
purpose of this study, then, was to research, develop and test a multicomponent, theo-
retically driven metric to assess the effectiveness of online discussion boards.
Literature review—developing a metric
A literature review of discussion boards and cognitive theory revealed 12 promising
areas for evaluating discussion boards: social learning, cognitive processing, quality of
discussion, the initial question in a thread, role of educator, navigation issues, chal-
lenges for students, types of users, attitude towards online discussion, response time,
learning outside of school and learning performance. Each area is discussed below.
Social learning
Vygotsky (1978) was a pioneer in exploring the role of language in thought. He noted
that conceptual learning was a collaborative effort requiring supportive dialogue. Slavin
(1995) added that extensive research supports the positive effects of cooperative learn-
ing on achievement. It is reasonable, then, to expect that online discussion has the
potential to support collaboration and concept development. A number of studies, how-
ever, have reported that true social interaction leading to cognitive development and
resolution to actual problems addressed in discussion boards is rare (eg, Berge &
Muilenburg, 2000; Biesenbach-Lucas, 2003; Garrison, Anderson & Archer, 2001;
Hara
et al
, 1998; Meyer, 2003; Son, 2002; Wickstrom, 2003). Other researchers have
observed significant social interaction leading to effective knowledge building (eg, Aviv
et al
, 2003; Fauske & Wade, 2003–04; Garrison
et al
, 2001; Grady, 2003; Poole, 2000;
Rourke & Anderson, 2002; Schallert, Reed & D-Team, 2003). A possible resolution to
this debate might lie in user acceptance of the medium, the amount of structure used
to guide the discussion board and the quality of measures used to assess social interac-
tion. There is some evidence to suggest that well-structured, focused, course-relevant
questions lead to more effective discussion (eg, Aviv
et al
; Ferdig & Roehler, 2003–04;
Greenlaw & DeLoach, 2003; Roblyer & Wiencke, 2003), particularly with students who
embrace the online mode of interaction (eg, Biesenbach-Lucas, 2003; Ferdig & Roehler,
2003–04; Loomis, 2000; Poole, 2000; Schallert
et al
, 2003; Wu & Hiltz, 2004). Roblyer
and Wiencke (2003) noted, however, that ‘the lack of definition as to what constitutes
observable, measurable interactive qualities [in discussion boards]... has prevented
transfer from theory and research to design practice’ (p. 77).

Developing a comprehensive metric
763
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Cognitive processing
While detailed content analyses of discussion boards focusing on cognitive processing
have been carried out by a number of investigators (Aviv
et al
, 2003; Berge & Muilen-
burg, 2000; Hara
et al
, 1998; Im & Lee, 2003–04; Knowlton & Knowlton, 2001;
Roblyer & Wiencke, 2003; Rourke & Anderson, 2002; Zhu, 1998), less than half of
these studies (Aviv
et al
; Berge & Muilenburg, 2000, Knowlton & Knowlton, 2001) have
used theoretically based taxonomies. In addition, most assessment tools appear biased
towards looking at higher-level, controversial, university-based topics. A more compre-
hensive metric, examining a full range of cognitive processes, would be useful in explor-
ing a more diverse population and range of topics. The current study will use a revised
version of Bloom’s Taxonomy (Anderson & Krathwohl, 2001) to look at both knowledge
and processing level of discussion board messages. Advantages of this revised model
over the original Bloom’s Taxonomy include an emphasis on actual use in planning,
delivery and assessment, a focus on a broader audience (the original model was aimed
at primary and junior years), and clearer descriptions of subcategories.
Quality of discussion
Aside from the level of social interaction and cognitive processing, the actual quality of
an individual message has been looked at from several angles—tone (Grady, 2003;
Knowlton & Knowlton, 2001), grammar (Edelstein & Edwards, 2002), number of words
(eg, Biesenbach-Lucas, 2003), reasoning (Edelstein & Edwards, 2002; Love, 2002),
level of controversy (Burstall, 2000) and content (Blignaut & Trollip, 2003; Edelstein
& Edwards, 2002; Grady, 2003; Im & Lee, 2003–04; Merryfield, 2001). The results
suggest that students need to be aware of their tone to avoid misunderstanding (Grady,
2003; Knowlton & Knowlton, 2001); messages are not always easily understood (Love,
2002); and interaction is improved with more controversial issues that do not have
specific, concrete answers (Blignaut & Trollip, 2003; Burstall, 2000).
Initial question
Previous research suggests that the initial question starting off a discussion board
thread is germane to the quality of subsequent interaction (Aviv
et al
, 2003; Berge &
Muilenburg, 2000; Ferdig & Roehler, 2003–04; Greenlaw & DeLoach, 2003; Hara
et al
,
1998; Roblyer & Wiencke, 2003; Savage, 1998; Wickstrom, 2003). Specifically, more
successful questions are clear and focused (Aviv
et al
; Berge & Muilenburg, 2000;
Fauske & Wade, 2003–04; Greenlaw & DeLoach, 2003), provocative or interpretive
(Greenlaw & DeLoach, 2003; Love, 2002), directly relevant to the course being taught
(Aviv
et al
; Ferdig & Roehler, 2003–04; Poole, 2000), authentic (Gold, 2001), lead
students towards producing specific products (Aviv
et al
; Roblyer & Wiencke, 2003) and
promote higher-level thinking (Savage, 1998).
Role of educator
The role of the educator in an online discussion has received considerable attention
(Berge & Muilenburg, 2000; Blignaut & Trollip, 2003; Burstall, 2000; Ferdig & Roehler,
2003–04; Figallo, 1998; Greenlaw & DeLoach, 2003; Hara
et al
, 1998; Knowlton &
Knowlton, 2001; Li, 2003; Love, 2002; Mazzolini & Maddison, 2003; Moller, 1998;

764
British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Poole, 2000; Rourke & Anderson, 2002; Wickstrom, 2003), although researchers have
yet to agree on the most appropriate strategy. One school of thought proposes that
educators are critical to the success of an online discussion (Blignaut & Trollip, 2003;
Ferdig & Roehler, 2003–04; Figallo, 1998; Greenlaw & DeLoach, 2003; Knowlton &
Knowlton, 2001; Love, 2002; Moller, 1998; Roblyer & Wiencke, 2003). The educator
is there to raise the level of discussion to a higher level (Figallo, 1998). Moreover, giving
students the responsibility to determine the direction is not a viable approach (Moller,
1998). The other school of thought claims that educators should take a back seat and
let students construct their own knowledge (Burstall, 2000; Li, 2003; Mazzolini &
Maddison, 2003; Poole, 2000; Rourke & Anderson, 2002). These researchers have
reported that peer messages are more effective than educator messages at stimulating
a discussion and that instructor presence can actually shut a discussion down (Li,
2003; Mazzolini & Maddison, 2003; Rourke & Anderson, 2002).
Navigation issues
Students and instructors can face considerable problems trying to navigate through a
typical discussion board. Specific problems observed include length of message (Hara
et al
, 1998), number of entries (Burstall, 2000; Hammond, 2000; Hara
et al
;
Knowlton & Knowlton, 2001; Meyer, 2003; Ross, Kukulska-Hulme, Chappel & Joyce,
2004; Son, 2002; Wickstrom, 2003), unclear subject lines (Hara
et al
), and lack of
organisation and focus (Chen & Hung, 2002; Li, 2003; Poole, 2000; Rourke &
Anderson, 2002). In other words, the number and length of messages can be over-
whelming, particularly if messages are not organised well. Chen and Hung (2002)
add that the traditional threaded discussion format may be inadequate for true
knowledge building.
Challenges for students
The challenges that students face in participating in online discussion are many—
perceived inability to participate (Beaudoin, 2002; Wickstrom, 2003); reticence of
many users, especially beginners, to add messages (Hammond, 2000; Mazzolini &
Maddison, 2003); writing being naturally more slower than talking (Hammond, 2000);
inability to change thoughts once they are written down and made public (Beaudoin,
2002; Hammond, 2000); extensive amount of time taken to participate (Loomis, 2000;
Meyer, 2003; Son, 2002); lack of personal interaction (Weiss, 2000); lack of organisa-
tion and self-discipline (Schrum & Hong, 2002), misinterpretation of humor or sarcasm
(Berge & Muilenburg, 2000); and the negative effect of being graded (Wickstrom,
2003). Most of these findings were observed in higher education, so it is unclear
whether the secondary students examined in this study would experience the same
concerns.
Types of users
There is some evidence to suggest that students assume specific and distinct roles as a
discussion board evolves (Aviv
et al
, 2003; Hammond, 2000; McGrath & Hollingshead,
1994; Palloff & Pratt, 1999; Wickstrom, 2003). These roles are based on level of par-

Developing a comprehensive metric
765
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
ticipation (Beaudoin, 2002; Ferdig & Roehler, 2003–04; Greenlaw & Deloach, 2003;
Hammond, 2000; Poole, 2000; Wickstrom, 2003), degree of reflection (Hara
et al
,
1998), mediation skills (Palloff & Pratt, 1999), learning style (Loomis, 2000) and
number of words used (Poole, 2000). Some students who have limited writing or verbal
skills may be at a distinct disadvantage (Hara
et al
). To date, there has been no system-
atic attempt to investigate individual differences in online discussion participants.
Attitudes towards discussion
Surprisingly little systematic research has been performed to examine student attitudes
towards online discussion. A number of researchers (Hammond, 2000; Schallert
et al
,
2003; Son, 2002; Wu & Hiltz, 2004) reported that students were generally positive
about using online discussion, but this conclusion was supported by anecdotal evidence
only. Other researchers have observed that students tend to do the bare minimum when
participation is mandatory, a possible reflection of negative attitude towards use (eg,
Hara
et al
, 1998; Schallert
et al
; Wickstrom, 2003; Wu & Hiltz, 2004). Clearly more
research is needed in this area.
Response time
Online dialogue differs from face-to-face conversation in several ways, but one key
difference is that there are inevitable delays between posting and replying to messages
within a thread. This means that students who post questions or responses may need
to check repeatedly the discussion board in order to continue the conversation. Son
(2002) and Yacci (2000) speculated that these delays could be a problem. A more
formal analysis of response times has yet to be performed.
Learning outside of school
A large number of discussion boards are used in conjunction with face-to-face learning
(eg, Hara
et al
, 1998; Love, 2002; Schrum & Hong, 2002), yet there is little research
on how much discussion actually goes on outside of school environment. Schallert
et al
(2003) reported that log-on time was important—students who logged on late made
few comments and received few responses from others. To date, no research has looked
at the ratio of messages posted at home versus those posted during school hours. The
assumption may be that students are spending time reflecting and posting messages at
home, yet there are no data to support this supposition.
Learning performance
Several researchers (Chen & Hung, 2002; Fabos & Young, 1999) have challenged the
conjecture that mere participation in an online discussion board guarantees the social
construction of knowledge and personal understanding. One way of examining the
impact of participation is to correlate it with actual learning performance. A number
of studies have evaluated the effect of online discussion use by assessing self-perception
of learning (eg, Hiltz, Coppola, Rotter, Turoff & Benbunan-Fich, 2000; Schallert
et al
,
2003; Swan, 2002). Only one study reported a significant increase in actual learning
performance (grades) as a result of participating in online discussions. It is critical to
continue testing the fundamental relationship between participation and learning.

766
British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Current study
After a thorough review of assessment methods used to evaluate discussion boards, a
multicomponent metric, comprising 12 key areas, was created, and it included social
learning, cognitive processing, quality of discussion, initial question, role of educator,
navigation, challenges for students, types of users, attitudes towards discussion,
response time, learning outside of school and learning performance. The purpose of this
study was to test the effectiveness of this metric in evaluating the impact of discussion
board use on learning.
Method
Sample
The sample tested consisted of 45 secondary school students enrolled in an introductory
computer science course at a secondary school in a metropolitan area. The students,
all males, ranging in age from 13 to 15 years old, were divided into two classes consist-
ing of 22 and 23 students. The assignment of students to a particular class was based
solely on their schedule at the beginning of the year. The data were collected and
analysed a year after the students finished the course.
Procedure
The students were asked to contribute messages in two consecutive asynchronous
online discussions used to supplement the learning of hypertext markup language
(HTML) (24 days) and beginning programming (36 days). The online discussions were
part of a regular face-to-face course that met every other day for 90 minutes. Partici-
pation in the online discussion was worth 10% of the final grade. Specific grading
guidelines were not provided in order to encourage as much participation as possible.
It was emphasised that messages consisting of questions or answers would be given
equal weighting. It is worth noting that the majority of discussion board research is
based on courses where participation is graded (eg, Burstall, 2000; Hara
et al
, 1998,
Li, 2003; Love, 2002; Mazzolini & Maddison, 2003; Schrum & Hong, 2002; Son,
2002; Wickstrom, 2003). The discussion board was intended to be student-led, and
the teacher would only intervene if there were problems that students could not
resolve. After each of the course topics was completed (HTML first, programming last),
the students were asked to fill in a survey consisting of two open-ended questions.
Data collection and analysis
Three sources of data collection were used in order to analyse all of the key components
identified in the literature review. These included (1) coding the online discussion board
messages, (2) analysing the statistics accumulated by the discussion board software
(Blackboard 5.0) on actual use and (3) distributing attitude surveys at the end of each
topic. Finally, where possible, two or more variables were used to evaluate specific
components in order to improve accuracy and validity.
After an extensive review of the literature, 12 key areas of analysis were identified as
promising for assessing online discussion use. The specific variables used to evaluate
these areas, along with supporting references, are provided in Appendix A.

Developing a comprehensive metric
767
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Coding of online discussion board messages
In order to make the coding scheme as transparent as possible, Appendix B provides a
detailed rubric for the key variables used this study.
Blackboard statistics
The Blackboard programme automatically collected the following statistics: time when
message was posted, number of times a message was read by others, number of visits
an individual student made to the discussion board, number of days an individual
student visited the discussion board and total number of posts an individual student
made.
Survey data
Two key questions were asked of students after they completed each course topic:
1. Did you use the discussion board? Please explain in detail why or why not.
2. Was the discussion board useful to you? Explain in detail why or why not.
The responses from students were examined to identify and categorise (1) reasons for/
for not participating and (2) why the discussion board was thought to be useful or not.
The frequencies of each category were then calculated.
Learning performance
An external measure of learning performance (grades for term project and term test)
was used to evaluate the overall effectiveness of participating in discussion boards. The
final term project grade and final exam score were correlated with the total number of
visits a student made to the discussion board, total number of days that a student visited
the discussion board and the total number of messages posted.
Results
Overview
Overall, a total of 260 messages were posted for both HTML and programming. The
mean length of a discussion thread was of 3.5 messages (
SD
= 2.3, range 1–11 mes-
sages), and the average number of words was 48.3 (
SD
= 46.2, range 1–263 words).
The subject lines were moderately clear (
M
= 1.68,
SD
= 0.9; scale range 0–3), and the
quality of messages was fair to good (
M
= 2.3,
SD
= 0.9; scale range 0–4). A typical
message was read an average of 29.5 times (
SD
= 11.3, range 2–77). The average time
to respond to a posted message was 3630 minutes or 2.5 days (
SD
= 7377 minutes,
range 1–49 109 minutes).
With respect to content, a majority of the messages were either course related or went
beyond the material covered in class (
n
= 223, 86%). The primary purpose of most
messages was to ask a question (
n
= 63, 24%) or to offer an answer (
n
= 175, 73%).
The discussion board was rarely used for nonacademic purposes (
n
= 15, 6%).

768
British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Social learning
The number of discussion threads containing five or more messages was 26/55 or 47%.
The mean number of times a typical message was read was 29.5 (
SD
= 11.3) and ranged
from 2 to 77 times. Specific responses to other students in the form of questions or
answers occurred in 66% of all message posted (
n
= 172). Finally, the students partic-
ipated in the same discussion thread two or more times in 13 out of 28 HTML discussion
threads (46%) and in 10 out of 35 programming discussion threads (29%).
Cognitive processing
According the Bloom’s revised taxonomy, the predominant knowledge type demon-
strated was procedural (
n
= 140, 57%) followed by conceptual (
n
= 51, 21%) and
factual (
n
= 50, 21%). Metacognitive knowledge was present in only 3 out of 244
messages evaluated. With respect to processing level, the students displayed under-
standing most (
n
= 85, 35%) followed by remembering (
n
= 66, 27%), applying (
n
=
52, 22%), analyzing (
n
= 31, 13%) and evaluating (
n
= 10, 4%). The creative use of
knowledge was observed only once.
Quality of discussion
A majority of the messages were clear (
n
= 174, 67%) or somewhat clear (
n
= 70, 27%).
Only 16 messages (6%) were unclear. The message quality was good or excellent 41%
of the time, fair 47% of the time and poor or incorrect 12% of the time. New knowledge
was added either indirectly (
n
= 69, 27%) or directly (
n
= 103, 40%) in the majority of
messages posted.
The content of the messages focused mostly on material beyond (
n
= 145, 56%) or
directly related to the curriculum (
n
= 78, 30%). Nonacademic (
n
= 24, 9%) and
administrative issues (
n
= 6, 2%) were discussed infrequently. Finally, the discussion
threads, as a whole, were resolved (ie, questions were answered correctly) or went above
and beyond the original topic 71% (
n
= 100) of the time.
Initial question
The impact of the initial question in a discussion thread was assessed by looking at two
dependent variables (number of times a message was read and the length of the discus-
sion) for five independent variables (whether the question was easily answered else-
where, subject line clarity, message quality, knowledge type and processing level) Ten
one-way ANOVAs revealed no significant differences. In other words, there appeared to
be no distinct quality in an initial question that caused students to read or post more
questions.
Role of educator
The philosophy of the teacher in this course was to allow the students the opportunity
to construct their own knowledge. Therefore, it is not surprising that the students
initiated questions in 95% (
n
= 50) of the discussion threads started. The students also
ended discussions majority of the time (
n
= 49, 89%).

Developing a comprehensive metric
769
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Overall, there were no significant differences between teacher and student messages
with respect to the number of times each were read, length of the message and response
time (how fast a message received a response).
Navigation
Navigation issues were examined by looking at the effect of subject line clarity and
location of message within a thread (message number) on how many messages were
read (reading rate) and how fast a message received a response (response time) The
clarity of a subject line was not significantly related to reading rate or response time.
However, the message number was significantly and negatively correlated with the
average number of times the message was read (
r
=
−
0.26,
p
< 0.001). There was a
steady drop in the average number of times a message was read from the initial message
(
M
= 39.18) to the message number 11 (M = 14.5). The message number was not
significantly related to how fast the students responded to messages (response time).
From the post-task survey data, navigation was reported as the number one problem
in using the discussion board in both the HTML and programming topics (n = 35, 54%).
Specific concerns included the following: (1) it was hard to find specific content, because
there were too many messages; (2) too much clicking was involved in negotiating
messages; (3) the discussion board was being diluted with messages, because the stu-
dents were being graded and (4) there should be greater division and classification of
topics to decrease navigation time.
Challenges for students
Aside from navigation difficulties, some students had technical or software problems
(n = 16/65, 25%), difficulty trusting the quality of their peers’ messages (n = 14/65,
22%) and felt inhibited by the use of grades to motivate participation (n = 7/65, 11%).
The students reported a variety of reasons for not using the discussion board including
using other methods (using a book, searching the Internet, asking a friend) that were
perceived as being faster or more efficient (n = 25/65, 38%), differences in learning
style (n = 8/65, 12%), lack of ability (n = 8/65, 12%), forgetting to post (n = 5/65, 8%)
and not having enough time (n = 4/65, 6%).
Types of users
Individual differences observed among active participants (those students who posted
five or more messages) were observed with respect to the average number of messages
read (p < 0.001), average response time (p < 0.001), number of words used (p < 0.001)
and message quality (p < 0.001). The students also differed with respect to the number
of messages that they posted ranging from 1 to 17. The students did not differ signifi-
cantly with respect to clarity of subject line, difficulty of questions they responded to,
knowledge type and processing level.
In the post-task survey, several students (n = 8/65, 12%) reported that they did not
participate regularly in the discussion board, because it did not match their individual

770 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
learning style. Some students preferred to use a book—others favored looking up
answers on the Internet or simply asking another student. One student learning HTML
summed up this issue with the following comment:
I did not use [the] discussion board because I [felt] that it [was] easier to look up the answers to
your own questions rather than rely and wait on someone else to answer them. If you just look
up your question on the internet, you have an unlimited supply of information at your disposal
and you can extract the amount or sections that apply to you. You also can access them at your
own time, at your own pace. In addition, if you are on a roll of good solid work, it is difficult to
break away, go to Blackboard, post, and then wait for the answer to continue working. If I were
to just wait for answers, I do not think I would get much done if I set myself a certain amount of
time for this subject.
On the other hand, a few students noted that it was easier for them to use the discussion
board for more course-specific or idiosyncratic questions, because books or the Internet
offered information that was too general.
Attitudes towards discussion
The measure of attitude in this study is based on the perceived usefulness of the discus-
sion board. In the post-task survey, over one third of the students thought that the
discussion board was an effective learning tool (n = 24/65, 37%). With respect to actual
use, 38% of the students used the discussion board frequently, 25% occasionally and
27% not at all. Almost two thirds (65%) of the students reported that they had received
useful information, while one third (39%) thought that they had provided helpful infor-
mation to others. Eighty-two percent of the students did not indicate grade as a key
motivator for participating in the discussion board.
Response time
The average time taken to respond to a message was 3630 minutes or two and a half
days (SD = 7338 minutes). The students responded as quickly as 1 minute and as slowly
as 34 days. After eliminating outliers (eg, response times greater than 10 000 minutes
or 1 week, n = 22 messages), the average response time was 1519 minutes or about 1
day (SD = 2204 minutes). The response time appears to jump after the third message,
from 1182 (19.7 hours) to 1853 minutes (31 hours) for the fourth message, although
the sample is too small to assess statistical significance reliably.
The response time was significantly and negatively correlated with the number of times
a message is read (r = −0.254, p < 0.01). In other words, the longer it took for someone
to post a reply, the less likely it was that the new message would be read.
Learning outside of school
Just over half (n = 142, 55%) of all messages posted on the discussion board were
completed outside of school hours. There were no significant differences between school
and home messages with respect to clarity of subject line, message quality, response,
time and number of words; however, message topics discussed at home were rated as
more difficult to answer than those posted in school (p < 0.05).

Developing a comprehensive metric 771
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Performance and discussion board participation
The learning performance for both HTML and beginning programming topics was
positively correlated with the number of visits, number of days visited and number of
messages posted with one exception—the number of visits the HTML discussion board
was not significantly correlated to the final web page project grade (see Table 1).
The results above are supported by the post-task survey where almost two-thirds of the
students reported learning significant concepts using the discussion board.
Discussion
The 12 areas used to assess the effectiveness of discussion boards presented useful data
for both educators and researchers.
Social learning, cognitive processing and learning performance
Tracking the number of threads exceeding five messages, the purpose of a message and
the number of times a student participated in the same discussion thread provided
concrete evidence to suggest that students were genuinely engaged in social activity.
However, social ‘activity’ does not necessarily guarantee that social ‘learning’ is taking
place (eg, Berge & Muilenburg, 2000; Hara et al, 1998; Son, 2002; Wickstrom, 2003).
The additional analysis of discussion board messages in terms of cognitive processing
provided evidence that students were actively and cooperatively trying to understand
and apply new concepts. Furthermore, the fact that more than two-thirds of all discus-
sion threads were resolved or went beyond what was originally asked shows that a
number of problems were being solved. Finally, correlating discussion board participa-
tion and actual performance in the course indicated that significant learning was
occurring at some level. In short, this combination of assessment variables was effective
in describing the level of social learning that occurred.
Quality of discussion
Examining the quality of discussion provided new and nontrivial information to the
discussion board knowledge base. Past research on higher education has suggested that
Table 1: Correlations among discussion board participation and learning performance measures
HTML
final project
HTML
final test
Programming
final project
Programming
final test
Number of visits r = 0.27
(n.s.)
r = 0.44
(p < 0.01)
r = 0.33
(p < 0.05)
r = 0.38
(p < 0.05)
Number of days visited r = 0.42
(p < 0.01)
r = 0.48
(p < 0.01)
r = 0.36
(p < 0.05)
r = 0.36
(p < 0.05)
Number of posts r = 0.31
(p < 0.05)
r = 0.44
(p < 0.01)
r = 0.33
(p < 0.05)
r = 0.35
(p < 0.05)
Note: n.s., not significant; HTML, hypertext markup language.

772 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
if discussion boards were to be successful, there had to be controversial and thought-
provoking topics to promote higher-level thinking and active discussion (eg, Blignaut &
Trollip, 2003; Burstall, 2000). The results of this study suggest that there may be
another role for discussion boards, one that supports more application-focused, con-
crete dialogue. The students learning HTML and beginning programming were able to
post and answer meaningful factual, conceptual and application-based knowledge
problems leading to improved learning performance. The students also posted relatively
clear and unambiguous messages, a finding that was not observed by Knowlton and
Knowlton (2001) or Love (2002).
It should be noted that this positive result is consistent with a number of studies in
higher education, which suggest that discussion board success is significantly related
to focusing on meaningful, authentic and course-related learning tasks or topics (Aviv
et al, 2003; Ferdig & Roehler, 2003–04; Gold, 2001; Poole, 2000; Roblyer & Wiencke,
2003).
Initial question
A detailed analysis of the initial question starting off a discussion thread, based on
subject line clarity, difficulty of the topic addressed, message quality and level of cogni-
tive processing, revealed no significant differences with respect to how often messages
were read or the length of the discussion. It appears for the topics looked at in this study
(HTML and programming) and the type of students (secondary level) that there are no
distinct characteristics of the initial question that encourage more participation. This
finding is inconsistent with past research (Aviv et al, 2003; Berge & Muilenburg, 2000;
Ferdig & Roehler, 2003–04; Greenlaw & DeLoach, 2003; Hara et al, 1998; Roblyer &
Wiencke, 2003; Savage, 1998; Wickstrom, 2003), although it is somewhat difficult to
compare given the difference in topics addressed, the educational level of students
participating and the absence of commonly defined assessment tools.
Role of educator
The teacher in this study did not dominate or excessively stimulate discussion. The
students initiated and ended the vast majority of discussion threads. This approach is
consistent with the ‘guide on the side’ philosophy advocated by Burstall (2000), Li
(2003), and Mazzolini and Maddison (2003). The students were not only successful at
interacting and building new knowledge, but their participation contributed to better
performance on final projects and tests. This result does not preclude the possibility that
they could have performed even better if the teacher had taken a more active role. It
does indicate, however, that students are capable of taking responsibility for a discussion
and learning new facts, concepts and applications without significant teacher interven-
tion and participation.
Because much of the knowledge covered in the discussion board went beyond the
standard curriculum, and students posted messages outside of class more than 50% of
the time, the online discussion board has the potential to be a powerful supplement to
a traditional classroom format. The effectiveness of students in guiding the discussion

Developing a comprehensive metric 773
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
in this study is also significant for educators who have noted the extensive time drain
that online discussion requires (eg, Berge & Muilenburg, 2000; Rourke & Anderson,
2002).
Navigation
Navigation issues were reported by students as the number one impediment to using
the discussion board. Somewhat surprisingly, subject line clarity, which is the only
initial guide to directing users in an online discussion board, was not significantly
correlated with whether a message was read or how fast the message was responded
to. However, messages at the end of long discussion threads were read less. This latter
finding supports the claim that the large number of entries in a discussion thread can
inhibit active participation.
The reading rate dropped sharply after the third message and then declined at a steady
rate. Two critical questions that still need to be answered are: how many messages are
users willing to read within a specific discussion, and why do they stop reading?
Chen and Hung’s (2002) speculation that the traditional online discussion format is
limited with respect to supporting true and personal knowledge building was not sup-
ported by the current results. Students, in spite of the navigation issues, managed to
participate regularly and learn effectively. Nonetheless, features, such as notifying the
author of a message when there is a response to that message or specific prompts to
encourage knowledge building, might improve learning (Chung, Severance & Chung,
2003; Scardamalia & Bereiter, 1999).
Challenges for students
Secondary students experienced some of the same challenges as their higher education
counterparts: reticence to use the discussion board (over 25% of all students posted two
or fewer messages), time challenges, lack of organisation and the negative effect of being
graded. While the discussion board was used frequently and improved learning for
many students, there are clear obstacles to using this tool.
Types of users and attitude towards use
To date, individual differences in using a discussion board have not been looked at in a
comprehensive way, although anecdotal evidence suggests that students assume spe-
cific roles (eg, Hammond, 2000; McGrath & Hollingshead, 1994; Palloff & Pratt, 1999;
Wickstrom, 2003). The results from the current study support previous anecdotal
observations. The students in this study could be significantly differentiated based on
the number of messages they read, how fast they responded to the messages, number
of words they wrote and the overall quality of their messages. These measures, when
viewed as a whole, could be seen as a reflection of general enthusiasm for using discus-
sion boards. In other words, students who read messages more, respond to messages
quickly and write detailed, high-quality messages could reasonably be considered as
enthusiastic participants. Not reading or responding to messages or providing short,
one-sentence messages might be a reflection of apathy. Lack of response could also mean

774 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
fear of committing a message to a public forum (Beaudoin, 2002) or an inability to
understand what is going on in a discussion (Ferdig & Roehler, 2003–04).
The results for identifying the distinct qualities of discussion board users were consistent
with students’ attitudes towards use. Roughly one third of all students said that they
used the discussion board sparingly or not at all. They noted that either the discussion
board did not match their personal style of learning or that there were more efficient
ways for them to learn (eg, using a book, talking with someone, using the Internet).
Another third of all students appeared to have a somewhat indifferent attitude and used
discussion boards on occasion. The final third were enthusiastic participants who
received and offered new ideas frequently.
These differences in use and attitude should be noted by educators. While some students
may thrive with this tool, others need more convincing or may not be prepared to use
the discussion board at all.
Response time
While response time has not been formally examined previously, the results of this study
support Son’s (2002) and Yacci’s (2000) speculation that delays in responding to mes-
sages could have a negative impact. The students responded relatively quickly to the
first three messages, after which the response time jumped by over 50%. When you
consider the significant and negative correlation between the response and number of
times a message is read, the response time becomes even more critical. In other words,
there appears to be a window of opportunity, roughly 19 hours, in which students will
follow up on a discussion board message, after which they start to lose interest. This
result is supported by research noting that the optimum response time for an instructor
should be 24 hours or less (Roblyer & Wiencke, 2003).
Delays in response time may be unavoidable with a traditional discussion board format,
which is relatively flat and serial. Chen and Hung (2002) proposed a new feature where
students have the ability to electronically indicate interest in a specific message or topic
and then are automatically informed of any future messages linked to their interests.
This kind of tool has the potential to increase the real time feel of a face-to-face discus-
sion. It could also reduce the time spent checking for a response to one’s message. It is
worthwhile to note that without some sort of automatic notification system, some
students will post messages a week and even a month after an original discussion is
started and that these messages will be rarely read.
Learning outside of school
The discussion board in this study was used as a supplement to teaching a secondary
course in HTML and programming, and the students were willing to use it outside of
school hours. This result is particularly noteworthy given that (1) a majority of the
topics covered in the discussion board went beyond the curriculum; (2) more difficult

Developing a comprehensive metric 775
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
topics were discussed at home; and (3) the use of the discussion board was significantly
correlated with learning performance.
Successful, meaningful and effective use of discussion boards outside of school hours
could prove to be beneficial to educators. In large classes, it is often not possible to
answer the range and number of questions during class (Ferdig & Roehler, 2003–04).
The use of discussion boards can clearly augment the traditional in-class model of
learning.
Limitations of the study
While it is argued that the metric used in this study is multidimensional and supported
by solid cognitive theory and extensive literature review, the specific results reported are
limited for several reasons.
First, the sample size was relatively small, consisting of only males. Second, the context
of learning was narrowly defined, focusing on the learning of a technical, procedure-
based subject, namely computer programming. Third, online discussion acted as a
supplement to a traditional face-to-face course. Therefore, in order to firmly establish
the effectiveness of the metric presented in this study, more research needs to be done
on larger, more diverse populations, in a broader range of subjects, looking at stand
alone as well as supplemental use of online discussion.
Summary
The assessment tool developed in this study proved to be useful in evaluating key
elements of online discussion use. The power of the metric rests in the wide range of
variables assessed based on a firm theoretical base. The highly selective and somewhat
idiosyncratic nature of previous discussion board metrics makes it difficult to compare
and fully evaluate results. The metric in this study uncovered valuable information
about asynchronous learning including the degree of social interaction and learning
that occurred, level of cognitive processing, quality of the discussion, effect of the initial
question, role of educator, challenges that participants experience while using a discus-
sion board, types of users, attitude towards discussion, effect of responding too slowly
to messages, impact of learning outside of school and effect of discussion boards on
learning performance. However, the metric needs to be further tested in a wider variety
of contexts with a larger, more diverse population.
References
Anderson, L. W. & Krathwohl, D. R. (Eds) (2001). A taxonomy for learning, teaching, and assessing:
a revision of bloom’s taxonomy of educational objectives. New York: Addison Wesley.
Aviv, R., Erlich, Z., Ravid, G. & Geva, A. (2003). Network analysis of knowledge construction in
asynchronous learning networks. JALN, 7, 3, 1–23.
Beaudoin, M. F. (2002). Learning of lurking? Tracking the ‘invisible’ online student. The Internet
and Higher Education, 5, 147–155.
Benigno V. & Trentin G. (2000). The evaluation of online courses. International Journal of
Computer Assisted Learning, 16, 3, 259–270.

776 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Berge, Z. & Muilenburg, L. (2000). Designing discussion questions for online, adult learning.
Educational Technology, September–October, 53–56.
Biesenbach-Lucas, S. (2003). Asynchronous discussion groups in teacher training classes: per-
ceptions of native and non-native students. Journal of Asynchronous Learning Networks, 7, 3,
24–46. Retrieved March 24, 2004, from http://www.aln.org/publications/jaln/v7n3/pdf/
v7n3_biesenbach-lucas.pdf
Blignaut, S. & Trollip, S. R. (2003). Developing a taxonomy of faculty participation in asynchro-
nous learning environments—an exploratory investigation. Computer and Education, 41, 149–
171.
Burstall, J. (2000). Learning communities for social change in forums on the web. Australian
Journal of Adult Learning, 40, 1, 33–52.
Chen, D. & Hung, D. (2002). Personalized knowledge representations: the missing half of online
discussions. British Journal of Educational Technology, 33, 3, 279–290.
Chung, S., Severance, C. & Chung, M.-J. (2003). Design of support tools for knowledge building
in a virtual university course. Interactive Learning Environments, 11, 1, 41–57.
Cooper, L. W. (March 2001). A comparison of online and traditional computer applications
classes. T. H. E. Journal, 28, 8, 52–58.
Edelstein, S. & Edwards, J. (2002). If you build it, they will come: building learning communities
through threaded discussions. The Online Journal of Distance Learning Administration, 5, 1, 21–
34.
Fabos, B. & Young, M. D. (1999). Telecommunications in the classroom: rhetoric and reality.
Review of Educational Research, 69, 3, 217–260.
Fauske, J. & Wade, S. E. (2003–2004). Research to practice online: conditions that foster democ-
racy, community, and critical thinking in computer-mediated discussions. Journal of Research
in Technology Education, 36, 2, 137–153.
Ferdig, R. E. & Roehler, L. R. (2003–2004). Student uptake in electronic discussions: examining
online discourse in literacy preservice classrooms. Journal of Research on Technology in Education,
36, 2, 119–136.
Figallo, C. (1998). Hosting web communities: building relationships, increasing customer loyalty, and
maintaining a competitive edge. New York: Wiley.
Garrison, D. R., Anderson, T. & Archer, W. (2001). Critical thinking, cognitive presence and
computer conferencing in distance education. American Journal of Distance Education, 15, 1, 7–
23.
Gold, S. (2001). A constructivist approach to online training for online teachers. Journal of
Asynchronous Learning Networks, 5, 1, 35–57. Retrieved January 10, 2004, from http://
www.aln.org/alnweb/journal/Vol5_issue1/Gold/Gold.pdf
Grady, D. B. (2003). Mapping online discussions with lexical scores. Journal of Interactive Learning
Research, 14, 2, 209–229.
Greenlaw, S. A. & DeLoach, S. B. (2003). Teaching critical thinking with electronic discussion.
Journal of Economic Education, 34, 36–53.
Hammond, M. (2000). Communication within on-line forums: the opportunities, the constraints
and value of communicative approach. Computers & Education, 25, 251–262.
Hara, N., Bonk, C. J. & Angeli, C. (1998). Content analysis of online discussion in an applied
educational psychology. Center for Research on Learning and Technology, No. 2–98. Retrieved
February 01, 2004, from http://crlt.indiana.edu/publications/techreport.pdf
Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collabora-
tive learning through computer conferencing: the Najaden papers (pp. 115–136). New York:
Springer.
Hiltz, S. R., Coppola, N., Rotter, N., Turoff, M. & Benbunan-Fich, R. B. (2000). Measuring the
importance of collaborative learning for the effectiveness of ALN: a multi-measure, multi-
method approach. Journal of Asynchronous Learning Networks, 4, 2, 103–125. Retrieved May
4, 2005, from http://www.aln.org/alnweb/journal/jaln-vol4issue2-3.htm
Im, Y. & Lee, O. (2003–2004). Pedagogical implications of online discussion for preservice teacher
training. Journal of Research in Technology Education, 36, 2, 155–170.

Developing a comprehensive metric 777
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Knowlton, D. S. & Knowlton, H. M. (2001). The context and content of online discussions:
making cyber-discussions viable for the secondary school curriculum. American Secondary
Education, 29, 4, 38–52.
Li, Q. (2003). Would we teach without technology? A professor’s experience of teach-
ing mathematics education incorporating the internet. Educational Research, 45, 1, 61–
77.
Loomis, K. D. (2000). Learning styles and asynchronous learning: comparing the LASSI model
to class performance. Journal of Asynchronous Learning Networks, 4, 1, 23–32. Retrieved May
4, 2005, from http://www.aln.org/publications/jaln/v4n1/pdf/v4n1_loomis.pdf
Love, K. (2002). Mapping online discussion in senior English. Journal of Adolescent & Adult
Literacy, 45, 5, 382–396.
McGrath, J. & Hollingshead, A. (1994). Groups interacting with technology. Thousand Oaks, CA:
Sage.
Mazzolini, M. & Maddison, S. (2003). Sage, guide or ghost? The effect of instructor interven-
tion on student participation in online discussion forums. Computers in Education, 40, 237–
253.
Merryfield, M. (2001). The paradoxes of teaching a multicultural education course online.
Journal of Teacher Education, 52, 4, 283–299.
Meyer, K. A. (2003). Face-to-face versus threaded discussions: the role of time and higher-order
thinking. Journal of Asynchronous Learning Networks, 7, 3, 55–65.
Moller, L. (1998). Designing communities of learners for asynchronous distance education.
Educational Technology, Research & Development, 46, 4, 115–122.
Palloff, R. M. & Pratt, K. (1999). Building learning communities in cyberspace—effective strategies for
the online classroom. San Francisco, CA: Jossey-Bass.
Poole, D. M. (2000). Student participation in a discussion-oriented online course: a case study.
Journal of Research on Computing in Education, 33, 2, 162–177.
Roblyer, M. D. & Wiencke, W. R. (2003). Design and use of a rubric to assess and encourage
interactive qualities in distance courses. The American Journal of Distance Education, 17, 2, 77–
98.
Ross, S. M., Kukulska-Hulme, A., Chappel, H. & Joyce, B. (2004). Taking e-moderating skills to
the next level: reflecting on the design of conferencing environments. Journal of Asynchronous
Learning Networks, 8, 2, 115–138.
Rourke, L. & Anderson, T. (2002). Using peer teams to lead online discussions. Journal of Interac-
tive Media in Education, 1, 1–21.
Savage, L. B. (1998). Eliciting critical thinking skills through questioning. The Clearing House, 71,
5, 291–293.
Scardamalia, M. & Bereiter, C. (1999). Schools as knowledge-building organizations. In D. Keat-
ing & C. Hertzman (Eds), Today’s children, tomorrow’s society: the development health and wealth
of nations (pp. 274–289). New York: Guilford.
Schallert, D. L., Reed, J. H. & D-Team, T. (2003). Intellectual, motivational, textual, and cultural
considerations in teaching and learning with computer-mediated discussion. Journal of
Research on Technology in Education, 36, 2, 103–118.
Schrum, L. & Hong, S. (2002). Dimensions and strategies for online success: voices from experi-
enced educators. Journal of Asynchronous Learning Networks, 6, 1, 57–67.
Slavin, R. E. (1995) Cooperative learning: theory, research, and practice. New York, NY: MacMillan.
Son, J. (2002). Online discussion in a CALL course for distance language teachers. CALICO
Journal, 20, 1, 127–144.
Swan, K. (2002). Building communities in online courses: the importance of interaction. Educa-
tion, Communication and Information, 2, 1, 23–49.
Vygotsky, L. S. (1978). Mind in society. Cambridge, MA: Harvard University Press.
Weiss, R. E. (2000). Humanizing the online classroom. New Directions for teaching and Learning,
84, 47–51.
Wickstrom, C. D. (2003). A funny thing happened on the way to the forum. Journal of Adolescent
& Adult Literacy, 46, 5, 414–423.

778 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Wu, D. & Hiltz, S. R. (2004). Predicting learning from asynchronous online discussions. Journal
of Asynchronous Learning Network, 8, 2, 139–152. Retrieved May 4, 2005, from http://
www.sloan-c.org/publications/jaln/v8n2/v8n2_wu.asp
Yacci, M. (2000). Interactivity demystified: a structural definition for distance education and
intelligent computer-based instruction. Educational Technology, 40, 4, 5–16.
Zhu, P. (1998). Learning and mentoring: electronic discussion in a distance learning course. In
C. J. Bonk & K. S. King (Eds), Electronic collaborators: learner-centered technologies for literacy,
apprenticeship, and discourse (pp. 233–259). Mahwah, NJ: Erlbaum.
Appendix A: Variables used to analyse discussion board message
Area Variables used Data source Reference
Social learning Number of threads with
more than five messages
DB Mazzolini & Maddison
(2003)
Average number of
messages read
DB None
Purpose of message (open
question, specific
question, answers,
independent comment,
not academic)
DB Aviv et al (2003); Blignaut &
Trollip (2003); Garrison
et al (2001); Henri (1992);
Meyer (2003); Son (2002);
Zhu (1998)
Number of times a
student posts two or
more messages in the
same discussion thread
DB Aviv et al (2003); Hammond
(2000); Roblyer & Wiencke
(2003); Wickstrom (2003)
Cognitive
processing
Knowledge type (facts,
concepts, procedures,
metacognitive
knowledge)
DB Anderson & Krathwohl
(2001); Aviv et al (2003);
Hara et al (1998); Henri
(1992)
Processing level
(remember, understand,
apply, analyse, evaluate)
DB Anderson & Krathwohl
(2001); Aviv et al (2003);
Hara et al (1998); Henri
(1992); Poole (2000)
Quality of
discussion
Message clarity (unclear,
somewhat clear, clear)
DB None
Message quality (poor,
fair, good, excellent)
DB Love (2002)
Presence of newknowledge
(no, yes-unrelated to
question, yes-related to
question)
DB Vygotsky (1978); Berge &
Muilenburg (2000);
Hara et al (1998);
Course knowledge in
message (none,
administrative, course
related, beyond course
curriculum, highly
advanced)
DB Grady (2003); Hara et al
(1998); Love (2002);
Son (2002)

Developing a comprehensive metric 779
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
External resources used
(none, teacher, another
message, computer
programme, web,
book)
DB Ferdig & Roehler (2003–04);
Im & Lee (2003–04);
Love (2002)
Resolution of discussion
(unresolved, partially
resolved, resolved,
resolved beyond what
was asked)
DB Garrison et al (2001); Slavin
(1995); Vygotsky (1978)
Initial question Easily answered with other
source (no, maybe, yes)
DB Berge & Muilenburg (2000);
Garrison et al (2001);
Hara et al (1998); Savage
(1998); Wickstrom
(2003)
Subject line clarity (unclear,
somewhat clear, pretty
clear, very clear)
DB Hara et al (1998)
Message quality (poor,
fair, good, excellent)
DB Aviv et al (2003); Love
(2002)
Knowledge type (facts,
concepts, procedures,
metacognitive
knowledge)
DB Anderson & Krathwohl
(2001); Aviv et al (2003);
Hara et al (1998); Henri
(1992)
Processing level (remember,
understand, apply,
analyse, evaluate)
DB Anderson & Krathwohl
(2001); Aviv et al (2003);
Hara et al (1998); Henri
(1992); Meyer (2003)
Role of educator Compare student vs.
teacher messages based
on the following
characteristics:
% of total threads
started
% of total threads
ended
DB Blignaut & Trollip (2003);
Burstall (2000); Hara et al
(1998); Knowlton &
Knowlton (2001);
Mazzolini & Maddison
(2003)
Number of times
messages were read
BB stats None
Length of message Word count Biesenbach-Lucas (2003);
Hara et al (1998); Poole
(2000)
Response time (how long
it takes for someone to
respond to posted
message)
BB stats Hammond (2000); Hara
et al (1998); Yacci (2000)
Navigation issues Reasons given for using/not
using discussion board
Survey Burstall (2000); Hara et al
(1998); Hammond
(2000)
Area Variables used Data source Reference

780 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Subject line clarity (unclear,
somewhat clear, pretty
clear, very clear)
correlated with how
often a message was read
(reading rate) and how
long it took to respond to
a message (response
time)
DB and BB
stats
Hara et al (1998)
Challenges for
students
Challenges identified in
open-ended survey
questions
Survey Burstall (2000); Hara et al
(1998); Hammond (2000);
Knowlton & Knowlton
(2001); Son (2002);
Wickstrom (2003)
Type of user Participants were
compared to each other
on ALL variables
assessed in the study,
except those based on
survey questions
DB and BB
stats
Av iv et al (2003); Hammond
(2000); McGrath &
Hollingshead (1994);
Palloff & Pratt (1999);
Wickstrom (2003)
Attitude towards
discussion
Explanation of why
discussion board was/
was not useful in
survey questions
Survey Burstall (2000); Hara et al
(1998); Hammond (2000);
Schallert et al (2003)
Response time Time taken to respond to
a posted message
BB stats Hammond (2000); Hara et al
(1998); Yacci (2000)
Learning location Compare inside school
hours (9 a.m. until
3:30 p.m.) vs. outside
school hours on the
following variables:
BB stats None
Subject line clarity
(unclear, somewhat
clear, pretty clear, very
clear) correlated with
reading rate and
response time
Message quality (poor,
fair, good, excellent)
Response time (How long
it takes for someone to
respond to posted
message)
Length of message
Easily answered with
other source (no,
maybe, yes)
Area Variables used Data source Reference

Developing a comprehensive metric 781
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Appendix B: Detailed rubric for assessing online discussion data
Learning
performance
Correlation of final test
score with total
number of visits,
number of days visited
and number of
messages posted
Correlation of final
project grade with total
number of visits,
number of days visited
and number of
messages posted
BB stats Beaudoin (2002); Hiltz et al
(2000); Wu & Hiltz
(2004)
Variable Rating Criteria
Subject line clarity 0—Unclear Confusing or cryptic subject line—often only
one or two words
1—Somewhat clear Subject line is vague but does capture the
general but not the specific topic area
2—Pretty clear You have a good idea what the specific topic is
based on the subject line; however, the exact
nature of the message content is not
completely evident
3—Very clear The subject line matches exactly what the
message is about
Question easily
answered from
other sources
0—No Question is asked that is idiosyncratic or very
challenging and requires social interaction
(eg, course or assignment-specific question)
1—Maybe If a student were resourceful, he might be able to
find an answer to the question using another
source (eg, searching the web, consulting a
book)
2—Yes Question can be easily answered without the
discussion board (eg, given in handout or
course web page)
Posting time
(learning location)
1—In school Message posted from 9:00 a.m. to 3:30 p.m.
2—Outside school Message posted from 3:31 a.m. to 8:59 p.m.
Response time Minutes Difference between the time a message is posted
and the time the following message is posted
Number of times
read
Blackboard calculates and posts the number of
times each message is read
Primary purpose
of a message
1—Open Open question directed at the class
2—Specific Specific question asked of a student in the
discussion
3—Reply Reply to a question asked
4—Independent Independent comment made that is not related
to the topic being discussed in a thread
Area Variables used Data source Reference

782 British Journal of Educational Technology Vol 37 No 5 2006
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
5—Nonacademic A comment made that is not course related and
adds no educational value
Message clarity 0—Unclear Message is unclear or confusing—it is typically
followed by a message asking for clarification
1—Somewhat clear Message is somewhat clear, but there are still
confusing or vague points that need
clarification
2—Clear The message is clear and appears to be
understood by the participants in the
discussion thread
New knowledge
added
1—No No new knowledge was added in a message
2—Yes (indirectly) New knowledge was added but was not related
to the initial question or discussion topic OR a
question is asked that implies that something
new can be done
3—Yes (directly) New knowledge added that is directly related to
the discussion thread topic
Message quality 0—Incorrect Incorrect information is provided
1—Poor Information provided is unrelated to the
discussion thread or course
OR
a question asked is confusing and hard to follow
2—Fair Information provided answers one aspect of
question with a thread, but not whole
question
OR
a question is asked out of context of the thread
3—Good Information is provided that answers most or all
parts of question within a thread
OR
a relevant and clear question is asked
4—Excellent Information provides clear and complete
response to a question asked and ADDS
additional and relevant details
OR
insightful question is asked on an advanced topic
(one that goes beyond the course curriculum)
often promoting considerable dialogue and/or
debate
Course knowledge 1—None No knowledge is provided (eg, social comment)
2—Unrelated Knowledge is provided that is unrelated to the
course
3—Administrative Administrative knowledge (eg, due dates or the
requirements for the final project)
4—Course Knowledge is provided that supports the course
curriculum
5—Beyond course Knowledge is provided that goes beyond the
course curriculum
6—Very advanced Knowledge that goes well beyond the course
curriculum—usually only a few students can
understand
Variable Rating Criteria

Developing a comprehensive metric 783
© 2006 The Author. Journal compilation © 2006 British Educational Communications and Technology Agency.
Knowledge type 1—Fact Student offers an isolated fact (eg, location of a
website, syntax of a specific command, code
for a colour)
2—Concept Student presents two or more connected facts
(eg, connecting facts with conjunctive
adverbs like because, consequently, therefore,
otherwise)
3—Procedure Student provides information on how to achieve
a specific task
4—Metacognitive Student is reflecting about a strategy to solve a
problem task or emotional state while
learning
Processing level 0—Clarification Student is asking what a question or comment
means—often referring to a specific element
or fact in a problem
1—Remember Evidence that student is recalling or trying to
recall a fact, concept or procedure
2—Understand The student understands or is trying to
understand a concept or a procedure
3—Apply A student is applying or trying knowledge which
typically involves the use of a procedure
4—Analyse A student is actively making connections
between two or more concepts
5—Evaluate Student provides comments about effectiveness
of a procedure or approach to solving a
problem
External resources
used
1—None/unknown No clear resources are noted or evident
2—Teacher/course
information
Reference is made to the teacher or course
information in a message
3—Another message Information from another posted message is
referenced
4—Programming or
HTML code
Reference is made to specific HTML or
programming code
5—Web A website is referenced
6—Book Reference is made to the course textbook
Resolution of
discussion thread
1—Unresolved Information was not given to solve the
question(s) raised in the thread
2—Partially resolved Information is offered that partially answers the
question (s) being asked in the thread
3—Resolved Complete and correct information is provided to
resolve the questions being asked in the
thread
4—Resolved beyond
what was asked
Information offered that answers and goes
beyond the question being asked in the thread
Variable Rating Criteria