The comparative effectiveness of Web-based and classroom instruction: A meta-analysis

Abstract

Meta-analytic techniques were used to examine the effectiveness of Web-based instruction (WBI) relative to classroom instruction (CI) and to examine moderators of the comparative effectiveness of the 2 delivery media. The overall results indicated WBI was 6% more effective than CI for teaching declarative knowledge, the 2 delivery media were equally effective for teaching procedural knowledge, and trainees were equally satisfied with WBI and CI. However, WBI and CI were equally effective for teaching declarative knowledge when the same instructional methods were used to deliver both WBI and CI, suggesting media effects are spurious and supporting Clark's (1983, 1994) theory. Finally, WBI was 19% more effective than CI for teaching declarative knowledge when Web-based trainees were provided with control, in long courses, and when trainees practiced the training material and received feedback during training. Study limitations and directions for future research are discussed.

Full text

peps˙049 Peps-xml.cls June 13, 2006 16:22
PERSONNEL PSYCHOLOGY
2006, 59, 623–664
THE COMPARATIVE EFFECTIVENESS
OF WEB-BASED AND CLASSROOM INSTRUCTION:
A META-ANALYSIS
TRACI SITZMANN
Advanced Distributive Learning
KURT KRAIGER
Colorado State University
DAVID STEWART
University of Tulsa
ROBERT WISHER
Department of Defense
Meta-analytictechniqueswere usedtoexaminetheeffectivenessofWeb-
based instruction (WBI) relative to classroom instruction (CI) and to
examine moderators of the comparative effectiveness of the two delivery
media. The overall results indicated WBI was 6% more effective than
CI for teaching declarative knowledge, the two delivery media were
equally effective for teaching procedural knowledge, and trainees were
equally satisfied with WBI and CI. However, WBI and CI were equally
effectiveforteachingdeclarativeknowledgewhenthesame instructional
methodswere usedto deliver bothWBI and CI, suggesting media effects
are spurious and supporting Clark’s (1983, 1994) theory. Finally, WBI
was 19% more effective than CI for teaching declarative knowledge
when Web-based trainees were provided with control, in long courses,
and when trainees practiced the training material and received feedback
during training. Study limitations and directions for future research are
discussed.
Web-based instruction (WBI) is becoming a favored training option
in industry, government, and higher education. WBI is a “hypermedia-
based instructional program which utilizes the attributes and resources of
the World Wide Web to create a meaningful learning environment where
learning is fostered and supported” (Khan, 1997, p. 6). WBI is delivered
The authors are indebted to Kenneth Brown, Kevin Ford, Richard Mayer, Ann Marie
Ryan, and three anonymous reviewers for comments and suggestions on earlier versions of
this manuscript.
This research was funded by the Department of Defense through contract number
DASW01-03-C-0010. The views expressed here are those of the authors and do not neces-
sarily reflect the views or policies of the Department of Defense.
Correspondenceand requests for reprintsshould be addressed toTraciSitzmann,4650 N
Washington Blvd #115, Arlington, VA 22201; sitzmant@adlnet.org.
COPYRIGHT C
2006 BLACKWELL PUBLISHING, INC.
623

peps˙049 Peps-xml.cls June 13, 2006 16:22
624 PERSONNEL PSYCHOLOGY
via a computer using the Internet, making it capable of instant updating,
distribution, and sharing of information (Rosenberg, 2001). In a survey
of organizations in the American Society of Training and Development’s
benchmarking service, the percentage of companies using technology-
delivered training increased from 8% in 1999 to 28% in 2004, and more
than half of the technology-delivered courses in 2004 were online (Sugrue
&Rivera, 2005). In addition, over 1,100 institutions of higher education in
the United States offer online courses (Newman & Scurry, 2001). Finally,
the Army uses online instruction as a retention tool, with over 40,000 sol-
diers in 50 countries pursuing advanced degrees online (Symonds, 2003).
Given its growing popularity, it is important to understand whether or
notthis deliverymedium iseffective,whetherit is moreeffectivethanother
delivery media, and what contextual or methodological factors moderate
its effectiveness. In this study, effectiveness is operationalized as both
learning from and reactions to delivery media. We examine the cumulative
evidence of the effectiveness of WBI relative to classroom instruction (CI)
andmoderatorsof the comparativeeffectiveness of thetwodeliverymedia.
Effectiveness of WBI as an Applied Issue
The rush to implement WBI preceded empirical evidence of its ben-
efits. Given the increasingly widespread implementation of WBI, it is
important to determine whether or not WBI is effective for imparting
useful knowledge and skills. WBI will have utility to organizations and
institutions if it results in learning and retention, is well received by users,
and is cost-effective to the sponsoring organization or institution. There
have been few studies of the cost-effectiveness of WBI, but a sufficient
number of primary studies have now been conducted to determine its ef-
fectiveness with respect to learning and user reactions. However, Arbaugh
(2005) questioned whether single studies are useful for understanding the
impact of technology and course characteristics on WBI effectiveness. By
examining trends across studies, we can draw quantitative conclusions of
WBIeffectivenessonly adecade afterits introduction. Given evidencethat
WBI is effective, more organizations and institutions will be able to justify
the expenditures necessary to adopt it. If evidence suggests that it is not
as effective as existing delivery media, organizations and institutions may
be more cautious about replacing traditional delivery media with WBI or
seek to develop more effective online training methods. Finally, if WBI is
effective under some conditions and not others, organizations and institu-
tions that place training online can use the results of this study to identify
optimal learning conditions.
Accordingly, the present study is a meta-analysis of studies that com-
parethe effectivenessofWBI and CIfor deliveringinstruction onthe same

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 625
topic. CI will be used as a basis for comparison as it is the most common
deliverymedia (Sugrue & Rivera,2005) and because there is stilla paucity
of studies comparing WBI to other delivery media.
Effectiveness of WBI as a Theoretical Issue
The question of whether or not WBI is more effective than other deliv-
ery media has theoretical importance. WBI has several advantages which
may result in WBI being more effective than other delivery media. WBI
represents a non-linear instructional medium that may encourage deeper
processing and cognitive flexibility in learners (Spiro & Jehng, 1990) by
allowing trainees to more effectively integrate new information with ex-
isting knowledge (Salomon, 1988). WBI may also be a superior medium
to the extent that it offers a cluster of instructional methods (e.g., text,
audio, graphics, synchronous and asynchronous communication) that can
be tailored to meet individual needs. Arbaugh (2005) detailed clusters of
WBI features that may lead to greater instructional effectiveness including
media variety, facilitation of Web exploration, learner ease, and flexibil-
ity of use. WBI can also provide beneficial features that are not easily
replicable in CI, such as immediate feedback (Kulik & Kulik, 1988; Phye
& Andre, 1989). To the extent that WBI incorporates these features, we
might expect WBI to be more effective than other delivery media.
On the other hand, educational psychologist Richard Clark (1983,
1994) has been a long-time critic of studies and reviews that purport to
show that newer, technologically based instructional media are superior
to existing media (e.g., Fletcher, 1990; Kulik, 1994). Although media is
often used to refer to the general method of delivering training, here media
refers to technological devices used for the purpose of instruction (Clark
& Sugrue, 1995). Clark has argued that delivery media, such as com-
puters, video-teleconferencing, and the Internet, are inconsequential in
affecting learning outcomes, especially when compared with more pow-
erful influences such as individual differences and instructional methods.
Instructional methods refer to techniques used within a course to convey
course content such as lecture, reading textbooks, assignments, or group
discussions.
Clark (1983, 1994) criticized media effectiveness research on two
grounds. First, most studies fail to institute experimental controls suffi-
cient to rule out alternative explanations for group differences. Specif-
ically, individuals may choose the delivery media with which they are
most comfortable due to a lack of random assignment into WBI and CI.
Second, Clark argued that most prior studies have failed to isolate instruc-
tional attributes that are unique to a single medium. For example, WBI
may provide more opportunities for learner customization than CI, but (a)

peps˙049 Peps-xml.cls June 13, 2006 16:22
626 PERSONNEL PSYCHOLOGY
CI can provide some customization in some situations and (b) opportuni-
ties for learner customization are not unique to WBI. Clark suggested that
if studies fail to isolate attributes unique to the medium, results of those
studies cannot be accepted as evidence of the superiority of the medium.
In short, Clark concluded that there is nothing uniquely beneficial about
any computer-aided instructional medium (including WBI).
Clark’s position has received broad support (e.g., Bernard et al., 2004;
Russell, 1999), but is not without its critics. Cobb (1997) argued that
certain delivery media reduce the cognitive demands placed on learners
and media should be chosen to maximize cognitive efficiency, allowing
learners to spend less time mastering the material. Kozma (1994) argued
thatalthough itmay bedifficult toisolate individualinstructional attributes
to any single medium, it is possible to identify clusters of attributes (e.g.,
customization and hyperlinking) that are more efficiently accomplished
in one medium than others. For example, compared to CI, WBI is more
likely to offer customization of instructional methods and content, as well
as continual access.
Insummary,therearetwo schools ofthought with respectto the relative
effectiveness of WBI and CI. Clark’s position argues that no instructional
medium is uniquely advantageous. On the other hand, pro-technology
researchersbelievethat WBIprovides greaterflexibility andgreater access
to multiple instructional methods such that it may be superior to media
that are grounded in a single instructional method (Dumont, 1996; Hiltz
&Wellman, 1997; Sullivan, 2001).
We intend to apply meta-analytic methods to address the important
theoretical question of whether instructional media matters. After Clark
(1983, 1984), we will do so in three ways. First, we examine a subset of Q1
all studies in which a true experimental design is used. Clark has argued
that past research supporting technology-assisted instruction has failed to
execute proper experimental procedures that control for participant mo-
tivation or prior experience with the technology. Our report will support
Clark’s position, if we find positive mean effect sizes for learning when
analyzing all studies and no effects for media when analyzing only true
experiments.
Second, we examine a subset of studies that equate instructional meth-
ods across delivery media. WBI and CI have similar instructional methods
when all of the instructional methods included in WBI have a comparable
instructional method in CI. For example, when lecture is provided in CI, a
comparable instructional method in WBI is an online video of the lecture.
WBI and CI have different instructional methods when an instructional
methodis present inWBI or CI,and there isnot a comparableinstructional
method in the other delivery media. For example, CI may use role-plays
but WBI does not. Clark has argued that media studies often confound

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 627
delivery media with instructional methods, making it impossible to deter-
mine whether main effects are due to differences in media or instructional
methods. In order to address the effect of delivery media when instruc-
tional methods are equivalent, we isolate studies that compare WBI to
CI when identical instructional methods are used. Our research will sup-
port the pro-technology position if greater learning from WBI relative to
CI occurs even when instructional methods do not differ across delivery
media. In turn, our research will support Clark’s position if no learning
differences between WBI and CI are found when instructional methods
are the same across delivery media.
Third, we will examine a subset of studies in which WBI is used as a
supplement to CI (henceforth WBI-S). Although Clark has not addressed
the additive effects of multiple delivery media, if media do not matter, we
will find no difference in the relative effectiveness of CI and WBI-S (pro-
vided content is identical across groups). In contrast, the pro-technology
position will be supported if there is greater learning or more positive re-
actions when course content is delivered via multiple media as in the case
of WBI-S.
Comparisonsof WBI-Sto CI provide preliminary cumulative evidence
of the relative effectiveness of blended learning programs. Blended learn-
ing programs are those that provide some combination of offline and on-
line learning (Singh, 2003). Note that in practice, instructional methods
might include face-to-face instruction, synchronous or asynchronous on-
line chat rooms or discussions, posted lecture notes, assignments, and
manyotherinstructional methods.While therehavebeen severalanecdotal
accounts of the effectiveness of WBI-S, there has not yet been a thorough
review of the delivery medium. Given the practical and theoretical issues
previously outlined, we propose a number of hypotheses and research
questions.
Main Effects Hypotheses
The first objective is to examine the effectiveness of WBI relative to
CI for teaching declarative and procedural knowledge and for training
reactions. Declarative knowledge refers to trainees’ memory of the facts
and principles taught in training and the relationship among knowledge
elements (Kraiger, Ford, & Salas, 1993). Declarative learning outcomes
include changes in verbal knowledge, how knowledge is organized, and
in cognitive strategies for accessing and applying knowledge. Procedural
knowledge refers to information about how to perform a task or action
(Kraiger et al., 1993). Procedural learning outcomes include compilation
(i.e., proceduralizing steps and mentally grouping the steps into a more
complex production) and automaticity (i.e., accomplishing tasks without

peps˙049 Peps-xml.cls June 13, 2006 16:22
628 PERSONNEL PSYCHOLOGY
conscious cognitive effort which enables simultaneous performance of
additional tasks).
Goldstein and Ford (2002) and Welsh, Wanberg, Brown, and Sim-
mering (2003) identified four advantages of WBI: consistent world-wide
training,training can bedeliveredjust-in-time to meetthe jobrequirements
of employees, reduced information overload, and training can be tailored
and refined to meet the needs, prior knowledge, and interests of individ-
ual learners. These advantages are both strategic (consistent world-wide
training) and pedagogical (individually tailored). Other pedagogical ad-
vantages include greater flexibility and access to multiple learning modes
(Dumont, 1996; Hiltz & Wellman, 1997; Sullivan, 2001). Welsh et al.
(2003) also summarized a number of instructional disadvantages of WBI,
includingthe potential forlack ofinteraction among peers(both duringand
followingtraining) andthe potential forstatic, non-interactiveinformation
processingreplacing more dynamic forms oflearning. Inaddition, learners
maylack Internetaccess orsufficient bandwidth to optimize training deliv-
ery or learners may lack the technical skills needed to access instructional
content or upgrade training software or hardware.
Zhao, Lei, Lai, and Tan (2005) conducted a meta-analysis to compare
the effectiveness of distance education courses (i.e., courses where the
instructor and students are physically separated) to face-to-face courses
and found no difference in the overall effectiveness of the two delivery
media. However, several previous meta-analyses have reported overall
positive effect sizes for various forms of technology-delivered instruction
compared to CI including videodiscs (Fletcher, 1990), computer-assisted
training (Kulik, 1994; Kulik & Kulik, 1991; Yaakub, 1998), and hyper-
media systems (Liao, 1999). In addition, earlier meta-analyses by Olson
and Wisher (2002) and Paul (2001) suggested WBI is more effective than
CI. Olson and Wisher (2002) reported a corrected mean effect size of .24
based on a meta-analysis of 15 studies, suggesting WBI is 9% more ef-
fective than CI for teaching declarative and procedural knowledge. Paul
(2001) reported a corrected mean effect size of .17 (k=27) for training
reactions and .24 (k=47) for learning criteria suggesting trainees react
7% more favorably toward WBI than CI and WBI is 9% more effective
than CI for learning criteria. However, both Olson and Wisher (2002) and
Paul (2001) completed their literature searches before 2003 (precluding
the inclusion of research reports written in the last 3 years) and averaged
across declarative and procedural knowledge outcomes (precluding an un-
derstanding of whether the effectiveness of WBI varies across factual and
skill-based knowledge). In addition, Olson and Wisher (2002) focused
exclusively on university courses and Paul (2001) did not distinguish be-
tween training reactions and learning criteria when testing the effective-
ness of WBI-S. The present meta-analysis will overcome these limitations

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 629
by examining 96 research reports from 1991 to 2005, including employee
andcollege training courses, and clearlydistinguishing between bothWBI
andWBI-S and amongdeclarative knowledge, procedural knowledge, and
training reactions. Based on the results of preliminary meta-analyses, we
hypothesize:
Hypothesis 1: WBI will be more effective than CI for teaching declarative
knowledge.
Hypothesis 2: WBI will be more effective than CI for teaching procedural
knowledge.
Hypothesis 3:Trainees will react more favorably toward WBI than CI.
In addition to studying the relative effectiveness of WBI, we are also
interested in the relative effectiveness of WBI-S compared to CI. Accord-
ingto media richnesstheory,richer media (i.e.,providing the same content
through multiple media) result in greater learning, particularly for equiv-
ocal or ambiguous tasks (Daft & Lengel, 1986). In other words, learners
benefit from exposure to both CI and WBI.
In both higher education and corporate training, WBI-S is known as
blended learning. Blended learning is perceived by many as a strong in-
structional approach that incorporates the benefits of both personal in-
teraction and facilitated instruction with self-study between instructional
meetingsusing the Web(Kerres&deWitt,2003;Masie,2002; Pratt, 2002).
In particular, blended learning is seen as advantageous in that it fosters
learningcommunities, extendsthe totallength oftraining, offersfollow-up
resources in a community of practice, can provide access to guest experts,
and offers timely mentoring or coaching via either face-to-face or online
laboratory and simulation activities (Bonk, Kim, & Zeng, 2005). In addi-
tion to these hypothesized advantages, there is preliminary empirical sup-
port for blended learning. Paul (2001) reported a meta-analytic corrected
effect size of .27 indicating WBI-S was 11% more effective than CI when
averaging across reactions and learning criteria. Thus, we hypothesize:
Hypothesis4: WBI-Swill be more effectivethanCI for teaching declarative
knowledge.
Hypothesis5: WBI-Swill be moreeffectivethanCIfor teaching procedural
knowledge.
Hypothesis 6:Trainees will react more favorably toward WBI-S than CI.
Research Design Moderators
Trainee population. A second objective of the study is to examine
moderators of the effectiveness of WBI relative to CI. The overwhelm-
ing majority of the research on the effectiveness WBI relative to CI has

peps˙049 Peps-xml.cls June 13, 2006 16:22
630 PERSONNEL PSYCHOLOGY
focused on declarative knowledge, precluding an analysis of the effect of
the moderator variables on the acquisition of procedural knowledge and
training reactions. Thus, the moderator hypotheses and research questions
focus exclusively on declarative knowledge.
The first moderator analysis examines differences in the effectiveness
of WBI relative to CI for college student and employee population, after
controlling for the age of trainees. We control for the age of trainees be-
cause trainees tend to be older in corporate training than college courses
(U.S. Bureau of the Census, 2006) and in WBI than CI (Bocchi, 2004;
Schneider & Germann, 1999; Tallent-Runnels et al., 2006). Research
also indicates trainees in their late 20s and 30s are more motivated, have
more positive attitudes toward training, are less anxious, and focus more
on achieving specific learning outcomes than younger trainees (Graham,
1991; Tallent-Runnels et al., 2006). Controlling for age effects allows us
to assess if there are differences in the effectiveness of WBI relative to CI,
after controlling for a demographic characteristic that is related to both
the trainee population and learning.
We focus exclusively on research reports where trainees are acquiring
knowledge to prepare them for their current or future employment oppor-
tunitiesin orderto generalizeto asample of working adults. Toaccomplish
this, we include both college and work-related training courses. There is
no theoretical reason to believe online instruction and is more or less
effective in university versus work settings and many corporations have
established training partnerships with universities, outsourcing employee
training to universities (e.g., Cisco, British Airways, and Merck outsource
to Duke University and Cardinal Health and the New Jersey State Police
outsource to The New Jersey Institute of Technology; Harris, 2005). Thus,
a clear distinction between university and corporate training courses may
no longer exist. However, we want to determine whether there is a differ-
ence in the relative effectiveness of WBI and CI for teaching declarative
knowledge across populations to ensure the results are applicable to both
university and corporate settings. Accordingly, the first research question
is:
Question 1:Will the population (student vs. employees) moderate the ef-
fectiveness of WBI relative to CI for teaching declarative knowledge, after
controlling for the age of trainees?
Similarity in instructional methods. The second moderator analysis
investigatestheeffectivenessof WBI relativeto CIafter eliminatingdiffer-
ences in instructional methods. To reduce experimental biases previously
discussed, we examine differences in the effectiveness of the two delivery
media when the same instructional methods are used to deliver both WBI
and CI. If Clark (1983; 1994) is correct, any observed effects of WBI

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 631
for teaching declarative knowledge should disappear when we control for
instructional methods.
Second, we examine a subset of studies that use true experimental de-
signs. This addresses Clark’s concern that media comparison studies often
confound instructional media with instructional quality, student motiva-
tion, and so forth. Random assignment of trainees to WBI and CI should
reduce differences between test groups that may confound observed ef-
fects on learning or reactions. If Clark’s position is correct, any observed
effects for WBI relative to CI should disappear when we control for the
experimental design. Consistent with Clark’s position, we propose the
following hypotheses:
Hypothesis 7: Similarity of the instructional methods used in WBI and CI
willmoderate learning declarativeknowledgefrom WBI relativetoCI. That
is, any effects observed for all studies will disappear when examining only
studies where similar instructional methods are used in WBI and CI.
Hypothesis 8: The research design will moderate learning declarative
knowledge from WBI relative to CI. That is, any effects observed for all
studies will disappear when examining only studies with true experimental
designs.
Learner control. Learner control refers to the extent to which trainees
have control over their learning experience by affecting the content, se-
quence, or pace of material (Friend & Cole, 1990). The absence of learner
control is characterized by program control in which the instructional
software controls most or all of the decisions in WBI.
A purported advantage of WBI is it typically provides trainees with
more control than CI (Welsh et al., 2003). However, research also shows
the effect of learner control on actual learning is negligible (Kraiger &
Jerden, in press; Niemiec, Sikorski, & Walberg, 1996). As prior research
has not consistently demonstrated an effect for learner control on learner
achievement, we cannot develop a directional hypothesis regarding the
moderatingeffect of learner controlon theeffectivenessof WBI.However,
given the potential for individual customization in online courses, we are
interested in the effect of learner control during WBI. Thus, we propose
the following research question:
Question 2:Will the level of learner control moderate learning of declar-
ative knowledge from WBI relative to CI? Relative to CI, will participants
learn more declarative knowledge with low or high levels of learner control
in WBI?
Human interaction. Human interaction refers to the extent to which
trainees interact with the instructor and other trainees throughout the
course. Although prevalent in CI, human interaction can also be built into
WBI through e-mails, chat rooms, group projects, and so forth. A number

peps˙049 Peps-xml.cls June 13, 2006 16:22
632 PERSONNEL PSYCHOLOGY
of studies have found that higher levels of interaction between instruc-
tors and learners or among learners result in greater learner motivation,
more positive attitudes toward learning or the instructional process, and
improved learning outcomes (e.g., Entwistle & Entwistle, 1991; Hackman
&Walker, 1990; Ritchie & Newbury, 1989; Wagner, 1994). In WBI, ver-
bal behaviors (e.g., text messages) that establish immediacy are associated
with greater participant learning (Freitas, Myers, & Avtgis, 1998; Rovai &
Barnum, 2003). Human interaction decreases the likelihood that trainees
will feel isolated in WBI and can help trainees remain motivated while
learning the material (Brown & Ford, 2002). Consequently, we hypothe-
size:
Hypothesis9: Human interaction willmoderate the extent towhich trainees
learndeclarativeknowledgefromWBIrelativetoCI.RelativetoCI,trainees
will learn more declarative knowledge from WBI with high levels of human
interaction than with little human interaction.
Practice and feedback. Both the opportunity to practice and the pro-
vision of feedback were included in Kraiger’s (2003) guidelines for de-
signing effective training. Practice is essential for skill acquisition and
feedback is needed for trainees to know whether they are effectively using
their newly acquired knowledge and skills (Brown & Ford, 2002). Further,
Azevedo and Bernard (1995) conducted a meta-analysis of 22 computer-
basedtrainingstudies andfound students whowere givenfeedback learned
more than students who were not given feedback. Thus, the relative effec-
tiveness of WBI and CI should be contingent upon whether one or both
delivery media incorporated practice and feedback during training. We
therefore hypothesize:
Hypothesis 10: Practice will moderate the extent to which trainees learn
declarative knowledge from WBI relative to CI. Relative to CI, trainees will
learn more declarative knowledge when they practice during WBI.
Hypothesis 11: Feedback will moderate the extent to which trainees learn
declarative knowledge from WBI relative to CI. Relative to CI, trainees
will learn more declarative knowledge when they receive feedback during
WBI.
Lengthof training. We alsoexplorethe effectsofthe lengthof training
on learning from WBI relative to CI. The training programs we reviewed
varied tremendously in length, ranging from 1 to 120 days. Course length
may differentially influence learning depending on whether trainees be-
come more proficient at learning or whether early novelty effects wear off.
Thus, we are curious as to whether the effectiveness of WBI relative to CI
decreases, increases, or remains the same as the course length increases.
Thus, we propose an additional research question:

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 633
Question 3:Will the length of training moderate learning declarative
knowledge from WBI relative to CI? Relative to CI, will trainees learn
more, less, or the same amount from WBI as course length increases?
Method
Literature Search
A computer-based literature search of PsycInfo and ERIC was used
to locate studies in the training and education literature from 1996 to
February 2005. The technology for online instruction is relatively new;
hence, we designated 1996 as a reasonable cutoff date for evaluations of
WBI. We scanned references of the obtained studies for earlier citations
and found only two relevant studies published prior to 1996.
In order to be included in the initial review of abstracts, each abstract
had to contain a term relevant to the Internet and reactions or learning out-
comes. To meet the search criteria, some combination of the keywords—
Web, online,orInternet and evaluate, learn, transfer, behavior, perfor-
mance, knowledge, satisfaction, dissatisfaction, reaction, achieve, or out-
come—had to be present. The initial computer search resulted in a list of
3,461 possible reports. A review of titles and abstracts reduced the list
to 249 reports potentially containing relevant information. Reading the
reports identified 59 relevant studies. The electronic search was supple-
mented with manual searches of the reference lists from Allen, Bourhis,
Burrell, and Mabry (2002), Bernard et al. (2004), Hsu (2003), Olson and
Wisher (2002), and Paul (2001), as well as a manual search of the Journal
of Asynchronous Learning Networks from 1996 to 2005. Manual searches
contributed an additional 33 studies to the present review.
We also searched for unpublished studies. First, a request was sent to
the Advanced Distributed Learning listserv of over 8,000 people working
in the area of training and development. Second, authors of annual review
chapters on training (Campbell, 1971; Goldstein, 1980; Latham, 1988;
Salas& Cannon-Bowers, 2001; Tannenbaum&Yukl,1992; Wexley,1984)
and training textbooks (Blanchard & Thacker, 2004; Goldstein & Ford,
2002; Noe, 2005; Saks & Haccoun, 2004; Wexley & Latham, 2002) were
asked to provide leads to unpublished work, as well as any manuscripts
they may have. Third, consultants who listed training evaluation as an
area of expertise on the Society of Industrial and Organizational Psychol-
ogy (SIOP) Consultant Locator (http://www.siop.org/sioplocator) were
contacted via e-mail. Fourth, the SIOP and Academy of Management
conference programs from 1996 to 2005 were manually searched to lo-
cate relevant studies. These efforts identified an additional four studies,
yieldinga totalof 96 studies that met the criteria for inclusionin thepresent
review.

peps˙049 Peps-xml.cls June 13, 2006 16:22
634 PERSONNEL PSYCHOLOGY
Inclusion Criteria
The goal of the literature search was to identify all research reports
where college students or employees were acquiring knowledge or skills
to prepare them for current or future employment opportunities. Initially
we also gathered research reports that reported gain scores from partici-
pating in WBI or that compared learning from or reactions following WBI
or WBI-S to CI. However, due to the upward bias in effect sizes from
gain score research (Lipsey & Wilson, 2001), the present report focuses
exclusively on studies that compared the effectiveness of WBI or WBI-S
to CI. WBI was defined as a course where the material is delivered via
the Internet. CI was defined as a course where the material is delivered
face-to-face via an instructor. WBI-S was a course that delivers material
via the Internet and face-to-face via an instructor.
Studieshad tomeet fivecriteria tobe included in the present review:(a)
thestudy comparedthe effectivenessofWBI or WBI-S to CIfor delivering
material on the same topic; (b) the article was written in English; (c) the
article reported results that allowed the calculation of a dstatistic (e.g.,
group mean values and standard deviations, a t-test, or univariate F-test)
orthe author compliedwith a requestto providethis information; (d)study
participants were non-disabled adults aged 18 or older; and (e) training
was conducted on a topic that provided job-related knowledge or skills.
The last two criteria were used to support generalization to a population
of adults participating in workplace training.
Data Set
Non-independence. Decisions about non-independent data points
(i.e., multiple effect sizes from one sample) should take into account
whether the effect sizes assess similar or different constructs (Arthur,
Bennett, & Huffcutt, 2001). Effect sizes calculated for different criteria
(i.e., training reactions, declarative knowledge, and procedural knowl-
edge) were considered to be independent and retained as separate data
points even if they were from the same sample. Occasionally a single
study would report data from two Web-based training groups and/or two
classroom groups. In these situations, an effect size was calculated for all
possible Web-classroom pairs and averaged by weighting each of the ef-
fect sizes by the sum of the sample size of the independent training group
and one half of the sample size of the non-independent group. Thus, the
non-independent sample was weighted according to its sample size in the
overall effect size. In addition, whenever a single study reported multiple
effect sizes based on the same sample for a single criterion, the effect size

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 635
that was most similar to the other assessments of that particular relation-
ship was used in the meta-analysis.
Coding and Interrater Agreement
In addition to recording all relevant effect sizes, sample sizes, and
reliabilities, the following information were coded from each study: (a)
reaction measures, (b) learning outcome criteria, (c) age, (d) population,
(e) similarity of instructional methods in WBI and CI, (f) experimental
design, (g) degree of learner control, (h) human interaction, (i) practice,
(j) feedback, and (k) length of training. Coding rules are described be-
low. Scales for each moderator were drafted prior to coding and modified
following initial attempts to code articles.
Training reactions. We initially sought to code and investigate the
comparative effectiveness of WBI and CI on specific dimensions of train-
ing reactions (e.g., affect vs. utility). However, research reports often av-
eraged across reaction dimensions when reporting the research results
and too few studies were available within certain reaction dimensions.
Accordingly, while separate dimensions of training reactions were coded
whenever possible, specific dimensions were treated as indicators of an
overall satisfaction construct by aggregating all studies that reported any
reaction effect size in a single analysis. To avoid violating the assumption
of independence, the effect sizes were averaged when multiple reactions
were reported in a single study.
Learning outcomes. Declarative and procedural knowledge were
coded based on the Kraiger et al.’s (1993) multi-dimensional framework
of learning. Declarative outcomes were defined as cognitive and struc-
tural knowledge assessments designed to assess if trainees remembered
concepts presented in training; they were always assessed with a writ-
ten test. Procedural outcomes were defined as the ability to perform the
skills taught in training. They were assessed by participating in an activ-
ity (e.g., simulation or role-play) or written test that required trainees to
demonstrate memory of the steps required to complete the skills taught in
training. For example, Browning (1999) taught an undergraduate course
on educational technology and evaluated the course with both declara-
tive and procedural knowledge assessments. The declarative knowledge
assessment consisted of a multiple-choice and fill-in-the-blank examina-
tion designed to assess understanding of the concepts taught in the course.
The procedural knowledge assessment required trainees to perform the
software application skills taught in training.
Age. We coded the average age of trainees in WBI and CI.
Population. We coded whether trainees were college students or em-
ployees.

peps˙049 Peps-xml.cls June 13, 2006 16:22
636 PERSONNEL PSYCHOLOGY
Similarity of instructional methods. Similarity of instructional meth-
ods across media was coded on a two-point scale. An instructional method
is a technique used to deliver training content (e.g., lecture, online tutori-
als, video, and textbooks). WBI and CI had similar instructional methods
whenallof the methodspresent in WBIhadcomparable methodspresentin
CI. An example is a management information systems course researched
by Carey (2001). Both WBI and CI included a textbook, practice ex-
aminations, and assignments. CI received lecture and discussed with the
instructor face-to-face, whereas WBI received a copy of the PowerPoint
slides from the lecture online and e-mailed with the instructor. WBI and
CI had different instructional methods when a method was present in WBI
or CI and there was not a comparable method in the other delivery me-
dia. An example is an introductory psychology course studied by Taylor
(2002). In this instance, CI was delivered via lecture, quizzes, and a text-
book, whereas WBI was delivered via quizzes, a textbook, assignments,
discussion board, and e-mail.
Experimental design. Research reports utilized an experimental de-
sign when trainees were randomly assigned to WBI and CI. Research
reports utilized a quasi-experimental design when trainees self-selected
into WBI or CI.
Learner control. Learner control was coded on a two-point scale
separating low from high levels of control. Learner control can include
control over the content, sequence, and pace of training (Friend & Cole,
1990). In the present study, learner control was low if trainees had little or
no control over the content, sequence, or pace. Examples of Web-based
courseswith a lowlevelofcontrol are non-interactivelecture-basedclasses
or a computer-controlled sequence of activities completed in a set amount
of time. Learner control was high when trainees had at least some control
overtwo of the three dimensions (content, sequence, or pace) or a high
level of control over one learner control dimension. Examples of courses
with a high level of control are managerial courses where trainees can
select material that is relevant to their jobs and courses where trainees
have several months to review the online content. Initially we coded for
pace,content, and sequence control separatelyand weattempted toinclude
three categories of control: high, medium, and low. However, because
many of the articles provided little description of the training programs
or did not provide information on all of the learner control dimensions, it
was difficult to assess the degree to which training provided control over
specific dimensions. Moreover, pace, content, and sequence control were
highly correlated. Thus, overall control was coded as either high or low.
Human interaction. Human interaction was coded on a two-point
scale where low indicated less than half of the course involved interacting
with people (instructor or other trainees). An example of a Web-based

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 637
coursewithlittle humaninteractionis acoursein whichtraineesparticipate
in an online discussion once a week or less. Human interaction was coded
as high when all or most of the course involved interacting with others.
An example of a Web-based course with a high level of interaction is a
course in which trainees frequently work on group projects and participate
in online discussions.
Practice. Practice was coded on a dichotomous scale to indicate
whether WBI and CI required trainees to practice the training material.
Practice activities include completing assignments, participating in role-
plays, taking practice exams, and writing papers.
Feedback. Feedback was coded on a dichotomous scale to indicate
whether WBI and CI provided feedback to trainees on whether they were
successfully learning the course material.1
Length of training. Length of training was coded as the number of
days trainees spent in WBI and CI.
Coding agreement. All articles were coded independently by two
trained raters. The initial mean level of agreement across all of the stud-
ies coded was 91%. The two coders then discussed discrepancies and
reached a consensus. After discussing all discrepancies, 100% agreement
was reached.
Calculating Effect Size Statistic (d) and Analyses
The Hedges and Olkin (1985) approach was used to analyze the data.
The effect size calculated for each study was d, the difference between
the Web and classroom training groups, divided by the pooled standard
deviation. When means and standard deviations were not available, effect
sizes were calculated from a t-test or univariate F-test based on the formu-
las reported in Glass, McGaw, and Smith (1981) and Hunter and Schmidt
(1990).
Effect sizes were corrected for small sample bias using formulas pro-
vided by Hedges and Olkin (1985). We then corrected the reactions effect
sizes for attenuation using the scale reliabilities reported in each study.
When a study failed to provide a coefficient alpha reliability estimate, we
used the average reliability for the variable across all samples from the
presentstudy andfrom Sitzmann,Casper,Brown, Witzberger,andPolliard
(2003).While we aggregatedall effectsizes for reaction measures, we cor-
rected effect sizes at the study level based on the type of reaction measure.
The average reliabilities were .83 for measures of affective, utility, and
1We attemptedtousemore sophisticated coding schemes for human interaction, practice,
and feedback, but lack of detail in primary studies prevented us from using more than a
dichotomous scale.

peps˙049 Peps-xml.cls June 13, 2006 16:22
638 PERSONNEL PSYCHOLOGY
difficulty reactions, .87 for instructor reactions, .79 for delivery reactions,
and .84 for reports that averaged across several reactions dimensions. We
did not correct the declarative or procedural knowledge effect sizes for
attenuation due to the lack of available test–retest or alternate forms re-
liability coefficients. Finally, 95% confidence intervals were calculated
around the weighted mean ds. Confidence intervals assess the accuracy of
theestimate of the meaneffect size and provide an estimate of the extentto
which sampling error remains in the weighted mean effect size (Whitener,
1990).
Outliers Analysis
We computed Huffcutt and Arthur’s (1995) sample-adjusted meta-
analytic deviancy (SAMD) statistic to identify outliers. This procedure
identified one declarative knowledge outlier reported by Vessell (2000).
Studentsin CIaccessed coursematerial that was intendedto be exclusively
utilized by students in WBI-S, providing students in CI with a competi-
tive advantage and resulting in CI outperforming WBI-S. The associated
SAMD value of 10.8 was more than twice the value of the next data point.
Inaddition, one reaction outlier reported by Stadtlander(1998) was identi-
fied in which students in WBI encountered extensive technical difficulties,
resulting in students being more satisfied with CI than WBI. The associ-
ated SAMD value of 10.76 was more than twice the value of the next
data point. All of the analyses were run with and without the outliers. The
results of the two sets of analyses were virtually identical. Thus, only the
results with outliers removed are included in the present report.2
Moderator Analysis
Hedges and Olkin’s (1985) homogeneity analysis was used to deter-
mine whether the effect sizes were consistent across studies. For main
effect analyses, the set of effect sizes was tested for homogeneity with the
QTstatistic. QThas an approximate χ2distribution with k–1degrees of
freedom, where kis the number of effect sizes. If QTexceeds the critical
value, then the null hypothesis of homogeneity is rejected. Rejection in-
dicates there is more variability in effect sizes than expected by chance
fluctuations, identifying the potential for moderators.
The goal of the moderator analysis was to focus exclusively on studies
that were consistent in their operationalization of WBI and CI. Oswald
and McCloy (2003) recommended narrowing the set of studies included
2Results with outliers included in the analyses are available upon request from the first
author.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 639
in meta-analyses to a subset of studies that are theoretically and rationally
similar to each other. We eliminated WBI-S studies, reducing the analysis
sample but increasing the interpretability of the results. In addition, only a
few studies examined the effectiveness of WBI relative to CI for teaching
procedural knowledge and for training reactions. Thus, insufficient data
were available to examine moderators of the relative effectiveness of the
delivery media for these two criteria, and the moderator analyses will
focus exclusively on the relative effectiveness of WBI and CI for teaching
declarative knowledge.
Forinvestigations of learner control and human interaction, we held
characteristics of CI constant. We focused our learner control moderator
analysis on classroom courses low in learner control. Only four reports
were based on CI that was high in learner control and we eliminated these
studies from the learner control analysis to increase the interpretability
of the results. This allowed us to compare effect sizes between WBI low
in learner control to WBI high in learner control. All of the classroom
courses were high in human interaction, allowing us to compare effect
sizes between WBI low and high in human interaction.
Inthe training length moderator analyses, we focusedon studies where
the number of days spent in training was the same for WBI and CI (elim-
inating four studies). This allowed us to examine the effect of varying
course length on the relative effectiveness of WBI and CI. For practice
and feedback, we utilized information from WBI and CI in the mod-
erator analyses. Thus, we will report results for four moderator cate-
gories: WBI high and low by CI high and low for these two moderator
variables.
The moderating effects of categorical variables were tested by clas-
sifying studies according to the moderator categories and testing for ho-
mogeneity between and within categories (Lipsey & Wilson, 2001). For
each categorical moderator, a between-class goodness-of-fit statistic, QB,
was calculated to test for homogeneity of effect sizes across moderator
categories. It has an approximate χ2distribution with p−1degrees of
freedom, where pis the number of moderator categories. If QBexceeds
the critical value, it indicates there is a significant difference across mod-
erator categories and is analogous to a significant main effect in analysis
of variance. In addition, a within-class goodness-of-fit statistic, Qw,was
calculated to test for homogeneity of effect sizes within each moderator
category. It has an approximate χ2distribution with m−1degrees of
freedom, where mis the number of effect sizes across all of the moderator
categories. If Qwexceeds the critical value, it indicates the effect sizes
within the moderator categories are heterogeneous.
Themoderating effects of ageand populationwere tested witha sample
size weighted hierarchical regression analysis. The mean ages of the Web

peps˙049 Peps-xml.cls June 13, 2006 16:22
640 PERSONNEL PSYCHOLOGY
and classroom training groups were entered in step one and the population
(student vs. employee) was entered in step two in order to predict the
declarative knowledge effect sizes across research reports. Finally, the
moderating effect of length of training was tested with a sample size
weightedcorrelation between the moderatorvariable and the effect sizes.
Results
Ninety-six research reports contributed data to the present meta-
analysis, including 65 published studies, 18 dissertations, and 13 unpub-
lished studies. These studies reported data gathered from 19,331 trainees
who took part in 168 courses. The topic of the training courses varied
greatly and included psychology, engineering, computer programming,
business, and technical writing courses. In 67% of research reports, the
trainees were undergraduates, while trainees were graduate students (18%
ofcourses) or employees(15%of courses)in the remainingstudies. Across
all studies providing demographic information, the average age of partic-
ipants was 24 years and 41% of the participants were male.
Relative Effectiveness of WBI and WBI-S
The first and second hypotheses predicted WBI would be more ef-
fective than CI for teaching declarative and procedural knowledge. As
shown in Table 1, across all studies, the declarative knowledge effect size
was .15 indicating that, on an average, WBI was 6% more effective than
CI for teaching declarative knowledge. Moreover, the confidence interval
for declarative knowledge excluded zero, supporting Hypothesis 1. The
WBI procedural knowledge effect size was near zero (d=−.07) and the
confidence interval contained zero, suggesting WBI and CI were equally
effective for teaching procedural knowledge and failing to support Hy-
pothesis 2. Thus, across all studies, there is evidence that WBI was more
effective than CI for teaching declarative knowledge, but not for teaching
procedural knowledge.
Hypothesis 3 predicted trainees would react more favorably toward
WBI than CI. The mean corrected effect size was zero, suggesting trainees
were equally satisfied with the two delivery media and failing to support
Hypothesis 3.
The fourth and fifth hypotheses predicted WBI-S would be more ef-
fective than CI for teaching declarative and procedural knowledge. The
WBI-S effect size was .34 for declarative knowledge and .52 for pro-
cedural knowledge, suggesting WBI-S was 13% more effective than CI
for teaching declarative knowledge and 20% more effective than CI for
teaching procedural knowledge. The 95% confidence intervals for both

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 641
TABLE 1
Meta-Analytic Results for Learning Outcomes and Reactions Comparing
Web-Based Instruction and Web Supplements to Classroom Instruction
95%
Confidence
Interval
Standard
dError kNLower Upper QT
Declarative knowledge
WBI vs. CI .15 .02 71 10,910 .11 .19 267.49∗
WBI-S vs. CI .34 .03 33 6,799 .29 .39 135.26∗
Procedural knowledge
WBI vs. CI −.07 .07 12 944 −.20 .06 61.15∗
WBI-S vs. CI .52 .09 6 507 .34 .70 23.33∗
Reactions
WBI vs. CI .00 .05 22 2,580 −.09 .09 51.78∗
WBI-S vs. CI −.15 .06 11 1,769 −.26 −.05 119.67∗
Notes. WBI =Web-based instruction; CI =classroom instruction; WBI-S =Web
supplement to classroom instruction; d=inverse variance weighted mean effect size;
k=number of effect sizes included in the analysis; N=sum of the sample sizes for each
effect size included in the analysis; QT=homogeneity statistic.
∗Indicates the QTvalue is statistically significant at the .05 level and the effect sizes are
heterogeneous.
declarativeandproceduralknowledge excluded zero,supporting Hypothe-
ses 4 and 5.
The sixth hypothesis predicted trainees would react more favorably
toward WBI-S than CI. The mean corrected effect size for the WBI-S
versus CI comparison was negative (d=−.15) and the 95% confidence
intervalexcludedzero. Traineesreacted6% morefavorablytowardCI than
WBI-S, and the results were in the opposite direction of Hypothesis 6.
The QTstatistic for all six effect sizes reported in Table 1 were sta-
tistically significant, suggesting there are potential moderators of the ef-
fectiveness of WBI and WBI-S relative to CI. When we looked for main
effects for each delivery media for reactions, declarative knowledge, and
procedural knowledge, as stated above, the moderator analyses will focus
exclusively on declarative knowledge comparing WBI to CI. Conducting
focused analyses will allow us to draw stronger conclusions regarding
moderators of the effectiveness of WBI relative to CI.
Moderator Analyses
The first research question addressed the moderating effect of the
trainee population on declarative knowledge, after controlling for the age

peps˙049 Peps-xml.cls June 13, 2006 16:22
642 PERSONNEL PSYCHOLOGY
of trainees. Sample size weighted hierarchical regression analysis was
used to run the analysis. The mean ages of the Web and classroom training
groups were entered in step 1 and the population (student vs. employee)
was entered in step 2 to predict the declarative knowledge effect sizes
across research reports. The mean ages of the WBI and CI groups ac-
counted for a significant 44.2% of the variance. In addition, both beta-
weights were significant, and the WBI beta-weight was positive and the
CI beta-weight was negative (beta-weights =.47 and −.58, respectively;
p<.05). Thus, the extent to which Web-based trainees learned more than
classroom trainees increased, as the age of the Web-based trainees in-
creased and the age of the classroom trainees decreased. It is important to
note that the average age was 29 years for WBI and 24 years for CI and
ages ranged from 20 to 46 years for both groups. Finally, trainee popu-
lation accounted for a non-significant .5% of the variance in declarative
knowledge effect sizes after controlling for the age of trainees. Thus, the
effectiveness of WBI relative to CI did not differ for student and employee
samples, after controlling for the age of trainees.3
Table 2 shows mean effect sizes and estimates of homogeneity within
moderator subgroups (QB) for the subgroup moderator analyses. A signif-
icant QBindicates the mean effect sizes across categories of the modera-
tor variable differ by more than sampling error, indicating the moderator
variable is having an effect (Lipsey & Wilson, 2001). The QBstatistic
was significant for all of the categorical moderators (i.e., similarity of in-
structional methods, experimental design, learner control, practice, and
feedback) except human interaction.
Experimental characteristics. The seventh hypothesis predicted WBI
and CI would be equally effective for teaching declarative knowledge
when the instructional methods were the same across delivery media. The
declarative knowledge effect size was near zero when the same instruc-
tional methods were used to deliver WBI and CI (d=.04), supporting
Hypothesis7. However,WBIwas 11% moreeffectivethan CI for teaching
declarative knowledge when different instructional methods were used to
deliver the two courses (d=.29). This pattern of results supported Clark’s
(1983, 1994) theory that instructional methods rather than delivery media
determined whether students learn during training.
Theeighthhypothesis predicted WBIand CI wouldbeequally effective
forteaching declarativeknowledgewhen trainees wererandomly assigned
3We alsoranourage–population hierarchical moderator analysis with population entered
in the equation first and age entered in the equation second. Population did not account for
a significant portion of the variance in the effectiveness of WBI relative to CI, even when
it was entered in the equation before age. Thus, we can be confident that age rather than
population is moderating the effectiveness of WBI relative to CI.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 643
TABLE 2
Meta-Analytic Moderator Results Comparing the Acquisition of Declarative
Knowledge From Web-Based Instruction Relative to Classroom Instruction
95% Confidence Homogeneity of
Interval Effect Sizes
Standard
dError kNLower Upper QBQw
Instructional methods
Same .04 .05 16 2,032 −.06 .13 17.43∗215.12∗
Different .29 .04 37 3,689 .22 .37
Experimental design
Experimental −.26 .09 11 529 −.43 −.08 22.96∗244.53∗
Quasi-experimental .18 .02 60 10,381 .13 .22
Learner control
WBI low, CI low .07 .04 31 2,721 −.01 .15 15.13∗227.07∗
WBI high, CI low .30 .04 25 3,304 .22 .38
Human interaction
WBI low, CI low .19 .05 19 1,719 .09 .29 .00 242.33∗
WBI high, CI low .18 .03 38 4,508 .12 .25
Practice
WBI yes, CI yes .16 .03 41 4,163 .10 .23 12.42∗173.47∗
WBI yes, CI no .31 .06 10 1,527 .20 .42
WBI no, CI yes −.27 .36 1 31 −.98 .44
WBI no, CI no −.25 .19 2 116 −.63 .12
Feedback
WBI yes, CI yes .16 .04 33 3,333 .08 .24 10.80∗152.53∗
WBI yes, CI no .33 .06 11 1,540 .22 .44
WBI no, CI yes −.27 .36 1 31 −.98 .44
WBI no, CI no .08 .07 9 933 −.05 .22
Notes. WBI =Web-based instruction; CI =classroom instruction; d=inverse variance
weighted mean effect size; k=number of effect sizes included in the analysis; N=sum
of the sample sizes for each effect size included in the analysis; QB=between-class
goodness-of-fit statistic; Qw=within-class goodness-of-fit statistic.
∗Indicates the Qvalue is statistically significant at the .05 level.
to courses. Although we found a positive mean effect size for declarative
knowledge in quasi-experimental studies (d=.18), CI was 10% more
effectivethanWBI forteaching declarativeknowledgewhen trainees were
randomly assigned to courses (d=−.26), failing to support Hypothesis 8.
More importantly, the positive effects for WBI were reversed when only
experimental designs were considered.
Training design characteristics. The second research question ad-
dressed the effect of learner control on the acquisition of declarative
knowledge from WBI relative to CI. Note that the level of learner con-
trol was low in all of the classroom courses, allowing us to examine the
effect of varying levels of learner control in WBI on training outcomes.

peps˙049 Peps-xml.cls June 13, 2006 16:22
644 PERSONNEL PSYCHOLOGY
The results indicated that the level of learner control moderated the ac-
quisition of declarative knowledge from WBI relative to CI. The extent
to which Web-based trainees learned more than classroom trainees was
greaterwhen theywere affordeda high(d=.30) thana lowlevelof control
(d=.07) during WBI.
Hypothesis 9 predicted, relative to CI, trainees would learn more with
a high than a low level of human interaction in WBI. The level of human
interactionwas high inall ofthe classroom courses, allowing us to examine
the effect of varying levels of human interaction in WBI on declarative
knowledge effect sizes. Relative to CI, trainees learned the same amount
with a low (d=.19) and a high (d=.18) level of human interaction in
WBI, failing to support Hypothesis 9.
The tenth hypothesis predicted classes that provided practice would be
more effective than classes that failed to provide practice. The effect size
was largest when WBI but not CI included practice (d=.31), indicating
WBI was 12% more effective than CI for teaching declarative knowledge.
WBI was also more effective than CI when both delivery media incorpo-
rated practice (d=.16), but was less effective than CI when WBI failed to
incorporate practice during training (d=−.27 when CI included practice,
and −.25 when CI did not include practice). It is important to note that
only one or two effect sizes were included in the analyses where WBI did
not include practice. Overall the results suggest practice is beneficial in
both WBI and CI, supporting Hypothesis 10.
Hypothesis 11 predicted feedback would moderate the relative effec-
tiveness of WBI and CI, and classes that provided feedback would more
effective than classes that failed to provide feedback during training. The
effect size was largest when WBI but not CI included feedback (d=.33)
followed by the effect size where both WBI and CI provided feedback
(d=.16). In both of these instances, WBI was more effective than CI for
teaching declarative knowledge. In addition, the effect size approached
zero (d=.08) when neither WBI nor CI provided feedback to trainees.
Thus, the results indicate feedback is beneficial during both WBI and CI,
supporting Hypothesis 11.
Thethird researchquestion addressed the effectof length of training on
learningfromWBI relativetoCI.Notethat ineachresearch report included
in the analysis, the number of instructional days was the same for WBI
and CI. A sample size weighted correlation was used to assess the effect
of the number of days of training on learning from WBI relative to CI.
The number of days of training was positively and significantly correlated
with the declarative knowledge effect sizes (weighted r=.33; p<.05),
indicatingWeb-basedtrainees gainedmore declarativeknowledge relative
to CI as the length of the class increased.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 645
TABLE 3
Correlation Among Web-Based Training Moderator Variables
1234567
1 Experimental design
2 Similarity of instructional methods −.24
3 Learner control −.10 .17
4 Practice .08 .19 .22
5 Feedback .21 .17 .15 .48∗
6 Population −.19 −.23 −.15 −.46∗−.07
7 Age of Web-based trainees −.37∗.07 −.14 −.07 .01 .39∗
Experimental design: quasi-experimental =1, experimental =0; Similarity of Q2
instructional methods: different =1, similar =0; Learner control: high =1, low =0;
Practice: 1 =yes,0=no; Feedback: 1 =yes,0=no; Population: employees =1, college
students =0.
Overallthe moderatorresults indicatedthat six of eight moderators had
an effect on the acquisition of declarative knowledge from WBI relative
to CI (the exceptions are human interaction and the population). However,
for all of the declarative knowledge categorical moderator results, the Qw
was significant, indicating there was more variation within the moderator
categories than would be expected by subject-level sampling error alone
(Lipsey & Wilson, 2001). That is, none of the moderator variables in-
dependently accounted for all of the variability in declarative knowledge
effect sizes across studies.
Correlationamongmoderatorvariables. A limitationofthe subgroup
approach for examining moderators is that it is restricted to test individual
hypotheses and does not control for possible confounds between corre-
lated moderators (Hedges & Olkin, 1985; Miller, Glick, Wang, & Huber,
1991). To address this concern, we tested the correlation among the Web-
based training moderator variables (see Table 3). This allowed us to asses
the extent to which the moderator variables overlap in their effects on
learning from WBI relative to CI. Seventeen of 21 correlations among the
moderators were less than .25 and were not statistically significant, sug-
gesting little overlap. However, four correlations were greater than .35 and
statistically significant: courses which incorporated practice also tended
to provide feedback to trainees (r=.48; p<.05), college students were
more likely to receive the opportunity to practice during training than em-
ployees (r=−.46; p<.05), employees tended to be older than college
students (r=.39; p<.05), and older Web-based trainees were more likely
toparticipate in anexperimental designwhile younger Web-based trainees
were more likely to participate in a quasi-experimental design (r=−.37;

peps˙049 Peps-xml.cls June 13, 2006 16:22
646 PERSONNEL PSYCHOLOGY
p<.05).Accordingly,it is possible that some observedmoderating effects
may have multiple determinants.
Discussion
Meta-analytic procedures were used to examine the training outcomes
of WBI and WBI-S compared to CI. More specifically, we examined the
relative effectiveness of these media for teaching declarative and proce-
dural knowledge and for training reactions. Additional analyses examined
experimental and training context variables that moderate the effects of
WBI relative to CI on the acquisition of declarative knowledge. We will
discuss both practical and theoretical implications of our results, as well
as limitations of the study and directions for future research.
Across all studies, the results indicated WBI was 6% more effective
than CI for teaching declarative knowledge. These results were based on
71 effect sizes and 10,910 learners. WBI and CI were equally effective for
teaching procedural knowledge and trainees were equally satisfied with
the two delivery media.
Theresultsdifferedwhenweexaminedinstances ofblendedlearning—
WBIused to supplementface-to-face instruction(WBI-S). Across allstud-
ies,the resultsindicated that WBI-S was moreeffectivethanstand aloneCI
for teaching trainees job-relevant knowledge and skills. WBI-S was 13%
more effective than CI for teaching declarative knowledge and 20% more
effectivethanCIforteachingprocedural knowledge. Similar meta-analytic
findings were reported by Zhao et al. (2005) who found “mixed method”
or blended distance courses result in better outcomes than distance educa-
tion or face-to-face instruction alone. WBI-S optimizes the instructional
advantages of both WBI and CI and meets the training requirements of
individual learners by employing multiple instructional methods (Dennis,
2002; Pratt, 2002). The instructional advantage of WBI-S may be due to
incorporating both the benefits of personal interaction typically found in
CI and self-study between instructional meetings using the Web (Kerres
& deWitt, 2003; Masie, 2002). Note, however, that trainees reacted 6%
more favorably toward stand alone CI than WBI-S. Although convert-
ing to WBI-S from CI may improve learning, there may be a tradeoff in
terms of trainee satisfaction. Additional research is needed to investigate
the underlying course characteristics which are driving this effect. It is
possible that blended learning courses are more demanding and require
a greater time commitment than CI due to incorporating both WBI and
CI components. The added time commitment may be frustrating for stu-
dents, decreasing satisfaction, but increasing the likelihood that trainees
will master the material by the end of training. However, this is conjecture
and should be tested empirically.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 647
Theoretical Implications
Advocates of WBI or technology-assisted instruction cite numerous
potential pedagogical benefits including the use of multi-media, learner
customization, and opportunities for guided learning (Bailey & Cotlar,
1994;Dumont, 1996; Hiltz& Wellman, 1997; Liaw,2001;Sullivan,2001).
However, other theorists argue that there is nothing uniquely advanta-
geous to any delivery medium; hence, we should expect no effects in
well-designed media comparison studies. This position is summarized by
Clark (1983) who wrote that media are “mere vehicles used to deliver in-
struction but do not influence student achievement any more than the truck
that delivers our groceries causes change in our nutrition” (p. 445). Thus,
a secondary purpose of our study was to capitalize on the large number of
studies analyzed and unique coding methods to investigate the veracity of
Clark’s frequently cited position.
Our results support Clark’s position that media effects in single study
research are largely spurious. We first note that across all studies, we
found relatively small differences between WBI and CI on both measures
of declarative knowledge and procedural (though confidence intervals for
the former outcome excluded zero). More importantly, we were able to
examine the effect of the research design on study outcomes for declara-
tive knowledge. We found that when trainees were randomly assigned to
delivery media, CI was more effective than WBI for teaching declarative
knowledge (d=−.26). However, this result is in the opposite direction
of the effect sizes for WBI relative to CI across all studies (d=.15) and
across studies using a quasi-experimental design (d=.18). Thus, consis-
tent with Clark’s arguments (1983, 1994; Clark & Sugrue, 1995), studies
are more likely to provide support for WBI when research participants are
allowed to self-select into courses.
The similarity of instructional methods moderator results added addi-
tional support for Clark’s position. Clark argued that media comparison
studies have confounded delivery media with instructional methods, mak-
ing it impossible to detect the true cause of differences in course effective-
ness. In the present meta-analysis, WBI and CI were equally effective for
teaching declarative knowledge when similar instructional methods were
used to deliver the two courses, supporting Clark’s position. This suggests
that unique instructional methods or learning conditions are driving ob-
served differences in the effectiveness of WBI relative to CI. In addition,
WBI was on average 11% more effective than CI for teaching declarative
knowledge when different instructional methods were used to deliver the
two courses.
A qualitative analysis of research reports identified two characteristics
of research reports where different instructional methods were used and

peps˙049 Peps-xml.cls June 13, 2006 16:22
648 PERSONNEL PSYCHOLOGY
trainees learned more from WBI than CI. First, the Internet courses tended
to incorporate more instructional methods than CI. Thus, trainees who
were having difficulty mastering the training content could utilize mul-
tiple instructional methods when reviewing the material to increase the
likelihood of mastering the course content. Second, the Internet courses
tendedto require studentsto be more activethan CI. Thisis consistent with
Webster and Hackley’s (1997) guidelines for teaching in distance learn-
ing, “learning is best accomplished through the active involvement of the
students” (p. 1284). Spending time practicing the key task components of
trainingshould help trainees developan understandingof thedeeper,struc-
tural features of the task (Newell, Rosenbloom, & Laird, 1989). Frequent
practice should also increase the likelihood that trainees will automate
skills by the end of training, leading to better performance at the end of
training (Rogers, Maurer, Salas, & Fisk, 1997). Thus, it is critical that
CI requires trainees to as active as they are in WBI and incorporates as
many instructional methods as WBI in order to promote similar learning
outcomes between the two delivery media.
Together,ourfindingsandobservationssuggest that instructionalmeth-
odsare moreimportant thandeliverymedia for ensuring effectivelearning.
Practical implications of study findings will be addressed in the following
section.
Practical Implications
The present meta-analytic results have several direct implications for
organizations and institutions considering implementing WBI. Advocates
of WBI (e.g., Galagan, 2001; Goodridge, 2001; Hall, 1997) suggest that it
canbeamore cost-effectivemeansoftrainingthanface-to-faceinstruction,
although well-controlled studies documenting the cost-effectiveness or
utility of WBI are rare (Welsh et al., 2003). Assuming that over time WBI
is less expensive than CI, even findings that show no mean differences
between WBI and CI training outcomes provide support for implementing
online instruction. The results we report can be used in conjunction with
accurate estimates of the cost of implementing and maintaining online
instructional programs to estimate the utility (see Mathieu & Leonard,
1987) of converting CI to WBI.
The results also indicate care should be taken whenever organizations
and institutions consider implementing WBI as the relative effectiveness
of training may depend on both the intended learning outcomes and the
training conditions. Given that WBI is at least as effective as CI for teach-
ing job-relevant knowledge and skills, the present results can be used by

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 649
organizations and universities to justify expenditures necessary to develop
online instruction. However, caution is warranted when considering
completely replacing CI with WBI. Researchers are beginning to un-
derstand that CI and WBI create very different learning environments
(Arbaugh, 2005; Dumont, 1996; LaRose & Whitten, 2000). Thus, care
should be taken to prevent trainees from being forced into online courses,
which could ultimately result in some trainees failing to master course
material. Accordingly, the moderator analyses we conducted are helpful
for understanding conditions that influence the effectiveness of WBI. Our
results indicated that learners acquired relatively more declarative knowl-
edge from WBI than CI when different instructional methods were used,
courses were longer, learners were afforded more control during WBI,
when learners had the opportunity to practice and received feedback dur-
ing WBI. We return to the issue of designing more effective Web-based
courses below.
Itis important to note that the positiveeffect size for declarativeknowl-
edgeacross all studies was reversedwhentrainees wererandomly assigned
tocourses. Thereare severalpossible explanations forthese findings. First,
it is possible that trainees who are higher in motivation or cognitive abil-
ity are self-selecting into WBI when they are allowed to choose between
Web and classroom versions of a course. Thus, preexisting differences be-
tween trainees who prefer WBI and trainees who prefer CI may result in
the appearance that WBI is more effective than CI. Second, trainees who
lack technical skills may be forced to participate in WBI when trainees
are randomly assigned to courses. Providing technically inept trainees
with a computer and Internet skills course prior to participating in WBI
may result in the two delivery media being equally effective for teaching
declarative knowledge. An experiment where half of the trainees are al-
lowed to self-select into WBI and CI and the other half of trainees are
randomly assigned to courses would allow researchers to disentangle dif-
ferences in the effectiveness of the delivery media for teaching declarative
knowledge.
Another important finding is the effectiveness of WBI relative to CI
did not differ across student and employee samples, after controlling for
the age of trainees. However, the age of trainees accounted for 44.2%
of the variance in the relative effectiveness of the two delivery media.
The extent to which Web-based trainees outperformed classroom trainees
increased as the age of online trainees increased and the age of classroom
trainees decreased. In addition, Web-based trainees, on an average, were
5 years older than classroom trainees. Research is needed to discover the
underlying causal mechanism driving this effect. We will elaborate on this
idea later in the Discussion section.

peps˙049 Peps-xml.cls June 13, 2006 16:22
650 PERSONNEL PSYCHOLOGY
Designing More Effective Online Training Courses
This study investigated the effect of several course design charac-
teristics on the effectiveness of WBI relative to CI. Across studies, the
extent to which Web-based trainees learned more than classroom trainees
was greatest when Web-based trainees were provided with control, when
trainees practiced the training material, when trainees received feedback
during training, and in long courses. Under these conditions, the declara-
tive knowledge effect size was .49, suggesting WBI was 19% more effec-
tive than CI. In contrast, it is also possible to design Web-based courses
in which learning levels will be inferior to CI. CI was 20% more effective
than WBI for teaching declarative knowledge when WBI failed to pro-
vide control, practice, and feedback to learners and in short courses (d=
−.51). Thus, attention to course design features is critical for maximizing
learning outcomes.
As WBI may be a new experience for many trainees, longer training
programs may give learners the opportunity to adapt to the technology
and the control they are afforded in WBI (DeRouin, Fritzsche, & Salas,
2004). That is, trainees may adapt their learning strategies in these envi-
ronments. One of the demonstrated advantages of WBI is the opportunity
to develop collaborative learning communities (e.g., Alavi, Wheeler, &
Valacich, 1995; Rovai, 2001), but it takes learners time to build and ben-
efit from collaborative contexts (Duffy & Kirkley, 2004; Garrison, 2003).
Accordingly, it would be interesting to test inexperienced participants at
multiple occasions in a Web-based training course to determine whether
they are using more adaptive learning strategies over time and how col-
laborative learning environments facilitate learning over time. In addition,
more research is needed to understand the effects of cohort size, peer-to-
peer interactions, and synchronous versus asynchronous communication
on the effectiveness of WBI.
We also found a moderating effect for learner control on declarative
knowledge effect sizes. Compared to classroom learners, participants in
WBIlearnedmore whengivenahigh leveloflearnercontrol. This outcome
is consistent with a recent finding of Kraiger and Jerden (in press) who
reportedthat positiveeffects for learner control are morelikely to be found
in recent studies than older studies. This trend toward more support for
learner control manipulations in recent research (such as those reviewed
here) suggests that there may be characteristics of the modern learning
environment or modern learner that interact with the provision of learner
control. More research is needed to isolate and understand these effects.
Learner control may be provided along a number of dimensions such
as content, sequence, or pace and research has suggested that various
dimensions of learner control may differ in their effects on learning from

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 651
WBI(Lunts, 1997). Dueto limiteddescriptions of trainingcourses inmany
research reports, we were unable to distinguish among the learner control
dimensions in our coding. Thus, future primary research should provide
moredetailed descriptions oftraining coursesto allow moreprecise coding
and evaluation of learner control in future meta-analyses. More research
is also needed to understand which specific learner control options online
learners prefer and which facilitate learning.
Contrary to our hypothesis, our meta-analysis found human interac-
tion did not affect learning from WBI relative to CI. However, there were
not sufficient studies to examine differences among various forms of com-
munication typically found in WBI (e.g., discussion boards, chat rooms,
and e-mail); hence, we were unable to determine whether various forms
of communication differ in their effects on learning. While we could not
investigate the specific communication channel, we coded the percent of
communication that was synchronous (i.e., 0 =0%, 1 =1–25%, 2 =26–
50%, 3 =51–75%, & 4 =76–100%) in each research report. After con-
trolling for the level of human interaction, synchronous communication
accounted for an additional 3.5% of the variance in the declarative knowl-
edge effect sizes and the beta-weight was positive suggesting synchronous
communication facilitates learning more than asynchronous communica-
tion in WBI. Additional primary research is needed to clarify whether
specific forms of synchronous and asynchronous communication differ in
their effects on learning outcomes.
Comparison to Previous Technology-Assisted Instruction Meta-analyses
It is worth noting that the overall positive effect size for WBI com-
pared to CI is smaller than those reported in meta-analyses of other types
of technology-assisted instruction (Fletcher, 1990; Kulik, 1994; Kulik &
Kulik, 1991; Liao, 1999; Yaakub, 1998), although similar to those in
a recent meta-analysis of distance education (Zhao et al., 2005). There
are several possible explanations for this. In contrast to previous meta-
analyses, ours used an adult population learning work-related knowledge
and skills, whereas others incorporated a wider age range of subjects and
had a heavier reliance on the acquisition of declarative knowledge. In ad-
dition note that WBI is a relatively new training platform, and as such,
its overall effectiveness may be compromised by several non-permanent
conditions.For example,in many studies, there may havebeen insufficient
bandwidth to optimize training delivery or trainees may have lacked the
technical skills needed to access the instructional content (Welsh et al.,
2003). Over time, instructional designers may make more informed deci-
sions about how to structure Web-based environments to ensure greater

peps˙049 Peps-xml.cls June 13, 2006 16:22
652 PERSONNEL PSYCHOLOGY
learning. Accordingly, our study results are valuable in that they identify
variables that influence the effectiveness of WBI.
In addition, our meta-analysis contained more published and unpub-
lished studies than prior meta-analyses of technology-assisted instruction.
It is possible that previous meta-analytic results reflect a publication bias
or other sampling problems not evident in the larger number of studies
we were able to locate and code. Regardless, as WBI, blended learning,
videodiscs, and single-work station computer-based training are each op-
tions for training delivery, other researchers may want to explore possible
differences in the relative effectiveness of various types of technology-
assisted instruction.
Study Limitations and Additional Directions for Future Research
Although we would have preferred to investigate the effectiveness of
WBIrelativetoCI basedon three categoriesof learning outcomes,we were
able to identify only 12 studies that assessed procedural knowledge and
even fewer studies that assessed affective learning. In the latter case, there
wasaninsufficient number of studies to determine an overall effect size,
and in the former case, there was an insufficient number of studies to ex-
amine potential moderators. Thus, we could not determine whether online
learning is more or less effective for affective outcomes than the overall
effect sizes reported for declarative and procedural knowledge. This is not
merely an academic question; an increasing number of organizations are
implementing WBI for diversity and sexual harassment training. Often
times changing participants’ attitudes toward groups of employees is the
desired outcome of these programs. Yet, little is known about the effec-
tiveness of WBI in this regard. It is also possible that the size or direction
of the moderating effects we found for declarative knowledge might differ
if the learning outcome was procedural knowledge. Additional primary
research is needed to examine the effectiveness of WBI for conveying
affective and procedural knowledge.
Itis also importantto notethat:(a) wefound different resultsdepending
on whether we isolated quasi-experimental designs or true experimental
designs; and (b) there were only 11 studies that used random assignment
of subjects to conditions. Accordingly, there were too few experimental
studies to replicate other moderator analyses within this set (e.g., does
holding instructional methods matter if all subjects are randomly assigned
to training condition?). In future studies, it is important to examine the
impact of moderators of WBI effectiveness when participants can and
cannot self-select into courses.
Another limitation is only 25 courses were identified that examined
the effectiveness of WBI relative to CI for employee training programs.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 653
Although the population did not moderate the relative effectiveness of the
two delivery media after controlling for age, additional research should be
conducted on the utility of WBI for delivering training for working adults.
The present results also suggest there are moderators of the effective-
ness of WBI that were not identified in the present study. We believe that
trainees’ previous experience with computers and the Internet may be one
of the best predictors of learning from the Web. In this study, we attempted
to code for this, but there was often a lot of variability among trainees
within a single course and few studies provided sufficient detail to code
forthis moderator.Despite this limitation, computer experience accounted
for 3.3% of the variance in the declarative knowledge effect size. In addi-
tion, the beta-weight was positive, indicating more experienced trainees
learned more from the Web. Another potential moderator may be the qual-
ity of the training course. We also attempted to code for course quality, but
again, many of the articles lacked sufficient details. This is another area
where more detailed descriptions of the training courses and participants
would advance meta-analytic research. Additional research is needed to
examine the impact of these moderator variables on the effectiveness of
WBI.
Research is also needed to assess why trainees aged 23–45 tended to
learn more declarative knowledge from WBI than CI while trainees ages
18–22 tended to learn more declarative knowledge from CI than WBI. It is
possiblethat, inaccordance with andragogical learning theories(Knowles,
1984), slightly older trainees are more adept at dealing with the autonomy
and learner control provided by WBI while younger trainees are more suc-
cessful in a structured classroom environment. Graham (1991) compared
attitudes toward tasks related to school, motivation, and anxiety levels of
traditional and non-traditional aged college students (mean ages 19 vs.
34). She found non-traditional students had more positive attitudes, were
more motivated, and experienced less anxiety than traditional students. In
addition, Tallent-Runnels et al. (2006) reviewed the literature on WBI and
concluded older trainees in WBI are more focused on achieving specific
learning outcomes than younger trainees. Additional research is needed
to assess if training attitudes, motivation, anxiety, or other factors are the
underlying causal mechanisms driving differences in the effectiveness of
WBI and CI for older and younger trainees.
Finally, research is needed to better understand the effect of course
design characteristics and process variables on the effectiveness of WBI
and the cost-effectiveness of WBI. Do the dimensions of learner control—
content,sequence, and pace—differ intheir effectson learningfrom WBI?
Do various forms of synchronous and asynchronous communication have
differential effects on learning from WBI? Is there an optimal group size
forfacilitating online peer-to-peer interaction in WBI? Dotrainees acquire

peps˙049 Peps-xml.cls June 13, 2006 16:22
654 PERSONNEL PSYCHOLOGY
moreadaptivelearning strategies overtimein Web-based training courses?
Research is also needed to determine why CI is more effective than WBI
when trainees are randomly assigned to delivery media. Is the underly-
ing mechanism driving this effect student motivation, cognitive ability, a
lack of technical skills, or another causal mechanism? Empirical data are
also needed to evaluate the cost-effectiveness of WBI. Is WBI more or
less cost-effective than other delivery media? What learning effect sizes
are necessary to offset the developmental costs of implementing WBI?
How does the utility of WBI change over time as: (a) costs move from
implementation to maintenance; (b) training designers learn how to better
delivery training online; and (c) learners adjust to learning in an online
environment?
Conclusion
The current meta-analysis identified 96 studies reporting data from
19,331 trainees who took part in 168 training courses. Across all of these
reports, CI was more effective than WBI for teaching declarative knowl-
edge when trainees were randomly assigned to courses and trainees were
equally satisfied with the two delivery media. However, trainees learned
the same amount of declarative knowledge from WBI and CI when the
same instructional methods were used to deliver training. Overall these
results support Clark’s (1983, 1994) argument that instructional methods
rather than delivery media determine learning outcomes. In addition, de-
signinglong training courses and providing traineeswith control, practice,
and feedback during WBI will maximize learning declarative knowledge
from WBI relative to CI.
REFERENCES
Aberson CL, Berger DE, Healy MR, Kyle DJ, Romero VL. (2000). Evaluation of an in-
teractive tutorial for teaching the central limit theorem. Teaching of Psychology,27,
289–291.
Q3
Aberson CL, Berger DE, Healy MR, Romero VL. (2000). Evaluation of an interactive
tutorial for teaching hypothesis testing concepts. Computers in Teaching,30, 75–78.
Alavi M, Wheeler BC, Valacich JS. (1995). Using IT to re-engineer business education: An
exploratory investigation of collaborative telelearning. MIS Quarterly,19, 293–312.
Allen M, Bourhis J, Burrell N, Mabry E. (2002). Comparing student satisfaction with
distance education to traditional classrooms in higher education: A meta-analysis.
The American Journal of Distance Education,16, 83–97.
Alzafiri FM. (2000). An experimental investigation on the effects of Web-based instruc-
tion/training on cognitive and psychomotor learning. Unpublished doctoral disser-
tation, University of North Texas, Denton.
Arbaugh JB. (2000). Virtual classroom versus physical classroom: An exploratory study
of class discussion patterns and student learning in an asynchronous Internet-based
MBA course. Journal of Management Education,24, 213–233.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 655
Arbaugh JB. (2005). Is there an optimal design for on-line MBA courses? Academy of
Management Learning and Education,4, 135–149.
Arthur W, Jr, Bennett W, Jr, Huffcutt AI. (2001). Conducting meta-analysis using SAS.
Mahwah, NJ: Erlbaum.
Arvan L, Ory JC, Bullock CD, Burnaska KK, Hanson M. (1998). The SCALE efficiency
projects. Journal of Asynchronous Learning Networks,2, 33–60.
Azevedo R, Bernard RM. (1995). A meta-analysis of the effect of feedback in computer-
based instruction. Journal of Educational Computing Research,13, 109–125.
Bailey EK, Cotlar M. (1994). Teaching via the Internet. Communication Education,43,
184–193.
BarkerBS.(2002).Acomparisonresearchmodelforadultdistributedtraining.Unpublished
doctoral dissertation, University of Nebraska, Lincoln.
Beile PM, Boote DN. (2002). Library instruction and graduate professional development:
Exploringtheeffectoflearningenvironmentson self-efficacyandlearningoutcomes.
Alberta Journal of Educational Research,68, 364–367.
Benedict JO, Anderton JB. (2004). Applying the just-in-time teaching approach to teaching
statistics. Teaching of Psychology,31, 197–199.
Bernard RM, Abrami PC, Lou Y, Borokhovski E, Wade A, Wozney L, et al. (2004). How
does distance education compare to classroom instruction? A meta-analysis of the
empirical literature. Review of Educational Research,74, 379–439.
Blanchard PN, Thacker JW. (2004). Effective training: Systems, strategies, and practices
(2nd ed). Upper Saddle River, NJ: Prentice Hall.
BocchiJ.(2004). Retaining the online learner: Profile ofstudentsin an online MBA program
and implications for teaching them. Journal of Education for Business,79, 245–
253.
BonkCJ, Kim KJ, Zeng T. (2005). Future directions of blended learningin higher education
and workplace learning settings. In Bonk CJ, Graham CR (Eds.). Handbook of
blended learning: Global perspectives, local designs (pp. 550–568). San Francisco,
CA: Pfeiffer Publishing.
BrownKG,FordJK. (2002). Using computer technologyin training: Building an infrastruc-
ture for active learning. InKraiger K (Ed.). Creating, implementing, and maintaining
effective training and development: State-of-the-art lessons for practice (pp. 192–
233). San Francisco, CA: Jossey-Bass.
Browning JB. (1999). Analysis of concepts and skills acquisition differences between Web-
delivered and classroom-delivered undergraduate instructional technology courses.
Unpublished doctoral dissertation, North Carolina State University, Raleigh.
Buchanan T. (2000). The efficacy of a World-Wide Web mediated formative assessment.
Journal of Computer Assisted Learning,16, 193–200.
Buerk JP, Malmstrom T, Peppers E. (2003). Learning environments and learning styles:
Non-traditional student enrollment and success in an Internet-based versus a lecture-
based computer science course. Learning Environments Research,6, 137–155.
Cahill D, Catanzaro D. (1997). Teaching first-year Spanish on-line. Calico Journal,14,
97–114.
Campbell JP. (1971). Personnel training and development. Annual Review of Psychology,
22, 565–602.
CampbellM,FloydJ,SheridanJB.(2002). Assessment of student performance and attitudes
for courses taught online versus onsite. The Journal of Applied Business Research,
18, 45–51.
Cardin LHL. (2002). Effects of individualized Web-based instructional modules on the
strategic learning strategies of students enrolled in a university study skills course.
Unpublished doctoral dissertation, Texas A&M University, College Station.

peps˙049 Peps-xml.cls June 13, 2006 16:22
656 PERSONNEL PSYCHOLOGY
Carey JM. (2001). Effective student outcomes: A comparison of online and face-
to-face delivery modes. Retrieved February 1, 2004, from http://teleeducation.nb.ca/
content/pdf/english/DEOSNEWS 11.9 effective-studentoutcomes.pdf.
Casanova RS. (2001). Student performance in an online general college chemstry
course. Retrieved April 30, 2003, from http://www.chem.vt.edu/confchem/2001/c/
04/capefear.html.
Caulfield JL. (2002). Examining the effect of teaching method and learning style on work
performance for practicing home care clinicians. Unpublished doctoral dissertation,
Marquette University, Milwaukee, Wisconsin.
Cheng H, Lehman J, Armstrong P. (1991). Comparison of performance and attitude in
traditional and computer conferencing classes. The American Journal of Distance
Education,5, 51–64.
Clark RE. (1983). Reconsidering research on learning from media. Review of Educational
Research,53, 445–460.
Clark RE. (1994). Media will never influence learning. Educational Technology Research
and Development,42, 21–29.
Clark RE, Sugrue BM. (1995). Research on instructional media, 1978–1988. In Anglin G
(Ed.). Instructional technology: Past present and future (pp. 348–364). Englewood,
CO: Libraries Unlimited.
Cobb T. (1997). Cognitive efficiency: Towards a revised theory of media. Educational
Technology Research and Development,45, 21–35.
Collins M. (2000). Comparing Web correspondence and lecture versions of a second-year
nonmajor biology course. British Journal of Educational Technology,31, 21–27.
Cooper LW. (2001). A comparison of online and traditional computer applications classes.
T.H.E. Journal,28, 52–58.
Daft RL, Lengel RH. (1986). Organizational information requirements, media richness, and
structural design. Management Science,32, 554–571.
Davies RS, Mendenhall R. (1998). Evaluation comparison of online and classroom in-
struction for HEPE 129—Fitness and lifestyle management course. Unpublished
manuscript, Brigham Young University, Provo, Utah.
Day TM, Raven MR, Newman ME. (1998). The effects of World Wide Web instruction
andtraditional instruction and learning styles on achievementandchanges in student
attitudes in a technical writing in agricommunication course. Journal of Agricultural
Education,39, 65–75.
DeBord KA, Aruguete MS, Muhlig J. (2004). Are computer-assisted teaching methods
effective? Teaching of Psychology,31, 65–68.
Dellana SA, Collins WH, West D. (2000). Online education in a management science
course—Effectiveness and performance factors. Journal of Education for Business,
76, 43–47.
Dennis C. (2002). A special blend. People Management,8, 151.
DeRouin RR, Fritzsche BA, Salas E. (2004). Optimizing e-learning: Research-based guide-
lines for learner-controlled training. Human Resource Management,43, 147–162.
DeSouza E, Fleming M. (2003). A comparison of in-class quizzes vs. online quizzes on
student exam performance. Journal of Computing in Higher Education,14, 121–
134.
DiBartola LM, Miller MK, Turley CL. (2001). Do learning style and learning environment
affect learning outcome? Journal of Allied Health,30, 112–115.
Duffy TM, Kirkley JR. (2004). Learning theory and pedagogy applied in distance learning:
The case of Cardean University. In Duffy TM, Kirkley JR (Eds.). Learner-centered
theory and practice in distance education: Cases from higher education (pp. 107–
141). Mahwah, NJ: Erlbaum.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 657
Dufresne R, Mestre J, Hart DM, Rath KA. (2002). The effect of Web-based homework
on test performance in large enrollment introductory physics courses. Journal of
Computers in Mathematics and Science Teaching,21, 229–251.
Dumont RA. (1996). Teaching and learning in cyberspace. IEEE Transactions on Profes-
sional Communication,39, 192–204.
Entwistle N, Entwistle A. (1991). Contrasting forms of understanding for degree examina-
tions. Higher Education,22, 205–227.
Eppler MA, Ironsmith M. (2004). PSI and distance learning in a developmental psychology
course. Teaching of Psychology,31, 131–134.
Fallah HM, Ubell R. (2000). Blind scores in a graduate test: Conventional compared
withWeb-basedoutcomes.Retrieved 2003, April 17fromwww.aln.org/publications/
magazine/v4n2/fallah.asp.
Faux TL, Black-Hughes C. (2000). A comparison of using the Internet versus lec-
tures to teach social work history. Research on Social Work Practice,10, 454–
466.
Fleetwood J, Vaught W, Feldman D, Gracely E, Kassutto Z, Novack D. (2000). MedEthEx
Online: A computer-based learning program in medical ethics and communication
skills. Teaching Learning in Medicine,12, 96–104.
Fletcher JD. (1990). Effectiveness and cost of interactive videodisc instruction in defense
training and education. Institute for Defense Analyses Paper, P-2372.
Fredda JV. (2000). Comparison of selected student outcomes for Internet- and campus-
based instruction at the Wayne Huizenga Graduate School of Business and En-
trepreneurship. ERIC Document Reproduction Service No. ED453743.
Freitas FA, Myers SA, Avtgis TA. (1998). Communication Education,42, 366–372. Q4
Friend CL, Cole CL. (1990). Learner control in computer-based instruction: A current
literature review. Educational Technology,20, 47–49.
Frisbie SH. (2002). Sexual harassment: A comparison of online versus traditional training
methods. Unpublished doctoral dissertation, Texas Tech University, Lubbock.
Galagan PA. (2001). 14 things CEOs should know about e-learning. Training Development,
55, 69–72.
Garrison DR. (2003). Self-directed learning and distance education. In Moore MG, An-
derson WG (Eds.). Handbook of distance education (pp. 161–168). Mahwah, NJ:
Erlbaum.
Gibbs GR. (1999). Learning how to learn using a virtual learning environment for philoso-
phy. Journal of Computer Assisted Learning,15, 221–231.
Glass GV, McGaw B, Smith ML. (1981). Meta-analysis in social research.Beverly Hills,
CA: Sage.
GlennAS.(2001).Acomparisonofdistancelearning and traditional learning environments.
ERIC Document Reproduction Service No. ED457778.
Goldstein IL. (1980). Training in work organizations. Annual Review of Psychology,31,
229–272.
Goldstein IL, Ford JK. (2002). Training in organizations: Needs assessment, development
and evaluation (4th ed.). Belmont, CA: Wadsworth.
Goodridge E. (2001). Feds turn to E-learning to cut costs. Information Week,840, 85.
Graham BW. (1991). Attitude, motivation, and anxiety in traditional and nontraditional
students at institutes of higher education. Unpublished doctoral dissertation, The
University of New Mexico, Albuquerque.
Hackman MZ, Walker KB. (1990). Instructional communication in the televised class-
room: The effects of system design and teacher immediacy on student learning and
satisfaction. Communication Education,39, 196–206.
Hall B. (1997). Web-based training cookbook. NY: Wiley.

peps˙049 Peps-xml.cls June 13, 2006 16:22
658 PERSONNEL PSYCHOLOGY
Hallgren RC, Parkhurst PE, Monson CL, Crewe NM. (2002). An interactive, Web-based
tool for learning anatomic landmarks. Academic Medicine,77, 263–265.
Harris P. (2005). Companies head back to college. Training & Development. Alexandria,
VA: American Society of Training and Development.
Hedges LV, Olkin I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic
Press.
Heines JM. (2000). Evaluating the effect of a course Website on student performance.
Journal of Computing in Higher Education,12, 57–83.
Hiltz SR. (1993). Correlates of learning in a virtual classroom. International Journal of
Man-Machine Studies,39, 71–98.
Hiltz SR, Wellman B. (1997). Asynchronous learning networks as a virtual classroom.
Communications of the ACM,40, 44–49.
Hoey JJ, Pettitt JM, Brawner CE, Mull SP. (1998). Project 25: First semester assessment—
Areport on the implementation of courses offered on the Internet. Retrieved March
8, 1999, from http://www2.ncsu.edu/ncsu/ltc/Project25/info/f97 assessment.html.
Hsu Y. (2003). The effectiveness of computer-assisted instruction in statistics education: A
meta-analysis. Unpublished doctoral dissertation, University of Arizona, Tucson.
Huffcutt AI, Arthur W. (1995). Development of a new outlier statistic for meta-analytic
data. Journal of Applied Psychology,80, 327–334.
Hunter JE, Schmidt FL. (1990). Methods of meta-analysis: Correcting for bias in research
findings. Thousand Oaks, CA: Sage.
Hurlburt RT. (2001). “Leclets” deliver content at a distance: Introductory statistics as a case
study. Teaching of Psychology,28, 15–20.
Johnson M. (2002). Introductory biology online: Assessing outcomes of two student pop-
ulations. Journal of College Science Teaching,31, 312–317.
Johnson SD, Aragon SR, Shaik N, Palma-Rivas N. (2000). Comparative analysis of learner
satisfaction and learning outcomes in online and face-to-face learning environments.
Journal of Interactive Learning Research,11, 29–49.
Khan B. (1997). Web-based instruction. Englewood Cliffs, New Jersey: Educational Tech-
nology Publications.
Kerres M, deWitt C. (2003). A didactical framework for the design of blended learning
arrangements. Journal of Educational Media,28, 101–114.
King FB. (2001). Asynchronous distance education courses employing Web-based instruc-
tion: Implications of individual study skills self-efficacy and self-regulated learning.
Unpublished doctoral dissertation, University of Connecticut, Storrs.
Klass G, Crothers L. (2000). An experimental evaluation of Web-based tutorial quizzes.
Social Science Computer Review,18, 508–515.
Knowles, MS. (1984). The adult learner: A neglected species (3rd ed.). Houston, TX: Gulf
Publishing Company.
Koory MA. (2003). Differences in learning outcomes for the online and F2F version of
“An introduction to Shakespeare.” Journal of Asynchronous Learning Networks,7,
18–35.
Kozma RB. (1994). A reply: Media and methods. Educational Technology Research and
Development,42, 11–14.
Kraiger K. (2003). Perspectives on training and development. In Borman WC, Ilgen DR,
Klimoski RJ, Weiner IB (Eds.). Handbook of psychology (pp. 171–192). Hoboken,
NJ: John Wiley & Sons, Inc.
Kraiger K, Ford JK, Salas E. (1993). Application of cognitive, skill-based, and affective
theories of learning outcomes to new methods of training evaluation. Journal of
Applied Psychology,78, 311–328.
Kraiger K, Jerden E. (in press). A new look at learner control: Meta-analytic results and
directions for future research. Chapter to appear In: Fiore SM, Salas E (Eds.). Where

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 659
is the learning in distance learning? Towards a science of distributed learning and
training.Washington, DC: American Psychological Association. Q5
Kulik C-LC, Kulik JA. (1991). Effectiveness of computer-based instruction: An updated
analysis. Computers in Human Behavior,7, 75–94.
Kulik JA. (1994). Meta-analytic studies of findings on computer-based instruction. In Baker
EL, O’Neil, HF, Jr. (Eds.). Technology assessment in education and training (pp.
9–33). Hillsdale, NJ: Lawrence Erlbaum.
Kulik JA, Kulik C-LC. (1988). Timing of feedback and verbal learning. Review of Educa-
tional Research,58, 79–97.
LaRose R, Whitten P. (2000). Re-thinking instructional immediacy for Web
courses: A social cognitive exploration. Communication Education,49, 320–
338.
Latham GP. (1988). Human resource training and development. Annual Review of Psychol-
ogy,39, 545–582.
LeasureAR,DavisL,ThievonSL.(2000).Comparisonofstudentoutcomesandpreferences
in a traditional vs. World Wide Web-based baccalaureate nursing research course.
Journal of Nursing Education,39, 149–154.
Liao Y-KC. (1999). Effects of hypermedia on students’ achievement: A meta-analysis.
Journal of Educational Multimedia and Hypermedia,8, 255–277.
Liaw S-S. (2001). Designing the hypermedia-based learning environment. International
Journal of Instructional Media,28, 43–56.
Liou HC. (1997). The impact of WWW texts of EFL learning. ComputerAssistedLanguage
Learning,10, 455–478.
Lipsey MW, Wilson DB. (2001). Practical meta-analysis: Applied social science research
methods series (Vol. 49). Thousand Oaks, CA: SAGE Publications.
Logan E, Conerly K. (2002). Students creating community: An investigation of student
interactions in a Web-based distance learning environment. Retrieved April 28,
2003, from www.icte.org/T01 Library/T01 253.pdf.
Long L, Javidi A. (2001). A comparison of course outcomes: Online distance learning
versus traditional classroom settings. Retrieved April 28, 2003, from http://www.
communication.ilstu.edu/activities/NCA2001/paper distance learning.pdf.
Lunts R. (1997). What does the literature say about the effectiveness of learner control in
computer-assisted instruction? Electric Journal for the Integration of Technology in
Education,2, 59–75.
MacGregor CJ. (2000). Does personality matter: A comparison of student experiences in
traditional and online classrooms. Unpublished doctoral dissertation, University of
Missouri, Columbia.
MakiWS,MakiRH.(2002).Multimediacomprehensionskillpredicts differentialoutcomes
of Web-based and lecture courses. Journal of Experimental Psychology: Applied,8,
85–98.
Maki WS, Maki RH, Patterson M, Whittaker PD. (2000). Evaluation of a Web-based
introductory psychology course: I. Learning and satisfaction in on-line versus
lecture courses. Behavior Research Methods, Instruments, Computers,32, 230–
239.
Masie E. (2002). Blended learning: The magic is in the mix. In Rossett A (Ed.). The ASTD
e-learning handbook best practices, strategies, and case studies for an emerging
field (pp. 58–63). New York: McGraw-Hill.
Mathieu JE, Leonard RL. (1987). An application of utility concepts to a supervisor skills
training program. Academy of Management Journal,30, 316–335.
Mehlenbacher B, Miller C, Convington D, Larsen J. (2000). Active and interactive learn-
ing online: A comparison of Web-based and conventional writing classes. IEEE
Transactions on Professional Communication,43, 166–184.

peps˙049 Peps-xml.cls June 13, 2006 16:22
660 PERSONNEL PSYCHOLOGY
Miller CC, Glick WH, Wang Y-D, Huber GP. (1991). Understanding technology-structure
relationships: Theory development and meta-analytic theory testing. Academy of
Management Journal,34, 370–399.
Molidor CE. (2000). The development of successful distance education in social work:
A comparison of student satisfaction between traditional and distance education
classes. Retrieved April 23, 2003, from www.nssa.us/nssajrnl/18-1/pdf/14.pdf.
Moorhouse DR. (2001). Effect of instructional delivery method on student achievement in
a master’s of business administration course at the Wayne Huizenga School of Busi-
ness and Entrepreneurship. Unpublished doctoral dissertation, Nova Southeastern
University, Ft. Lauderdale, Florida.
Murphy TH. (2000). An evaluation of a distance education course design for general soils.
Journal of Agricultural Education,41, 103–113.
Musgrove AT. (2002). An examination of the Kolb LSI and GEFT and their relationship to
academic achievement in Web-based and face-to-face nursing courses. Unpublished
doctoral dissertation, Florida Atlantic University, Boca Raton.
Mylona ZH. (1999). Factors affecting enrollment satisfaction and persistence in a Web-
based video-based and conventional instruction. Unpublished doctoral dissertation,
University of Southern California, Los Angeles.
Navarro P, Shoemaker J. (2000). Performance and perceptions of distance learners in cy-
berspace. Web-Based Communications, the Internet, and Distance Education,14,
1–15.
Newell A, Rosenbloom PS, Laird JE. (1989). Symbolic architectures for cognition. In
Posner M (Ed.). Foundations of cognitive science (pp. 93–131). Cambridge, MA:
MIT Press.
Newman F, Scurry J. (2001). Online technology pushes pedagogy to the forefront. The
Chronicle of Higher Education,47, B7–B1O.
Niemiec RP, Sikorski C, Walberg HJ. (1996). Learner-control effects: A review of re-
views and a meta-analysis. Journal of Educational Computing Research,15, 157–
174.
Noe RA. (2005). Employee training and development (3rd ed.). Boston, MA: McGraw Hill.
Olson TM, Wisher RA. (2002). The effectiveness of Web-based instruction: An initial
inquiry. International Review of Research in Open and Distance Learning,3(2).
Retrieved November 10, 2003, from http://www.irrodl.org/content/v3.2/olsen.html.
OswaldFL,McCloyRA.(2003).Meta-analysisandtheart of the average.InMurphyK(Ed).
Validity generalization: A critical review (pp. 311–338). Mahwah, NJ: Erlbaum.
Pankey RB. (1998). Piloting exercise physiology in the Web-based environment. T.H.E.
Journal,26,62–64.RetrievedNovember10,2003,fromhttp://www.thejournal.com.
Parker P. (2001). Inside online learning: Comparing conceptual and techniques learning
performance in place-based and ALN formats. Journal or Asynchronous Learning
Networks,5, 64–74.
Partrich DE. (2004). An analysis of learning style and grade achievement in relation to
Web-based and on-campus courses. Unpublished doctoral dissertation, Southwest-
ern Baptist Theological Seminary, Fort Worth, Texas.
Paul DS. (2001). A meta-analytic review of factors that influence the effectiveness of Web-
based training within the context of distance learning. Unpublished doctoral disser-
tation, Texas A&M University, College Station.
Phye GD, Andre T. (1989). Delayed retention effect: Attention, preservation, or both?
Contemporary Educational Psychology,14, 173–185.
Piccoli G, Ahmad R, Ives B. (2001). Web-based virtual learning environments: A research
framework and a preliminary assessment of effectiveness in basic IT skills training.
MIS Quarterly,25, 401–426.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 661
Poirier CR, Feldman RS. (2004). Teaching in cyberspace: Online versus traditional instruc-
tion using a waiting-list experimental design. Teaching of Psychology,31, 59–62.
Pratt, JR. (2002). The manager’s role in creating a blended learning environment. Home
Health Care Management and Practice,15, 76–79.
Ragan RE, Kleoppel JW. (2004). Comparison of outcomes on like exams administered to
in-residence and asynchronous distance-based Pharm.D. students. Journal of Asyn-
chronous Learning Networks,8, 15–24.
Ritchie H, Newbury TJ. (1989). Classroom lecture/discussions vs. live televised instruc-
tion: A comparison of effects on student performance, attitude, and interaction. The
American Journal of Distance Education,3, 36–45.
Rivera J, Rice M. (2002). A comparison of student outcomes and satisfaction between
traditional and Web-based course offerings. Online Journal of Distance Learning
Administration,5.RetrievedMay13,2003, from http://www.westga.edu/∼distance/
ojdla/fall53/rivera53.html.
Rogers W, Maurer T, Salas E, Fisk A. (1997). Task analysis and cognitive theory: Con-
trolled and automatic processing task analytic methodology. In Ford JK, et al. (Eds.).
Improving training effectiveness in work organizations (pp. 19–46). Mahwah, NJ:
Erlbaum.
Rosenberg MJ. (2001). e-Learning: Strategies for delivering knowledge in the digital age.
New York: McGraw Hill.
RossJL. (2000). An exploratoryanalysisof post-secondary student achievementcomparing
aWeb-basedanda conventionalcourselearning environment.Unpublished doctoral
dissertation, University of Calgary, Canada.
Rovai AP. (2001). Building classroom community at a distance: A case study. Educational
Technology Research Development,49, 33–48.
Rovai AP, Barnum KT. (2003). Online course effectiveness: An analysis of student in-
teractions and perceptions of learning. Journal of Distance Education,18, 57–
73.
Russell TL. (1999). The no significant difference phenomenon: As reported in 355 research
reports, summaries and papers. Chapel Hill, NC: Office of Instructional Telecom-
munications, North Carolina University.
Sain R, Brigham TA. (2003). The effect of a threaded discussion component on student
satisfactionandperformance. Journalof Educational Computing Research,29, 419–
430.
Saks AM, Haccoun RR. (2004). Managing performance through training and development
(3rd ed.). Scarborough, Ontario: Nelson.
Salas E, Cannon-Bowers JA. (2001). The science of training: A decade of progress. Annual
Review of Psychology,52, 471–499.
SalomonG.(1988).AIinreverse:Computer tools thatturncognitive.Journalof Educational
Computing Research,4, 123–134.
Sandercock GRH, Shaw G. (1999). Learners’ performance and evaluation of attitudes to-
wards Web course tools in the delivery of an applied sports science module. Asyn-
chronous Learning Networks Magazine,3, 1–10.
Sankaran SR, Sankaran D, Bui TX. (2000). Effect of student attitude to course format on
learning performance: An empirical study in Web vs. lecture instruction. Journal of
Instructional Psychology,27, 66–73.
Schaad DC, Walker EA, Wolf FM, Brock DM, Thielke SM, Oberg L. (1999). Evaluating
the serial migration of an existing required course to the World Wide Web. Academic
Medicine,74, S84–S86.
Scheetz NA, Gunter PL. (2004). Online versus traditional classroom delivery of a course
in manual communication. Exceptional Children,71, 109–120.

peps˙049 Peps-xml.cls June 13, 2006 16:22
662 PERSONNEL PSYCHOLOGY
SchmeeckleJM.(2000).Onlinetraining:Anevaluationoftheeffectivenessand efficiency of
training law enforcement personnel over the Internet. Journal of Science Education
and Technology,12, 205–260.
SchmidtK. (2002). Classroom action research:A case study assessing students’perceptions
and learning outcomes of classroom teaching versus on-line teaching. Journal of
Industrial Teacher Education,40, 45–59.
Schneider SP, Germann CG. (1999). Technical communication on the Web: A profile of
learners and learning environments. Technical Communication Quarterly,8, 37–
48.
Schoenfeld-Tacher R, McConnell S, Graham M. (2001). Do no harm—A comparison of
the effects of on-line vs. traditional delivery media on a science course. Journal of
Science Education and Technology,10, 257–265.
Schulman AH, Sims RL. (1999). Learning in an online format versus an in-class format:
An experimental study. T.H.E. Journal,26, 54–56.
Sexton JS, Raven MR, Newman ME. (2002). A comparison of traditional and World Wide
Web methodologies, computer anxiety, and higher order thinking skills in the in-
service training of Mississippi 4-H extension agents. Journal of Agricultural Edu-
cation,43, 25–36.
Shafer ME, Lahner JM, Calderone WK, Davis JE, Petrie TA. (2002). The use and effec-
tiveness of a Web-based instructional program in a college student success program.
Journal of College Student Development,43, 751–757.
Singh H. (2003). Building effective blended learning programs. Educational Technology,
43,6.
Sitzmann TM, Casper WJ, Brown KG, Witzberger K, Polliard C. (2003). Moderators of
the relationship between reactions to training and learning outcomes. Presented at
the meeting of the Society for Industrial/Organizational Psychology Conference,
Orlando, FL.
Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM.
(2005). We don’t train in vain: A dissemination trial of three strategies of training
clinicians in cognitive-behavioral therapy. Journal of Consulting Clinical Psychol-
ogy,73, 106–115.
Smith SB. (1999). The effectiveness of traditional instructional methods in an online
learning environment. Unpublished doctoral dissertation, University of Nevada, Las
Vegas.
SoleML,LindquistM.(2001).Enhancingtraditional, televised, andvideotapedcourseswith
Web-basedtechnologies:Acomparisonofstudent satisfaction. Nursing Outlook,49,
132–137.
SpiresHA, Mason C, CrissmanC, Jackson A. (2001).Exploring the academic selfwithin an
electronic mail environment. Research and Teaching in Developmental Education,
17, 5–14.
SpiroRJ, Jehng J. (1990). Cognitive flexibility and hypertext: Theory andtechnology for the
nonlinear and multidimensional traversal of complex subject matter. In Nix D, Spiro
R (Eds.). Cognition, education, and multimedia: Exploring ideas in high technology
(pp 163–205). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Spiros RK. (2003). Individual differences in motivation during distance training: The in-
fluence of goal orientation and self-efficacy on learning outcomes. Unpublished
doctoral dissertation, University of California, San Diego.
Stadtlander LM. (1998). Virtual instruction: Teaching an online graduate seminar. Teaching
of Psychology,25, 146–148.
Sugrue B, Rivera RJ. (2005). State of the industry: ASTD’s annual review of trends in
workplace learning and performance. Alexandria, VA: ASTD.

peps˙049 Peps-xml.cls June 13, 2006 16:22
TRACI SITZMANN ET AL. 663
Sullivan P. (2001). Gender differences and the on-line classroom. Male and female college
students evaluate their experiences. Community College Journal of Research and
Practice,25, 805–818.
Symonds WC. (2003). eArmyU. Business Week, 106.
Tallent-Runnels MK, Thomas JA, Lan WY, Cooper S, Ahern TC, Shaw SM, et al. (2006).
Teaching courses online: A review of the research. Review of Educational Research,
76, 93–135.
Tannenbaum SI, Yukl G. (1992). Training and development in work organizations. Annual
Review of Psychology,43, 399–441.
Taylor KK. (2002). Evaluation of an online psychology class. Unpublished doctoral disser-
tation, University of Memphis, Tennessee.
Tucker S. (2001). Distance education: Better, worse, or as good as traditional
education? Retrieved February 5, 2004, from www.westga.edu/∼distance/ojdla/
winter44/tucker44.html.
U.S. Bureau of the Census. (2006). Statistical abstract of the United States (112th ed.)
Washington, DC: U.S. Government Printing Office.
Vessell DC. (2000). Comparing the WWW and WebCT to traditional methods of supporting
an undergraduate psychology course: Is it effective? Unpublished doctoral disserta-
tion, University of Missouri, Columbia.
Wagner ED. (1994). In support of a functional definition of interaction. American Journal
of Distance Education,8, 6–26.
Wang AY, Newlin MH. (2000). Characteristics of students who enroll and succeed
in psychology Web-based classes. Journal of Educational Psychology,92, 137–
143.
Waschull SB. (2001). The online delivery of psychology courses: Attrition, performance,
and evaluation. Teaching of Psychology,28, 143–146.
WebsterJ, HackleyP.(1997).Teachingeffectivenessintechnology-mediateddistancelearn-
ing. Academy of Management Journal,40, 1282–1309.
Weems GH. (2002). Comparison of beginning algebra taught onsite versus online. Journal
of Developmental Education,26, 10–18.
Wegner SB, Holloway KC, Garton EM. (1999). The effects of Internet-based instruc-
tion on student learning. Journal of Asynchronous Learning Networks,3, 98–
106.
WelshLT,WanbergCR,BrownKG,Simmering MJ. (2003). E-learning: Emerginguses,best
practices, and future directions. International Journal of Training and Development,
7, 245–258.
Wexley KN. (1984). Personnel training. Annual Review of Psychology,35, 519–551.
Wexley KN, Latham GP. (2002). Developing and training human resourcesinorganizations
(3rd ed.). Upper Saddle River, NJ: Prentice-Hall.
WhiteSE. (1999). The effectiveness of Web-based instruction: A casestudy.Paper presented
at the Central States Communication Association Conference, St. Louis, Missouri.
Whitener EM. (1990). Confusion of confidence intervals and credibility intervals in meta-
analysis. Journal of Applied Psychology,75, 315–321.
Wideman HH, Owston RD. (1999). Internet based courses at Atkinson College:
An initial assessment. Retrieved 2003, May 13 from http://www.yorku.ca/irlt/
reports/techreport99-1.htm.
WilsonSP,HarrisA.(2002).Evaluationofthepsychologyplace:AWeb-basedinstructional
tool for psychology courses. Teaching of Psychology,29, 165–168.
Woo MA, Kimmick JV. (2000). Comparison of Internet versus lecture instructional meth-
ods for teaching nursing research. Journal of Professional Nursing,16, 132–
139.

peps˙049 Peps-xml.cls June 13, 2006 16:22
664 PERSONNEL PSYCHOLOGY
Woodward DK. (1998). Virtual-recitation: A World Wide Web based approach to active
learninginclinicalpharacokinetics.AmericanJournalof Pharmaceutical Education,
62, 329–332.
Yaakub MN. (1998). Meta-analysis of the effectiveness of computer-assisted instruction in
technical education and training. Doctoral dissertation, Virginia Polytechnic Insti-
tute and State University, Blacksburg.
ZhaoY, Lei J, LaiBYC, TanHS.(2005). What makes the difference?Apractical analysis of
research on the effectiveness of distance education. Teachers College Record,107,
1836–1884.

peps˙049 Peps-xml.cls June 13, 2006 16:22
Queries
Q1 Author: Please add reference “Clark (1984)” to the list.
Q2 Author: Please provide footnote for “*” in Table 3.
Q3 Author: Many of the references in the list are not cited in the text. Please
cite them.
Q4 Author: Please provide article title for reference “Freitas et al. (1998).”
Q5 Author: Please update reference “Kraiger and Jerden (in press)” in the
list.
665