Content uploaded by Tuan Nguyen
Author content
All content in this area was uploaded by Tuan Nguyen on Sep 19, 2016
Content may be subject to copyright.
Content uploaded by Tuan Nguyen
Author content
All content in this area was uploaded by Tuan Nguyen on Sep 16, 2016
Content may be subject to copyright.
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
309
The Effectiveness of Online Learning:
Beyond No Significant Difference and Future Horizons
Tuan Nguyen
Leadership, Policy, and Organization
Peabody College, Vanderbilt University
Nashville, TN 37203 USA
tuan.d.nguyen@vanderbilt.edu
Abstract
The physical “brick and mortar” classroom is starting to lose its monopoly as the place of
learning. The Internet has made online learning possible, and many researchers and
educators are interested in online learning to enhance and improve student learning
outcomes while combating the reduction in resources, particularly in higher education. It
is imperative that researchers and educators consider the effectiveness of online learning
compared to traditional face-to-face format and the factors that influence the
effectiveness of online courses. This study examines the evidence of the effectiveness of
online learning by organizing and summarizing the findings and challenges of online
learning into positive, negative, mixed, and null findings. Particular attention is paid to the
meta-analyses on the effectiveness of online learning, the heterogenous outcomes of
student learning and the endogenous issue of learning environment choice. Taken as a
whole, there is robust evidence to suggest online learning is generally at least as
effective as the traditional format. Moreover, this body of literature suggests that
researchers should move beyond the “no significant difference” phenomenon and
consider the next stage of online learning.
Key words: no significant difference, online learning, hybrid learning, blended learning,
higher education, selection bias
Introduction
The physical “brick and mortar” classroom is starting to lose its monopoly as the place of learning. The
Internet and the World Wide Web have made significant changes to almost all aspects of our lives
ranging from a global economy, personal, and professional networks to sources of information, news, and
learning. The Internet has made online learning possible, and many researchers and educators are
interested in online learning to enhance and improve student learning outcomes while combating the
reduction in resources, particularly in higher education (Farinella, Hobbs, & Weeks, 2000; Kim & Bonk,
2006; Pape, 2010). Moreover, there have also been increases in demand for online learning from
students from all walks of life. Given the exponential—some would say precipitous—growth of online
education and its potential in higher education, it is imperative that researchers and educators examine
the effectiveness of online learning in educating students compared to traditional face-to-face learning.
Thus, this paper addresses the question of “To what extent does the body of work on online learning
indicate that online learning is as least as effective in educating students as the traditional format?”
Definitions
Online learning is a form of distance learning or distance education, which has long been a part of the
American education system, and it has become the largest sector of distance learning in recent years
(Bartley & Golek, 2004; Evans & Haase, 2001). For the purpose of this literature review, both hybrid or
blended learning and purely online learning are considered to be online learning as much of the literature
compares these two formats against the traditional face-to-face. Purely online courses are courses
delivered entirely over the Internet, and hybrid or blended learning combines traditional face-to-face
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
310
classes, learning over the Internet, and learning supported by other technologies (Bliuc, Goodyear, &
Ellis, 2007; Hoic-Bozic, Mornar, & Boticki, 2009; Osguthorpe & Graham, 2003).
The Benefits and Uses of Online Learning
One reason why there is so much discussion around online learning is that there are many purported
benefits and uses of online learning. Some of the most important ones are: its effectiveness in educating
students, its use as professional development, its cost-effectiveness to combat the rising cost of
postsecondary education, credit equivalency at the postsecondary level, and the possibility of providing a
world class education to anyone with a broadband connection (Bartley & Golek, 2004; De la Varre,
Keane, & Irvin, 2011; Gratton-Lavoie & Stanley, 2009; Koller & Ng, 2014; Lorenzetti, 2013). What has
received most of the attention for online learning is the postsecondary education arena. The rising cost of
postsecondary education and the importance of a postsecondary degree are well documented in the
literature. The lifetime earning gap between high school graduates and college graduates is continuing to
widen (Dynarski & Scott-Clayton, 2013). At the same time, the cost of college tuition is rising faster than
inflation and the student loan debt is rapidly increasing. As of 2014, the total national student loan debt is
over one trillion dollars (Finaid.org, 2014). Many scholars and educators believe that online learning can
be an effective tool in combating the rising cost of postsecondary education by spreading the cost of a
class over a much larger number of students compared to the traditional setting, dividing the cost by tens
or hundreds of thousands of students as opposed to dozens (Bowen, 2013; Bartley & Golek, 2004; Jung
& Rha, 2000; Koller & Ng, 2014; Tucker, 2007). Moreover, the marginal cost of a student in an online
setting is negligible relative to the traditional setting, necessarily constrained by a number of factors such
as the size and availability of the physical classroom.
Intimately connected to this issue of cost and postsecondary education are the required credits to obtain
a postsecondary degree. Traditionally, students have to earn most of the college credits at an institution
before they are awarded bachelor degrees at that institution. The point of contention is how online
classes will play a role in awarding credits or credentials, and many educators connected to online
learning are hoping that there will be credit equivalency for some online classes. For instance, Daphne
Koller and Andrew Ng, creators of Coursera, had worked with the American Council on Education to
recommend credit-equivalency for some online courses (Koller & Ng, 2012). The goals of this endeavor
are to increase completion rate, reduce time to degree attainment, reduce costs to postsecondary
education, and offer more access to non-traditional students. As of 2013, the American Council of
Education had approved five online courses for college credit (Kolowich, 2013). However, there is
concern over whether colleges will accept the recommendation, and there is also concern about the
dilution of a traditional degree due to the transition (Kolowich, 2013; Lorenzetti, 2013).
Last but not least, there is the hope that online learning will be able to provide a world class education to
anyone, anywhere, and anytime as long as they have access to the Internet. A number of websites and
companies—Khan Academy, Udacity, edX, and Coursera are some of the most prominent ones—are
built on this premise, and many well-respected scholars and entrepreneurs have high hopes and
expectations for online learning, particularly for massive open online courses (Bowen, 2013; Fisher, 2012;
Koller & Ng, 2012; Lewin, 2012; Selingo, 2013). Central to this particular benefit—in fact, to most of the
purported benefits of online learning—is the effectiveness of the online format in educating students. If
online learning is generally less effective than the conventional face-to-face format, then some of the
aforementioned purported claims and benefits of online learning are highly suspect. Therein lies the crux
of the issue, the fundamental concern of online learning and the focus of this paper: the effectiveness of
the online format in educating students compared to the traditional format. To address this issue, the
positive, negative, and mixed and null findings of the effectiveness of online learning as compared to the
traditional format will be examined.
The Positive Findings
There are a large number of studies that find positive statistically significant effects for student learning
outcomes in the online or hybrid format compared to the traditional face-to-face format. Some of the
positive learning outcomes are improved learning as measured by test scores, student engagement with
the class material, improved perception of learning and of the online format, stronger sense of community
among students, and reduction in withdrawal or failure. Consider the following illustration based on a
study by Riffell and Sibley (2005). Jean-Luc was an archeologist who needed to fulfill a general science
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
311
course to graduate. He had not performed well in a traditional science course and when he saw there was
a hybrid environmental biology course that included bi-weekly online assignments in lieu of the traditional
lecture, he thought this might work better for him. He found that the online assignments gave him time to
think and reflect about the materials better than the traditional lectures. This led him to understand the
ideas more thoroughly, which allowed him to participate more during face-to-face active-learning
exercises. He also felt that he had more meaningful online and in-person interactions with the professor
since he was able to participate more than he usually did in a science class. As a result, Jean-Luc had a
deeper understanding of environmental biology and he did well in the class, above the average
performance of his face-to-face counterpart and well above what he expected from himself. This simple
example illustrates the kind of stories that can be told in these positive studies.
From a more systematic analysis, Navarro and Shoemaker (2000) found that student learning outcomes
for online learners were as good as or better than traditional learners regardless of background
characteristics and that the students were greatly satisfied with online learning. Rovai and Jordan (2004)
examined the relationship of sense of community between traditional classroom and the blended format,
and they found that students in the blended format had a stronger sense of community than students in
the traditional format. In a study that compares learning outcomes for students who self-selected into the
online format for a macroeconomics course, researchers found that after correcting for sample selection
bias, test scores for the online format students were four points higher than for the traditional format
(Harmon & Lambrinos, 2006). In a methodologically rigorous study conducted at Ithaka (Bowen & Ithaka,
2012), students were randomly assigned to the traditional format (control) and a hybrid interactive online
learning format that met once a week where students did most of the work online (treatment). The
researchers found that there are comparable learning outcomes for both groups and that there was the
promise of cost savings and productivity gains over time for the hybrid course. Furthermore, these
learning improvement and cost saving gains are expected to increase as new tools and software for
online learning are being developed and tested continually.
In a large political science course, using mixed methods, researchers found that students using
PeerWise—a recently created online pedagogical tool that enables students to write, share, answer,
discuss and rate multiple choice questions with little to no input from the instructor—had better learning
outcomes and improved perceptions of learning as well as motivation to learn (Feeley & Parris, 2012). To
further develop the use and effectiveness of PeerWise, a study on the effect of virtual achievements, a
badge-based achievement system in PeerWise, in a large randomized control trial found that there was a
significant positive effect on the quantity of students’ contributions without a corresponding loss of quality
(Denny, 2013). As online learning grows, more and more aspects of “gamification,” the use of game
mechanics and virtual achievements in non-game contexts to engage users, are being added to the
virtual environment to increase task engagement and decrease attrition (Deterding, Dixon, Khaled, &
Nacke, 2011; Huotari & Hamari, 2012; Kapp, 2012).
Even though there are positive findings for the effectiveness of online learning, it is still unclear that this
generally holds true across studies. Funded by the U.S. Department of Education, a team of researchers
at Stanford Research Institute International conducted a systematic search of the literature from 1996 to
2008 and identified more than a thousand empirical studies of online learning (Means et al., 2010). In the
meta-analysis which used stringent criteria for selecting studies that utilized a rigorous research design,
compared online learning with the traditional format, quantitatively measured student learning outcomes,
and provided enough information to calculate an effect size, the researchers analyzed 45 studies and on
average, they found that students in an online format performed modestly better than those in the
traditional format. The difference in student learning outcomes was larger in the studies where online
elements were blended with face-to-face instruction, and these blended conditions often included
additional learning time and instructional elements not received by students in the control conditions. The
variations in how online learning was implemented did not affect student learning outcomes significantly,
but it should be noted that there is a small number of studies for this particular finding (N=13). The
researchers concluded that the combination of time spent, curriculum, and pedagogy in the online format
produced the observed difference in learning outcomes, but there was no evidence that online learning is
superior as a medium for learning, which is consistent with prior literature (Bernard et al., 2004; Clark,
1994). The researchers noted that there were few rigorous K-12 studies and so their findings are not
necessarily generalizable to K-12 settings.
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
312
It must be emphasized that this seminal work by Means et al. is one of the most cited and well-respected
meta-analyses to date (Lack, 2013). It sets a very high standard for meta-analytical work, and its main
finding is student learning outcomes are better for online learning than the traditional format, modest, but
significant nonetheless.
The Null Findings
In comparison to the number of positive studies, there are many, many more studies that found null
findings for the effects of online learning. One of the most cited (1900 citations!) and well-known studies
for the effects of distance and online education on student learning outcomes is the seminal work by
Thomas Russell (1999). The author compiled over 350 studies on distance and online education dating
back from 1928 that suggested that there is no significant difference in the learning outcomes for the
traditional face-to-face format versus mediated instruction. The author has continued this work by
soliciting and compiling studies on distance education in its various formats—most of the current studies
are now on online learning—at http://www.nosignificantdifference.org. This website contains one of the
largest collections of studies comparing the effects of distance and online learning versus the traditional
format. Of all the positive, mixed, null, and negative findings on the site, about 70 percent of the studies
found no significant differences. However, one of the most common criticisms of Russell’s work is that the
majority of the original studies have poor methodology: they often lack control groups, random
assignment, experimental controls for confounding variables, and little to no discussion of attrition.
Subsequent meta-analyses, such as Bernard et al. (2004) and Means et al. (2010), have used more
rigorous selection criteria.
In a meta-analysis in higher education, Bernard et al. (2004) found that overall there was no significant
difference in achievement, attitude, and retention outcomes between distance education, which included
online education, and the traditional face-to-face education. However, there was significant heterogeneity
in student learning outcomes for different activities. Separating student learning outcomes based on
synchronous and asynchronous activities, activities that have to be done at the same time or at each
person’s convenience respectively, showed that the mean achievement effect sizes for synchronous work
were better for the traditional format, but asynchronous work favored distance education. In other words,
there are better learning outcomes in the traditional format for activities that have to be done
simultaneously and better outcomes in the mediated distance format for activities that can be done at
various times. Moreover, researchers also found, using weighted multiple regression, that the
methodology of the studies accounts for most of the variations in learning outcomes followed by
pedagogy and media (Bernard et al., 2004). Otherwise stated, the medium of distance education, whether
it is mail correspondence or the TV or the Internet, explains the least of the variation in learning
outcomes, which supports Clark’s (1994) claim and is later confirmed by Means et al. (2010). Other
studies have also arrived at similar conclusions. For instance, a recent systematic review comparing the
learning of clinical skills in undergraduate nurse education between the online format and the traditional
found that there was no significant difference between the two formats (McCutcheon, Lohan, Traynor, &
Martin, 2015).
In 2005, a year after Bernard et al. published their study, another group published an analysis on the
effectiveness of distance education. Zhao et al. (2005) analyzed prior literature, which included the
Russell’s 1999 study among other meta-analyses, and found that the overall mean effect size was close
to zero, but there was a modest size standard deviation. They then used a rigorous methodology to trim
studies with weak methodology or ones that did not provide adequate information and arrived at some
rather interesting findings. Zhao et al. found the presence of the Hawthorne effect where there was a
tendency to find favorable findings for distance or online education if the researcher was also the
instructor of the course. They also found that the “right” mixture of human and technology, i.e., hybrid or
blended learning, was particularly effective. Implications of this study are that courses that can combine
the strengths of online learning and traditional learning are more effective than courses that use mainly
one format and it is possible that as digital and online technologies improve and mature they will become
more effective in helping students learn.
One unexpected finding from the Zhao et al. study was that the publication year was a significant
moderator for the effectiveness of distance education. Studies published before 1998 do not find
significant difference between distance education and traditional education while studies published in and
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
313
after 1998 generally find significant differences in favor of distance education. It is perhaps useful to think
of online classes before the turn of the millennium as first-generation online courses and those after as
second-generation online courses. The second-generation online courses are able to build upon the first-
generation courses and improved student learning. It remains to be seen if massive open online courses
(MOOCs), due to the sheer numbers of users and open access feature, are substantially different enough
to be classified as third-generation or if it is simply a continuation of the second-generation. Most of the
current conversations and studies in the literature, including this paper, are focused on the second
generation of online courses.
In summary, most of the no significant difference studies found that overall there are no significant
differences. However, other studies find the effectiveness of online learning is not positive or equivalent
compared to the traditional format and some find that certain groups of students benefit from online
learning while others benefit from the traditional format. To get a more complete picture, there needs to
be an examination of the mixed and negative findings to arrive at a more nuanced conclusion.
The Mixed and Negative Findings
Compared to the number of studies that found positive or no significant effects for student learning
outcomes in the online format, the number of studies that found mixed or negative significant effects is
much smaller, by a full order of magnitude. Some of these studies are direct contradictions of the studies
with positive results: they find that students performed worse in the online format compared to the
traditional format. Some studies’ findings are more nuanced. They find that there are negative effects for
certain groups of students and null findings for others. There are studies discussed in this section that
systematically examine the ubiquitous self-selection bias of online learning: the endogeneity of learning
environment choice. Most studies on distance or online learning do not examine this selection bias, which
some researchers posit as a culprit for the “no significant difference” phenomenon.
In a study that compares student learning outcomes in a microeconomics course, Brown and Liedholm
(2002) found that students in the online format performed significantly worse on tests than the students in
the traditional format even though they had better GPA and ACT scores. This difference was most
pronounced for complex questions and least pronounced for basic questions. One possible explanation
was that half of the online students reported to spend less than three hours per week and none claimed to
spend more than seven hours per week, while half of the students in the traditional format attended every
class, a minimum of three hours per week. The differences in time devoted to class or active engagement
resulting in differential outcomes were also found in another study (Hiltz et al., 2000). Brown and
Liedholm (2002) also found that female students performed significantly worse, six percentage points
worse, than male students in the traditional format, but there was no significant difference for the sexes in
the online format. Other studies have also found that sex is a moderating variable for student learning
outcomes when comparing online and traditional formats (Figlio, Rush, & Yin, 2010; Xu & Jaggars, 2013).
For instance, Xu and Jaggars (2013) used a dataset of about 500,000 courses taken by over 40,000
students in Washington state, and they found that there were detrimental effects for all types of students
in the online format, but most particularly for male students, younger students, black students, and lower-
achieving students.
In one of the first experimental studies on the effects of traditional instruction versus online learning where
students were randomly assigned to live lectures versus watching the same lectures online while
supplemental materials and instructions were the same, Figlio et al. (2010) found modest evidence that
the traditional format has a positive effect compared to the online format. This difference was more
pronounced for Hispanic students, male students, and lower-achieving students. One possible and very
likely significant internal validity threat, which the authors fully acknowledge, was treatment diffusion for
the “live-only” students since they could look at the online lectures using a friend’s account, while “online”
students were prevented from attending live lectures. Moreover, there were at least two sources of
external validity threats: volunteer effect and grade incentive (half a grade boost to students who
volunteered to be in the experiment). Thus, researchers should be cautious in interpreting this study’s
findings or generalizing them to other settings.
Perhaps the most mixed finding of all the research thus far is the most recent meta-analysis by Kelly Lack
at Ithaka S&R (2013). Using a similar set of criteria as the DOE meta-analysis by Means et al. with an
additional criterion for the studies to involve one or more undergraduate for-credit college course(s), Lack
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
314
found an additional 30 studies that were published after the DOE report and/or simply missed. The
researcher found that most of the studies had mixed results. In some studies, students in the online or
hybrid format performed better, but in others, they performed worse, and for some, there were no
significant differences between the two groups. She concluded that these studies did not provide enough
evidence for assessing whether online learning is significantly more or less effective than the traditional
face-to-face format.
In short, there is not a constant effect for the effectiveness of online learning relative to traditional
learning. There is strong evidence for the heterogeneous outcomes of the effects of online learning, and
in particular, a number of student characteristics such as sex, race/ethnicity, and ability, can moderate the
learning outcomes. Moreover, there are other factors such as the existence and structure of learning
communities, the type of online learning activities, varied materials, formative assessment, and the level
of students’ active engagement also play critical roles in determining the outcomes of the two formats
(Blitz, 2013; Brown & Liedholm, 2004; Hiltz et al., 2000; Tsai, Tsai, & Lin, 2015; Wang et al., 2006). For
instance, Brown and Liedholm (2004) found that there was considerable diversity in both the order in
which students used course materials (ranging from textbook, media-enhanced PowerPoint slides, video
lectures, interactive and individualized Excel-based practice problems, and repeatable, low-stakes
practice questions) and the value they placed on different materials for learning. They concluded that
additional tools and variegated materials in a course would be more beneficial than the exclusion of them.
Lastly, there is the ubiquitous threat of selection bias: the endogeneity of learning environment choice. If
students self-select into the online format, then the achievement differences between the online and
traditional format are potentially biased as a result of the characteristics of the students. Therefore, this
particular issue must be examined thoroughly.
Selection Bias
In terms of selection bias, one of the first studies that address this issue is a study by Anstine and
Skidmore (2005). The researchers examined the effectiveness of online learning versus traditional
learning for MBA students at a comprehensive university. The courses were taught by two instructors who
developed the course for the online environment first and then each instructor taught both formats. A
comparison of test scores indicated no differences in learning outcomes for the traditional and online
format. However, students were not randomly selected into control or treatment group. Instead, students
selected their learning environment. The concern was that students with higher human capital
endowment self-selected into the online format, resulting in an increased in the mean test score. Anstine
and Skidmore found that when other factors other than online format and tradition format were controlled,
the outcomes in the online format were inferior to the traditional format. They compared the results of a
two-stage least squares analysis against ordinary least squares and found similar results, and a switching
regression showed that the online format was substantially less effective than the traditional format. They
concluded that self-selection bias masked the true effectiveness of the traditional format relative to the
online format, and once corrected, the learning outcomes for the online format were inferior.
This point is well taken as the majority of studies and meta-analyses do not account for the endogenous
selection bias, so it is unclear and unknown how many “no significant difference” studies would come to a
different conclusion once selection bias is accounted for. However, it must also be acknowledged that
there is a small sample size in Anstine and Skidmore’s work.
Nevertheless, there have been other studies that have found similar results as Anstine and Skidmore
(Brown & Liedholm, 2002; Coates et al., 2004; Gratton-Lavoie & Stanley, 2009). For instance, in a study
comparing student learning outcomes between online and traditional formats in a Principle of Economics
undergraduate course, Coates et al. (2004) found that the achievement differences bias toward zero if
self-selection was not taken into account. In the selection-corrected model, the students in the online
format scored significantly worse than the students in the live format. An endogenous switching model
predicted that had online students selected the live format instead of the online format, ceteris paribus,
they would have performed better.
The story, however, does not end there. It should be noted that the many of the studies that find student
learning outcomes in the online format are inferior to the traditional format are done at the undergraduate
level in Principle of Economic courses (Brown & Liedholm, 2002; Coates et al., 2004; Figlo, Rush, & Yin,
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
315
2010; Gratton-Lavoie & Stanley, 2009). Harmon and Lambrinos hypothesized that the outcomes might be
different for graduate students who might be more mature and have better independent learning skills
(2012). Using panel data and fixed effects model to correct for bias from unobservable variables, the
researchers found that the effect of online learning was not significantly different than the traditional
format for graduate-level students and may have a positive effect on learning outcomes (a 23% increase
in correctly answering a question at the .10 level). The researchers believed this finding suggested that
more mature students with better independent learning skills were better candidates for online learning.
Summary of Findings
Using the studies found on Nosignificantdifference.org as indicator of the effectiveness of distance and
online learning, it would be observed that about 92% of all distance and online education studies find that
distance and online education is at least as effective, if not better, than traditional education. About 3% of
the studies compiled by the site show the reverse, that traditional face-to-face format is more effective,
and about 4% show mixed findings. However, given the issues of selection bias that later studies pointed
out and the lack of rigorous methodology of the earlier studies, it is difficult to say how meaningful these
numbers really are. Moreover, this repository is subject to selection issues related to voluntary submittal
to the site. In terms of high standard meta-analyses, Means et al. (2010) found there is positive but
modest significant difference in favor of online learning, and Lack (2013) concluded that there is not
enough evidence one way or another. Given these findings, there are two different but intertwining paths
that researchers and educators can take at this juncture.
Beyond No Significant Difference and Future Horizons
The first path is the ever ubiquitous “more research needed” approach to determine the heterogeneity
effects of online learning. Researchers and educators should conduct more research on the effectiveness
of the online learning format, employing rigorous research designs and report adequate information to
contribute to the literature. In particular, there needs to be a focus on the factors that have been observed
to have an impact on the effectiveness of online education: self-selection bias, blended instruction, active
engagement with the materials, formative assessment, varied materials and repeatable low-stake
practice, collaborative learning communities, student maturity, independent learning skills, synchronous
and asynchronous work, and student characteristics. Moreover, there is pressing need for more research
in the learning sciences in regards to how students learn online and how adaptive learning software can
mediate and individualize student learning.
An interesting research question in this area would be: To what extent do blended instruction, active
engagement with class materials, formative assessment, varied materials, low-stake practice,
collaborative learning communities, and synchronous and asynchronous work influence student learning
outcomes? These factors are (presumably) under the direct control of the course instructor and can be
adjusted to improve student learning. One challenging aspect of this work would be how to measure each
of those factors, as they are not all simply descriptive and easily quantifiable. A possibility is using a panel
of experts who have little to no direct link with the courses to rank each variable for each course
independently and use a combined score as an indirect measure for each variable. This is simply a
skeletal design of the research and needs to be fully fleshed out, but it could provide invaluable insight
into what makes an online course effective.
The second path is to move beyond the no significant difference phenomenon. Twigg and Learning
(2001) posited that the key to move beyond no significant difference was to individualize student learning
and determine the most efficient and effective learning pathways for different learners in particular
courses. Since then, there has been the development and growth of many software and courses that can
and have been used to individualize student learning (Bowen & Ithaka, 2012; Feeley & Parris, 2012; Hoic-
Bozic, Mornar, & Boticki, 2009; Mihai, Stanciu, & Aleca, 2011). For instance, in an experimental study of
228 university students, Xu et al. (2014) found that personalized virtual online learning environments
improved students’ exam performance, satisfaction and self-efficacy compared to non-personalized
virtual learning environments. Moreover, in conjunction with learning sciences, scholars and researchers
should use the massive amount of data collected from MOOCs and analyze student learning, click by
click, as they go through lessons and quizzes. This should greatly increase what is known about how
students learn and it should be used by researchers, educators, and entrepreneurs to design better online
courses aimed squarely at improving student learning outcomes.
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
316
For individual instructors, there are a considerable number of available resources that can support the
transition from the traditional format to online learning. The Handbooks of Research on Educational
Communications and Technology cover considerable ground on online education, ranging from the
theoretical foundations, different types of technologies, instructional design approaches, instructional
strategies, and learning models (Jonassen & Driscoll, 2004; Spector et al., 2008). There are also practical
resources that offer and provide innovative ideas to promote active learning online with ready-made
adaptable activities, specific examples of what can be done, case studies detailing actual teaching
practices, tips for effective pedagogy and technologies that are based on traditional theories integrated
with the latest research in cognitive learning (Bennett, Marsh & Killen, 2007; Boettcher & Conrad, 2010;
Thomas, 2011).
A most intriguing project that combines the work from both prospective paths would be the creation of one
or two online courses that leverage the factors that are most effective in improving learning outcomes,
individualize student learning using adaptive learning software, and incorporate non-trivial best practices
of “gamification.” Gamification has been found to increase engagement, motivation and productivity in
solving problems and task engagement in a variety of non-game contexts including learning (Deterding et
al., 2011; Hamari, Koivisto, Sarsa, & Hamari, 2014; Kapp, 2012; Landers & Callen, 2011; Tsai, Tsai, &
Lin, 2015). The creators of the course should consist of experts on “gamification”, online learning and
learning sciences, and instructors of the most highly rated online courses. The goal would be to create an
online course that maximizes student learning.
Concluding Remarks
It would be too easy altogether to jump on the online learning bandwagon or to dismiss it as a fad that will
go away (and come back as many educational fads have been known to do). Overall, there is strong
evidence to suggest that online learning is at least as effective as the traditional format, but the evidence
is, by no means, conclusive. Online learning is a story that is still being written, and how it progresses will
likely depend on those present.
References
Anstine, J., & Skidmore, M. (2005). A small sample study of traditional and online courses with sample
selection adjustment. The Journal of Economic Education, 107–127.
Bartley, S. J., & Golek, J. H. (2004). Evaluating the Cost Effectiveness of Online and Face-to-Face
Instruction. Educational Technology & Society, 7(4), 167–175.
Bennett, S., Marsh, D., & Killen, C. (2007). Handbook of online education. Continuum International
Publishing Group, Incorporated. Retrieved from http://dl.acm.org/citation.cfm?id=1526291
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., … Huang, B. (2004). How
Does Distance Education Compare With Classroom Instruction? A Meta-Analysis of the Empirical
Literature. Review of Educational Research, 74(3), 379–439.
Blitz, C. L. (2013). Can Online Learning Communities Achieve the Goals of Traditional Professional
Learning Communities? What the Literature Says. REL 2013-003. Regional Educational
Laboratory Mid-Atlantic. Retrieved from http://eric.ed.gov/?id=ED544210
Boettcher, J. V., & Conrad, R.-M. (2010). The online teaching survival guide: Simple and practical
pedagogical tips. John Wiley & Sons.
Bowen, W. G. (2013). Higher education in the digital age. Princeton University Press.
Bowen, W. G., & Ithaka, S. (2012). Interactive learning online at public universities: Evidence from
randomized trials. Ithaka S+ R. Retrieved from http://mitcet.mit.edu/wp-
content/uploads/2012/05/BowenReport-2012.pdf
Brown, B. W., & Liedholm, C. E. (2002). Can web courses replace the classroom in principles of
microeconomics? The American Economic Review, 92(2), 444–448.
Brown, B. W., & Liedholm, C. E. (2004). Student preferences in using online learning resources. Social
Science Computer Review, 22(4), 479–492.
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
317
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and
Development, 42(2), 21–29. doi:10.1007/BF02299088
Coates, D., Humphreys, B. R., Kane, J., & Vachris, M. A. (2004). “No significant distance” between face-
to-face and online instruction: evidence from principles of economics. Economics of Education
Review, 23(5), 533–546.
De la Varre, C., Keane, J., & Irvin, M. J. (2011). Enhancing Online Distance Education in Small Rural US
Schools: A Hybrid, Learner-Centred Model. Journal of Asynchronous Learning Networks, 15(4),
35–46.
Denny, P. (2013). The Effect of Virtual Achievements on Student Engagement. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (pp. 763–772). New York, NY,
USA: ACM.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness:
defining gamification. In Proceedings of the 15th International Academic MindTrek Conference:
Envisioning Future Media Environments (pp. 9–15). ACM.
Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011). Gamification. using game-design
elements in non-gaming contexts. In PART 2———–Proceedings of the 2011 annual conference
extended abstracts on Human factors in computing systems (pp. 2425–2428). ACM.
Evans, J. & Haase, I. (2001). Online business education in the twenty‐first century: an analysis of
potential target markets.Internet Research, 11(3), 246–260.
http://doi.org/10.1108/10662240110396432
Farinella, J. A., Hobbs, B. K., & Weeks, H. S. (2000). Distance delivery: The faculty perspective. Financial
Practice and Education, 10, 184–194.
Figlio, D. N., Rush, M., & Yin, L. (2010). Is it live or is it internet? Experimental estimates of the effects of
online instruction on student learning. National Bureau of Economic Research.
FinAid | Loans | Student Loan Debt Clock. (n.d.). Retrieved March 5, 2014, from
http://www.finaid.org/loans/studentloandebtclock.phtml
Fisher, D. (2012, November 6). Warming Up to MOOC’s. The Chronicle of Higher Education Blogs:
ProfHacker.
Gratton-Lavoie, C., & Stanley, D. (2009). Teaching and learning principles of Microeconomics online: An
empirical assessment. The Journal of Economic Education, 40(1), 3–25.
Hamari, J., Koivisto, J., Sarsa, H., & Hamari, J. (2014). Does Gamification Work. In A Literature Review of
Empirical Studies on gamification. In proceedings of the 47th Hawaii International Conference on
System Sciences, Hawaii, USA.
Harmon, O. R., & Lambrinos, J. (2006). Online Format vs. Live Mode of Instruction: Do Human Capital
Differences or Differences in Returns to Human Capital Explain the Differences in Outcomes?
Harmon, O. R., & Lambrinos, J. (2012). Testing the Effect of Hybrid Lecture Delivery on Learning
Outcomes. Retrieved from http://ideas.repec.org/p/uct/uconnp/2012-36.html
Hiltz, S. R., Coppola, N., Rotter, N., Toroff, M., & Benbunan-Fich, R. (2000). Measuring the Importance of
Collaborative Learning for the Effectiveness of ALN: A Multi-Measure. In Learning Effectiveness
and Faculty Satisfaction: Proceedings of the 1999 Sloan Summer Workshop on Asynchronous
Learning Networks (Vol. 1, p. 101). Olin College-Sloan-C.
Hoic-Bozic, N., Mornar, V., & Boticki, I. (2009). A Blended Learning Approach to Course Design and
Implementation. IEEE Transactions on Education, 52(1), 19–30. doi:10.1109/TE.2007.914945
Huotari, K., & Hamari, J. (2012). Defining gamification: a service marketing perspective. In Proceeding of
the 16th International Academic MindTrek Conference (pp. 17–22). ACM. Retrieved from
http://dl.acm.org/citation.cfm?id=2393137
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
318
Jonassen, D., & Driscoll, M. (2004). Handbook of research for educational communications and
technology (Vol. 2). Routledge.
Jung, I., & Rha, I. (2000). Effectiveness and Cost-Effectiveness of Online Education: A Review of the
Literature. Educational Technology, 40(4), 57–60.
Kapp, K. M. (2012). The gamification of learning and instruction: game-based methods and strategies for
training and education. John Wiley & Sons.
Kim, K., & Bonk, C. J. (2006). The future of online teaching and learning in higher education: The survey
says. Educause Quarterly, 29(4), 22.
Kolowich, S. (2013, February 7). American Council on Education Recommends 5 MOOCs for Credit. The
Chronicle of Higher Education. Retrieved from http://chronicle.com/article/American-Council-on-
Education/137155/
Lack, K. A. (2013). Current status of research on online learning in postsecondary education. Retrieved
from http://apo.org.au/sites/default/files/docs/Ithaka-
sr_OnlineLearningPostSecondaryEducation_May2012.pdf
Landers, R. N., & Callan, R. C. (2011). Casual social games as serious games: The psychology of
gamification in undergraduate education and employee training. In Serious games and
edutainment applications (pp. 399–423). Springer.
Lewin, T. (2012, July 18). Anant Agarwal Discusses Free Online Courses Offered by a Harvard/M.I.T.
Partnership. The New York Times. Retrieved from
http://www.nytimes.com/2012/07/20/education/edlife/anant-agarwal-discusses-free-online-
courses-offered-by-a-harvard-mit-partnership.html
Lorenzetti, J. (2013.). Academic Administration - Running a MOOC: Secrets of the World’s Largest
Distance Education Classes - Magna Publications.
McCutcheon, K., Lohan, M., Traynor, M., & Martin, D. (2015). A systematic review evaluating the impact
of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse
education. Journal of Advanced Nursing, 71(2), 255–270.
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010, September). Evaluation of Evidence-
Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies.
Monograph. Retrieved February 1, 2014, from
http://www.ed.gov/about/offices/list/opepd/ppss/reports.html
Mihai, F., Stanciu, A., & Aleca, O. (2011). Changing learning environment through technology. Annales
Universitatis Apulensis Series Oeconomica, 13(1), 48–56.
Navarro, P., & Shoemaker, J. (2000). Performance and perceptions of distance learners in cyberspace.
American Journal of Distance Education, 14(2), 15–35.
No Significant Difference - Presented by WCET. (n.d.). Retrieved February 8, 2014, from
http://www.nosignificantdifference.org/
Osguthorpe, R. T., & Graham, C. R. (2003). Blended Learning Environments: Definitions and Directions.
Quarterly Review of Distance Education, 4(3), 227–33.
Pape, L. (2010). Blended Teaching & Learning. School Administrator, 67(4), 16–21.
Riffell, S., & Sibley, D. (2005). Using web-based instruction to improve large undergraduate biology
courses: An evaluation of a hybrid course format. Computers & Education, 44(3), 217–235.
Rovai, A. P., & Jordan, H. (2004). Blended Learning and Sense of Community: A Comparative Analysis
with Traditional and Fully Online Graduate Courses. The International Review of Research in
Open and Distance Learning, 5(2). Retrieved from
http://www.irrodl.org/index.php/irrodl/article/view/192
MERLOT Journal of Online Learning and Teaching Vol. 11, No. 2, June 2015
!
!
319
Russell, T. L. (1999). The no significant difference phenomenon: A comparative research annotated
bibliography on technology for distance education: As reported in 355 research reports,
summaries and papers. North Carolina State University.
Spector, J. M., Merrill, M. D., Elen, J., & Bishop, M. J. (2008). Handbook of research on educational
communications and technology. Springer. Retrieved from
http://link.springer.com/content/pdf/10.1007/978-1-4614-3185-5.pdf
Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types
on game-based formative assessment in an online learning environment. Computers &
Education, 81, 259–269.
Thomas, M. (2011). Digital education: Opportunities for social collaboration. Palgrave Macmillan.
Tucker, B. (2007). Laboratories of reform: Virtual high schools and innovation in public education.
Education Sector Reports. Retrieved from
http://heartland.org/sites/all/modules/custom/heartland_migration/files/pdfs/28154.pdf
Twigg, C. A., & Learning, P. (2001). Innovations in online learning: Moving beyond no significant
difference. Center for Academic Transformation, Rensselaer Polytechnic Institute.
Wang, K. H., Wang, T. H., Wang, W.-L., & Huang, S. C. (2006). Learning styles and formative
assessment strategy: enhancing student achievement in Web-based learning. Journal of
Computer Assisted Learning, 22(3), 207–217.
Xu, D., & Jaggars, S. (2013). Adaptability to online learning: Differences across types of students and
academic subject areas. Retrieved from
http://academiccommons.columbia.edu/catalog/ac:157286
Xu, D., Huang, W. W., Wang, H., & Heales, J. (2014). Enhancing e-learning effectiveness using an
intelligent agent-supported personalized virtual learning environment: An empirical
investigation. Information & Management, 51(4), 430–440.
Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, S. (2005). What makes the difference? A practical analysis of
research on the effectiveness of distance education. The Teachers College Record, 107(8),
1836–1884.
This work is published under a Creative Commons Attribution-Non-Commercial-Share-Alike License
For details please go to: http://creativecommons.org/licenses/by-nc-sa/3.0/us/
