Content uploaded by Wilfried Admiraal
Author content
All content in this area was uploaded by Wilfried Admiraal on Apr 30, 2017
Content may be subject to copyright.
Students’ Learning Outcomes in Massive Open Online
Courses (MOOCs): Some Suggestions for Course Design
Ö¤rencilerin kitlesel aç›k eriflim çevrimiçi derslerdeki kazan›mlar›: Ders tasar›m›na yönelik baz› öneriler
Olga Pilli1, Wilfried Admiraal2
1Faculty of Education, Girne American University, Girne, TRNC
2ICLON, Leiden University, Leiden, The Netherlands
MMassive Open Online Courses (MOOC) are built on
the impression that “information is everywhere”
by extending access to education. A MOOC is a
course, but it is open, distributed, participatory, and part of
lifelong network learning. The underlying idea of a MOOC is
accessibility, since anyone can participate by working collabo-
ratively either to acquire new knowledge or to expand existing
knowledge. This implies that MOOCs create a pathway for
lifelong learning processes. MOOCs are online classes in
which anyone can participate, regardless of location, in most
cases for free. They are comprised of short video lectures, sim-
ulations, and online labs combined with computer-graded tests
and online forums where participants can discuss the course
content or get help (Hoy, 2014). Basically, MOOCs are a form
of online learning that share some common features: open
access using the Internet, free of charge, asynchronous, inter-
active user forums, and the opportunity to receive a certificate
upon successful completion (EDUCAUSE, 2011). Student
Üçüncü nesil uzaktan e¤itim kapsam›nda kitlesel aç›k eriflim çevrimiçi dersler
(massive open online courses, MOOC’lar) sayesinde yüksek ö¤renimde herkes is-
tedi¤i yerden ücretsiz e¤itim alabilmektedir. Son y›llarda, e¤itimde MO-
OC’lar›n yeri üzerine birçok çal›flma yap›lm›flt›r, ancak ö¤rencilerin kazan›m-
lar› üzerine olan çal›flmalar s›n›rl›d›r. Bu çal›flmada, aç›k eriflim çevrimiçi ders-
lerin tasarlanmas›na yönelik birtak›m önerileri belirlemek amac›yla, ö¤rencile-
rin MOOC’lardaki kazan›mlar›na iliflkin literatürü gözden geçirildi. ‹nceleme,
bilimsel literatür veritabanlar›n›n sistematik olarak araflt›r›lmas›n›n ard›ndan,
3P (presage [öngörü], process [süreç] ve product [ürün]) ö¤retim ve ö¤renim mode-
linin temel bileflenlerine yönelik elefltirel bir analizle gerçeklefltirildi (Biggs,
2003). 56 yay›n›n bulgular› sentezlenerek, ö¤rencilerin kat›l›m›n› ve akademik
baflar›y› gelifltirmek ve terk etme oranlar›n› düflürmek amac›yla 13 ders tasar›-
m› önerisi gelifltirildi. Gerek ileriki araflt›rmalarda incelenmek üzere gerek ise
de MOOC’lar›n mevcut içeri¤ini gelifltirerek ve zenginlefltirerek ö¤renim ka-
zan›mlar›n› en iyi hale getirmek için baz› uygulama önerileri sunuldu.
Anahtar sözcükler: 3P modeli, baflar›, de¤erlendirme, kat›l›m, ö¤renim
kazan›mlar›.
Massive open online courses (MOOCs) as a third generation distance edu-
cation enable anyone anywhere to study for free in higher education. In
recent years, various studies have been conducted on the position of
MOOCs in education, but studies on students’ learning outcomes are lim-
ited. In this study, literature concerning students’ learning outcomes in
MOOCs was explored with the aim of identifying a set of suggestions to
design open online courses. The review was accomplished through a sys-
tematic search within scientific literature databases followed by a critical
analysis with the main components of 3P (presage-process-product) model of
teaching and learning (Biggs, 2003). Findings of the 56 publications were
synthesized which resulted in the formulation of 13 course design sugges-
tions in order to enhance students’ engagement, academic achievement and
lower attrition rate attrition. Some implications are proposed for further
research and for providers to improve and enrich the current context of
MOOCs to optimize students’ learning outcomes.
Keywords: 3P model, achievement, assessment, engagement, learning
outcomes.
‹letiflim / Correspondence:
Olga Pilli
Faculty of Education, Girne American
University, University, via Mersin 10,
Girne, TRNC
e-mail: olgapilli@gau.edu.tr
Yüksekö¤retim Dergisi 2017;7(1):46–71. © 2017 Deomed
Gelifl tarihi / Received: Eylül / September 21, 2016; Kabul tarihi / Accepted: Ocak / January 15, 2017
Bu çevrimiçi makalenin at›f künyesi / Please cite this online article as: Pilli, O., Admiraal, W. (2017).
Students’ learning outcomes in MOOCs: some suggestions for course design. Yüksekö¤retim Dergisi,7(1),
46–71. doi:10.2399/yod.17.001.
Özet Abstract
Çevrimiçi eriflim / Online available at: www.yuksekogretim.org • doi:10.2399/yod.17.001 • Karekod / QR code:
Derleme / Literature Review
www.yuksekogretim.org
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
47
learning outcomes in a MOOC platform may not be the same
as those in regular online or on-campus education, which
makes a significant contribution to ensuring the quality of
MOOCs. Understanding which factors account for students’
learning outcomes in open online courses, including student
characteristics, teaching context and learning activities, is an
important step toward designing efficacious courses and
improving open online learning. Recent attempts to use learn-
ing analytics and data mining to understand learners’ behav-
iour provide ambiguous findings on learning outcomes in
MOOCs. The similarity of behavioural patterns among stu-
dents who fail and pass in the course context compels
researchers to ask further questions and to conduct deeper
analyses of students’ learning behaviours and experiences (Wen
and Rose, 2014). On the other hand, other research findings
that evaluate the value of the MOOC phenomena indicate that
students’ learning experiences and study behaviours in
MOOCs fluctuate (Yuan, Powell, & Olivier, 2014).
Furthermore, although the low retention rate in MOOCs has
been extensively debated and pointed out as a failure, research
on the pedagogical aspects of MOOCs provides more insights
about the deficiencies of the instructional model used in open
learning environments (Fasihuddin, Skinner, & Athauda,
2013). That is to say, efforts to increase completion rates
should be designed and implemented in light of learning and
teaching theories, as well as learners’ preferences and needs.
Despite the enthusiasm for and expectations of MOOCs as
new learning platforms, many studies are based on personal
observations and/or experiences of researchers either as
instructors or participants in MOOCs (Fisher, 2014; Kop,
2011; Stefanic, 2014; Zutshi, O’Hare, & Rodafinos, 2013).
There are also auto ethnographic studies in which the
researcher acts as a participant observer (Wasson, 2013). Since
2013 several empirical studies have been published in peer-
reviewed journals, which mainly focused on effectiveness, par-
ticipation, reasons for low completion rates or high drop-out
rates, and assessment. The small number of empirical studies is
likely related to the difficulty of examining the huge amount of
complex data generated by MOOCs (Fischer, 2014; Fournier,
Kop, & Durand, 2014). At the same time, researchers have also
began to point out the advantages of analysing huge digital data
in the context of assessment, process of learning, and social
interaction (Thille et al., 2014). In addition, although most
research on MOOCs is quite recent, some review studies have
already been published. The reviews are mainly oriented
towards providing a general idea of the state-of-the-art in
MOOC phenomenon from various perspectives (Ebben &
Murphy, 2014; Gasevic, Kovanovic, Joksimovic, & Siemens,
2014; Hew & Cheung, 2014; Koutropoulos & Zaharias, 2015;
Liyanagunawardena, Adams, & Williams, 2014). Nevertheless,
these reviews provide limited practical implications for stu-
dents’ learning outcomes. Therefore, as Reich (2015) empha-
sized, additional research must be conducted to explore factors
that promote students’ learning. In addition to other research
reviews, the current study adds a new perspective to the
MOOC literature by drawing on findings of published MOOC
studies to identify the course design principles that impact stu-
dents’ learning outcomes.
Purpose of the Study
Even though MOOCs are rooted in online learning, scholars
suggest that pedagogical aspects of these massive courses may
have a distinguishable nature in laissez-faire environments
with rich data (Bayne & Ross, 2014; Redfield, 2015).
Grounded on a diversity of students’ backgrounds and inten-
tions, outcomes of teaching and learning processes in
MOOCs can be misleading if metrics from conventional in-
class or online education are applied. As the traditional vari-
ables in higher education might play out quite differently in
MOOCs, a systematic review of the MOOC literature could
provide essential insights to understand new, diverse concepts
including achievement, assessment, retention, and participa-
tion as crucial ingredients for students’ learning outcomes
(DeBoer, Ho, Stump, & Breslow, 2014). Understanding how
these concepts are related to students’ learning outcomes is
important since these are crucial elements for MOOC course
design, which helps enhance the pedagogical aspects of
MOOCs as well as provide concrete perspective for MOOCs
(Glance, Forsey, & Riley, 2013; Perna et al., 2014). For this
purpose, the 3P Model (Fig. 1) of teaching and learning in
universities by Biggs (2003) was used as a framework to pro-
vide an organized way of structuring findings identified in the
literature that appear to explain students’ learning outcomes.
According to Biggs (2003), teaching and learning in uni-
versities are considered an interacting system of four compo-
nents: students, learning environment, learning processes,
and learning outcomes. Previous studies effectively used this
model as a framework to review the literature (Han, 2014;
Noroozi, Weinberger, Biemans, Mulder, & Chizari, 2012;
Spelt, Biemans, Tobi, Luning, & Mulder, 2009). In the cur-
rent review, Biggs’s 3P Model is used to structure the find-
ings into each component, thereby presenting a comprehen-
sive model for successful learning outcomes in MOOCs. This
model might enable curriculum and course developers in
open online learning platforms to gain a holistic understand-
ing of factors influencing students’ learning outcomes.
Explicitly, this study aims to review existing MOOC research
in order to answer the following research questions:
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
48
“Which student characteristics are related to students’
learning outcomes in MOOCs?”
“Which teaching context is related to students’ learning
outcomes in MOOCs?”
“Which learning activities are related to students’ learn-
ing outcomes in MOOCs?”
Methods
This review covers literature published in or before the year
this study started (2015). The digital catalogue search of
Leiden University was used to conduct a research that
spanned multiple databases related to educational and social
sciences: Academic Search Premier (EBSCO), ProQuest,
Annual Reviews, ScienceDirect, Cambridge Journals, DOAJ,
SAGE, Web of Science, SSRN (Social Science Research
Network), and Wiley Online Library.
Inclusion and Exclusion Criteria
The following criteria were formulated to determine if previ-
ous studies should be included in the literature review: (a)
published in peer-reviewed journals, (b) reported empirical
findings, (c) reported in English, and (d) related with learning
outcomes in MOOCs. Online databases were searched using
Boolean logic with the keywords; MOOC, MOOCs, massive
open online course, and learning outcomes. This search gen-
erated 203 hits. The first author subsequently read all studies
and identified whether each article matched the criteria men-
tioned above. After the first scan for appropriateness, 46 were
not published in peer-reviewed journals, leaving 157 studies.
Among them, 84 did not provide empirical findings, leaving
73 articles. Only 56 of these research studies were selected for
this review since the others were not related to student learn-
ing outcomes (Liyanagunawardena et al., 2014, Noroozi et
al., 2012). The Appendix I summarizes the 56 studies, show-
ing the authors, publication date, purpose, research question(s),
method, sample, results, and implications for research and
practice.
Data Analysis
Initially the first author read all text segments of the Results and
Discussion sections of the selected articles that related to stu-
dents’ learning outcomes to identify the factors influencing stu-
dents’ learning outcomes. Following careful reading of the
Results and Discussion sections of each reviewed study, the
critical analysis was executed guided by research questions
based on Biggs’ (2003) 3P Model. The factors identified as con-
tributing to students’ learning outcomes were refined in an iter-
ative manner during which alternative classifications were con-
sidered. An outside researcher conducted the same analysis
procedure in order to ensure the internal consistency of the
research. This selection was then categorized into four inter-
related components (i.e., student characteristics, learning envi-
ronment, learning process, and learning outcomes) based on
Biggs model (Fig. 2).
Fig. 1. The 3P model of teaching and learning (Biggs, 2003).
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
49
In the present study, the first factor that presaged learning
outcomes was student characteristics, which includes academic
(i.e., prior-knowledge, prior-experience, and expertise) and
personal (i.e., self-motivation, self-confidence, and participa-
tion) student characteristics. The other factor that presaged
learning outcomes was course features. These features are part
of the learning environment in which MOOCs are set, which is
established by instructors or providers in terms of pedagogy,
tools, and assessment. In terms of factors that portend learning
outcomes, some of the student characteristics and course fea-
tures were related to each other. For example, course assess-
ments were related to student characteristics and some student
characteristics may have affected the efficiency of tools used in
MOOCs. The learning process component consists of findings
related to learning activities while. The final component (i.e.,
learning outcomes) includes students’ engagement, achieve-
ment, and attrition. As Fig. 2 suggests, the adopted 3P
model from Biggs (2003) identifies the relationship among
and/or between these four components and provides a com-
prehensive framework of how factors that emerged from pub-
lished studies interacted and related to students’ learning out-
comes.
Results
The factors related to learning outcomes extracted from the
reviewed publications were clustered into four inter-related
components from Biggs model (2003; Fig. 2):
Students’ characteristics
Learning environment
Learning process
Learning outcomes
Fig. 2. Framework of the factors account for learning outcomes in MOOCs (adapted from the original 3P model of Biggs, 2003).
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
50
The component of students’ characteristics was divided
into academic (i.e., prior-knowledge, prior-experience, expert-
ise, academic achievement, and matriculation) and personal
(i.e., self-motivation, self-confidence, intrinsic motivation, par-
ticipation, social economic statute, and task-oriented) student
characteristics. The course features component addressed
course design elements of MOOCs that characterize the learn-
ing environment including pedagogy, tools, tasks, duration,
feedback, and assessment. The component process factors ref-
ered to students’ learning activities in MOOCs and the com-
ponent product factors included students’ engagement,
achievement, and attrition.
Presage Factors
Students’ Academic Characteristics
Student’ academic characteristics referred to learning goals (of
an individual or a group of individuals), prior-experience, prior-
knowledge, expertise, academic achievement, procrastination,
matriculation, and task-orientation. Many of the reviewed stud-
ies highlighted that the students who participated in forums,
discussion groups, and blogs were well-educated and taking the
courses to gain professional skills (Gillani & Eynon, 2014).
Moreover, students with task-oriented skills tended to be suc-
cessful in MOOCs (Liu et al., 2014).
Students’ prior experiences with e-learning were found to
be positively related to their participation level. Experienced
students in networked learning participated at a higher level in
MOOCs (Greene, Oswald, & Pomerantz, 2015; Kop,
Fournier, & Mak, 2011). The experienced students tended to
participate and to contribute more than novice learners in dis-
cussion forums, blogs, and learning networks; new students
tended to use the ready-made materials in MOOCs (Fournier
et al., 2014; Milligan, Littlejohn, & Margaryan, 2013).
Moreover, one recent study indicated the gap between novice
and experienced MOOCers as a possible ‘dark side’ of MOOCs
since the novice MOOC participants of Rhizo 14 cMOOC felt
isolated, which limited their engagement (Mackness & Bell,
2015).
Although findings of the reviewed studies (Breslow et al.,
2013; Greene et al., 2015; Konstan, Walker, Brooks, Brown, &
Ekstrand, 2015) did not indicate any significant correlation
between either age or gender with student learning outcomes,
the authors found a relationship between student level of
schooling and outcomes, as higher level of schooling is associ-
ated with higher participation and lower attrition.
For student retention in MOOCs students’ prior achieve-
ment also seemed to be an influential factor (de Freitas,
Morgan, & Gibson, 2015), although findings about this rela-
tionship were ambiguous. For example, Jiang, Williams,
Warschauer, He, & O’Dowd (2014) found that students with a
poor academic background were the ones who completed and
received the certificate. On the other hand, other research indi-
cated that matriculated students were more likely to complete a
MOOC (Chen & Chen, 2015; Firmin et al., 2014) since they
are more task-oriented (Jiang et al., 2014). Although students
enrol in MOOCs for degree purposes (Chen & Chen, 2015),
those who score high on procrastination on academic tasks
(Diver & Martinez, 2015) tended to dropout of the course.
Student Personal Characteristics
The second category of student characteristics, personal char-
acteristics, refer to non-academic characteristics including self-
motivation, self-confidence, intrinsic motivation, intentions,
self-commitment, and socioeconomic status. In general, these
individual student characteristics were related to how students
engaged with MOOC activities and their completion of the
course. For example, Kizilcec & Schneider (2015) found that
students’ intentions and their level of intrinsic motivation were
positively related to the extent to which students watched
videos and their assessment completion in MOOCs. Similarly,
students with high self-motivation were more engaged in
cMOOCs (Castaño-Garrido, Maiz-Olazabalaga, & Garay-
Ruiz, 2015; Dillahunt, Wang, & Teasley, 2014). This also was
the case with students who reported a relatively high self-con-
fidence (Milligan et al., 2013). Finally, students with a low
socioeconomic status who self-identified as being unable to
afford a formal education seemed to put more effort into being
successful in the course compared to other students (Dillahunt
et al., 2014).
Course features: Pedagogy
Many of the reviewed studies explicitly explained the design
and implementation process of the MOOCs, but only a limited
number of studies examined how the design of MOOCs was
related to students’ learning activities or outcomes. The pio-
neering empirical studies concentrated on only two philosoph-
ical MOOC designs: cMOOCs and xMOOCs (Rodriguez,
2012). After several years, however, the research shows that
more varieties of xMOOC and cMOOC had emerged (Clark,
2013). It is what actually happens in these courses, however,
rather than the specific pedagogical beliefs, that are essential for
students’ learning outcomes.
Students’ learning mostly results from an interface
between the provided content and pedagogical strategies
when these engage the learner’s interest (Khine &
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
51
Lourdusamy, 2003). Learners seem to feel more interactive,
open, connected, and autonomous in small cMOOC [e.g.,
SPOCs (Small Private Open Courses) or SCOOCs (Small
Connectivist Open Online Courses)] platforms (Mackness,
Waite, Roberts, & Lovegrove, 2013). Some other factors,
including ‘flexibility to do and read,’ ‘course design,’ and
‘receiving feedback from a knowledgeable person,’ are also
identified as influential factors on students’ learning in
cMOOCs (Fournier et al., 2014). However, many MOOC
students (i.e., achievers, non-achievers, live, and archive) fol-
low the course content and watch videos in the sequential
order specified by the instructor (Campbell, Gibbs, Najafi, &
Severinski, 2014; Perna et al., 2014). Furthermore, most of
the MOOCs follow the objectivist-individual teaching
method, which actually contradicts basic features of MOOCs
such as active learning and connectivisim (Toven-Lindsey,
Rhoads, & Lozano, 2015). MOOC platforms can facilitate
both online and offline communication, which is suitable for
designing social learning experiences and many studies con-
nect the pedagogy of a MOOC and the interaction and com-
munication of students. The lack of student-student and stu-
dent-instructor interaction in many MOOCs generally can-
not provide engaged learning experiences (Hew & Cheung,
2014), whereas MOOCs that facilitate student-student inter-
action by asking students to collaborate with their peers pos-
itively influenced students’ engagement (Trumbore, 2014)
and their satisfaction with the course (Al-Atabi & DeBoer,
2014). These findings are confirmed by Kizilcec & Schneider
(2015) who found that students show relatively more engage-
ment when they are enrolled in MOOCs with their col-
leagues and/or friends.
Some authors claim that MOOCs lack a coherent instruc-
tional design process including learning objectives, instruc-
tional activities, and assessment (Margaryan, Bianco, &
Littlejohn, 2015; Spector, 2014). In fact, there is a strong pos-
itive relationship between developing a curriculum that is
consistent with learning objectives and assessments
(Falchikov & Goldfinch, 2000). This means that in many
cases, a lack of instructional objectives in MOOCs makes
them insufficient to achieve the expected learning outcomes.
If we compare xMOOCs and cMOOCs, connectivist orient-
ed MOOCs seem to provide more quality in terms of instruc-
tional principles, such as students’ activation, authentic
resources, application and integration of learning activities,
collaboration between peers, development of collective
knowledge, and differentiation between various student
groups (Margaryan et al., 2015). But this doesn’t prove that
xMOOCs are inappropriate for student learning. In MOOC
environments, the flexibility of students to follow individual-
ized learning pathways is sometimes incompatible with the
course providers’ or instructors’ pre-determined course
design structure. Therefore, researchers should think of new
metric system to evaluate the design quality of MOOCs.
Course Features: Tools
Materials are the backbone of teaching-learning activities by
supporting students with different learning styles in meaning-
ful learning (Klimova & Poulova, 2013). MOOCs utilize
commonly used teaching materials such as instruction videos,
e-resources, e-books, and exercise sets. In addition, mostly in
cMOOCs, social media tools such as discussion groups,
blogs, web forums, social network sites (SNSs), Wikis, and
podcasts encourage students to participate, contribute, and
collaboratively construct knowledge (Veletsianos, Collier, &
Schneider, 2015). Some authors found positive relationships
between the use of social media tools in MOOCs and learn-
ing outcomes (e.g., the use of Google+) (Vivian, Falkner, &
Falkner, 2014). In addition, some exclusive learning activities
such as challenge-lesson-resolution, the daily, and brain rewiring
facilitated students’ participation and discussion, which
resulted in students being more satisfied with the course (Al-
Atabi & DeBoer, 2014; Kop et al., 2011). In addition to the
potential beneficial results associated with integrating social
media tools into the learning process, learners can empower
themselves and contribute more autonomously to their own
learning. Similar to open educational resources (OER) in
education, availability and accessibility of learning tools and
materials put MOOCs in an advantageous position, which
means that the openness and flexibility of MOOCs are two
major incentives for participation (Yousef, Chatti, Wosnitza,
& Schroeder, 2015).
Whereas the pedagogical quality of instructional materi-
als in online learning has been investigated by many
researchers (Klimova & Poulova, 2013), only a few
researchers have done so in MOOCs. Research on instruc-
tional materials in MOOCs indicates that readings (50%) and
videos (40%) are the most used supportive materials; among
other materials the discussion forums are cited by only 6% of
the students as a useful learning resource (Giannakos,
Jaccheri, & Krogstie, 2014; Liu et al., 2014). Pre-recorded
videos are quite popular in open education platforms, and
some authors show positive evaluations of pre-recorded video
based on xMOOCs (Adams, Yin, Madriz, & Mullen, 2014;
Firmin et al., 2014). However, students generally prefer to
watch MOOC videos in a group and with individual control
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
52
over videos (Gasevic et al., 2014). This mode of watching
videos increases student concentration and engagement, and
balances synchronicity, video interactivity, and group discus-
sion. Yet, simply incorporating interactive videos into an
online learning environment may not always result in
enhanced learning. Research shows that embedding topic
related questions in a video-based online learning environ-
ment promotes meaningful student learning, improves the
amount of student interaction, and increases the time stu-
dents spend on the learning materials (Adams et al., 2014).
Thus, MOOC platforms using question-embedded videos
may help students be more active and consequently promote
meaningful learning. Finally, including the instructor’s face
in the videos has no significant effect on students’ recall and
transfer learning, which would help students connect previ-
ous experiences to new learning contexts (Kizilcec et al.,
2015).
Course Features: Duration
Generally, the popular standard for MOOC length changes
between 6-8 week classes. Longer MOOCs can make both
developers and students feel overwhelmed. This may be why
the duration of the MOOC is negatively associated with the
completion rate. As Jordan (2014, 2015) indicated, students
tend to dropout of the course when the duration is extended.
Course Features: Assessment and Feedback
Assessment is one of the most criticized issues in MOOCs
(Clarà & Barberà, 2014), with studies mainly focused on the
credibility of e-assessment as well as self and peer-assessment.
Self and peer-assessment are distinguishing features of
MOOCs since they relieve instructors from grading huge
number of assignments and quizzes, and support learners in
enhancing their learning and understanding.
Use of self and peer-assessment as formative evaluation
helps students see their progress throughout the course.
Using self and peer-assessment as an assessment for learning
can be useful if proper feedback or assessments with rubrics
are provided to students during the formative assessment
processes; otherwise students cannot become aware of their
biases and/or misunderstanding (Admiraal, Huisman, & Pilli,
2015; Admiraal, Huisman, & Van de Ven, 2014). Peer and
self-assessment is eventually needed and will be an enduring
quality of MOOCs since it is one of the most beneficial ways
to cope with disadvantages of having so many students
enrolled in the same course simultaneously. Thus, it would be
useful to increase the effectiveness, credibility, and usability
of self and peer-assessment (Vista, Care, & Griffin, 2015).
Moreover, providing feedback and guidance (i.e., a rubric)
on peer and self-assessment rating biases can help enhance
students’ learning. Using predetermined rubrics enable stu-
dents to recognize their mistakes and misunderstandings,
which provides a more accurate learning experience and bet-
ter serves the purpose of assessment (Balfour, 2013; Kulkarni
et al., 2013). Students learn in meaningful ways when they
receive feedback from peers in discussion forums since they
feel more comfortable and open when interacting with each
other (Comer, Clark, & Canelas, 2014; Liu et al., 2014).
To improve assessment accuracy in MOOCs, machine-
based assessment would be an alternative method to peer and
self-assessment. Thus, some research studies have investigat-
ed the usability of machine assessment to evaluate students’
learning outcomes. However, MOOC instructors have criti-
cized Automated Essay Scoring (AES) tools because the way
in which they score writing assignments in MOOCs is unsat-
isfactory. The reason is that AESs can be less accurate and
reliable for evaluating students’ writing assignments when
they include complex metaphors and humour when com-
pared to instructor grading (Reilly, Stafford, Williams, &
Corliss, 2014). Finally, not the types of assessment, but the
design and clarity of assessment, are important. For instance,
poorly designed assessments decrease students’ attention to
the topic (Zutshi et al., 2013).
Process Factors
Learning Activities
In MOOC environments, understanding the learners’ activities
is mostly limited by log and clickstream analysis. For instance,
Liang et al. (2014) analysed students’ learning records using
data mining technology to discover students’ learning out-
comes. Other researchers, however, have attempted to use
qualitative data in order to reach the answer the question of
how learners approach their tasks in MOOC environments.
Veletsianos et al. (2015) distinguished four categories of stu-
dents’ activities in MOOCs: (1) digital activities, which mostly
occur in outside MOOC platforms such as social networking
sites, (2) non-digital activities such as note taking, (3) social
activities, and (4) individual activities such as locating a study
space at home.
Based on the reviewed studies, it can be concluded that
there are various learning activities. Firstly, there is a need for
equilibrium between collaborative and individual work. For
instance, in cMOOC environments, students’ learning
approaches are oriented towards collaborative learning such
as sharing, creating, and making mutual ways for learning
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
53
instead of following individual paths (Bali, Crawford, Jessen,
Signorelli, & Zamora, 2015). Findings also showed that, apart
from collaborative learning, query- and game-based learning
also are highly preferred learning approaches in MOOCs
(Chang, Hung, & Lin, 2015). Some studies indicated that
learning activities are mainly structured on principles of self-
directed learning (Bonk, Lee, Kou, Xu, & Sheu 2015; Hew &
Cheung, 2014). Learning routines can help students build
confidence, which in turn fosters commitment to the course
(Castaño-Garrido et al., 2015). Thus, the amount of collabo-
rative and individual learning activities should be balanced
since too many collaborative activities might make students
feel frustrated and contribute to incomplete submissions that
result in dropout (Saadatmand & Kumpulainen, 2014).
Secondly, both synchronous and asynchronous learning
activities should be balanced since learners might have some
difficulties following synchronous activities. Thirdly, a robust
balance between active learning and reproductive learning
activities should be created. For instance, Miller (2015) sug-
gested that active learning activities help students engage
with course content easily while other studies have indicated
that the opportunity to work on practical examples provides
meaningful learning by requiring learners to apply theoreti-
cal knowledge (Park, Jung, & Reeves, 2015; Stefanic, 2014).
Product Factors
Engagement
Coates (2006, p. 122) defines engagement as encompassing
“the active and collaborative learning, participation in chal-
lenging academic activities, formative communication with
academic staff, involvement in enriching educational experi-
ences, and feeling legitimated and supported by university
learning communities.” In online education, active and
authentic learning environments, interactive learning activi-
ties, and learner-centred communities provide the foundation
for a high level of student cognitive engagement (Katuk &
Kim, 2013).
In MOOCs, engagement refers to learner participation
with peers, instructors, and materials on the network/web.
Interaction, an active learning environment, as well as clear
instructions and guidance are effective for increasing student
engagement in MOOCs (Chang et al., 2015). Participation and
engagement in MOOCs can have different forms as students’
interaction with MOOC resources happens at various times, in
unique orders, and in different amounts (DeBoer et al., 2014).
Thus, different forms of participation and engagement should
be taken into consideration while developing MOOC curricu-
lums, teaching-learning activities, organizing learning environ-
ments, and creating assignments to increase the quality of
learning outcomes in MOOCs (Ahn, Butler, Alam, & Webster,
2013).
Achievement
Academic achievement can be defined as fulfilling course
requirements and making satisfactory progress on the way to
receiving a diploma. However, this might manifest quite dif-
ferently in MOOCs since there is still disagreement on
appropriate measures of academic achievement between
MOOC researchers and providers (Hew & Cheung, 2014).
When MOOCs are considered as an open and large-scale
course context, course certification rates can be misleading
and counterproductive indicators of their real impact and
potential.
Likewise, it may not be useful to evaluate students’
achievement with traditional metrics and methods. The defi-
nition of student success might be reformulated in terms of if
students are able to reach their own goals or realize their own
intentions (DeBoer et al., 2014; Ho et al., 2014). Furthermore,
Ho et al. (2014, p. 2) specifically stated that “Pressure to increase
certification rates may decrease the impact of open online courses, by
encouraging instructors and administrators to suppress or restrict
registration, lower certification standards, deemphasize recruitment
of target subpopulations, or disregard interventions that may dispro-
portionately increase numbers of non-certified registrants over certi-
fied registrants”.
The current review showed that being assignment-orient-
ed and well-structured, having sequential course structure
and well-designed assessments, task-oriented MOOCs, small
cMOOCs, as well as the quality of materials (e.g., videos) are
important portents of student success (Forsey, Low, &
Glance, 2013). Mainly, assignments play a significant role in
students’ achievement. For instance, Daza, Makriyannis, and
Rovira Riera (2014) revealed that learning tasks called chal-
lenge–lesson–resolution, which introduce simple real-life prob-
lems to students that are then explained and solved during the
lesson, can help students comprehend course content.
Apart from the underlying course design, some key fea-
tures of courses positively affect students’ achievement. For
instance, group projects, e-learning activities, tutorials and
online quizzes, discussion sessions such as brain rewiring,
which require students to post daily positive experiences,
result in increased student success (Al-Atabi & DeBoer,
2014). In addition, integrating other social media tools (e.g.,
Skype, Facebook, Google+) that enable students to work col-
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
54
laboratively with discussion boards and blogs are also effec-
tive for ameliorating students’ understanding and success
(Comer et al., 2014; Firmin et al., 2014; Zutshi et al., 2013).
Moreover, instructor support (e.g., providing feedback) of
student effort, which increases course engagement, may have a
substantial positive impact on achievement in MOOCs
(Hernández-Carranza, Romero-Corella, & Ramírez-Montoya,
2015). Some studies pointed that participation, motivation,
intention to complete the course, and level of course satisfac-
tion are all related to students’ achievement (Castaño-Garrido
et al., 2015; Liyanagunawardena, Lundqvist, & Williams, 2015;
Milligan et al., 2013).
Attrition
The dropout rate is a critical issue in the MOOC literature.
Thousands sign up for courses, but a very small percentage
finish with a passing grade. The literature showed that
notwithstanding the huge enrolment rate of MOOCs, the
retention rate is generally quite low (Jordan, 2014). The vast
gap between enrolment and completion is caused by several
factors such as ‘lack of time,’ ‘bad time management,’ and
‘limited time-on-task’ (Fini, 2009; Liu et al., 2014).
Some course design features are understood as strong pre-
dictors of student retention in MOOCs (Castaño-Garrido et
al., 2015; Macleod, Haywood, Woodgate, & Alkhatnai, 2015).
For instance, courses with flexible structure, support from and
monitoring by the instructor, high student cognitive engage-
ment, and high quality course materials positively influence
student retention (Campbell et al., 2014; Hernández-Carranza
et al., 2015; Liu et al., 2014; Yang, Wen, Kumar, Xing, & Rose,
2014). Finally, Perna et al. (2014) suggested that attending the
first lecture and the first quiz are two significant predictors of
course completion.
Discussion and Conclusion
It is clear that research on MOOCs is undergoing rapid
development. As this review underlines, there is a new grow-
ing body of empirical research that supports the notion that
instructional quality and learning analytics play a significant
role in the MOOC phenomenon. Criticisms of MOOCs
regarding their low completion rates, lack of pedagogical
infrastructure, and unreliable assessment methods have led
recent research to focus on students’ learning outcomes in
MOOCs (Mackness & Bell, 2015). Thus, knowing more
about what and how students learn would provide data for
designing ways to address the challenges faced in MOOCs.
As called for by many researchers, the current study aimed to
explore MOOCs by examining factors involved in students’
learning outcomes (Castaño-Garrido et al., 2015; Reich,
2015). Thus, literature on MOOCs was reviewed to identify
students’ characteristics, course features, and learning
processes related to significant learning outcomes. The
selected studies were systematically analysed with respect to
the components of Biggs (2003) 3P model. The students’
characteristics, teaching context, and learning activities relat-
ed to students’ learning outcomes (see Fig. 2.) were synthe-
sized in order to formulate a set of suggestions for designing
significant learning outcomes in MOOCs. Applying the fol-
lowing suggestions for the design of MOOCs might be ben-
eficial to both MOOC providers and instructors:
Ensure that all students with different personal and aca-
demic characteristics are able to follow the course infor-
mation. Conducting need assessment could be helpful to
identify the students’ needs, preferences, and expectations
as a basis for organizing course design. For instance, stu-
dents who have prior experience with online learning
might be more active and ready to participate in open
online courses compared to those who have no or limited
experience.
Course resources and tools should encourage students to
participate. These may include social networking tools,
authentic tasks, project-based assignments, and collabora-
tive projects.
Providing unique features (e.g., authentic e-learning
activities) within the courses increases students’ commit-
ment and participation.
Use peer and self-assessment for formative evaluation in
conjunction with rubrics or other form of guidance to
improve both students’ learning and the accuracy of their
assessments.
Provide clear and structured assessments, and design the
assessments by taking into account the students’ profile
and preferences in order to capture the students’ atten-
tion.
Ensure that feedback is personalized and contextualized
to stimulate students’ participation and engagement.
Facilitate learner-centred communities using group proj-
ects or collaborative study groups to encourage students’
participation and engagement.
Provide opportunities for students to contribute in discus-
sion forums and blogs in order to sustain their motivation
to participate and complete the course.
Ensure that MOOCs are prepared based on a well-struc-
tured instructional design models that include learning
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
55
tasks, quality materials (e.g., videos) and tools, SNSs,
aligned assessments, and personalized learning environ-
ments.
Provide opportunities for students to manage their own
time in order to develop their intrinsic motivation and
commitment to the course.
Ensure that the duration of the course is no longer than 8
weeks; students tend to remain in and complete shorter
MOOCs.
Provide alternatives for students to accredit MOOCs to
increase the retention. There should be an option to
transfer credits from MOOCs into institutional degree
programs.
Foster self-directed learning environments to expand stu-
dents’ autonomy, encourage them to complete their week-
ly assignments, and provide opportunities for students with
limited computer and language skills.
Based on the current review study several conclusions can
be highlighted. Firstly, the MOOC studies reviewed rein-
force the message that proper course design, which considers
students’ individual differences and intentions, may provide a
solution to current problematic issues that make the higher
education committee sceptical of MOOCs. No one denies
the reality that the mounting MOOC phenomenon brings
vital change and development to higher education, but this
innovation must not change the real purpose of providing
effective learning environments. Therefore, the needs and
requirements of those who follow and lead MOOCs should
be fulfilled by MOOC providers to continue their existence
and enhance efficiency. It is further important to note that
these needs and requirements are evolving and changing in
very different patterns compared to traditional education
(DeBoer et al., 2014; Liyanagunawardena et al., 2015).
Secondly, there is widespread agreement that students’
learning outcomes are more difficult to explore and analyse in
open online learning environments than in campus environ-
ments because of the difficulty, discrepancies, and fertility of
data in open online learning environments. More research is
needed to fully comprehend factors related to significant
learning outcomes in MOOCs by conducting research that
goes beyond counting ‘clicks.’
Thirdly, this review study revealed that there are many
MOOCs without sufficient pedagogical infrastructure.
Although teaching and learning practices including instruc-
tional design, teaching materials, and assessment might be
problematic, many students who participated in MOOCs
especially in miniMOOCs expressed a high level of satisfac-
tion (Khalil & Ebner, 2013). Even more surprisingly, this
positive attitude towards MOOCs is not related to course
completion (Mackness & Bell, 2015). Some distance educa-
tion researchers claim that the MOOC phenomenon is just a
fad that will never challenge or alter in-class higher education
and that they are going to lose their popularity in the near
future. Other researchers, however, claim that MOOCs will
continue to provide new insights and opportunities for high-
er education.
MOOCs promote a great opportunity for lifelong learn-
ing (Liyanagunawardena, 2015; Macleod et al., 2015;
Milligan & Littlejohn, 2014; Steffens, 2015). Albeit that stu-
dents differ in reasons why they attend a MOOC (e.g., life-
long learning, personal development or credits), MOOCs
should be developed on the basis of instructional design mod-
els. To this end, the set of implications mentioned above
which were based on empirical findings from the literature,
offers an opportunity to develop open online courses for sig-
nificant students’ learning outcomes.
Implications for Future Work
Although the research literature defines general issues that
could be addressed in research on MOOCs, only a few studies
focused on teaching and learning aspects. More research is
needed on how MOOCS impact students’ learning outcomes
and performance, and their connection with aspects of instruc-
tion and teaching. Finding ways to increase student’ comple-
tion rates would not automatically translate to definitely estab-
lishing the quality of MOOCs. Like in face-to-face education,
passing rates are not always good indicators of students’ mean-
ingful learning. This means that MOOC stakeholders must
develop additional indicators of MOOC quality.
Firstly, we suggest investigating issues related to pedagog-
ical aspects of MOOCs, such as how to align with students’
needs and how various course designs (e.g., personalized
learning, e-activity-based learning, game-based learning, and
project-based learning) impact students’ engagement, satis-
faction, achievement, and retention rates in MOOCs. One of
these pedagogical aspects is feedback. Timely feedback that is
formulated to be “to the point” is positively related to stu-
dents’ meaningful learning and future research could investi-
gate how to incorporate this kind of feedback into MOOCs.
Secondly, we suggest examining alternative assessment
methods that are aligned with learners’ needs and motiva-
tions, and to also assess aspects of performance that are more
relevant for MOOC platforms (e.g., collaboration, openness,
active involvement) compared to traditional learning envi-
ronments.
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
56
Thirdly, it might be useful to examine the differences in
learning outcomes of experienced and novice MOOCers, and
how these differences are related to the learning behaviors they
exhibit during a MOOC. Experience could also be a research
topic in terms of the instructors to examine differences between
experienced and novice MOOC instructors.
Fourthly, further research could focus on testing hypothet-
ical relationships between students’ characteristics, course fea-
tures, learning and teaching activities, and students’ learning
outcomes in MOOCs. This kind of research can support teach-
ers and designers in decisions regarding how to plan MOOC
components.
References
Adams, C., Yin, Y., Madriz, L. F. V., & Mullen, C. S. (2014). A phenomenol-
ogy of learning large: The tutorial sphere of xMOOC video lectures.
Distance Education, 35(2), 202–216.
Admiraal, W., Husiman, B., & Pilli, O. (2015). Assessment in massive open
online courses. Electronic Journal of e-Learning, 13(4): 207–216.
Admiraal, W., Huisman, B., & Van de Ven, M. (2014). Self- and peer
assessment in massive open online courses. International Journal of
Higher Education, 3(3), 119–128.
Ahn, J., Butler, B. S., Alam, A., & Webster, S. A. (2013). Learner participa-
tion and engagement in open online courses: Insights from the Peer 2
Peer University. MERLOT Journal of Online Learning and Teaching,
9(2), 160–171.
Al-Atabi, M., & DeBoer, J. (2014). Teaching entrepreneurship using
massive open online course (MOOC). Technovation, 34(4), 261–264.
Balfour, S. P. (2013). Assessing writing in MOOCs: Automated essay scor-
ing and calibrated peer review. Research & Practice in Assessment, 8,
40–48.
Bali, M., Crawford, M., Jessen, R. L., Signorelli, P., & Zamora, M. (2015).
What makes a cMOOC community endure? Multiple participant per-
spectives from diverse MOOCs. Educational Media International,
doi:10.1080/09523987.2015.1053290
Bayne, S., & Ross, J. (2014). The pedagogy of the Massive Open Online
Course: the UK view. York, UK: The Higher Education Academy.
Biggs, J. (2003). Teaching for quality learning at university (2 ed.).
Berkshire, UK: Open University Press.
Bonk, C. J., Lee, M. M., Kou, X., Xu, S., & Sheu, F.-R. (2015).
Understanding the self-directed online learning preferences, goals,
achievements, and challenges of MIT OpenCourseWare subscribers.
Educational Technology & Society, 18(2), 349–368.
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., &
Seaton, D. T. (2013). Studying learning in the worldwide classroom
research into edX’s First MOOC. Research & Practice in Assessment, 8,
13–25.
Campbell, J., Gibbs, A. L., Najafi, H., & Severinski, C. (2014). A compar-
ison of learner intent and behaviour in live and archived MOOCs. The
International Review of Research in Open and Distance Learning, 15(5),
235–262.
Castaño-Garrido, C., Maiz-Olazabalaga, I., & Garay-Ruiz, U. (2015).
Design, motivation and performance in a cooperative MOOC course.
Comunicar, 22(44), 19–26.
Chang, R. I., Hung, Y. H., & Lin, C. F. (2015). Survey of learning experi-
ences and influence of learning style preferences on user intentions
regarding MOOCs. British Journal of Educational Technology, 46(3),
52819–26541.
Chen, Y.-H., & Chen, P.-J. (2015). MOOC study group: Facilitation strate-
gies, influential factors, and student perceived gains. Computers &
Education, 86, 55–70.
Clarà, M., & Barberà, E. (2014). Three problems with the connectivist con-
ception of learning. Journal of Computer Assisted Learning, 30(3),
197–206.
Clark, D. (2013). MOOCs: Taxonomy of 8 types of MOOC. Accessed through
<http://donaldclarkplanb.blogspot.co.uk/search?q=MOOCs: +taxono-
my> on March 24th, 2015.
Comer, D. K., Clark, C. R., & Canelas, D. A. (2014). Writing to learn and
learning to write across the disciplines: Peer-to-peer writing in intro-
ductory-level MOOCs. The International Review of Research in Open and
Distance Learning, 15(5), 26–82.
Daza, V., Makriyannis, N., & Rovira Riera, C. (2014). MOOC attack:
Closing the gap between pre-university and university mathematics.
Open Learning: The Journal of Open, Distance and e-Learning, 28(3),
227–238.
de Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform
learning and teaching in higher education? Engagement and course
retention in online learning provision. British Journal of Educational
Technology, 46(3), 455–471.
DeBoer, J., Ho, A. D., Stump, G. S., & Breslow, L. (2014). Changing
“Course”: Reconceptualizing educational variables for massive open
online courses. Educational Researcher, 43(2), 74–84.
Dillahunt, T. R., Wang, B. Z., & Teasley, S. (2014). Democratizing high-
er education: Exploring MOOC use among those who cannot afford a
formal education. The International Review of Research in Open and
Distance Learning, 15(5), 177–196.
Diver, P., & Martinez, I. (2015). MOOCs as a massive research laboratory:
Opportunities and challenges. Distance Education, 1-21.
Ebben, M., & Murphy, J. S. (2014). Unpacking MOOC scholarly dis-
course: a review of nascent MOOC scholarship. Learning Media and
Technology, 39(3), 328–345.
EDUCAUSE. 7 things you should know about MOOCs. (2011). Accessed
through <http://net.educause.edu/ir/library/pdf/ ELI7078.pdf/> on
February 13th, 2015.
Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher
education: A meta-analysis comparing peer and teacher marks. Review
of Educational Research, 70(3), 287–322.
Fasihuddin, H. A., Skinner, G. D., & Athauda, R. I. (2013). Boosting the
opportunities of open learning (MOOCs) through learning theories.
GSTF Journal on Computing, 3(3), 112–117.
Firmin, R., Schiorring, E., Whitmer, J., Willett, T., Collins, E. D., &
Sujitparapitaya, S. (2014). Case study: Using MOOCs for convention-
al college coursework. Distance Education, 35(2), 178–201.
Fini, A. (2009). The technological dimension of a massive open online
course: The case of the CCK08 course tools. The International Review of
Research in Open and Distance Learning, 10(5).
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
57
Fischer, G. (2014). Beyond hype and underestimation: Identifying research
challenges for the future of MOOCs. Distance Education, 35(2),
149–158.
Fisher, D. H. (2014). Educational advances in artificial intelligence leverag-
ing AI teaching in the cloud for AI teaching on campus. Ai Magazine,
35, 98–100.
Forsey, M., Low, M., & Glance, D. (2013). Flipping the sociology class-
room: Towards a practice of online pedagogy. Journal of Sociology, 49
(4), 471–485.
Fournier, H., Kop, R., & Durand, G. (2014). Challenges to research in
MOOCs. MERLOT Journal of Online Learning and Teaching, 10(1),
1–15.
Gasevic, D., Kovanovic, V., Joksimovic, S., & Siemens, G. (2014). Where
is research on massive open online courses headed? A data analysis of
the MOOC research initiative. The International Review of Research in
Open and Distance Learning, 15(5), 134–176.
Giannakos, M., Jaccheri, L., & Krogstie, J. (2014). Looking at MOOCs
rapid growth through the lens of video-based learning research.
International Journal of Emerging Technologies in Learning (IJET), 9(1),
35–38.
Gillani, N., & Eynon, R. (2014). Communication patterns in massively
open online courses. The Internet and Higher Education, 23, 18–26.
Glance, D. G., Forsey, M., & Riley, M. (2013). The pedagogical founda-
tions of massive open online courses. First Monday, 18(5).
Greene, J. A., Oswald, C. A., & Pomerantz, J. (2015). Predictors of reten-
tion and achievement in a Massive Open Online Course. American
Educational Research Journal, 52(5), 925–955.
Han, J. H. (2014). Closing the missing links and opening the relationships
among the factors: A literature review on the use of clicker technolo-
gy using the 3P model. Educational Technology & Society, 17(4),
150–168.
Hernández-Carranza, E. E., Romero-Corella, S. I., & Ramírez-Montoya,
M. S. (2015). Evaluation of digital didactic skills in massive open online
courses: A contribution to the latin american movement. Comunicar,
XXII(44), 81–89.
Hew, K. F., & Cheung, W. S. (2014). Students’ and instructors’ use of mas-
sive open online courses (MOOCs): Motivations and challenges.
Educational Research Review, 12, 45–58.
Ho, A. D., Reich, B. J. F., Nesterko, S. O., Seaton, D. T., Mullaney, T. P.,
Waldo, J. H., & Chuang, I. (2014). HarvardX and MITx: The first year
of open online courses, Fall 2012-Summer 2013. Working Paper, 1,
1–33.
Hoy, M. B. (2014). MOOCs 101: An introduction to massive open online
courses. Medical Reference Services Quarterly, 33(1), 85–91.
Jiang, S., Williams, A. E., Warschauer, M., He, W., & O’Dowd, D. K.
(2014). Influence of incentives on performance in a pre-college biology
MOOC. The International Review of Research in Open and Distance
Learning, 15(5), 99–112.
Jordan, K. (2014). Initial trends in enrolment and completion of massive
open online courses. The International Review of Research in Open and
Distance Learning, 15(1), 133–160.
Jordan, K. (2015). Massive open online course completion rates revisited:
Assessment, length and attrition. The International Review of Research in
Open and Distributed Learning, 16(3), 341–358.
Katuk, N., & Kim, J.H. (2013). Experience beyond knowledge: Pragmatic
e-learning systems design with learning experience. Computer Human
Behaviour, 29, 747–758.
Khine, M. S., & Lourdusamy, A. (2003). Blended learning approach in
teacher education: Combining face-to-face instruction, multimedia
viewing and on-line discussion. British Journal of Educational Technology,
34(5), 671–675.
Khalil, H., & Ebner, M. (2013). “How satisfied are you with your MOOC?”
- A research study on interaction in huge online courses. Paper presented at
the Proceedings of World Conference on Educational Multimedia,
Hypermedia and Telecommunications 2013, Chesapeake, VA, USA.
Kizilcec, R. F., & Schneider, E. (2015). Motivation as a lens to understand
online learners: Towards data-driven design with the OLEI scale. ACM
Transactions on Computer-Human Interaction, 22(2), doi:10.1145/2699735
Kizilcec, R. F., Bailenson, J. N., & Gomez, C. J. (2015). The instructor’s
face in video instruction: Evidence from two large-scale field studies.
Journal of Educational Psychology, 107(3), 724–739.
Klimova, B. F., & Poulova, P. (2013). Impact of a form of online materials
on the quality of education: A case study. International Journal of Digital
Information and Wireless Communications (IJDIWC), 3(1), 43–49.
Konstan, J. A., Walker, J. D., Brooks, D. C., Brown, K., & Ekstrand, M. D.
(2015). Teaching recommender systems at large scale: Evaluation and
lessons learned from a Hybrid MOOC. ACM Transactions on Computer-
Human Interaction, 22(2), doi:10.1145/2728171
Kop, R. (2011). The challenges to connectivist learning on open online net-
works: learning experiences during a massive open online course. The
International Review of Research in Open and Distance Learning and
Instruction, 12(3), 19–38.
Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or
a pedagogy to support human beings? Participant support on massive
open online courses. The International Review of Research in Open and
Distance Learning, 12(7), 74–93.
Koutropoulos, A., & Zaharias, P. (2015). Down the rabbit hole: An initial
typology of issues around the development of MOOCs. Current Issues
in Emerging eLearning, 2(1), Article 4.
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J.,
Koller, D., & Klemmer, S. R. (2013). Peer and self assessment in mas-
sive online classes. ACM Transactions on Computer-Human Interaction,
20(6), 1–31.
Li, N., Himanshu, V., Skevi, A., Zufferey, G., Blom, J., & Dillenbourg, P.
(2014). Watching MOOCs together: Investigating co-located MOOC
study groups. Distance Education, 35(2014), 217–233.
Liang, D., Jia, J., Wu, X., Miao, J., & Wang, A. (2014). Analysis of learn-
ers’ behaviors and learning outcomes in a massive open online course.
Knowledge Management & E-Learning, 6(3), 281–291.
Liu, M., Kang, J., Cao, M., Lim, M., Ko, Y., Myers, R., & Schmitz Weiss,
A. (2014). Understanding MOOCs as an emerging online learning
tool: perspectives from the students. American Journal of Distance
Education, 28(3), 147–159.
Liyanagunawardena, T. R. (2015). Massive open online courses.
Humanities, 4(1), 35–41.
Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2014).
MOOCs: A systematic study of the published literature 2008–2012. The
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
58
International Review of Research in Open and Distance Learning, 14(3),
202–227.
Liyanagunawardena, T. R., Lundqvist, K. Ø., & Williams, S. A. (2015).
Who are with us: MOOC learners on a FutureLearn course. British
Journal of Educational Technology, 46(3), 557–569.
Mackness, J., Waite, M., Roberts, G. & Lovegrove, E. (2013). Learning in
a small, task-oriented, connectivist MOOC: Pedagogical issues and
implications for higher education. The International Review of Research in
Open and Distributed Learning, 14(4), 140–159.
Mackness, J., & Bell, F. (2015). Rhizo14: A rhizomatic learning cMOOC
in sunlight and in shade. Open Praxis, 7(1), 25–38.
Macleod, H., Haywood, J., Woodgate, A., & Alkhatnai, M. (2015).
Emerging patterns in MOOCs: Learners, course designs and direc-
tions. TechTrends: Linking Research & Practice to Improve Learning, 59(1),
56–63.
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of
massive open online courses (MOOCs). Computers & Education, 80,
77–83.
Miller, S. L. (2015). Teaching an online pedagogy MOOC. MERLOT
Journal of Online Learning and Teaching, 11(1), 104–119.
Milligan, C., & Littlejohn, A. (2014). Supporting professional learning in a
massive open online course. The International Review of Research in Open
and Distance Learning, 15(5), 197–213.
Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of engage-
ment in connectivist MOOCs. MERLOT Journal of Online Learning and
Teaching, 9(2), 149–159.
Noroozi, O., Weinberger, A., Biemans, H. J. A., Mulder, M., & Chizari,
M. (2012). Argumentation-based computer supported collaborative
learning (ABCSCL): A synthesis of 15 years of research. Educational
Research Review, 7(2), 79–106.
Perna, L. W., Ruby, A., Boruch, R. F., Wang, N., Scull, J., Ahmad, S., &
Evans, C. (2014). Moving through MOOCs: Understanding the pro-
gression of users in massive open online courses. Educational Researcher,
43(9), 421–432.
Redfield, R. J. (2015). Putting my money where my mouth is: The useful
genetics project. Trends in Genetics, 31(4), 195–200.
Reich, J. (2015). Rebooting MOOC research. Science, 347(6217), 34–35.
Reilly, E. D., Stafford, R. E., Williams, K. M., & Corliss, S. B. (2014).
Evaluating the validity and applicability of automated essay scoring in
two massive open online courses. The International Review of Research in
Open and Distance Learning, 15(5), 83–98.
Rodriguez, C. O. (2012). MOOCs and the AI-Stanford like courses: two suc-
cessful and distinct course formats for massive open online courses. The
European Journal of Open, Distance and E-Learning, 2012/II. Accessed
through <http://www.eurodl.org/?p=archives&year=2012&halfyear=2&
abstract=516> on November 12th, 2014.
Saadatmand, M., & Kumpulainen, K. (2014). Participants’ perceptions of
learning and networking in connectivist MOOCs. MERLOT Journal of
Online Learning and Teaching, 10(1), 16–30.
Seaton, D. T., Bergner, Y., Chuang, I., Mitros, P., & Pritchard, D. E.
(2014). Who does what in a massive open online course?
Communications of the ACM, 57(4), 58–65.
Spector, J. M. (2014). Remarks on MOOCS and Mini-MOOCS.
Educational Technology Research and Development, 62(3), 385–392.
Spelt, E. H., Biemans, H. A., Tobi, H., Luning, P., & Mulder, M. (2009).
Teaching and learning in interdisciplinary higher education: A system-
atic review. Educational Psychology Review, 21(4), 365–378.
Stefanic, N. M. (2014). Creativity-based music learning: Modeling the process
and learning outcomes in a massive open online course. Unpublished PhD
thesis, (3631459 PhD), University of South Florida, Ann Arbor, MI,
USA. Accessed through <http://search.proquest.com/docview/
1564785347?accountid=12045> on April 16th, 2015.
Steffens, K. (2015). Competences, learning theories and MOOCs: Recent
developments in lifelong learning. European Journal of Education, 50(1),
41–59.
Thille, C., Schneider, E., Kizilcec, R. F., Piech, C., Halawa, S. A., &
Greene, D. K. (2014). The future of data-enriched assessment. Research
& Practice in Assessment, 9(2), 5–12.
Toven-Lindsey, B., Rhoads, R. A., & Lozano, J. B. (2015). Virtually unlim-
ited classrooms: Pedagogical practices in massive open online courses.
The Internet and Higher Education, 24, 1–12.
Trumbore, A. (2014). Rules of engagement: Strategies to increase online
engagement at scale. Change: The Magazine of Higher Learning, 46(4),
38–45.
Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging deeper into
learners’ experiences in MOOCs: Participation in social networks out-
side of MOOCs, notetaking and contexts surrounding content con-
sumption. British Journal of Educational Technology, 46(3), 570–587.
Vista, A., Care, E., & Griffin, P. (2015). A new approach towards marking
large-scale complex assessments: Developing a distributed marking sys-
tem that uses an automatically scaffolding and rubric-targeted interface
for guided peer-review. Assessing Writing, 24, 1–15.
Vivian, R., Falkner, K., & Falkner, N. (2014). Addressing the challenges of
a new digital technologies curriculum: MOOCs as a scalable solution
for teacher professional development. Research in Learning Technology,
22, 24691.
Wasson, C. (2013). “It was like a little community”: An ethnographic study
of online learning and its implications for MOOCs. Ethnographic Praxis
in Industry Conference Proceedings, 2013(1), 186–199.
Wen, M., & Rose, C. P. (2014). Identifying latent study habits by mining learn-
er behavior patterns in massive open online courses. Paper presented at the
23rd ACM Conference on Information and Knowledge Management
(CIKM’14), November 3–7, 2014, Shanghai, China.
Yousef, A. M. F., Chatti, M. A., Wosnitza, M., & Schroeder, U. (2015). A
cluster analysis of MOOC stakeholder perspectives. RUSC, Universities
and Knowledge Society Journal, 12(1), 74–90.
Yang, D., Wen, M., Kumar, A., Xing, E. P., & Rose, C. P. (2014). Towards
an integration of text and graph clustering methods as a lens for study-
ing social interaction in MOOCs. The International Review of Research in
Open and Distance Learning, 15(5), 214–234.
Yuan, L., Powell, S., & Olivier, B. (2014). Beyond MOOCs: Sustainable
online learning in institutions. Lancaster, UK: Centre for Educational
Technology, Interoperability and Standards (CETIS LLP). Accessed
through <http://publications.cetis.ac.uk/2014/898> on June 6th,
2015.
Zutshi, S., O’Hare, S., & Rodafinos, A. (2013). Experiences in MOOCs:
The perspective of students. American Journal of Distance Education,
27(4), 218–227.
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
59
Appendix I. Summary of reviewed studies.
No
1
2
3
4
5
Adams, Yin,
Madriz, & Mullen
(2014)
Admiraal,
Huisman, & Van
de Ven (2014)
Ahn, Butler, Alam,
& Webster (2013)
Al-Atabi &
DeBoer (2014)
Bali et al. (2015)
Examine the
students’ accounts
of their everyday
experiences of
learning in
MOOCs.
Examine the
quality of self- and
peer assessments
in three MOOCs.
Understand the
notions of learner
participation and
engagement in
open online
courses.
Explore the
students’ learning
achievement
outcomes in
entrepreneurship
course as a
MOOC.
To provide
insight into the
thrill and depth
of learning and
connection
possible through
participation
in cMOOCs.
What are
completers’
experiences of
learning in an
xMOOC?
1. What is the
reliability of
self- and peer
assessment
implemented in
MOOCs?
2. What is the
relationship
between self-
and peer assess-
ment and
quizzes?
3. To what extent
do self- and
peer assessment
and quizzes
explain
differences in
students’ final
exams scores?
How have learners
participated and
engaged with
open online
learning in P2PU?
What is the
effectiveness of
using the MOOC
to teach
entrepreneurship?
Not provided
Qualitative
research
methodology,
phenomenology
of practice
Case study
Case study
Case study/
Survey
Collaborative
autoethnography
4 current
participants in
MOOC and 6
MOOC completers
Students from
three MOOCs from
Leiden University:
The Law of the
European Union:
An Introduction
and Terrorism and
Counterterrorism:
Comparing theory
and practice
Dataset from
entire history of
P2PU
80 students
Five cMOOCers
Intimacy developed
for xMOOC
instructor, most
especially in the
context of the
pre-recorded
instructional videos
No significant
correlation
between the
students’ peer and
self- assessment,
self-assessments
might not be a
valid way to assess
students’
performance in
MOOCs
Participation and
engagement take
on varied forms in
cMOOCs
MOOC is a
suitable platform
to students'
collaborative
learning,
opportunity
recognition and
resource
acquisition.
MOOCs are the
mentalequivalent
of working on
a 3D puzzle
that is in
constant
motion rather
than a static
workbook.
Need to explore
the everyday
activities of
xMOOC students
The reliability of
self- and peer-
assessment should
be conducted in
more learner-
centred MOOCSs
pedagogy. Need
to have new
metric to consider
the relationship
between self- and
peer assessment.
Examine
longitudinal
patterns of the
learning tasks of
P2PU members,
the relationship
between factors
such as course
design, social
interaction,
dropout and
retention
Not provided
Not provided
Using video in
xMOOC can be
helpful to develop
engagement both
to the course and
instructor.
Self-assessments
and peer
assessments
should be
improved if they
are used as
summative
indicators of one’s
achievements
(assessment of
learning).
Course,
assessment, and
learning
environment must
be developed
considering the
engagement and
participation
patterns.
Brain rewiring
exercises is a good
way to initiate
discussions among
students.
MOOCs are a
good way to
develop or hone
digital literacies.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
60
Appendix I. [Continued] Summary of reviewed studies.
No
6
7
8
9
Bonk et al. (2015)
Breslow et al.
(2013)
Campbell,
Gibbs, Najafi, &
Severinski (2014)
Castaño-Garrido,
Maiz-Olazabalaga,
& Garay-Ruiz
(2015)
To explore the
self-directed and
informal learning
experiences of
subscribers to the
monthly MIT
OpenCourseWare
(OCW) online
newsletter.
Understand the
modes of learning
which work well
under the specific
situations of
6,002x MOOC?
Explore and
compare the
demographics,
intent, and
behavior differ-
ences of live-and
archived-learners.
Focus on the
pedagogical
design of a
cooperative
MOOC and its
influence on
motivation and
academic results
Not provided
Who the students
were in 6.002x?,
how they utilized
course resources?,
what contributed to
their persistence?,
and what advanced
or hindered their
achievement?
Are there any
potential and
purpose for
archived MOOCs
to be used as
learning resources
beyond and
between
instructor-led
live-sessions?
1. Is there a
relationship
between
academic
performance and
the pedagogical
design of the
course?
2. Is there a
relationship
between student
motivation and
the pedagogical
design of the
course?
3. Is there a
relationship
between
academic
performance and
student
motivation?
Survey
Case study
Survey, activities
and clickstream
of all learner
actions
Survey, analyzing
of e-activities
1429 people
Data set of
155,000 people
who enrolled
6.002x
Learners in the
live- and
archived-sessions
of “Statistics:
Making Sense of
Data” (STATS) and
“Learn to Program:
The Fundamentals”
(LPT1)
186 participants of
744 students who
enrolled on the
MOOC
Motivational factors
are curiosity, interest,
and internal need for
self-improvement.
Success or personal
change factors
included freedom to
learn, resource
abundance, choice,
control, and fun.
In terms of
achievements,
respondents were
learning both
specific skills as well
as more general skills
that help them
advance in their
careers.
The certificate
earners used the
forum at a much
higher rate than
other students. High
achievers are
studying offline with
another person.
Positive relationship
between highest
degree earned
and achievement
The archived-
learners interact
with the course in
much the same
way as live-learners.
Learners follow the
course sequentially
who intended to
complete all
required work.
Course design
(cooperative and
social network
learning) influences
students'
performance.
No global
significance
between motivation
and performance,
but positive
relationship
between satisfaction
and performance
Not provided
How different
representations of
complex concepts
and phenomena
(textual, graphical,
mathematical) can
best be used to
help students
master them?
Exploration of the
pace at which
archived-learners
access videos
and complete
assessments may
have valuable
implications for
course design.
How is the
relationship
between academic
performance and
students’ learning
experiences?
Educators,
instructional
designers, and
online learning tool
and resource
designers need to
embed a sense of
choice and control
when creating or
enhancing OCW,
OER, and MOOCs.
Collaborating with
another person,
whether novice or
expert, strengthens
learning
MOOCs to be
beneficial as
self-study courses,
students in
archived-MOOCs
have potentials
to fulfill students’
needs.
Increase the level
of student
satisfaction can
help to cut
dropout rates.
Course design
should be
integrated with
students’
motivational
factors.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
61
Appendix I. [Continued] Summary of reviewed studies.
No
10
11
12
13
Chen & Chen
(2015)
Chang et al.
(2015)
Comer, Clark, &
Canelas (2014)
Daza, Makriyannis,
& Rovira Riera
(2014)
Investigate the
effectiveness and
sustainability of
face-to-face study
groups for
MOOCs.
Explore whether
learning styles can
influence the use
of MOOCs and
determine the
learning style
related to use
intentions.
Evaluate how
peer-to-peer
interactions
through writing
impact student
learning in
introductory-level
MOOCS.
Create a useful
tool for students
that enter
university.
RQ1: What are
MOOC students’
perceived gains
from the face-to-
face study group?
RQ2: What are the
key factors that
influence the
dynamic/
effectiveness of the
MOOC study
group? RQ3:
What are MOOC
students’
suggestions to
improve the face-to-
face study group?
Not provided
1. How do peer-to-
peer interactions
through writing
impact student
learning in
introductory-level
writing and
chemistry
MOOCs?
2. What is the
impact of peer-to-
peer writing on
engaging
students in MOOC
coursework who
identify as less
academically-
prepared and less
self-motivated?
3. How can peer-to-
peer writing
function as a
metric to assess
student success in
MOOC delivered
introductory
writing and
science
coursework?
Not provided
Interpretive case
study approach
Survey
Qualitative
coding analysis
Survey
Four audiences
who attended a
2-h guest speech
entitled “From
OCW to MOOCS:
Implication for
college students”
184 undergraduate
students
English
Composition I:
Achieving Expertise
and Introduction to
Chemistry
194 students
Cognitively,
participants
broadened their
perspective of
thinking, raised
cultural awareness,
and shared many
learning strategies
Innovative learning
styles influence
the learning
experience in
MOOCs.
Peer-to-peer
interactions in
writing through
the forums and
through peer
assessment
enhance learner
understanding, link
to course learning
objectives, and
generally
contribute
positively to the
learning
environment.
Not provided
To explore study
group dynamics,
perceived gains
and challenges,
and key influential
factors across
culture.
Increase the
sample size and
include more
important personal
traits to enhance
the effect of
personalized
learning in MOOCs.
Not provided
To collect more
data at the end of
the academic year.
We would like to
know whether this
year’s students will
be more successful
with linear algebra
compared with
previous years.
Study group may
serve as an ideal
approach to help
MOOC learners
develop requisite
skills, share feelings
and thoughts, and
strengthen their
self-determination
to continue.
Collaborative, game-
based and query-
based learning
should be taken into
account for
increasing the quality
and enrolment of
MOOCs.
The peer-feedback
process, discussion
forums, and
writing through
the forums con-
tribute to students’
learning, especially
in understanding.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
62
Appendix I. [Continued] Summary of reviewed studies.
No
14
15
16
17
DeBoer et al.
(2014)
de Freitas et al.
(2015)
Dillahunt, Wang, &
Teasley (2014)
Diver & Martinez
(2015)
Propose the
redefine the some
conventional
variables: enrollment,
participation,
curriculum, and
achievement to be
useful for
description and
evaluation of the
educational
experience in
MOOCs.
Explore how
course retention
can be improved
in online provision.
Compare the
MOOC learners
who self-identified
as being unable to
afford to pursue a
formal education
(the target group)
with other learners
(the comparison
group) in terms of
demographic data
and motivation.
Investigate
dropout rates and
how students who
decide to drop out
differ from those
who continue
courses.
Not provided
1.How did
students view
the course after
completing it?
2.What were the
main retention
patterns of
students?
3.How did student
activity relate to
final grade?
4.Did the quality
of the MOOC
and its links to
Astronomy
degree level
education have
an impact upon
retention?
5. Did the role of
“gamified”
elements have
an impact upon
engagement
and retention?
How MOOCs
might better serve
those who feel
financially unable
to pursue a more
traditional path
to post-secondary
education studies?
Not provided
Data analyzing
Case study
Survey
Data analysis
Data from the first
MOOC offered by
MIT, “Circuits and
Electronics”
Students in
Astronomy MOOC
Six Coursera
MOOCs related
with Humanities,
Economics and
Finance, and
Technology,
N=3812
MOOC data from
two Coursera
Reconceptualization
of conventional
variables in terms
of individualized
and informed user
intentions
Higher levels of
engagement,
creativity and
experimentation
can be used to
decrease the of
high dropout
rates.
There were no
significant
differences between
the two groups’
engagement in
terms of watching
videos, accessing
course materials
and/or conducting
assessments. The
comparison group
had a higher
percentage of
course completion
but the
achievement level is
significantly high in
target group.
Procrastination, as
measured by
delays in taking
quizzes is negatively
correlated with
achievement on
quizzes.
Analysis and
mining of log files
to identify the
actions users are
taking, may also
surface the
intentions of the
students.
Not provided
Whether and how
MOOC platforms
can capture more
detailed
information about
learners during
their activity and
engagement in
the courses
Determine how
students change
behavior when
they are offered
certificates of
different values.
Not provided
Experience
presenting video
and audio
materials,
activities including
interactive media,
quizzes and
assignments,
and for social
interactions
should be in
MOOCs.
Students who have
educational
affordability are
more likely to earn
a certificate with
distinction than
those who enrolled
in the MOOC for
reasons other
reasons.
Establishing or
altering deadlines
or sending
students almost
costless emails to
reduce
procrastination
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
63
Appendix I. [Continued] Summary of reviewed studies.
No
18
19
20
21
22
Fini (2009)
Firmin et al.
(2014)
Forsey, Low, &
Glance (2013)
Fournier, Kop, &
Durand (2014)
Gillani & Eynon
(2014)
Investigate lifelong
learners attitudes
towards learning
network
technologies.
Identify reasons
for students’
achievement and
to discover
patterns to inform
future online
course planning.
Critically
re-examine
pedagogy and
practice in the
sociology
classroom
Explore the
cPLENK MOOC,
and to highlight
the challenges in
the research and
analyze process.
Understand the
ways that learners
from around the
world interact in
MOOCs.
What are the
learners’ views
about the multi-tool
environments in
CCK08 cMOOC?
1. Who engaged
and who did not
engage in a sus-
tained way and
who passed or
failed in the
remedial and
introductory
AOLE courses?
2. What student
characteristics and
use of online
material and
support services
are associated
with success?
3. What do key
stakeholders
(students, faculty,
online support
services,
coordinators, and
leaders) tell us
they have learned
from the AOLE
experiment?
How is the
pedagogy and
practice in sociology
flipped classroom
with MOOC?
What are the
learning and
activity levels in an
open learning
environment ?
1. What are the
demographic
characteristics of
students that
participate in
MOOC discussion
forums?
2. What are the
discussion
patterns that
characterize their
interactions?
3. How does
participation in
discussion forums
relate to students’
final scores?
Survey
Case study
Survey and
focus group
interviews
Qualitative and
quantitative-
survey method
Case study,
survey
Eighty-three
students in
cMOOC (49 males,
34 females)
213 students in
SJSU Plus by
Udacity
74 completed
surveys
PLENK participants
87,000 individuals
from one MOOC
Various opinions
about the tools
related with needs,
purposes and self-
organization skills.
Social networks
that were external
to the course was
perceived as
unnecessary.
Student effort
was the strongest
success indicator,
suggesting
criticality of early
and consistent
student
engagement.
No statistically
significant
relationships with
student
characteristics
(age and gender)
were found.
AOLE support
effectiveness was
compromised
with staff time
consumed by the
least prepared
students.
Students were
experienced an
increase in the
amount of learn-
ing time, materials
were clear and
well-structured in
MOOC platform.
Motivation, past
experienced and
reflective are three
factors to self-
directed learning.
“Freedom to do
and read as I felt
like” and “how
the course
organized” are the
two main factors
on learning.
Forum participants
tend to be well-
educated adults
from the Western
world. Engaged
students in the
discussion forums
are often higher-
performing than
those that do not,
although the huge
majority of forum
participants receive
“failing” marks.
Issues related to
sustainability and
the workload of
instructors should
be studied.
How some factors
(e.g. early support,
high degree of
structure and use
of learning
analytics) are
related with
students’
engagement and
persistence in
MOOCs.
Not provided
Usefulness of the
tools that help
students to
develop self-
directed learning
How MOOC
participants use
the discussion
forums? How
learning occurs in
MOOC using both
qualitative and
quantitative
methods?
Use of unique tools
and/or activities
such as The Daily
can be a useful
tool for learning.
Matriculation of
students and soft-
ware development
for early warning
to students can
increase the
achievement,
contextualization
of MOOCs
provides more
advantage for
both engagement
and authentic
learning.
MOOC can be an
good alternative
for blended
learning.
The importance of
human factors
such as motivation,
incentives, support
in creating high-
quality learning
experiences in
MOOC
Higher-performing
students use the
discussion forums,
but they do not
only interact with
other higher-
performing
students.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
64
Appendix I. [Continued] Summary of reviewed studies.
No
23
24
25
26
Greene, Oswald, &
Pomerantz (2015)
Hernández-
Carranza, Romero-
Corella, & Ramírez-
Montoya (2015)
Hew & Cheung
(2015)
Ho et al. (2014)
Examine which
student
characteristics,
relevance, prior
experience with
MOOCs,
self-reported
commitment, and
learners’ implicit
theory of
intelligence
predicted retention
and achievement.
Present an
evaluation of
digital teaching
skills in a project
funded by the
National Distance
Education System
(SINED) in Mexico
conducted on a
Massive Open
Online Course
(MOOC) which
was designed
to develop
competences in
teachers in the
distance learning
or classroom
setting for the
integration of
open educational
resources (OER).
Propose a model
of engaging
students in online
learning courses,
based on six major
instructional design
elements.
Describe the
registrant and
course data
provided by
edX in the
context of the
diverse efforts
and intentions of
HarvardX and
MITx instructor
teams.
Not provided
Not provided
1. What are
the main
recommendations
offered by
professional
councils for
designing an
online course?
2. What specific
instructional
design factors
related to highly
rated MOOC
may have
engaged students
to complete an
online course?
Not provided
Survey
Qualitative and
quantitative
approaches
Content analysis
Data analysis
‘Metadata:
Organization and
Discovering
Information’
MOOC
1126 students
from 11 Latin
American
countries, Spain
and Portugal and
58 MOOC
teachers officially
enrolled on the
course
910 participants’
comments on two
most highly-rated
MOOCs
17 courses from
the first year of
HarvardX and
MITx
Learners’ expected
investment,
including level of
commitment,
expected number
of hours devoted
to the MOOC, and
intention to obtain
a certificate,
related to
retention. Prior
level of schooling
and expected
hours devoted to
the MOOC
predicted
achievement
MOOC participants
were able to
develop digital
teaching skills,
identify how to
use OER and how
the training
process occurs
in the open
education
movement.
Instructional design
elements can play
major role in
engaging online
students.
Course
certification rates
are misleading.
New metrics, far
beyond grades
and course
certification,
are necessary to
capture the
diverse usage
patterns in the
data.
Finding out why
MOOC droppers
did not persist,
how learners do
and do not interact
in discussion
forums and
synchronous
meetings
Analysing the
contributions of
MOOCs, the open
education
movement and
the development
of digital
competences for
education
Analyzing the
viewpoints of other
silent participants
and investigate
what factors may
predict student
learning outcomes.
Future research
designs including
pretesting and
experiments
should focus on
what and how
registrants are
learning.
Provide specific
intervention for
students with
low commitment
or intentions to
ensure the
retention and
success
MOOC virtual
learning scenarios
are highly suitable
for the design
and use of OER
to develop
digital didactic
competences.
Not provided
High-quality
and scalable
assessments to
understand
what and how
registrants are
learning
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
65
Appendix I. [Continued] Summary of reviewed studies.
No
27
28
29
30
Jiang, Williams,
Warschauer, He, &
O’Dowd (2014)
Jordan (2014)
Jordan (2015)
Kizilcec, Bailenson,
& Gomez (2015)
Explore the
factors influencing
enrollment and
completion in
a pre-college
preparatory
MOOC.
Explore factors
affecting
enrolment and
completion.
Extend a previous
study on initial
trends in MOOC
completion rate
(Jordan, 2014).
Examine the
behavior and
attitudes of adult
learners in MOOC
under the different
presentation styles
(Video watching
with instructor
face or w/o
instructor face).
1. How did UC Irvine
(UCI) Bio 93
students perform
in the MOOC
compared to the
general population
students?
2. Among UCI Bio 93
students, were
underprepared
students more
likely to enroll in
the MOOC given
an explicit
incentive?
3. Among UCI Bio
93 students, were
underprepared
students more
likely to complete
the course?
Can we learn
anything about
factors which might
affect enrolment
numbers and
completion rates?
Not provided
1. Is cognitive load
higher in the
strategic or the
constant
condition?
2.Is there a learning
period in the
strategic condition?
3. Is social presence
higher in the
strategic or the
constant
condition?
4. Are learning
outcomes higher
in the strategic
or the constant
condition?
5. Is attrition lower in
the strategic or the
constant condition?
6. Do individual
differences in
learning preference
moderate the
effect of the
strategic
presentation
relative to the
constant
presentation of
the face on
(a) cognitive load
and (b) attrition
Descriptive
assessment
and logistic
regression
model
Linear
regression of
the data from
internet
searches and
crowdsourcing
information
Multiple
regression
analysis of
factors that
affects
completion
rates
Longitudinal
field experiment
382 students from
Pre-College Biology
MOOC
279 MOOCs
221 MOOCs
11% response
rate out of 44,432
learners
Two groups of UCI
students had a
much higher
percentage of
completion and
Distinction
compared to
non-UCI group.
Weak math UCI
students (n=156)
more earned a
Distinction certifi-
cation (39%) than
a Normal
certificate (30%).
Enrolment numbers
are decreasing over
time and are
positively correlated
with course length.
Completion rates
are consistent across
time, university rank,
and total enrolment,
but negatively
correlated with
course length.
Factors that
significantly
predicted
completion rate
included start date,
course length and
assessment type
No significant main
effects of the
strategic relative to
the constant
presentation were
found on attrition,
social presence,
and recall and
transfer learning.
Impact of the
MOOC for
students’ academic
performance in the
onsite Bio 93
course
Whether the
underlying
pedagogy of
MOOCs
(transmissive or
connectivist)
influential on the
enrolment and
completion rates
or not.
Educator should
carefully consider
whether to use this
as an assessment
mechanism, or
whether automated
assessments would
meet their
educational goals.
The potential
benefits and costs
of showing the
instructor’s face in
multimedia
learning in a more
controlled setting
MOOC can
provide alternative
learning
environment for
students who have
disadvantages such
as weak math
background.
The relationship
between
completion rate
and course length
is critical issue for
course designers.
To examine the
effects in practice
Learners’ verbal
or nonverbal
preferences are
important factor
to consider before
utilizing instructor
face on videos.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
66
Appendix I. [Continued] Summary of reviewed studies.
No
31
32
33
34
35
Kizilcec &
Schneider (2015)
Konstan et al.
(2015)
Kop (2011)
Kop, Fournier, &
Mak (2011)
Kulkarni et al.
(2013)
Examine the how
learners’ initial
motivations shape
sub-sequent
actions in MOOCs.
Experiment with
MOOC based
hybrid
instruction
Examine the
levels of learner
autonomy,
presence, and
critical literacies
required in active
connectivist
learning.
Examine how
emergent
technologies
might influence
the design of the
learning
environment.
Use peer
assessment over
two iterations in
the first large-scale
class and to
improve
assessment
accuracy and
encourage
qualitative
feedback.
1. What motivates
learners in
MOOCs?
2. How pronounced
are individual
differences in
motivations
between demo-
graphics or across
courses?
3. Which
motivations are
predictive of
behaviors in
MOOCs and
how predictive
are they?
1. Do students learn
in this MOOC?
2. Do students in a
face-to-face
recommender
systems course,
who have access
to MOOC
resources, learn
more than a
comparable
group of MOOC
students who
have access to
recorded face-to-
face instructional
sessions?
Whether the four
activities highlighted
as being crucial
to learning
(aggregating,
relating, creating,
and sharing) were
actually as
important as
envisaged by the
course planners
What are the
roles of educators
and learners in
creating learning
experiences on
online networked
learning
environments?
How to improve
the assessment
accuracy with
peer-assessment?
Survey
Single-group
cross-sectional
and pretest-
posttest
nonequivalent
groups design
Mixed-method
approach:
Surveys,
observations,
discourse
analysis, and
learning
analytics
Survey and
virtual
ethnography
Case study
71,475 from 14
MOOCs.
39 students in
the face-to-face
section and 4844
online-section of
CSci 5980:
Recommender
Systems hybrid
MOOC
PLENK and CritLit
participants
Personal Learning
Environments
Networks and
Knowledge course
(PLENK2010) and
the Connectivism
and Connective
Knowledge course
(CCK11)
5,876 students-
online HCI class
Earn a certificate,
to improve English
skills, and a
variety of social,
academic,
vocational, and
interest-driven.
Significant
knowledge gains
and retention,
and the MOOC
was successful in
reaching across
age, sex, and
other demographic
categories.
The four activities
mentioned in the
introduction—
aggregation,
relation, creation,
and sharing—were
not achieved by
the majority of
participants.
The more
experience in
networked
learning and
through MOOCs,
the higher the
level of
participation.
Providing feedback
to students about
their grading bias
increased
subsequent
accuracy. “Fortune
cookie” is a
method for peers
to provide each
other personalized
feedback.
Report enrollment
intentions using the
OLEI scale as a
standardized
metric
The effects of
changes of
comparative
assessment and
better ways of
integrating a live
class with the
MOOC
To find out if this
“creation” stage is
really necessary to
enhance learning
in a connectivist
learning
environment
Explore the role
educators and
learners should play
in adding value to
the learning
experience in
MOOC. Support
from facilitator or
knowledgeable
students play critical
role on learning in
cMOOCs.
To explore if
fortune cookies
confer differential
benefits to different
students
Design of MOOCs
should be based
on learner
motivations.
Generation of a
class-specific
dataset for the
assignments is a
successful and
motivating activity.
That people needs
time to feel
comfortable and
confident to get
involved in
activities, while it
also seems that
people needs some
time to digest
readings and
resources.
Novice MOOCers
can be supported
through a series of
activities that are
structured on
connectivist
learning principles.
Feedback is an
effective way to
decrease the
students rating
bias in peer-
assessment.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
67
Appendix I. [Continued] Summary of reviewed studies.
No
36
37
38
39
Li et al. (2014)
Liang et al. (2014)
Liu et al. (2014)
Liyanagunawardena
et al. (2015)
Investigate how
co-located study
groups watch and
study MOOC
videos together.
Explore the
relationship
among learners’
perceived learning
experience,
learning behaviors,
and learning
outcomes with
MOOC.
Examine
participants’
learning
experiences in
the context of a
six-week massive
open online
course (MOOC)
in journalism.
To identify
the learner
groups and their
perception of the
MOOC
1. Do study groups
tend to watch
videos
asynchronously
(independent
watching within
the time frame
when the students
meet to study) or
is synchronicity a
desirable attribute
of group video
watching?
2. Are there
discussions while
watching a video
or after finishing a
video- lecture?
Does the video
watching
configuration
influence the
discussion patterns
of the group
members?
3. Do video watching
styles lead to a
difference in the
amount of
interactivity with
the video lectures?
Not provided
1. Who are the
students and why
are they enrolled
in this MOOC?,
2. How much time
have the students
spent in taking this
MOOC and have
they completed all
the assignments?,
and,
3. What have they
learned and what
aspects of this
MOOC do the
students find
most helpful?
Not provided
Longitudinal
study
Survey and data
mining
Both
quantitative
and qualitative:
survey, interview
and course
activity data
Pre-course and
post-course
survey
54 engineering
students
312 participants
Five thousand
students from
137 countries
Students of the
Begin programming:
build your first
mobile game
MOOC with two
iterations
Students like to
stay synchronized
in the group while
watching MOOC
videos. Overall
high satisfaction
with the study
group style.
Learners’perceived
usefulness
rather than
perceived ease of
use of the MOOC,
positively
influences
learners’use of the
system, and
consequentially,
the learning
outcome.
Most participants
reported a positive
learning
experience, but
lack of feedback
and/or poor
quality were
reported as
negative
experiences.
Games can be
used to teach
introductory
programming
over MOOC
when sufficient
support is provided
to the participants.
Measuring the
learning outcomes
of the study group
participants under
different video
watching style
To discover
more learning
mechanism of
MOOC users with
bigger data and
more reliable
survey
Not provided
Further
investigations
should focus on
insight about
learner
participation.
Video watching
style that has
shared display
and distributed
individual controls
might enable
study groups to
find a fine
balance between
synchronicity,
video interactivity,
and discussions.
It essential to
attach more
importance to the
dissemination of
the course, not
merely for
increasing the
registrants
MOOC
environment
requires learners
to be more self-
directed, self-
disciplined, and
intrinsically
motivated than in a
typical face-to-face
course.
Concentrate on
the target learner
group while
designing a
course with
providing possible
pathways for
other learners
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
68
Appendix I. [Continued] Summary of reviewed studies.
No
40
41
42
43
44
Mackness, Waite,
Roberts, &
Lovegrove (2013)
Mackness & Bell
(2015)
Macleod,
Haywood,
Woodgate, &
Alkhatnai (2015)
Margaryan,
Bianco, &
Littlejohn (2015)
Miller (2015)
Provide evidence
about how people
learned in FSLT12
MOOC and
consider wider
implications for
teaching and
learning in higher
education.
Focus on the
participant
experiences in
Rhizo 14 MOOC
Understanding
who Edinburgh
MOOC learners
are, who elects to
participate and the
aspirations of that
population, and
the place that the
MOOC will
occupy in the
University’s online
learning ecology.
Assess and
compare the
instructional
design quality of
MOOCs (xMOOC
and cMOOC).
Understanding
the differences
in quality online
pedagogy
between
MOOCs and
quality online
learning as
currently defined
1. How did cMOOC
design principles
and activities in
FSLT12 enable
participant
learning?
2. What are
the deeper
implications for
learning of the
principles and
activities used
in the design of
FSLT12?
3. What are the
possible
implications of
small task-oriented
cMOOCs for
higher education?
Not Provided
1. Who are the
tens of thousands
of individuals who
sign up to learn on
short, free, online
courses that offer
no qualification or
credits, and what
are they hoping to
achieve?
2. Are they
attracting an
‘unusual
audience’, and
if so, will a stable
audience arise and
if so, when?
Whether or not and
to what extend the
design of MOOCs
reflects the
fundamental
principles of
instruction?
1. Do MOOCs
represent quality
online pedagogy?
2. Are students
able to stay
and learn
effectively in
MOOCs?
Case study
Survey
Survey
Course scan
questionnaire
Survey
21 participants out
of 206 students
47 survey
participants and
35 follow up
survey participants
150k participants
76 MOOCs
500 participants
Learning is
happening in
distributed
platforms in
cMOOCs. Social
construction of
knowledge is key
component of
cMOOCs for active
learners. Technical
skills are also
important to
participate and
be in MOOCs.
Light side; some
highlighted
positive experience
i.e. learner-
autonomy, self-
organization. Dark
side; some felt not
connected, less
experienced
MOOCers were
felt isolated.
Providing
educational
opportunities to
the disadvantaged;
global uptake of
online learning;
growth of an
‘educational
imperialism’; and
the claim that
‘MOOCs are for
male geeks’
The majority of
MOOCs scored
poorly on most
instructional
design principles.
MOOCs can
conflict with
certain established
best practices in
online learning.
Not provided
Interrelated
processes of
community and
curriculum
formation in
Rhizo14. The
positive and
negative effects
of emotion and
alienation
Not provided
Investigation
of institutions’
and individual
academics’ and
instructional
designers’ rationale,
goals and
motivations
underpinning
their involvement.
Not provided
Learner autonomy
play critical role
in cMOOC
environment
especially for the
design of tasks and
completion rates.
Small task-oriented
cMOOCs can be
better option in
higher education.
Rhizomatic
learning, can
benefited adult
learner by forming
community, and
creating curriculum
in a community
setting
Not provided
Ten-principles
and course scan
instrument serve
as an evaluation
framework for
quality control and
improvement of the
implementation.
Not provided
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
69
Appendix I. [Continued] Summary of reviewed studies.
No
45
46
47
48
49
Milligan &
Littlejohn (2014)
Milligan, Littlejohn,
& Margaryan
(2013)
Perna et al. (2015)
Reilly, Stafford,
Williams, & Corliss
(2014)
Saadatmand &
Kumpulainen
(2014)
Investigate
the learning
behaviours
of health
professionals
within Fundamentals
of Clinical Trials, a
MOOC offered
by edX.
Determine patterns
of engagement
and to find out
factors influencing
engagement
in MOOC
environment.
Report the
progress of users
through 16
Coursera courses
taught by
University of
Pennsylvania.
Examine the
effectiveness of an
(automated essay
scoring) AES tool
to score writing
assignments in
two MOOCs.
Examined
participants’
experiences and
perceptions of
learning in
cMOOCs
How do professionals
prepare for learning
in a MOOC? This
question explores
the motivations and
expectations of
professional learners
as well as their goal
setting and strategic
planning during the
forethought phase.
1. What patterns of
engagement exist
within the
Change11
cMOOC course?
2.What principal
factors mediate
this engagement?
1. Do MOOC
users progress
through a course
sequentially in the
order identified by
the course
instructor, or do
users determine
their own
approach to
accessing content?
2. What are the
milestones that
predict course
completion?
To what extent is
the current edX
machine-graded
assessment system
(both holistic and
rubric-total) valid,
reliable and
comparable to
instructor grading?
Do the AES-graded
assignments (AES-
Holistic and AES-
Rubric total)
correlate with non-
essay assignment
grades in the course?
1. How do
participants in
cMOOCs use tools
and resources for
their learning?
2. What networking
activities take place
in cMOOCs?
3. What is the nature
of participation
and learning in
MOOCs, and
how is it
perceived by
MOOC learners?
Survey
Qualitative
study/Interview
sessions
Descriptive
analysis
Causal-
comparative,
a non-
experimental
research design
Online
ethnography
design; survey,
interview and
autoethno-
graphic insight
MOOC participants
29 participants
from out of 2300
16 first-generation
MOOCs
Randomly selected
206 of the
AES-scored essays
PLENK10, CCK11
and EC&I 831
participants
A professional
learning MOOC
could support
professional
learners to reflect
on the knowledge
gained from the
course.
Three levels of
engagement: active
participants, lurkers,
and passive
participants.
Confidence, prior
experience and
motivation are key
factors for
engagement.
Users accessed
course content in
the sequential
order identified by
the instructor.
Accessing the first
lecture and the
fourth quiz are
strong predictors
for retention to
the course.
AES and instructor’s
scores are
significantly related,
but that the
instructor assigned
significantly higher
grades than either
AES-scoring system.
AES-Holistic Total
and AES-Rubric
Total were most
highly correlated
Participation in
MOOCs challenges
learners to develop
self-organization,
selfmotivation,
and a reasonable
amount of
technological
proficiency to
manage the
abundance of
resources and the
more open format.
Explore the same
research questions
in a different
MOOC context.
Compare the
learning
experience offered
by different
cMOOCs, target
specific types of
learners, to
gain a better
understanding of
how critical
literacies for
learning in
cMOOCs develop.
Examination of
users’ course
experiences with
‘better data’.
Understanding
how design
elements and
pedagogical
practices effect
user products.
To determine
the types of
assignments that
are most relevant
for this scoring tool
(length, topic,
number of rubri
categories, range
of rubric scores,
etc.)
Results of the
study should be
further exploration
in different
MOOCs.
Not provided
Course organizers
can accommodate
students’ diverse
learner profiles
when design a
learning experience.
Support can be
provided those
students who
has no prior
experiences on
online learning.
Not provided
MOOC providers
and faculty
members can
use AES systems
in MOOCs.
cMOOCs are
learner-controlled
environments in
which learners
participate in
the flow and
generation of
knowledge.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Yüksekö¤retim Dergisi
|
Journal of Higher Education
Olga Pilli, Wilfried Admiraal
70
Appendix I. [Continued] Summary of reviewed studies.
No
50
51
52
53
Seaton, Bergner,
Chuang, Mitros, &
Pritchard (2014)
Toven-Lindsey,
Rhoades, & Lozano
(2015)
Trumbore (2014)
Veletsianos et al.
(2015)
Overview of how
the 108,000
participants behaved
in 6.002x - Circuits
and Electronics, the
first course in MITx
Explore the range
of pedagogical
tools used in
MOOCs, to
consider the
extent to which
these courses
provide
students with
high-quality,
collaborative
learning
experiences.
Determine what
course design
elements were
successful in
increasing their
engagement.
Describes
MOOC learner
activities around
notetaking and
interactions in
social networks
outside of
the MOOC
platform.
How various course
components, and
transitions among
them, influence
learning in MOOCs?
1.What
instructional
tools and
pedagogical
practices
are using
MOOCs?
2.How are
new digital
and networked
technologies
impacting
the delivery
of MOOCs?
3.To what
extent
are MOOCs
able to provide
a space for
critical inquiry
and active
student
engagement
in the learning
process?
What course
design elements
are successful in
increasing their
engagement?
Not provided.
Case study
Qualitative
multi-case study
analysis
Survey and
datasets of
MOOCs
Qualitative
study
230 million
interactions
were logged in
38,000 log files
24 university-level
MOOCs
8 MOOCs
hosted in the
NovoEd social
41 students
surveyed and
13 students
interviewed
Students spent
the most time
per week
interacting with
lecture videos
and homework,
followed by
discussion
forums and
online
laboratories.
The range of
pedagogical
practices currently
used in MOOCs
tends toward an
objectivist-individual
approach, a few
number of MOOCs
are oriented by
constructivist-group
approaches.
MOOCs hosted
on the NovoEd
social learning
site produce
sustained
student
engagement,
leading to
increased
persistence
and completion
rates. Three
critical conditions
for engagement
are student
collaboration;
cohesive,
open-ended
assignments;
and learning
communities.
Interactions in
social networks
outside of the
MOOC platform,
notetaking and
consuming are
most common
experiences and
activities.
The correlation
studies between
resource use and
learning
Not provided
How can we
better define
learning outcomes
for complex
open-ended
assignments?,
How can we
measure learning
outcomes so that
we can improve
students’ learning
while they are in
the course?
Use diverse
methodologies
to generate
a greater
understanding
of learner
experiences and
activities in
MOOCs.
Course component
can improve
students’ learning
both in MOOC
and traditional
on-campus
environment.
For better benefit
MOOC should be
design in a
creative and
empowering
structure.
Engagement is
necessary for
learning, course
designer should
use strategies for
better student
engagement
both in online
and on-campus
education.
Design digital
notebooks live
outside of a
particular
course
so that learners
can use them
for multiple
related
courses.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
Cilt / Volume 7|Say› / Issue 1|Nisan / April 2017
Students’ Learning Outcomes in MOOCs: Some Suggestions for Course Design
71
Appendix I. [Continued] Summary of reviewed studies.
No
54
55
56
Yang, Wen,
Kumar, Xing, &
Rose (2014)
Yousef, Chatti,
Wosnitza, &
Schroeder (2015)
Zutshi, O’Hare, &
Rodafinos (2013)
Analyze the
emergent social
structure in
massive open
online courses
(MOOCs).
Cluster and
analyze the
different objectives
of MOOC
stakeholders to
build a deeper
and better
understanding of
their behaviors.
Examine the
experiences of
students who
have participated
in massive open
online courses
(MOOCs).
How the novel
exploratory
machine learning
modeling approach
is able to identify
emerging social
structure in
threaded
discussions?
Not provided
Social network
structure and
thematic
structure of
text
Action
research/Survey
Google blog
search
1146 active
users and 5107
forum posts; 771
active users and
6250 posts;
3590 active
users and 24,963
forum posts.
76 professors and
82 learners
Higher attrition
demonstrate
lower comfort
with course
procedures and
lower expressed
motivation and
cognitive
engagement
with the course
materials.
Blended learning,
flexibility, high
quality content,
instructional
design and
learning
methodologies,
life-long learning,
network learning,
openness, and
student-centered
learning
Integration of
other networks
in addition to
text would be
another approach
for further
research
Investigate a set
of specific criteria
related to each
emerged cluster.
More dropout
when students
have not yet
found a personal
connection
between their
interests and
goals and the
specific content
provided by the
course
Put more
emphasis on the
hybrid MOOCs
which combine
both xMOOCs
and cMOOCs to
meet the goals
of a wide range
of participants.
Author/Date Purpose Research
question(s)
Methodology Sample Results Implications for
future research
Implications
for practice
