Learning in MOOCs and learning about MOOCs: Reflections on investigating completion rates

Abstract

This is a transcript of a keynote presentation given at 'MOOC: Lokala möjligheter till globalt lärande' conference, Campus Skellefteå, Skellefteå, Sweden, 19 May 2016.

Full text

Learning in MOOCs and learning about MOOCs: Reflections
on investigating completion rates
Katy Jordan
Institute of Educational Technology, The Open University, UK
This is a transcript of a keynote presentation given at 'MOOC: Lokala
möjligheter till globalt lärande' conference, Campus Skellefteå, Skellefteå,
Sweden, 19 May 2016.
Good afternoon all. I am Katy Jordan. I’m a PhD student in the Institute of
Educational Technology, at the Open University, in the UK. I’m currently
writing up my doctoral research, which has focused on the use of academic
social networking sites, but today I’m going to be talking about one of my
other research projects, which has looked at enrolment figures and
completion rates of MOOCs. So just to give you a brief overview of the
structure of my talk today;
- First, I will talk about the background and how I came to study MOOC
completion rates (as a separate project alongside my formal doctoral
research, although it has become much more well known);
- Second, I will give you the key results from my research into completion
rates;
- And in the final part I will discuss some of the reflections that I have had on
MOOC completion rates, and how this work fits into the wider MOOC
research agenda, and what I think are the open challenges for the field
moving forward.
1. Background
My interest in MOOCs began in Spring 2012, which was around the time that
Coursera and Udacity were being formally launched. I was an Educational
Research Masters student at the time. My research interests had been
focused on e-learning and technology-enhanced learning, which is an
interdisciplinary field. My Masters course covered the social science side well
but I don’t have any qualifications in Computer Science; I had been looking
into whether I could study for a postgraduate certificate in Computer Science
to fill this gap, but not had much luck.
So this was my motivation for joining MOOCs: to learn about Computer
Science. And fortunately, most of the early courses were exclusively
Computer-Science based. Because of my research interests in e-learning, I
wanted to try to capture what I was learning about MOOCs themselves, so I
started blogging my reflections at the end of completed courses at
MoocMoocher from July 2012 (Figure 1). In the Autumn, I started my PhD at
the OU and so focused on social network analysis courses, as they were of
direct relevance to my PhD.

Figure 1: Timeline of events.

In 2013, I continued with my MOOC studies, although I was tending to choose
things which sounded interesting rather than being part of my original
Computer Science goal.
I also started experimenting with other platforms. A key point was when I was
a student on the Infographics and Visualisation MOOC, run by the Knight
Center for Journalism. This course required submission of a final project, and
you could choose your own topic. I decided that I wanted to look at MOOC
completion rates, as through my participation in earlier courses, I had been
developing ideas about how the use of assessments impacted on
engagement with the course. However, I was met with an immediate
challenge – this is a course about data visualisation, but where was the data?
From the courses that I had taken, I knew that course leaders would
sometimes share figures about the numbers of students who enrolled and
completed at the end of the course. At the time, MOOC hype was really taking
off, so these were also the figures most frequently used by MOOC academics
when interviewed by local and international news outlets. So, I started to
piece these sources together, using my blog as a platform to collate and
crowdsource data. I posted my visualisation (Figure 2) on my website,
submitted my project, passed the MOOC and thought that was the end of it.
Figure 2: Screen capture of the MOOC completion rate (as percentage who complete plotted
against the number who enrol) data visualisation.
However, it was just the start. My blog post was picked up by Phil Hill, who is
a well-known blogger about educational technology, on the E-literate blog. He
blogged about my work and how it represented the largest study of MOOC
completion rates to date (I think my sample size was about 25 at that point).
Through social media, it was thrust into the public eye, and I began to be
contacted by journalists about my work.

In due course, the call for proposals for the MOOC Research Initiative came
out, and because of the raised profile of my small study, I was invited to
submit a research proposal, which was successful. This transition from small-
scale, pilot project, conducted in the open, with publicly-available data, to a
formal research project, can be viewed as an example of a Guerrilla research
project, and is discussed in these terms by my supervisor Martin Weller in his
book ‘the Battle for Open’ (Weller, 2014).
2. MOOC completion rates
Having the MRI funding allowed me to extend my data collection in a more
formalised and systematic way (by this point, more academic outputs were
emerging which included completion data), and formal peer-reviewed papers
followed.
I’m now going to review the headline results from the two IRRODL papers
(Jordan, 2014a; Jordan, 2015); the middle one is slightly different in that it
was a cross-over between my interests in MOOCs and my doctoral studies on
network analysis and interdisciplinarity (Jordan, 2014b). It also started as a
blog post, and focused upon exploring the courses that students sign up to
together on Coursera, as a social network graph.
Figure 3: Summary of key findings from Jordan (2014a).
The goals of my first paper, which appeared in the International Review of
Research in Online and Distributed Learning (IRRODL), sought to provide an
empirical answer to some of the very basic questions about ‘how massive is
massive?’ in the context of MOOCs (Figure 3). This was quite descriptive and
not a very sophisticated study, but it helped to cut through the hype that was
surrounding the emerging field of MOOCs at the time.

The definition of completion used here was ‘the percentage of enrolled
students who achieved a certificate or met the criteria to be awarded a
certificate’. This definition has limitations, but it was simply the only type of
data that was available in the early days. I will talk a bit more about this later.
Figure 4: Summary of key findings from Jordan (2015).
The second paper in IRRODL (Jordan, 2015) developed the completion rate
data collection and analysis. This paper was published after the MRI project,
which had facilitated further building the dataset. It gave an indication of how
enrolment and completion was changing over time, and also allowed
categorising of courses according to assessment types, and allowed some
analysis from a course design perspective.
First, that lower completion rates were associated with longer courses (Figure
5). Second, that significant differences in completion rate were associated
with the types of assessment used in courses, specifically around the use of
peer grading (Figure 6).
Figure 5: Percentage completion rate plotted against course length in weeks.

Figure 6: Boxplots of percentage completion rates according to assessment category.
In categorising courses, assessments such as multiple choice questions and
maths or code-based problem solving, which were graded automatically, were
categorised as auto grading. Peer grading was defined as assessments which
were marked by other students on the course, typically using a pre-defined
rubric. The assessments themselves included short form essays, or longer
project-based submissions.
Few courses in my sample exclusively use peer grading (10, compared to 23
which used both, and 92 auto-graded courses); it is more common to be used
in combination with auto-graded assessments. But in both instances, the
completion rates observed were lower than those which only used auto
grading.
This is not to say that peer grading should not be used, but the evidence here
suggests that it discourages some students, so careful consideration should
be given to how it is used, and further research would be valuable in this area.
Depending on the type of peer grading used, I believe that it can be a
valuable pedagogical strategy. For me, I found that I learned a lot from taking
part in peer assessed projects, but it might be better not to use it in situations
where the benefits are not outweighed by the loss of engagement. For
example, short form essays that mainly test factual recall may be better
addressed by auto-grading.
There is also an open research question of why we see lower completion
associated with peer assessment; whether the concept of being assessed by
your peers puts students off taking part in assessments, or if marking is more
harsh. I think it is more likely that some students are put off taking part, as this
can be an uncomfortable experience for traditional students, and may relate to
some of the assumptions of educational systems in different cultures.

A further related note of caution about peer assessments is that in a study
based on the HCI course at Coursera, students were found to give higher
marks to anonymous students based in the same country as them (Kulkarni et
al., 2013). So I think more work is needed to figure out the best ways of
utilising peer assessment in a way which is fair and inclusive. In addition to
the practical implications of these findings, I think they make the point that
despite its limitations as a measurement, you can learn things about MOOCs
and learning design by looking at completion rates.
3. Reflections and ways forward
In this final part of my talk, I am going to discuss what has emerged for me as
the implications of this, limitations and key challenges for MOOCs moving
forward, including:
- Arguments against studying completion rates;
- Openness of data;
- Moving from student behaviour to an open, inclusive MOOC pedagogy;
- Pathways to expertise - combining individual MOOCs.
There is a certain resistance to engaging with completion rates. But this
resistance stops useful conversations from happening about how MOOCs
might be improved, for the ultimate benefit of both learners and educators.
The arguments that I have encountered polarize around two particular
perspectives.
The first perspective argues that looking at completion rates misses the point,
and it is more about the free access to educational materials. This comment
(Figure 7) was one of the peer reviews I received on my infographics and
visualization MOOC assessment. I wouldn’t actually disagree with most of the
sentiment here – but I do not see how it is a logical argument against thinking
about how to improve completion rates. The two things are not related, let
alone mutually exclusive.
Figure 7: Peer-review comment on the MOOC completion rate visualisation.
Yes, free access is a fundamental benefit of MOOCs, but that’s not a reason
to not think about how to make the courses better. It is arrogant to think that
just taking ‘elite’ university courses and giving access to them online is all that
is needed. Why should MOOCs still be designed in the same way that they
are run as formal courses?

The second perspective argues that most students do not sign up with the
intent of completing courses (Koller et al., 2013). I take issue with this position
as it places the onus upon the student for their persistence and success in the
course, and removes all pedagogic responsibility. However, studies examining
this perspective have found that motivation is not well correlated with success,
and completion rates remain low even taking this position (Breslow et al.,
2013; Reich, 2014). That is, there is work to be done!
Linked to this, the argument is also often made that learners are highly self
directed and take what they want from courses without needing to complete
them. While this undoubtedly applies to a subset of MOOC students, I am not
sure that it applies to the majority.
Figure 8: Percentage of enrolled students who remain active across the course of MOOCs.
This chart (Figure 8) shows part of the work I did during the MRI project,
where I looked at the proportion of students still active on a week by week
basis across courses. These are the average curves – data was available
from 54 courses in terms of those submitting assessments, and 59 courses in
terms of accessing course materials - but the pattern is fairly consistent
across individual courses. We see this drop in the first two weeks of a course;
it seems unlikely that these weeks just happen to contain the material that all
the students were looking for! To me, this drop is fascinating and represents a
really key challenge to MOOCs, and understanding it calls for a move away
from analytics and a focus back to pedagogy.
To recap, in response to the arguments against completion rates, I would
argue that they can be a useful way of thinking about course design. MOOCs
are largely designed like traditional courses, in that there is a progression and
completion at the end; but as we had seen, factors relating to course design
can affect completion rates. If completion represents a significant educational
goal, there is a fundamental tension between this and the concept that
completion rates do not matter. To this end, I think it really highlights the need
to rethink course design; a move to shorter, more modular courses, but with
more explicit links between modules and progression, would benefit both
perspectives.
But, openness of data, or lack of, is a big barrier to progress here. When I set
out, most of the sources I used were from media reports, but as the MOOC
hype subsided, so did the news stories. There was a progression then to
university reports, or peer-reviewed journal articles, as sources for data. This
was good, in that the data were more reliable and more detail was available,
but a smaller proportion of courses undertook this more in-depth analysis, and
there is a time lag associated with the peer review and academic publishing
process.
I would really like to revisit my work now and explore different definitions of
completion though; I used percentage of enrolment as it was simply the only
metric for which data was available when I started. Whereas now, there is
more data available which uses completion as a percentage of active

students, who viewed some of the course material at all. This would improve
the accuracy of exploring course design decisions.
In the past year or so, we have also seen more of a move towards exclusively
fee-based certification, first through Futurelearn and increasingly at Coursera.
This reinforces the barrier to free data sharing. But I believe that it is by
comparing trends in MOOCs across the field, rather than individual courses,
which would be the most illuminating.
Conclusions
To conclude, there are two main ways in which I have come to view as being
critical issues for MOOCs, moving forward, as a result of both my experiences
as a MOOC student myself, and considering MOOC completion rates. Whilst
completion rates are not the whole story in terms of how students can benefit
from MOOCs, they do provide some indication of levels of engagement with
the course, and areas for improvement.
Courses will require redesign from their original traditional higher education-
style formats. It isn’t enough as educators simply to make things available
online. So this is what I mean when I call for a move toward inclusive, open
pedagogy; yes, one of the great benefits of MOOCs is the removal of formal
qualifications –based requirements as a barrier to entry. But by teaching in the
same way, it will select those who meet the requirements anyway. Numerous
studies have shown that those who are most likely to complete MOOCs are
learners who have already been through higher education. Having hidden
prerequisites like this undermines openness in this sense. This is not
something which can be easily derived from electronic data traces or
analytics, and represents a major challenge.
However, one way forward, would be for greater signposting between
courses. This would go hand-in-hand with shorter courses, as my research
suggests, split into individual topics. As a novice, it is difficult to see where to
go once you have completed a MOOC, in order to advance to a more
substantial level of expertise in a field. Where there are these hidden
prerequisites, there is now usually ‘a MOOC for that’. But this modelling of
knowledge pathways, and links between individual MOOCs, could be a
practical and useful way forward. Possible technological solutions exist, such
as recommender systems and semantic web-based technologies, which if
combined with expert knowledge modelling, could fill this gap, also potentially
opening a new role for educators as curators of open knowledge.
References
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A.D., & Seaton, D.
T. (2013). Studying learning in the worldwide classroom: Research into edX’s
first MOOC. Research and Practice in Assessment, 8, 13-25

Jordan, K. (2014a) Initial trends in enrolment and completion of massive open
online courses. The International Review of Research in Open and Distance
Learning, 15(1), 133-160.
Jordan, K. (2014b) Exploring co-studied massive open online course subjects
via social network analysis. International Journal of Emerging Technologies in
Learning 9(8), 38-41.
Jordan, K. (2015) Massive Open Online Course completion rates revisited:
Assessment, length and attrition. The International Review of Research in
Open and Distributed Learning 16(3), 341–358.
Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in
massive open online courses: In depth. Educause Review. Retrieved from
http://www.educause.edu/ero/article/retention-and-intention-massive-open-
online-courses-depth-0
Kulkarni, C., Koh, P. W., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller,
D., & Klemmer, S.R. (2013). Peer and self-assessment in massive online
classes. ACM Transactions on Computer-Human Interactions, 9(4) Article 39.
Reich, J. (2014) MOOC completion and retention in the context of student
intent. Educause Review. Retrieved from
http://www.educause.edu/ero/article/mooc-completion-and-retention-context-
student-intent
Weller, M. (2014) The battle for open: How openness won and why it doesn’t
feel like victory, London, Ubiquity Press.