Content uploaded by Martin Ebner
Author content
All content in this area was uploaded by Martin Ebner on Feb 11, 2016
Content may be subject to copyright.
Content uploaded by Michael Kopp
Author content
All content in this area was uploaded by Michael Kopp on Feb 11, 2016
Content may be subject to copyright.
393
How MOOCs can be used as an instrument
of scientific research
Claudia ZIMMERMANN
1
, Michael KOPP
2
& Martin EBNER
3
1
University of Graz, Department of Sociology,
claudia.zimmermann@uni-graz.at
2
University of Graz, Academy of New Media and Knowledge Transfer,
michael.kopp@uni-graz.at
3
Graz University of Technology, Educational Technology,
martin.ebner@tugraz.at
Abstract
Massive Open Online Courses (MOOCs) are an increasingly important
phenomenon in the world of technology-enhanced learning. This development
opens many opportunities for interdisciplinary interaction, not only for the purpose
of researching MOOCs themselves, but for integrating them into various research
settings. In this publication, we adress the question of how MOCCs can be used as
instruments in scientific research. Our suggestions are illustrated on the example of
the “Dr. Internet” project, which allowed us to gain practical experience in this area.
Keywords
Case study, experiences, Higher Education, Dr. Internet
Experience Track / Institutions Track / International Track / Research Track
Proceedings of the European MOOC Stakeholder Summit 2016
394
1 Introduction
During recent years, MOOCs have been established as a rather substantial form of
knowledge transfer in various academic fields. However, their didactic potential as
well as their profitability is still being called into question. Therefore, MOOCs are an
interesting object of continuous scientific investigation. This research usually focuses
on how MOOCs may support lecturers and students, how they enhance technology-
based teaching and learning and how much effort it takes to produce and to offer a
MOOC. High numbers of participants produce copious amounts of data, which are
routinely collected and analyzed. Previous research mostly deals with demographic
issues (NEUBÖCK et al., 2015), (rather low) completion rates (KHALIL & EBNER,
2014) or the didactic value provided by MOOCs (LACKNER et al., 2014).
In all these cases, MOOCs themselves are the objects of research. While there are cer-
tainly many inherent aspects to MOOCs that warrant further investigation, they contin-
ue to play an increasingly important role in higher education (KOPP et al., 2014).
Hence, they should not only receive consideration as targets of scientific research, but
also as potential tools to be integrated in a variety of research designs to the avail of
several academic disciplines. In this context, the following research question is the
most apparent: How can we use MOOCs as an instrument of scientific research and
what are the benefits and drawbacks of integrating MOOCs in a specific research set-
ting? This paper describes the first case (at least in Austria) where a MOOC is part of a
comprehensive, multi-disciplinary scientific research project. More precisely, this par-
ticular MOOC supports the investigation of people’s behavior regarding Internet use to
find diagnoses for diverse disease patterns.
Unfortunately, the MOOC “Dr. Internet – determining diseases with the help of Inter-
net searches” is scheduled to start after the submission deadline for this paper. There-
fore, the authors will primarily describe the objectives and the structure of the MOOC
as well as its relevance for the chosen research setting, finishing off with a preview of
the expected results (which will be available when the summit takes place).
How MOOCs can be used as an instrument of scientific research
Claudia Zimmermann, Michael Kopp & Martin Ebner
395
2 “Dr. Internet”-project: Description and
objectives
The “Dr. Internet”-project focuses strongly on online searching for health-related in-
formation. The aim of the project is to investigate how the increasingly common prac-
tice of using the Internet to answer medical questions affects the doctor-patient rela-
tionship, and what benefits and risks are associated with this behavior. Recent experi-
ences of general practitioners show that more and more patients visit their doctor’s
office with previously acquired medical knowledge, obtained from online sources like
popular websites, patients’ forums etc. The acquired information can be extensive, but
has often been found to be inconsistent and difficult to evaluate (BENIGERI &
PLUYE, 2003). Possible negative consequences include unsettled patients and over-
whelmed doctors: First, Internet searches might lead patients to under- or overestimate
the severity of their perceived illness. Second, some patients may only consult with
their physician to get a confirmation of the self-diagnosis they have already attained
with the help of the Internet, and it can prove difficult for doctors to convince them of a
differing diagnosis.
Therefore, it is important to raise awareness for a more balanced and critical approach
to online health information. The “Dr. Internet” project is providing a MOOC that in-
cludes six different medical case studies, all of which will be assessed and diagnosed
by the participants. These case studies are presented in the form of short videos in
which a patient describes or exhibits various symptoms. After watching the video, the
participants are encouraged to make full use of the Internet in order to find possible
diagnoses, and to discuss their assumptions with other participants in the forum of the
MOOC. A special quiz format is used to gather data on the participants’ diagnosing
preferences and to provide feedback on medical opinions about the subject (which will
be described in more detail in chapter 3).
An experienced general practitioner designed the medical case studies for the MOOC.
In addition, each module includes a video, where this doctor explains which one of the
suggested diseases he believes to be the most likely one and how he came to this con-
clusion, as well as what to do if these symptoms were to occur in real life.
Experience Track / Institutions Track / International Track / Research Track
Proceedings of the European MOOC Stakeholder Summit 2016
396
The design of the MOOC thus enables participants to question their search behavior on
the Internet and evaluate their skills in the context of determining diseases. Simultane-
ously, the participants increase their knowledge about certain disease symptoms and
receive useful instructions on what to do (e.g. searching on the Internet versus consult-
ing a doctor) when particular symptoms occur.
The MOOC is available on the first and only Austrian MOOC-platform called
“iMooX”. In contrast to many other platforms, all course materials on “iMooX” are so
called Open Educational Resources (OER). This means that all videos are licensed
under a Creative-Commons-License and may be accessed and used by anybody who is
interested in using them (as long as this is not done for commercial purposes). Moreo-
ver, all materials stay available after the initial MOOC and the research project have
finished, so that future participants may still benefit from the course experience.
3 MOOCs as tools of research – a case study
The description of this case study is intended to illustrate some of the opportunities and
challenges of using MOOCs as a research instrument. The “Dr. Internet” research pro-
ject was contrived to integrate the MOOC within an already elaborate research design,
which uses a triangulation of traditional qualitative and quantitative research methods
in order to build a multifaceted database. We will first give an overview of how the
MOOC is involved on various levels of data generation, followed by a preview of the
expected results. Finally, we will briefly discuss the particular qualities of MOOCs in a
research setting and offer a few preliminary conclusions.
3.1 The “Dr. Internet” MOOC as part of a research design
The first and most traditional axis of analysis is the accumulated data that is generated
solely from the participants’ activity on the MOOC itself. In addition to person-level
variables like video completion, number of “clicks”, number of threads read and post-
ing activity in the forum etc., there is a short compulsory questionnaire which partici-
pants have to fill out during the first week of the MOOC. Questions include but are not
limited to the individual health situation, any experience of researching symptoms
How MOOCs can be used as an instrument of scientific research
Claudia Zimmermann, Michael Kopp & Martin Ebner
397
online and the perceived trustworthiness of medical information on the Internet. There
is also a qualitative arm of analysis that will focus on the MOOC’s forum, with the aim
not just to observe and analyze the postings, but also to start discussion threads and
provide stimuli for debate.
The second level uses data that is generated both by the MOOC users as well as by an
outside comparison group that is researched before the start of the MOOC. As with
most MOOCs, there are quizzes to be completed as part of each medical case. Howev-
er, contrary to most conventional quizzes that are constructed to test the participant’s
acquired knowledge and therefore serve as an indicator of learning. The quizzes of the
“Dr. Internet” MOOC are better described as polls: for each medical case, the partici-
pants are asked to assess the likelihood of eight potential diagnoses that the patient in
the video could be suffering from. This likelihood is rated on a four-part scale (com-
prising the categories unlikely, little likely, more likely, very likely) and can only be
submitted once per participant. While there is no direct feedback on whether or not the
likelihood estimations are correct, the participants are not only able to see the average
results of all users who have already submitted their choices, but also the average rat-
ings of a similar survey conducted among trained physicians.
The research activity on the third level can be seen as a complimentary exploration of
one of the main research questions of the Dr. Internet project, which is focused on
changes in the doctor-patient-relationship due to the increase of medical information
available on the Internet. Therefore, both sides need to be heard and appropriate re-
search tools have to be employed in order to provide sufficient material. Most of the
subjects will be recruited through a pyramid scheme of contacting general practitioners,
asking them for interviews as well as permission to approach their patients, who will be
prompted to fill out a questionnaire. In order to highlight potential differences in this
population and the Internet users who participate in the MOOC, there will also be sev-
eral interviews with participants of the MOOC, which will roughly follow the same
outline as the questionnaire. An online version of said questionnaire will also be made
available to the MOOC users at a later stage of the course, but its completion will be
optional.
Experience Track / Institutions Track / International Track / Research Track
Proceedings of the European MOOC Stakeholder Summit 2016
398
3.2 Preview of the expected results
Starting in the last week of October 2015, the “Dr. Internet” MOOC will last for six
weeks. After course completion, the analysis of data collected through or in addition to
the MOOC should allow for a more profound illustration of the following aspects:
Information on the course participants and their previous experiences with
online searches for medical information. While the sample is obviously not
representative of the general population, the sample size is still quite substan-
tial (expected N=400) and relatively conclusive with regard to the target popu-
lation (internet users who search for medical information online). The compul-
sory questionnaire as well as the forum discussions will yield data on the soci-
odemographic background of the course participants, their perceived physical
condition, their experiences with online searches for diagnosing purposes and
their interactions with medical personnel regarding their findings.
Cross analysis of participants’ characteristics and their course activities. One
reason for including the compulsory questionnaire was to be able to find out
more about who performs well on the diagnosing test. Additionally, we will be
able to do a cross-sectional data analysis with regard to the users’ personal
characteristics and several relevant course parameters (forum posts etc.).
Comparison of diagnosing preferences of course participants and physicians.
The previously conducted survey among doctors used the same videos that the
MOOC users will get to see as part of the course. We are thus collecting data
that will allow us to compare the diagnosing preferences of doctors and lay-
people (who are encouraged to use the Internet). This kind of comparison is
not only highly relevant to the main research question of the project, but also
an innovative way of administering the same test for two very different and
hard-to-reach populations.
3.3 Benefits and drawback of MOOCs as research tools
As demonstrated on the example of the Dr. Internet MOOC, there is a lot of potential
with regard to the incorporation of MOOCs into larger and multi-level research de-
How MOOCs can be used as an instrument of scientific research
Claudia Zimmermann, Michael Kopp & Martin Ebner
399
signs. The advantages are relatively clear: MOOCs provide a simple and convenient
way of collecting data, and they feature innovative options of presenting materials and
tasks or administer tests on a large scale, which can all be pertinent to many research
settings. They allow for both observation and engagement of participants, who might
be more inclined to put some effort into their tasks if they feel like they receive an
interesting learning experience in return. Compared to standardized questionnaires that
are administered online, MOOCs make it easier to test (or question) the same sample of
participants more than once. Further possible applications of these technical opportuni-
ties also include the large area of social intervention research, where there is a wide
range of conceivable interventions that could be delivered and tested online.
The disadvantages include the focus on a tech-savvy and Internet-affine population,
where several important characteristics might be different from the general population.
Thus, there are some research areas that are more suited to use MOOCs than others
(like the “Dr. Internet” project, where the phenomenon under consideration was closely
tied to Internet use in and of itself). There are also a few general characteristics of
MOOCs that are detrimental to some research designs, for example the low completion
rate and an overall declining frequency of activities of most course users. This is a
problem for most experimental settings, where a high attrition rate severely compro-
mises the comparability of outcomes in different groups. While some incentives that
are used in traditional research settings might also prove useful here (like monetary
compensation or course credits for students), it is likely that new ways of encouraging
ongoing participation will have to be developed and tested.
Finally, there is the slightly ambivalent aspect of data generated through MOOCs:
while they usually produce massive amounts of data, a meaningful analysis is not al-
ways possible, since “big data sets do not, by virtue of their size, inherently possess
answers to interesting questions.” (REICH, 2015) Perhaps more so than with other
research tools, the instrumentation of MOOCs should always go hand in hand with a
clearly structured research design and a set of realistic research questions, with full
consideration of the above limitations. Additionally, the incorporation of MOOCs into
complex research designs requires a high degree of coordination on an interdisciplinary
level, though some might argue that this could also constitute an advantage.
Experience Track / Institutions Track / International Track / Research Track
Proceedings of the European MOOC Stakeholder Summit 2016
400
4 Conclusion
The case study of the “Dr. Internet” project demonstrates one promising approach to
the question of how MOOCs can be integrated in larger research settings, and the pre-
liminary experiences are all but favorable. More insights and actual results are sure to
be available at the time of the EMOOCs 2016.
Overall, a wide range of possibilities with regard to MOOCs as research tools is yet to
be explored, which no doubt will lead to the development of more recommendations as
well as best practice examples. We can, however, already conclude that the instrumen-
tal use of MOOCs in scientific research will give rise to new ways of collecting and
analyzing data that shows every indication of being a useful addition to the existing
array of scientific methods.
References
Benigeri, M., & Pluye, P. (2003). Shortcomings of health information of the internet. Health
Promotion International, 18(4), 381-386.
Khalil, H., & Ebner, M. (2014). MOOCs Completion Rates and Possible Methods to
Improve Retention – A Literature Review. In Proceedings of World Conference on
Educational Multimedia, Hypermedia and Telecommunications 2014 (pp. 1236-1244).
Chesapeake, VA: AACE.
Kopp, M., Ebner, M., & Dorfer-Novak, A. (2014). Introducing MOOCs to Middle European
Universities – is it worth it to accept the challenge? International Journal for Innovation and
Quality in Learning, 2(3), 46-52.
Lackner, E., Kopp, M., & Ebner, M. (2014). How to MOOC? – A pedagogical guideline for
practitioners. In I. Roceanu (Ed.), Proceedings of the 10
th
International Scientific Conference
"eLearning and Software for Education", Bucharest, April 24 - 25, 2014. Editura Universitatii
Nationale de Aparare "Carol I”.
Neuböck, K., Kopp, M., & Ebner, M. (2015). What do we know about typical MOOC
participants? First insights from the field. In M. Lebrun, I. de Waard, M. Ebner, & M. Gaebel
(Eds.), Proceedings of eMOOCs 2015 conference (pp. 183-190). Mons, Belgium.
Reich, J. (2015). Rebooting MOOC Research. Improve assessment, data sharing, and
experimental design. Science, 347(6217), 34-35.
