Content uploaded by Sarah Edelsbrunner
Author content
All content in this area was uploaded by Sarah Edelsbrunner on Sep 10, 2021
Content may be subject to copyright.
1
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
QUIZ FEEDBACK IN MOOCS FROM THE PERSPECTIVE OF
LEARNING ANALYTICS: ROLE OF FIRST QUIZ ATTEMPTS
Sandra Schön
Philipp Leitner
Martin Ebner
Sarah Edelsbrunner
Katharina Hohla
Graz University
of Technology
September 2021
ICL conference
2
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
License
These slides may be used under the terms of the CC
licence CC BY 4.0 International
(https://creativecommons.org/licenses/by/4.0/ ). Please
use the links and references as sources for individual
illustrations, if applicable.
The slides are available online at ResearchGate.
TU Graz
3
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Full paper Will be published in the ICL Proceedings:
Sandra Schön, Philipp Leitner, Martin Ebner, Sarah Edelsbrunner &
Katharina Hohla (to be published in 2021); Quiz Feedback in Massive
Open Online Courses from the Perspective of Learning Analytics: Role
of First Quiz Attempts. Will be published in: ICL2021 Proceedings,
Lecture Notes in Networks and Systems, Springer.
Available online at:
https://www.researchgate.net/publication/354462866_Quiz_Feedback_i
n_Massive_Open_Online_Courses_from_the_Perspective_of_Learning
_Analytics_Role_of_First_Quiz_Attempts
4
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Agenda
•Quiz results and feedback from perspective of learning analytics: What
we know
•Method: How we wanted to get new insights
•Quizzes at iMooX and the selected MOOC
•Results
5
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Quizzes in MOOCs as predictor
•
Compared with other assessment forms in MOOCs (self-
assessment, peer-assessment), quizzes are the strongest
explanatory variable
for the final exam (e.g. Admiraal, Huisman
& Pilli, 2015; 3 MOOCs, nearly 100,000 participants)
•
Learners’ satisfaction in MOOCs related to quiz functionality
(N=116, edX, Coursera)
•Papthoma et al. (2015): perception of automatic feedback or peer
assessment may also be related to cultural background and
previous experiences (based on qualitative interviews)
6
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Quiz feedback as a sort of learning analytics
Do quizzes and their
feedback influence future
activities of a learner in a
MOOC?
7
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Methodology
•Explorative quantitative analysis of available data of participants
•MOOC platform (iMooX)
•single MOOC (with 1,484 participants)
8
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
iMooX.at – the Austrian MOOC platform
8
TU Graz Universität Wien
9
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
iMooX.at – the Austrian MOOC-platform
free
OER
(Copyright)
GDPR-compliant
(ACRONET-Partner)
university level
founded in 2013 by
10
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
iMooX – quizzes as key for assessment
•each unit has a quiz
•quizzes can be repeated to 5 times
•best result counts
•75 % quota of positive quiz results needed for certificate
(some MOOCs use additional or other forms of assessment)
11
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Selected MOOC
•lecture with practical exercise “Teaching and learning with new
media“ (https://imoox.at/course/luliss21)
•co-designed by six Austrian universities
•lecture part offered as a MOOC in summer semester 2019
12
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Selected MOOC
13
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
MOOC participation
Activity
Number of
Participants
Registered participants in MOOC (6 weeks)
1,484
Participants who did no quiz
540
Participants who did at least one quiz
944
Participants who did at least one quiz but were not successful (did not
reach 45 of 60 points in quizzes)
192
Participants who were successful (reached 45 of 60 points in quizzes)
752
Certification for successful MOOC participation
690
Successful participation at lecture and exercise (MOOC, group work,
exam)
605
Number of successful participants in the MOOC “Learning with new media” in 2019 and the parallel “lecture and exercise” of six
Austrian partner universities. Source: Ebner et al. 2020, Table 1, p. 73 and additional data (02/2020).
14
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Quiz attempts
Quiz no of
questions
Quiz
attempts in
sum
Number of
Participants Quiz Attempts
per Participant
Quiz 1 5 1,956 781 2.50
Quiz 2 6 1,896 812 2.33
Quiz 3 4 1,665 764 2.18
Quiz 4 2 1,198 938 1.28
Quiz 5 4 1,403 759 1.85
Quiz 6 2 1,244 750 1.66
Sum 23 9,362 944 9.92
15
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Activities concerning quizzes on a timeline
16
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Distribution of quiz results
17
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Quiz attempts and their results
quiz
no
not
successful
attempts
successful
attempts
not
successful
attempts in %
successful
attempts in % sum of
attempts
1 516 1440 26% 74% 1956
2 609 1287 32% 68% 1896
3 574 1091 34% 66% 1665
4 293 905 24% 76% 1198
5 226 1177 16% 84% 1403
6 424 820 34% 66% 1244
All 2642 6720 28% 72% 9362
18
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Results in the first quiz attempt
failed
attempts successful
attempts
rate of success
at first attempt
number of
participants
quiz 1382 546 58,8 % 928
quiz 2 6 5 45,5 % 11
quiz 3 1 0 0,0 % 1
quiz 4 0 1 100,0 % 1
quiz 5 1 1 50,0 % 2
quiz 6 0 0 / 0
sum 390 553 58,6 % 943
19
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Correlation of quiz first attempt and
final assessment
Scatterplot Depicting the Correlation
Between Quiz Result
of First Attempt (x-axis, positive with
7.5-10 points)
and Final Assessment (y-axis, positive
with 45-60 points)
Data collected during the 2019 summer
term.
20
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Discussion
statistically positive, but moderate effect of first attempt feedback on
future success is interesting (r = 0,2765)
study gives some ideas on how to dive in into data in future analysis
21
WE CARE ABOUT EDUCATION
TU Graz, www.tugraz.at
Acknowledgements
.
„iMooX – the MOOC platform as a service for
all Austrian universities” (2020-2023), co-
funded by the Federal Ministry Republic of
Austria Education Science and Research.
Parts of the presented development and research was carried out
for the following projects:
"Learning Analytics: Auswirkungen der
Datenanalyse auf den Lernerfolg", Partner:
Graz University of Technology and University
of Graz, Sponsor: Provice of Styria (Land
Steiermark)
12th Future Fund Styria