Content uploaded by Gillian Nicola Saunders-Smits
Author content
All content in this area was uploaded by Gillian Nicola Saunders-Smits on Sep 10, 2021
Content may be subject to copyright.
THE EFFECT OF THE COVID-19 PANDEMIC ON A MOOC IN AEROSPACE
STRUCTURES AND MATERIALS
Dr. Ir. I. Jivet
Leiden-Delft-Erasmus Center for Education and Learning
Delft, The Netherlands
0000-0002-8715-2642
Dr. Ir. G.N. Saunders-Smits
1
Faculty of Aerospace Engineering, TU Delft
Delft, The Netherlands
0000-0002-2905-864X
Conference Key Areas: Methods, formats and essential elements for
online/blended learning, Social aspects and communication in online/blended
learning
Keywords: MOOC, Aerospace Engineering, Learning Analytics, Lifelong Learning
ABSTRACT
In March 2020 COVID-19 brought the world and with that aviation to a standstill. Also
in March 2020, the third run of the DelftX MOOC Introduction to Aerospace
Structures and Materials started on edX. This MOOC generally attracts a mixture of
young aviation enthusiasts (often students) and aviation professionals. Given the
large interest MOOCs have received as the pandemic hit, we investigate how the
new global context affected the motivation and the way learners interact with our
course material. For this project, we will use learning analytics approaches to
analyse the log data available from the edX platform and the data from pre- and
post-course evaluations of two runs of the same MOOC (2019 and 2020).
With the insights gathered through this analysis, we wish to better understand our
learners and adjust the learning design of the course to better suit their needs. Our
paper will present the first insights of this analysis.
1
Corresponding Author
G.N.Saunders-Smits
G.N.Saunders@tudelft.nl
1 INTRODUCTION
After the first reported case of COVID-19 in December of 2019 in China, the virus
quickly spread throughout the world causing travel to come to an almost complete
standstill. By mid-April 2020, more than two-thirds of the 22,000 passenger airliners,
had been grounded and associated staff either furloughed or made redundant, by
April 2021 aviation data analysts still report 8,684 aircraft in storage [1]. Already in
mid-March 2020 most higher education institutes in the world had closed their
campuses [2] and switched where possible to online teaching, either creating their
own or using existing online resources, a situation persisting on and off until today.
1.1 MOOC Aerospace Structures and Materials
The MOOC Introduction to Aerospace Structures and Materials (ASM MOOC) has
been running on edX since August 2018 [3], and is currently in its fourth run. This
MOOC is an introductory course, requiring only basic knowledge of physics, and is
aimed at anyone interested in aerospace structures and materials. On 10 March
2020, the third run of the MOOC Aerospace Structures and Materials opened on edX
for a 12-months run, one month after the previous run of the course, running for 10
months, had finished. The first run in 2018 was excluded from our analysis as it was
not self-paced and only ran for 12 weeks. Within the MOOC, learners have a choice
to try the course for free with limited access (9 weeks) or to upgrade to edX’s
‘Verified Track’ for $49 giving unrestricted access and the opportunity to earn a
certificate by taking online exams and doing online assignments during the course.
1.2 MOOC learners and COVID-19
With so many people associated with the aviation industry unable to work, as well as
many students and educators switching to online learning, the question arose how
the new global context affected the motivation and the way learners interact with our
course material compared to learners in the earlier run of the MOOC. In this paper,
we used data collected in the pre- and post-course surveys carried out by our
institution and learner data extracted from edX trace logs in order to understand how
learners interact with the platform. Ethical permission was sought and granted by the
TU Delft’s Ethics Board for this research and learners were asked for informed
consent on the gathering of their data both by TU Delft and by edX.
2 METHODOLOGY
With the rise of online education, the field of learning analytics was born. Learning
Analytics is “the measurement, collection, analysis and reporting of data about
learners and their contexts, for purposes of understanding and optimising learning
and the environments in which it occurs” [4]. Learning analytics can help educators
to understand and optimise learning and form an important tool in the field of online
education research. Especially MOOCs, with their relatively large number of
enrolments provide great data sources to better understand the behaviours of
learners in online courses and are as yet often underused [5].
2.1 Research questions
Our main research question for this paper is: How does the COVID-19 pandemic
affect the motivation and the way learners interact with the course material in the
MOOC introduction to Aerospace Structures and Materials on edX? To answer this
question, we compared the data from the 2019 (collected 9 April 2019 - 20 June
2020) and the 2020 run (collected 10 March 2020 to 21 March 2021) of the ASM
MOOC in terms of (i) the number of enrolments, and the professional and
educational background of enrolled learners, (ii) the completion rates of verified
certificate holders, (iii) the motivation in taking the course and (iv) the level of
interaction with the course material.
2.2 Data sets and data analysis
For the analysis, we used the anonymised edX learner data sets to determine the
overall number of enrolments in the run and the self-reported age, gender and
education level of the population and course completion. On the edX platform, formal
course completion is defined as obtaining a Verified Certificate but that only applies to
those learners who chose to pay to upgrade. Therefore, we defined course completion
for the audit track as students who attempted all quizzes in all 7 modules. The second
data set used in this paper is the answers offered by learners to the pre- and post-
course survey which included detailed questions about their motivation for enrolling in
the course, their background, expectations and evaluations of the course.
Table 1. Cohort and sample size.
*percentage calculated with respect to total enrolment
**percentage calculated with respect to number of Verified Enrolments
Run 2019
Run 2020
Total enrolment
11987
26329
Verified Enrolment
663
5.5%*
2533
9.6%*
Verified Certificate
301
45.4%**
1027
40.5%**
Countries represented
151
168
Pre-course survey
Agreed consent
2318
5807
Full Surveys
1944
4978
Net Response Rate*
16%
19%
Post-course survey
Agreed consent
269
957
Full Surveys
226
802
Net Response Rate*
2%
3%
Response Rate Verified Track**
27%
53%
All data was analysed using JAMOVI (jamovi.org). In table 1 the description of the
population of both runs is given as well as the response to the pre- and post-
questionnaire. For all analyses, only complete surveys were used.
3 RESULTS
3.1 Learners
As can be seen from table 1, there was a sharp increase in learners in COVID times.
The number of learners in the course more than doubled compared to 2019. Figure
1shows the normalised enrolment of both courses plotted over time, revealing a
much steeper increase in growth of learners of the 2020 run in the first 6 weeks after
the world-wide shutdown began (vertical grey line). This trend is confirmed in reports
by other MOOC makers [6]. Also, more learners opted to purchase access to the
Verified Track and the number of countries learners originated from also increased.
Fig. 1. Normalised Enrolment over Course Length
Using the edX learner data sets, the self-reported age of both runs was compared
using an Mann-Whitney U test, U = 4.08·107 and p < .001 with a small effect size z =
0.134 showed that in the 2020 run the learners were significantly younger (Median =
23 and N = 12908) than in the 2019 run (Median = 25, N = 7301). In terms of self of
self-reported gender, a - analysis showed that there is a significant difference in
gender distribution, with with a 3.4% increase in the
participation of women in the 2020 run. In 2019, 17.5% of enrolled learners were
female compared to 20.9% in the following year. The overall share of women taking
part in both runs is higher than the yearly influx in the BSc aerospace engineering of
TU Delft. The number of enrolled learners that reported their gender as "other" did
not vary across cohorts: 0.5% in 2019 and 0.4% in 2020. A - analysis of the self-
reported level of education of the learners in the MOOC, see table 2, also showed
significant differences with with increases of learners with just
high school education or lower and decreases in learners holding a masters or PhD,
again indicating that the major increase is among undergraduates.
Table 2. Level of education
Run 2019 (N=6470)
Run 2020 (N=10387)
Total (N=16857)
PhD
2.2%
1.7%
1.9%
Master
17.9%
14.7%
15.9%
Bachelor
36.8%
36.7%
36.7%
High School or lower
35.7%
39.2%
37.9%
Other
7.7%
7.7%
7.6%
We also looked at the differences between the self-reported employee situation of both runs
in the pre-course survey. Again, significant differences were found between the 2019 and
the 2020 run, . There is a sharp decrease of almost 10% in the
number of people classing themselves as working, recent graduates or looking for a job and
a sharp increase in students (Table 3). An investigation into the average age of these
students did not indicate that the average age of this group was rising so the increase in
students is not due to a return to education of people who were working. Surprisingly in
COVID times, the share of parents and care-givers remains the same.
Table 3. Current job situation
Run 2019 (N=1934)
Run 2020 (N=4962)
Total (N=6896)
Working
37.0%
27.3%
30.0%
Looking for a job
7.1%
6.3%
6.6%
Retired
0.7%
0.7%
0.7%
Student
42.7%
56.2%
52.4%
Recently Graduated
9.8%
7.3%
8.0%
Parent/care-giver
0.5%
0.4%
0.4%
other
2.3%
1.8%
1.9%
For those working, we analysed what industry sector and industry branch they
worked in. Significant differences were reported, , between
both runs, with a 45% increase in 2020 in learners reporting to work in
Transportation, indicative of the reported shutdown of aviation (full contingency table
omitted due to lack of space). The top 5 represented industry branches in table 4,
show that particularly in aerospace-related industry, there is an increase in the
absolute number of learners in COVID times with particularly airlines/aviation
standing out. This may be indicative of a culture of Lifelong Learning in the
aerospace sector.
Table 4. Reported Industry
Industry (sector)
2019 #
2020 #
Increase %
1. Aviation & Aerospace (manufacturing)
144
229
59%
2. Airlines/Aviation (transportation)
90
255
183%
3. Mechanical or Industrial Engineering (manufacturing)
46
64
39%
4. Military (government)
33
65
97%
5. Defence & Space (High Tech)
33
53
61%
3.2 Motivation
Learners were asked for their motivation to enrol in the course. An overview of both
runs is given in table 5. For both runs, the most named motivation is (prospective)
career, followed by (prospective) studies and personal interest. A - analysis
revealed significant differences in motivation to enrol between the two runs with
, which seems to stem from less people reporting taking the
MOOC for their (prospective) career, but more people reporting taking the MOOC in
view of their (prospective) studies. The explanation for this, combined with the
results reported in tables 3 and 4, may be that as universities and schools were
mostly shut down students were looking for alternative courses to take, were
encouraged by their own schools to do so or taking these courses in lieu of being
able to visit open days to help them decide on their future. Sadly, no COVID specific
questions were asked in the 2020 course questionnaires.
Table 5. Motivation to enrol
Run 2019 (N=1923)
Run 2020 (N=4929)
Total (N=6852)
(Prospective) career
41.2%
37.1%
38.2%
(Prospective) studies
30.2%
34.0%
32.9%
Personal interest
25.5%
26.4%
26.1%
(Prospective) teaching
2.0%
1.6%
1.7%
Other
1.0%
1.0%
1.0%
3.3 Challenges
Learners were asked in the pre-course survey what they felt was their biggest
expected challenge and in the post-course survey what they felt was the biggest
challenge they faced during the course. The pre-course survey showed significant
differences between the expected challenges between the two runs
and the post course survey confirms these findings with significant
differences in the challenges faced: . If we take a closer look
at the results as listed in table 6, it can be seen that, compared to 2019, in 2020
learners indicated that they expected time to be less of a challenge and this was
confirmed in the post-course survey. This may be indicative of more people being
able to make time during the pandemic as they followed the advice to stay at home
as much as possible.
Table 6. Expected challenges in taking this online course
pre course
post course
Challenges
2019
(n=1946)
2020
(n=4992)
total
(n=6938)
2019
(n=199)
2020
(n=692)
total
(n= 891)
Finding sufficient time
57.2%
47.8%
50.5%
50.3%
39.9%
42.2%
Grasping the content
13.8%
18.0%
16.8%
13.6%
19.7%
18.3%
I expect no challenges
8.5%
10.8%
10.2%
18.1%
25.6%
23.9%
Meeting the deadline
12.2%
13.7%
13.3%
9.0%
6.4%
7.0%
Using the platform
6.1%
7.0%
6.8%
1.5%
2.5%
2.2%
Other
2.1%
2.6%
2.8%
7.5%
6.1%
6.4%
3.4 Interaction and engagement with course material
As can be seen from table 1, there is an almost 80% increase in the number of
learners opting to buy access to the Verified Track in the course, which can be an
indication of learners during COVID times wishing to engage longer with the course
material. This is in part supported by the lower certificate completion rate of the 2020
learners compared to 2019. This can be an indication that they are more interested
in engaging with the material for longer than in obtaining a qualification. When
comparing the reported participation level in the post-course survey no significant
differences between the two runs were found .
However, participation can also be measured in interaction and engagement using
the learning activities: Video Lectures, Reading, Discussion Forum and Exercises.
To do so, we first look at the reported pre-course levels of importance learners
placed on these activities as well as the post-course reported levels of satisfaction
and value of these activities. Using a Mann-Whitney U test to check if there were
significant differences between the two runs on the importance learners placed on
these learning activities, only significant differences between the 2019 run (N = 1934,
Mean = 3.15) and the 2020 run (N = 4963, Mean = 3.23)) were found for the
Discussion Forum (U= 4.62.106, p = 0.013 and a small effect size z = 0.038),
indicating that interaction with other learners is more important to learners in the
2020 run. When looking at post-course satisfaction of learning activities, we again
see significant differences between the 2019 (N = 59, Mean = 3.73) and 2020 (N =
237, Mean = 4.19) run for the Discussion Forum satisfaction: U = 5267, p = .002 and
a medium effect size z = 0.25 and similarly for the value of the Discussion Forum
between 2019 (N = 60, Mean = 4.27) and 2020 (N = 237, Mean = 4.38): U = 5460, p
= .004 and a medium effect size z = 0.23. Learners also reported a significantly
higher satisfaction of the exercises between 2019 (N = 199, Mean = 4.27) and 2020
(N = 675, Mean = 4.37) with U = 61354, p = .039 and a small effect size of z = 0.09.
In terms of hours worked per week, in the post-course survey learners reported an
average of 7.14 hours/week in the 2019 run (N = 207) against 7.91 hours/week in the
2020 run. A Mann-Whitney U analysis showed a borderline significant difference: U =
69338, p = 0.05 and a small effect size, z = 0.089.
As self-reported levels can have issues [7], we also analysed the edX learner data to
look at how many learners interacted with each type of activity. In table 7, the
number of learners is listed that engaged at least once with videos, assignments or
the forum. Significant differences were found with learners engaging in larger
numbers than in the 2019 run. In table 8 we compare the extent of the engagement
between the two runs and again we notice a significantly higher engagement with the
course material with regard to videos watched, problems attempted and activity on
the forum in 2020.
Table 7. Learners that engaged at least once with videos, assignments or the forum
Activity (N = 38316)
2019
2020
Total
χ2 (1)
p
Watched at least one video
40.2%
44.1%
42.9%
52.0
<.001
Submitted at least one problem
36.2%
38.1%
37.5%
13.2
<.001
Posted in the forum at least once
10.0%
12.7%
11.9%
61.3
<.001
Table 8. Comparing the extent of engagement with videos, assignments and the forum for
learners that engaged at least once with these activities
Activity
Year
N
Mean
Med
SD
Mann Whitney
U
p
z
# of videos
watched
2019
4813
10.5
4
15
2.69·107
<.001
0.04
2020
11607
12.2
5
16.8
# of problems
attempted - audit
2019
3707
15.4
4
27
1.41·107
.004
0.03
2020
7856
18.3
4
32.3
# of problems
attempted - verified
2019
627
81.7
106
54.9
6.67·105
.407
0.02
2020
2173
80.9
98
55.2
# of posts in the
forum
2019
1193
1.31
1
1.5
1.98·106
.450
0.01
2020
3355
1.34
1
1.63
If we look in more detail at videos, we see that in 2020, significantly more learners
watched at least one video (χ2 (1) = 52.0, p<.001) and they watched significantly
more videos than in 2019: U = 2.69·107, p = <.001 (see Table 8). A similar pattern is
also seen when looking at the interaction with problems of learners that are auditing
the course. In 2020, more learners attempt at least one problem and these learners
attempt to solve more problems on average. This trend is not visible among learners
on the ‘verified’ track, i.e., learners who purchased access to the course. Our data
does not show significant differences between 2020 and 2019 with regards to the
number of problems attempted by the learners that paid for the course (U = 6.67·105,
p=.407). Finally, our data shows that although a significantly higher number of
learners posted on the MOOC forums in 2020 ( ), most learners
did not post more than 1 message on the discussion board.
3.5 Course satisfaction levels
Looking at overall indicators of course satisfaction in the post-course evaluation in
terms of overall course rating, the likelihood of recommending the course and the
ratings learners gave the course for uniqueness, usefulness, being interesting and
difficulty, no significant differences between the two runs were found. This may be in
part that some of these ratings were already very high in the first run.
4 CONCLUSION
Our data showed significant differences between the 2020 “COVID” run of the
MOOC and the 2019 run. It appears the 2020 run not only attracted a larger overall
audience but it also attracted a younger audience, consisting of significantly more
students and significantly more females than the year before and a decrease in the
percentage of people who are working. In absolute numbers however, the aerospace
sector bucks that trend, which is not surprising given the standstill in aviation due to
COVID and the topic of the MOOC.
When looking at the motivation to enrol, we see here that focus in the 2020 run shifts
more towards (prospective) studies than towards (prospective) careers even though
this still makes up for 37% of the motivation to enrol. This is not surprising given the
increase in the number of undergraduate and graduate students enrolling in this
course. We also observed lower course completion rates, even though learners
reported significantly less problems with allocating sufficient time. This may be
indicative of less interest in obtaining a qualification and more interest in interacting
with the course material.
We also see a greater need, value and satisfaction for more interactive course
activities such as Discussion Forums in the MOOC, which is not strange given that
most learners will have been stuck at home with less opportunities for social
interaction elsewhere. Looking in more detail into the learning data of edX confirmed
that during COVID times learners engaged far more with the material than learners
in non-COVID times and seemed to be genuinely interested in gaining more
knowledge about the topic than gaining a qualification.
5 ACKNOWLEDGMENTS
The authors would like to acknowledge Willem van Valkenburg and Nardo de Vries
of the Extension School for their help in providing them with the data.
REFERENCES
[1]
The Cirium Core (2021) Fleets Data, filed 6 April 2021, Cirium, London United
Kingdom. https://www.cirium.com/thoughtcloud/cirium-weekly-covid-19-
updates/ Last accessed 23 April 2021.
[2]
UNESCO, (2020) Education from disruption to discovery,
https://en.unesco.org/covid19/educationresponse#schoolclosures
Last accessed 23 April 2021.
[3]
Saunders-Smits, G.N., Rans, C.D., Teuwen, J.J.E., Sinke, J., Alderliesten,
R.C., and van Valkenburg, W.F., (2020). Teaching Aerospace Structures and
Materials to the World: Analysis of the edX MOOC Introduction to Aerospace
Structures and Materials, AIAA SciTech 2020 Forum: American Institute of
Aeronautics and Astronautics Inc. (AIAA), 10 p. AIAA 2020-0326. 6-10
January 2020, Orlando, Florida. DOI: 10.2514/6.2020-0326
[4]
SoLAR, (2011). 1st International Conference on Learning Analytics and
Knowledge, Banff, Alberta, Canada.
[5]
Liyanagunawardena, T., Adams, A. A., & Williams, S. A. (2013), MOOCs: A
systematic study of the published literature 2008–2012. The International
Review of Research in Open and Distance Learning, 14, 202–227.
[6]
Impey, C., (2020), Massive online open courses see exponential growth
during COVID-19 pandemic. The Conversation.
https://theconversation.com/massive-online-open-courses-see-exponential-
growth-during-covid-19-pandemic-141859 Last accessed 3 May 2021
[7]
Kruger, J. and Dunning, D., (1999), Unskilled and unaware of it: How dif-
ficulties in recognizing one’s own incompetence lead to inflated self-
assessments. J. Pers. Soc. Psychol., vol. 77, no. 6, pp. 1121–1134.
