ArticlePDF Available

Educational Gains of In-Person vs. Distance Learning in Primary and Secondary Schools: A Natural Experiment During the COVID-19 Pandemic School Closures in Switzerland

Authors:

Abstract

Using data from a computer-based formative feedback system, we compare learning gains in the eight weeks of school closures related to the COVID-19 pandemic in Switzerland with learning gains in the eight weeks before these school closures. The school performance in mathematics and language of N = 28,685 pupils is modelled in second-order piecewise latent growth models with strict measurement invariance for the two periods under investigation. While secondary school pupils remain largely unaffected by the school closures in terms of learning gains, for primary school pupils learning slows down, and at the same time interindividual variance in learning gains increases. Distance learning arrangements seem an effective means to substitute for in-person learning, at least in an emergency situation, but not all pupils benefit to the same degree.
COVID-19 SCHOOL CLOSURES - 1
Educational Gains of In-Person vs. Distance Learning in Primary and Secondary
Schools: A Natural Experiment During the COVID-19 Pandemic School Closures
in Switzerland
Martin J. Tomasik
University of Zurich, Switzerland, and University of Witten-Herdecke, Germany
Laura A. Helbling and Urs Moser
University of Zurich, Switzerland
Corresponding Author
Dr Martin J. Tomasik
Institut für Bildungsevaluation, Wilfriedstrasse 15, 8032 Zurich, Switzerland
Tel.: +41 (78) 904 01 36
Email: martin.tomasik@ibe.uzh.ch
Statement of Authors’ Contribution
MT developed the concept of the study, drafted the manuscript and was responsible
for the statistical hypothesis testing; LH prepared the data and supported MT in the
data analyses; UM developed the computer-based formative feedback system,
provided the data, and revised the manuscript.
COVID-19 SCHOOL CLOSURES - 2
Abstract
Using data from a computer-based formative feedback system, we compare learning
gains in the eight weeks of school closures related to the COVID-19 pandemic in
Switzerland with learning gains in the eight weeks before these school closures. The
school performance in mathematics and language of N = 28,685 pupils is modelled in
second-order piecewise latent growth models with strict measurement invariance for
the two periods under investigation. While secondary school pupils remain largely
unaffected by the school closures in terms of learning gains, for primary school
pupils learning slows down, and at the same time interindividual variance in learning
gains increases. Distance learning arrangements seem an effective means to
substitute for in-person learning, at least in an emergency situation, but not all pupils
benefit to the same degree.
Keywords: COVID-19; distance learning; learning progress; school achievement;
school closures
COVID-19 SCHOOL CLOSURES - 3
Educational Gains of In-Person vs. Distance Learning in Primary and Secondary
Schools: A Natural Experiment During the COVID-19 Pandemic School Closures
in Switzerland
In an attempt to contain the spread of the COVID-19 virus, most governments around
the world have temporarily closed schools and other educational institutions,
affecting more than 1.2 billion pupils and students or almost three-quarters of the
learner population (United Nations Sustainable Development Group, 2020). Learners
are probably the single largest group to experience the pandemic’s indirect effects.
Some researchers and organisations (e.g., Burgess & Sievertsen, 2020; Education
Endowment Foundation [EEF], 2020; Kuhfeld et al., 2020) have projected that school
closures during the pandemic could have detrimental effects on learning gains and
social disparities in learning. To the best of our knowledge, there is no empirical
evidence yet on the school closure effect’s actual direction and size. There is mainly
more or less substantiated speculation, as educational researchers, like many other
stakeholders, were unprepared for and overwhelmed by the situation. Few expected
the pace and scope of the pandemic’s development in Spring 2020.
In this paper, we analyse a coincidentally ongoing data collection to provide
timely empirical evidence on the impact of distance learning in schools. If there was
an effect of the school closures on learning and if this effect was sufficiently large to
be reliably measured during the relatively short period of time, this would not only be
relevant on its own or helpful to identifying pupils at-risk that would require special
attention in the case of another school closure potentially to follow. We also know
that educational achievement can have cascading effects into other developmental
domains such as employment or health and affect other developmental outcomes
COVID-19 SCHOOL CLOSURES - 4
such income or civic engagement even years later. Of course, we cannot provide
any evidence on these long-term effects with the data at hand. However, we wanted
to mention them in order to indicate the potentially broad impact of the findings for
the individual and the society as a whole. Because education is correlated with
virtually every psychological trait and because it moderates many psychological
processes, this paper is meant as a more general contribution to the broader
discussion on the psychological effects of the pandemic.
Pandemic Policy on Education in Switzerland
In Switzerland, all educational institutions closed on 16 March 2020 and
reopened again on 11 May 2020. During these eight weeks, schools virtually
overnight switched to distance learning. It is safe to assume that most pupils, parents
and teachers were unprepared for the situation. The school closures were
announced at very short notice on the Friday afternoon before taking effect the
following Monday. School authorities at first did not provide any guidance for parents
and teachers on how to deal with the situation’s challenges, resulting in high
uncertainty on all sides. In the first days, for instance, it was not clear whether pupils
were allowed to pick up their workbooks that they had left in school. By the end of
March, however, school authorities had put together and distributed information on
best practice in distance instruction, and many EdTech companies had made their
distance learning applications available to schools. Moreover, decisions had been
made on how to handle attendance, grading and progression in the second half of
the school year. While the school closures were mandated by federal directive, the
definition of specific regulations and their implementation in school practice were
decided and supervised by the 26 individual cantons. This makes it difficult to
COVID-19 SCHOOL CLOSURES - 5
summarise them in brief. In most cantons, grading was suspended sooner or later,
and special regulations came into effect regarding attendance and progression.
Pupils often received no grades until the end of the school year, even after schools
reopened. Many pupils therefore received only a pro forma school report at the end
of the 2019/20 school year.
Potential Impact of School Closures on Educational Gains
Although the situation leading to the school closures in the second term of the
2019/20 school year and its scope are unprecedented in recent history, studies on
school non-attendance for reasons other than a pandemic may be informative for
estimating the effects of school closures and the rapid transition to distance learning
due to COVID-19. In reviewing these studies, we orientate ourselves by the few
highly topical works that have been published in a remarkably short time: the
projection study by Kuhfeld et al. (2020), the short article by Burgess and Sievertsen
(2020) and the meta-analysis by the EEF (2020). In all three publications, based on
previous research, the authors expect that the school closures will have an impact on
learning gains, although they disagree on its order of magnitude. Notably, none of
these papers analyse data collected during the actual pandemic. Evidence is cited
from studies on seasonal learning and school closures during natural disasters,
comparative studies on instructional time and school absenteeism studies.
Seasonal Learning Studies
There is a long tradition of studies scrutinising the institutional effects on
learning from the learning loss during regular school vacations. Early works on
summer learning loss, summarised and meta-analysed, for instance, by Cooper et al.
(1996) suggest not only that achievement declines to an extent equivalent to one
COVID-19 SCHOOL CLOSURES - 6
month of school learning but also that there are social disparities in this effect that
contribute to a growing heterogeneity in achievement. More recent studies, however,
question not only the summer learning loss but also the social disparity effect (e.g.,
von Hippel & Hamrock, 2019) and argue with the methodological issues related to
scaling, the use of different test forms, the choice between a regressor variable
model or a change score model and the way of modelling measurement error,
asserting that all these issues can jeopardise the valid interpretation of findings from
studies on learning loss. In any case, a large variance of effect sizes is reported, with
estimates ranging from no loss at all up to d = .010 per vacation day. Extrapolating
these figures to the eight weeks or 40 days of school closures during the COVID-19
pandemic would result in learning losses of .000 d .400.
Comparative Studies on Instructional Time
As Burgess and Sievertsen (2020) argue, studies investigating differences in
instructional time on educational outcomes might be informative for estimating the
impact of school closures related to COVID-19. The authors cite as exemplars two
such studies that try to establish causality by investigating dosage-response patterns.
In the study by Carlson et al. (2015), Swedish males took a battery of cognitive tests
in preparation for military service that randomly varied in date and hence in the time
for preparation in school. In this study, just ten days of extra schooling raised the
scores of the recruits on the crystallised intelligence test by d = .010. Lavy (2015)
took another approach by explaining international achievement gaps found in the
Programme for International Student Assessments by differences in schools’
instruction time in the various countries. He found that one more hour per week in
the main subjects increased test scores by about d = .060. Of course, this correlative
COVID-19 SCHOOL CLOSURES - 7
study can only statistically control for all the other differences that exist between the
educational systems in the various countries. Taking these two effect sizes and
extrapolating them to the 40 days of school closures in Switzerland would result in a
learning loss of only d = .040.
Studies on School Absenteeism
Another strand of research relevant in the present context deals with school
absenteeism and compares the learning progress of those who attend school with
the learning progress of those who miss some lessons, hence also applying a
dosage-response paradigm. There are numerous reasons why pupils do not attend
school, including their own illness or lack of access to reliable transportation. Minority
status and low family income are also important correlates for school absenteeism.
Research regularly reports a linear association between the number of days missed
in school and end-of-year test scores, although the range of effect sizes reported
tends to be large. While Aucejo and Romano (2016), Gershenson et al. (2017) and
Liu et al. (2019) consistently report effect sizes of .006 d .008 for each school day
missed, Goodman (2014) finds that one single day of absence reduces the pupils’
mathematics scores by as much as d = .050. Extrapolating this to the 40 days of
school closures during the COVID-19 pandemic would amount to a learning loss of
.240 d 2.000.
School Closures During Natural Disasters
Research on school closures due to natural disasters is less frequent, as these
events rarely occur. Kuhfeld and colleagues (2020) identify three recent studies that
observe the impact of severe weather events on learning. Hansen (2011) found that
each day of school closures due to snow in Colorado reduced achievement by .013
COVID-19 SCHOOL CLOSURES - 8
d .039. Goodman (2014), analysing the effects of school closures due to snow in
Massachusetts, found a comparable effect only for poor schools. The third study by
Sacerdote (2012), which is not directly comparable in terms of the effect size,
investigated learning loss in the aftermath of hurricane Katrina and found an overall
effect of d = .100 due to displacement. Extrapolating the effect sizes of the snow
studies to the 40-day school closures during the COVID-19 pandemic would yield
effect sizes of .510 d 1.560, which is much larger than the overall effect reported
for hurricane Katrina.
Generalisability of Existing Evidence to School Closures During COVID-19
The generalisability of all four types of studies to the situation during the
COVID-19 pandemic is somewhat limited for several reasons. Evidence from
seasonal learning studies has the advantage of relying on numerous studies and
sometimes very large samples. However, school vacations are predictable events
that do not resemble sudden school closures. Parents and their children can prepare
and often spend at least some vacation time together, especially when the children
are still young. Comparative studies on instructional time are useful to estimate the
effect of schooling but hardly control for cultural, curricular and other differences
between single countries, so a number of alternative explanations are possible.
Evidence from studies on school absenteeism probably offers the most direct way of
studying institutional influence on learning, and at the same time large control groups
of pupils attending school are available. However, a high degree of self-selection and
low equivalence between attending and absent pupils on virtually all behavioural,
cognitive and socio-demographic variables threaten the valid interpretation of the
results as effects of schooling. Evidence from studies on school closures during
COVID-19 SCHOOL CLOSURES - 9
natural disasters is probably best suited to generalise to the COVID-19 pandemic
situation because such events usually occur suddenly, allow for little if any
preparation (in contrast to seasonal learning studies), are locally confined, which
limits the influence of cultural factors (in contrast to studies on instructional times),
affect every single pupil and hence reduce self-selection bias (in contrast to studies
on absenteeism). However, fortunately, such disasters are rare as is empirical
evidence from large samples. Furthermore, the educational situation is different, as
no systematic distance learning has so far been implemented as a means to
compensate for in-person learning. All these reasons, together with the
methodological uncertainties and the broad range of extrapolated effect size
estimates (.000 d ≤ 2.000), make it extraordinarily difficult to provide reliable point
estimates for the actual effect of the COVID-19 school closures.
Another substantial factor that significantly limits the comparability of existing
studiesperhaps with the notable exception of some of those related to natural
disastersis that the COVID-19 pandemic brought about a high degree of
psychological uncertainty for parents and their children, confronted many families
with social, emotional and economic strain, and compromised mental health in the
general population (e.g., Serafini et al., 2020). Many working parents had to educate
and care for their children besides meeting their jobs’ old and new demands. Existing
social ties, for instance those to grandparents and other supportive networks, were
disrupted. The risks of income or job loss during one of the major economic
downturns in recent history together with tangible health threats were ubiquitous in
many families. It is well known that such strains immediately translate from the distal
to the proximal developmental contexts of individuals (Tomasik & Silbereisen, 2016)
COVID-19 SCHOOL CLOSURES - 10
and can challenge the functioning of families, particularly those who are most
vulnerable (Elder & Caspi, 1988; Silbereisen & Tomasik, 2011). All these
considerations suggest that the effect of the school closures due to the COVID-19
pandemic could be larger than in previous studies and produce more heterogeneity
in the learning trajectories.
At the same time, learning continued (see Chamberlain et al., in press) and
the institutional impact of schooling was not totally diminished during the COVID-19
school closures. On the contrary, teachers, in collaboration with the school
authorities, quickly implemented digital forms of distance learning. It is possible that
some teachers were incapable, unmotivated or both to develop pedagogically
effective instruction in the digital space (see also Iivari et al., in press). Research from
the United States suggests that a concerning number of teachers lost contact with
their pupils and did not interact with their pupils on a daily basis (Lieberman, 2020).
This is consistent with models of instruction in which the coordination of pupils is the
central challenge that teachers face. However, instruction continued, albeit under
very different and sometimes quite demanding conditions (see also Basilaia &
Kvavadze, 2020).
Present Study
The unique situation of school closures related to the COVID-19 pandemic
enables testing the effects of in-person vs. distance learning in an unselected sample
to estimate the potential loss in learning progress that could have occurred during
this time. Two competing hypotheses can be scrutinised. On the one hand, one
could assume that the lack of institutional schooling results in slower progress or
even a decline in competence. This effect should be particularly pronounced for at-
COVID-19 SCHOOL CLOSURES - 11
risk pupils, such as those from disadvantaged social backgrounds, those with
learning difficulties or very young learners (see Cooper et al., 1996; Lee, 2020). On
the other hand, one could argue that the institutional influence was not totally
diminished and that learning took place in another form and maybe even at the same
pace. Some studies even suggest that distance learning, if implemented well, can
have advantages compared to traditional classes (e.g., Allen et al., 2006), although
other studies report a negative impact of distance education (e.g., Ahn & McEachin,
2017). In the given emergency situation, one would not assume that distance
learning would outperform in-person learning. However, advantages and
disadvantages could be more or less balanced. The current pandemic situation offers
a unique natural experiment to test these competing hypotheses by comparing
learning progress before and during the school closures.
Even if the potentially negative effects of the school closures and the
potentially positive effects of distance learning balance out or if the time intervals for
observing any positive or negative effects are too small, one would nevertheless
expect the variance of educational gains to significantly increase during school
closures compared to regular in-person learning. As the institutional influence of
school decreases, the effect of the family environment tends to increase, introducing
another source of variance in learning. Furthermore, while some teachers and pupils
might have found themselves completely unprepared for online distance learning,
others might have already been using digital tools in their classrooms for some time.
Teachers, on the one hand, not only differed in their experience but also in their
motivation for using distance learning arrangements. Pupils, on the other hand,
differed in their personal characteristics, such as age or self-regulatory learning skills,
COVID-19 SCHOOL CLOSURES - 12
and found themselves in different contexts that made distance learning more or less
effective. While some pupils were very well equipped at home and supported by their
parents, others lacked basic material such as a working personal computer or their
own desk or had to deal with dysfunctional family arrangements. Social background
seems to play a pivotal role for support provided by schools, as surveys conducted
with parents (Andrew at al., 2020) or teachers (Cullinane & Montacute, 2020)
suggest. All these considerations are consistent with surveys from the UK that found,
for instance, a huge variance in the proportion of students getting involved in school
work at home as a function of the level of deprivation (Lucas et al., 2020). Taken
together, one might assume that the variance in learning gains during the eight
weeks of the school closures would be larger than the variance in learning gains
during the eight weeks of regular in-person school attendance (see also van Lancker
& Parolin, 2020).
We also hypothesised that the impact of the school closures would be more
profound for younger compared to older pupils, as their capabilities for self-regulated
distance learning are less developed, they require more cognitive scaffolding during
learning and are probably more severely affected by the socioemotional strains that
the COVID-19 pandemic brought about for families, as previous research on the
effects of macrostructural disruption (e.g., Elder & Caspi, 1988) suggests. Also,
limited physical activity due to restrictions requiring physical distancing, limited
community interactions, as well as sports facilities, playground, and park use (Moore
et al., 2020) might be particularly detrimental for the younger children.
Methods
Participants and Procedure
COVID-19 SCHOOL CLOSURES - 13
All active users of the MINDSTEPS system who completed at least one
teacher-generated assessment between 19 January 2020 and 11 May 2020 were
included in the statistical analyses. MINDSTEPS is a computer-based formative
feedback system developed at the Institute for Educational Evaluation in Zurich and
deployed to all pupils in the cantons of Aargau, Basel-Stadt, Basel-Landschaft and
Solothurn and to many pupils from other German-speaking cantons of Switzerland. It
serves pupils from grade 3 to grade 9 and covers the subjects of mathematics,
German (the instructional language), English and French (the two foreign languages
taught at schools), according to the official school curriculum. The system allows
teachers to set up both linear and adaptive assessments as well as pupils to practice
ad lib in these subjects. For the present analyses, we focused on teacher-generated
assessments only in mathematics and German (in the reading comprehension and
grammar domains) because instruction in these two subjects is most aligned
between the single cantons. There were almost 15,000 items in mathematics and
almost 10,000 items in German that could have been used to set up the single
assessments. Although the system was originally developed to provide formative
feedback to teachers and pupils, it can also be used to obtain ability estimates of
pupils over time. Details of the system’s theoretical rationale can be found in Tomasik
et al. (2018). König et al. (2020) provide a hands-on demonstration of the system’s
capabilities. A total of N = 28,685 pupils (N = 13,134 in primary school and N =
15,551 in secondary school) were considered in the following analyses, although this
sample size cannot be compared to a traditional study design, as the data are
relatively sparse both between domains and over time. The numbers of pupils and
assessments are summarised in Table 1. On average, pupils were M = 9.20 (SD =
COVID-19 SCHOOL CLOSURES - 14
1.69) years old in grade 3, M = 10.20 (SD = 1.46) years old in grade 4, M = 11.21
(SD = 0.82) years old in grade 5, M = 12.20 (SD = 1.55) years old in grade 6, M =
13.17 (SD = 1.93) years old in grade 7, M = 14.49 (SD = .90) years old in grade 8,
and M = 15.39 (SD = 1.63) years old in grade 9. There were 50.3% boys and 30.8%
non-native speakers in primary school as well as 49.1% boys and 32.7% non-native
speakers in secondary school.
We divided the eight weeks before the school closures and the eight weeks
during the school closures into eight intervals of 14 days. For each interval and each
domain, we obtained the WLE ability estimates based on all items completed during
that time. For WLE estimation, we used grade-by-grade one-parameter logistic
models based on probabilistic measurement theory that were then vertically linked
using the Stocking-Lord equation method. All the item parameters needed for this
procedure were previously obtained from a much larger sample collected between
September 2017 and February 2020 and fixed during WLE estimation. Berger et al.
(2019) provide a justification of this procedure together with a validation of the item
parameters obtained on the curriculum contents for mathematics.
Statistical Modelling Approach
School achievement was operationalised with eight latent factors, each
representing one of the eight intervals, four before and four during the school
closures. Within each interval, we used the respective three ability estimates in the
three domains as manifest indicators of latent school achievement. Using these eight
latent factors, we set up a multigroup second-order piecewise latent growth model
(see Isiordia & Ferrer, 2018) with a joint intercept for the two periods under
investigation but with separate estimates for primary school (grades 3 to 6) and
COVID-19 SCHOOL CLOSURES - 15
secondary school (grades 7 to 9) pupils. The two central growth components we
focused on in our analyses were the in-person learning slope and the distance
learning slope.
Before conducting the substantive analyses on these two, we first tested the
measurement invariance properties of the latent factor model incorporating the eight
single intervals. Reverting to a procedure suggested, for instance, by Ferrer et al.
(2008), we started with a model representing configural invariance and subsequently
tested models representing weak, strong and strict invariance across the eight
intervals. Unlike Ferrer and colleagues, however, we parameterised weak invariance
in terms of tau-equivalence by fixing all factor loadings to one. Following Chen (2007;
as cited in Putnick & Bornstein, 2016), the criteria we used for assessing invariance
were ΔCFI < .01, ΔRMSEA < .015 and ΔSRMR < .030 (for weak invariance only) or
ΔSRMR < .015 (for strong and strict invariance). We used this model to statistically
compare the two slopes and their associations with other variables by setting up
equality constraints one by one.
All computations were conducted using the lavaan package for R, considering
the nesting of pupils in school classes, employing the robust maximum likelihood
estimator for parameter estimation and using full information maximum likelihood
(FIML) for dealing with missing data. The decision for the latter, given the sparse data
we faced, was made against the findings reported by Xiao and Bulut (in press), who
found that FIML outperforms all other methods compared under most conditions.
Results
Baseline Model and Measurement Invariance
The baseline model with a common intercept across all eight intervals and
COVID-19 SCHOOL CLOSURES - 16
with two different slopes for either in-person learning or distance learning fit the data
well: χ2(508) = 910.74, χ2/df = 1.79, CFI = .972, RMSEA = .010, 90% CIRMSEA = .009
.011, SRMR = .070 (all indices reported here and hereafter are robust estimates). We
had to set the variance of some first-order factors to zero to prevent the estimation of
an inadmissible solution but otherwise did not encounter any further computational
problems despite the sparse data provided. Although the coverage of the variance-
covariance matrix was very low, with most of the cells representing less than five
percent of all cases, simulation studies suggest the effectiveness of structural
equation modelling techniques under similar conditions (e.g., Willse et al., 2008).
Starting with this configural invariance model, we first established weak
invariance in a tau-equivalent model by fixing all factor loadings to one. In addition,
we needed to fix the variance of the distance learning slope factor to zero in the
primary school group because this component was very close to zero (ψatt < .001,
SEψ = .002, p = .90), which would have resulted in inadmissible solutions in the
following analyses. This weak invariance model did not fit the data significantly worse
than the configural invariance model: ΔCFI = .004, ΔRMSEA = .000, ΔSRMR = .012.
To arrive at a strong invariance model, we first fixed the intercepts within the
two groups and then also between them. With the first set of constraints, one can test
whether invariance is given for each individual group and with the second whether
the measurement property constrained is the same in both groups. Fixing the
intercepts within groups did not significantly deteriorate the model fit compared to
the weak invariance model: ΔCFI = .007, ΔRMSEA = .010, ΔSRMR = .002.
Subsequently, fixing the intercepts between the two groups also did not harm the
model fit: ΔCFI = .001, ΔRMSEA = .000, ΔSRMR = .001.
COVID-19 SCHOOL CLOSURES - 17
Finally, we fixed the error variances to be equal for the respective indicators,
first within and then also between the two groups. In addition, we had to fix two error
variances in the secondary school group to zero because otherwise they would have
been estimated as negative. The within-groups strict invariance model did not fit the
data significantly worse: ΔCFI = .002, ΔRMSEA = .000, ΔSRMR = .003. Compared to
this model, the following between-group strict invariance model deteriorated in terms
of ΔCFI = .011 but not in terms of ΔRMSEA = .001 and ΔSRMR = .004. We retained
this model for further hypothesis testing, as model fit deterioration was only marginal
and, and overall the model fit was satisfactory: χ2(622) = 1317.94 χ2/df = 2.12, CFI =
.947, RMSEA = .012, 90% CIRMSEA = .011.013, SRMR = .092.
Means and Variances of the Learning Slopes
In this strict invariance model, the mean of the learning slope for primary
school pupils was estimated as νinp = .042 (SE ν = .007, p < .001) for in-person
learning and νdis = .018 (SE ν = .004, p < .01) for distance learning. The learning
progress of primary school pupils during in-person learning was more than twice as
high as during school closures, and this difference was highly significant: Δχ2(1) =
8.86 (p < .001). For secondary school pupils, the in-person learning slope was νinp =
.012 (SE ν = .005, p < .05), and the distance learning slope was νdis = .008 (SE ν =
.004, p = .05). These slopes did not differ significantly from each other: Δχ2(1) = 1.01
(p = .31).
For primary school pupils, the variance of the in-person learning slope was
deliberately set to ψinp = .00, and the variance of the distance learning slope was
estimated as ψdis = .009 (SEψ = .001, p < .001), so that the two significantly differed.
For secondary school pupils, the variance of the in-person learning slope was ψinp =
COVID-19 SCHOOL CLOSURES - 18
.002 (SEψ = .001, p = .27), and the variance of the distance learning slope was ψdis =
.002 (SEψ = .001, p < .05). These two did not differ significantly: Δχ2(1) = 0.99 (p =
.32). Clearly, the heterogeneity in learning progress only increased in primary school
pupils and not in secondary school pupils.
Covariances of the Learning Slopes
Because for primary school pupils the in-person learning slope was set to
zero, no covariance between this and other growth components was computed. In
this group, we found a significant negative correlation between the intercept and the
distance learning slope (r = -.27, p < .001). Primary school pupils generally achieving
higher made slower learning progress during the school closures and vice versa.
We did not find this correlation to be significant for secondary school pupils,
for whom the intercept and the distance learning slope were uncorrelated (r = .00, p
= .98). By contrast, we found a strong positive correlation between the intercept and
the in-person learning slope (r = .57, p < .001). The two slopes were not significantly
correlated with each other (r = .23, p = .75).
Discussion
Among the findings reported here, some are particularly noteworthy. First, the
overall quality of our data, as indicated by the many insignificant measurement
invariance tests performed and the appropriate modelling approach chosen as
indicated by the excellent model fits, seems highly suitable for testing differences in
learning progress between in-person and distance learning. The unique design of our
study with unbiased observations before the school closures and continued
observations during this event allowed us to draw strong causal conclusions about
the two different types of learning. However, cautious interpretation is necessary
COVID-19 SCHOOL CLOSURES - 19
because we were not comparing best in-person learning with best practice distance
learning but rather usual in-person learning with ad hoc distance learning rapidly
implemented in an emergency situation of high societal and individual uncertainty.
Nevertheless, the natural experiment setup of this study in combination with large
and unselected samples allows us to draw unique conclusions from the data.
We observed that the heterogeneity in learning progress significantly
increased for primary school pupils during the school closures. In the eight weeks
before the school closures, learning in primary schools took place rather uniformly
and with hardly observable differences between single pupils, but during the school
closures interindividual differences skyrocketed. These findings are compatible with
those of parents’ (Andrew et al., 2020) and teachers’ survey (Cullinane & Montacute,
2020) that were conducted in the UK and found that pupils from the most affluent
households were being offered active assistance (e.g., online tutoring) from their
schools during the lockdown more frequently than pupils from the least affluent
households. Although we have no data on the socio-economic status of pupils
participating in this study, the very same social disparity might explain the growing
heterogeneity in learning outcomes. At the same time, learning slowed down for this
particular group. We found that primary school pupils learned more than twice as fast
attending school in person compared to the distance setup. In contrast, secondary
school pupils were not significantly affected in their learning pace by the school
closures.
From a developmentalist perspective, the increased variance in and the
decreased pace of learning progress in primary school pupils can probably be
explained by cognitive, motivational and socio-emotional factors. The younger the
COVID-19 SCHOOL CLOSURES - 20
pupils, the more they need to rely on cognitive scaffolding during instruction. In
addition, their executive functioning and hence their capabilities for self-regulated
learning might not yet be fully developed. Finally, younger pupils might be
particularly vulnerable to the stress and strains related to the pandemic. Similar age-
differential effects were reported for a completely different situation in the seminal
work of Glen Elder (Elder & Caspi, 1988), who found that while older children gained
in terms of autonomy and competence development during the great recession of
the 1930s, younger children suffered more from the economic hardship of their
families and more often became victims of marital discord or even family violence.
A final noteworthy effect is that the pace of learning during school closures
could not be predicted by the pace of learning during in-person learning. The
correlations between the intercept and the distance learning slope and between the
two slopes were either not significant or negative. This could mean either that two
completely different processes were measured before and during the school
closures (which would be a validity issue) or that the same process was measured
but the situations of in-person and distance learning were so different that learning
progress was driven by different factors before and during school closures (which
would be a substantial finding). Given the strict invariance measurement properties,
the former explanation seems unlikely.
Potential Long-Term Repercussions
Academic achievement can have cascading effects into other domains of life.
While Masten and colleagues (2005) have demonstrated such “developmental
cascades” on internalizing and externalizing problem behaviour, most existing
studies usually focus on academic or achievement-related outcomes such as
COVID-19 SCHOOL CLOSURES - 21
academic self-concept (Guay, Marsh, & Boivin, 2003), the choice of a college major
(Trautwein & Lüdtke, 2007), the choice of an occupation (Heckhausen & Tomasik,
2002), or earnings later in life (Zax & Rees, 2002). Studies based on economic
models are particularly interesting, as they provide a tangible description of the
expected effect size. Taking such economic models as a foundation, Azevedo and
colleagues (2020), for instance, estimated a life-time permanent loss in yearly
earnings ranging from USD 355 to 1,408 as a function of the duration of the school
closures.
Although studies predicting more general, non-achievement related aspects of
development by school achievement are not very common, there are good reasons
to assume a positive association if one considers some basic concepts of human
motivation. From a macro perspective, competence can be considered a
fundamental need “to experience satisfaction in exercising and extending one’s
capabilities” (Levesque et al., 2004, p. 68) and its successful fulfilment is associated
with intrinsic motivation, effective self-regulation, positive social development, and
well-being (Ryan & Deci, 2000). Following this reasoning, Tomasik and colleagues
(2019) have demonstrated that steeper educational gains across compulsory
schooling predicted successful development as indicated by notions of competence,
confidence in oneself, strong character, caring for others, and positive connections
with others (“Five Cs Model”; see Lerner et al., 2015). Notably, these constructs in
turn are predictive for contribution to society (Lerner et al., 2014).
This finding allows us to speculate about long-term repercussions for society
as a whole. Besides the direct impact of the pandemic in terms of death toll or
economic losses, one could almost certainly expect indirect consequences that are
COVID-19 SCHOOL CLOSURES - 22
mediated by slower educational gains and lower academic attainment. These
consequences comprise, but are not limited to, lasting effects on economic growth
and tax revenues, lower job satisfaction, more prevalent health issues, higher crime
rates, and lower cohesion in society. Whether or not the eight weeks of school
closures are sufficient to produce measurable effects has to remain an open question
here.
Theoretical Implications
Silbereisen and Tomasik (2011) argue that circumstances and events that in
Bronfenbrennian terms are located in the macro context of individual development
become psychologically effective only insofar as they are able to translate into the
most proximal micro contexts such as the family or the school. Within these micro
contexts, habits and routines are disturbed and require some form of adaptation. This
is exactly the conceptual blueprint needed to understand why the outbreak of the
COVID-19 pandemic could have had a measurable effect on pupils’ learning.
Against this backdrop, at least three theoretical insights can be gained from
the present analyses. First, the increased heterogeneity in the individual learning
trajectories points to the existence of unobserved factors might have moderated the
impact of the macro level transformation on individual adaptation and development.
These moderating factors might either be conceptualized as “institutional filters” (see
Schoon & Bynner, 2019), which prevent that events at the macro level become
manifest on the more subordinate levels. They might also be conceptualized at the
level of individual or social resources that strengthen the resilience of children and
their families (Tomasik & Silbereisen, 2009). Finally, more or less effective coping
strategies might be responsible for the large variance in developmental outcomes
COVID-19 SCHOOL CLOSURES - 23
(Pinquart & Silbereisen, 2008). Which of these factors and in which combination
were particularly relevant here needs to investigated in future research.
Second, our research demonstrated that macro level events can impact
specific outcomes (here: educational gains) also outside the specific developmental
context that is most proximal to the specific outcome (here: educational institutions
such as schools). We were able to present a case in point for a transmission
mechanism for which already Bronfenbrenner (1979) has coined the term “meso
context”. Although we have no direct evidence, it is plausible to assume that features
in the family context (such as the education of the parents or their occupational
uncertainty during the pandemic) were at least in part responsible for the educational
gains of children when the influence of schools was partly muted. Understanding and
predicting such meso effects is an intriguing endeavour for future psychological and
sociological work.
Third, our data provides convincing evidence that schools effectively attenuate
social disparities in learning, at least in primary school pupils. During in-person
schooling, we were able to observe a rather uniform learning progress, which is a
finding that also seems to generalize across much longer periods of time (e.g.,
Helbling et al., 2019). This is not only good news for educational policy but also might
help understanding how the participation in social contexts more generally shapes
individual developmental trajectories.
Limitations of the Study
Our study has some limitations, of which the questionable generalisability to
other countries and educational settings is probably the least severe. Of course, we
cannot say anything about the effects in other countries that were different in the
COVID-19 SCHOOL CLOSURES - 24
organization of their school system, in their cultural values, in their economic
standing, or in the social and economic impact caused by the pandemic. Therefore,
we refrain from speculating about any potential cross-cultural differences that might
or might not be found in comparative studies, although there is some elaborated
reasoning about the potentially differential impact of the pandemic in low- and
middle-income countries (Zar et al., 2020). Furthermore, data from international
large-scale assessments suggest striking similarities in the factors and mechanisms
associated with school achievement across countries with a quite diverse cultural
and economic background (e.g., Lee, 2014). We do not want to argue for a
universalist interpretation of our findings, but also do not see plausible reasons to
assume that they would be completely different somewhere else (but see Guan et al.,
2020, for a discussion of differential impact of the COVID-19 pandemic on career
development from a cultural psychology perspective).
More problematic is the establishment of causal effects of distance learning,
given the plethora of other factors that simultaneously comprised the situation during
the school closures. The three probably most important to mention are the high level
of strain for the families that could have undermined the teachers’ educational efforts
to provide good instruction, the teachers’ lack of expertise and time for preparation
and the lack of grading during and after the school closures that might have
undermined extrinsic motivation in pupils. These and other factors threaten the
internal validity of interpreting any differences as differences between in-person and
distance learning. At the same time, the external validity of our study could not have
been superior, and the results are informative for understanding both the short- and
long-term causal effects of this specific historical event.
COVID-19 SCHOOL CLOSURES - 25
Another limitation of the present study is that it does not allow empirically
answering the question whether the effect of reduced learning gains will translate
into disadvantageous developmental trajectories within other domains of life in the
short and in the long run. Not only does it not comprise developmental outcomes in
these domains nor does it cover sufficiently long time spans to make substantial
statements about such repercussions. We also cannot draw an any conclusions
about potentially aggravating or compensating factors in the other development
context of these youth.
A final limitation of our study is that it did not cover any transition phase in the
lives of the pupils affected by the school closures. None of our sample was entering
school, transitioning between different school types, or graduating from school,
although it is known that the timing of such events can make a huge difference for
their effect (e.g., Schoon et al., 2002). Research conducted during German
unification in the 1990s, for instance, suggests that those pupils who just graduated
from school during the system change were much worse off as compared to those
who have successfully entered the labour market or who had some “protected time”
while still at school. We are not able to investigate these interesting effects in the
present study because such transitions are not covered with the data available and
because the duration of the school closures was way too short to produce
meaningfully large comparison groups.
Conclusion
Empirical studies that could establish causal relations between the macro and
the micro are extremely rare. For both practical and ethical research reasons, one
has to rely on naturally occurring experimental designs and hence be in the right
COVID-19 SCHOOL CLOSURES - 26
place at the right time to be able to study the links between societal events and
individual adaptation and development. The COVID-19 pandemic provides an
opportunity for such a natural experiment. With MINDSTEPS we were in the right
place at the right time to provide solid evidence-based findings that can inform both
developmental science and educational policy. The main message that these findings
convey is that while older pupils are seemingly able to compensate for school
closures in terms of a sustained learning progress, the effects are dramatically
different for the younger ones. The learning gains of the younger children do not only
slow down, with potential long-term repercussions for future development, but also
become more heterogeneous. While some primary school pupils even seem to profit
from school closures, others’ school performance markedly deteriorates within a very
short period of time. These children are at risk for loosing track in the academic
domain and we may not leave them behind (see also Masonbrink & Hurley, 2020).
COVID-19 SCHOOL CLOSURES - 27
Ethical Compliance Section
The authors have no funding to disclose. All procedures performed in
studies involving human participants were in accordance with the ethical standards
of the institutional research committee and with the 1964 Helsinki Declaration and its
later amendments or comparable ethical standards. The authors declare that they
have no conflict of interest. In line with the American Psychological Association's
Ethical Principles and Code of Conduct, as well as with the Swiss Psychological
Society's Ethical Guidelines, written informed consent from pupils and their parents
was not required because this study was based on the assessment of normal
educational practices and curricula in educational settings. The Institute for
Educational Evaluation as a contractor of the cantonal educational authorities signed
and committed to obeying the laws of the four cantons involved to ensure strict data
confidentiality. In line with the laws of the four cantons approval from an ethics
committee was not required for this study.
COVID-19 SCHOOL CLOSURES - 28
References
Ahn, J., & McEachin, A. (2017). Student enrollment patterns and achievement in
Ohio’s online charter schools. Educational Researcher, 46, 4457.
Andrew, A., Cattan, S., Costa-Dias, M., Farquharson, C., Kraftman, L., Krutikova, S.,
Phimister, A., & Sevilla, A. (2020). Learning during the lockdown: Real-time data
in children’s experiences during home learning. London, United Kingdom:
Institute for Fiscal Studies.
Aucejo, E. M., & Romano, T. F. (2016) Assessing the effect of school days and
absences on test score performance. Economics of Education Review, 55, 70-
87.
Allen, M., Mabry, E., Mattrey, M., Burhis, J., Titsworth, S., & Burrell, N. (2006).
Evaluating the effectiveness of distance learning: A comparison using meta-
analysis. Journal of Communication, 54, 402-420.
Azevedo, J. P., Hasan, A., Goldemberg, D., Iqbal, S. A., & Geven, K. (2020).
Simulating the potential impacts of COVID-19 school closures on schooling and
learning outcomes: A set of global estimates (Policy Research Working Papers
No. 9284). Bruxelles, Belgium: World Bank.
Basilaia, G., & Kvavadze, D. (2020). Transitions to online education in schools during
a SARS-Cov-2 coronavirus (COVID-19) pandemic in Georgia. Pedagogical
Research, 5, em0060.
Berger, S., Verschoor, A. J., Eggen, T. J. H. M., & Moser, U. (2019). Development and
validation of a vertical scale for formative assessment in mathematics. Frontiers
in Education, 4, 103.
Bronfenbrenner, U. (1979). The ecology of human development. Cambridge, MA:
COVID-19 SCHOOL CLOSURES - 29
Harvard University Pres
Burgess, S., & Sievertsen, H. H. (2020, April 1). Schools, skills, and learning: The
impact of COVID-19 on education. VoxEU. https://voxeu.org/article/impact-
covid-19-education
Carlsson, M., Dahl, G. B., Öckert, B., & Rooth, D.-O. (2015). The effect of schooling
on cognitive skills. Review of Economics and Statistics, 97, 533547.
Chamberlain, L., Lacina, J., Bintz, W. P., Jimerson, J., Payne, K., & Zingale, R. (in
press). Literacy in lockdown: Learning and teaching in COVID-19 school
closures. The Reading Teacher.
Chen F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement
invariance. Structural Equation Modeling, 14, 464-504.
Cooper, H., Nye, B., Charlton, K., Lindsay, J., & Greathouse, S. (1996). The effects of
summer vacation on achievement test scores: A narrative and meta-analytic
review. Review of Educational Research, 66, 227-268.
Cullinane, C., & Montacute, R. (2020). School shutdown (COVID-19 and Social
Mobility Brief No. 1). London, United Kingdom: Sutton Trust.
Education Endowment Foundation (2020). Impact of school closures on the
attainment gap: Rapid evidence assessment. London: EEF.
Elder, G. H., Jr., & Caspi, A. (1988). Economic stress in lives: Developmental
perspectives. Journal of Social Issues, 44, 25-45.
Ferrer, E., Balluerka, N., & Widaman, K. F. (2008). Factorial invariance and the
specification of second-order latent growth models. Methodology, 4, 22-46.
Gershenson, S., Jacknowitz, A., & Brannegan, A. (2017). Are student absences worth
the worry in US primary schools? Education Finance and Policy, 12, 137-165.
COVID-19 SCHOOL CLOSURES - 30
Goodman, J. (2014). Flaking out: Student absences and snow days as disruptions of
instructional time (NBER Working Paper No. 20221). Cambridge, MA: National
Bureau of Economic Research.
Guan, Y., Deng, H., & Zhou, X. (2020). Understanding the impact of the COVID-19
pandemic on career development: Insights from cultural psychology. Journal of
Vocational Behavior, 119, 103438.
Guay, F., Marsh, H. W., & Boivin, M. (2003). Academic self-concept and academic
performance: Developmental perspectives on their causal ordering. Journal of
Educational Psychology, 95, 124-136.
Heckhausen, J., & Tomasik, M. J. (2002). Get an apprenticeship before school is out:
How German adolescents adjust vocational aspirations when getting close to a
developmental deadline. Journal of Vocational Behavior, 60, 199-219.
Helbling, L. A., & Tomasik, M. J. (2020). Ambiguity in the social disparities in summer
setback effects: Methodological issues in European seasonal comparative
research. Manuscript submitted for publication.
Helbling, L. A., Tomasik. M. J., & Moser, U. (2019). Long-term trajectories of school
performance in the context of social disparities: Longitudinal findings from
Switzerland. Journal of Educational Psychology, 111, 1284-1299.
Iivari, N., Sharma, S., & Ventä-Olkkonen, L. (in press). Digital transformation of
everyday life: How COVID-19 pandemic transformed the basic education of the
young generation and why information management should care? International
Journal of Information Management.
Isiordia, M., & Ferrer, E. (2016). Curve of factors model: A latent growth modelling
approach for educational research. Educational and Psychological
COVID-19 SCHOOL CLOSURES - 31
Measurement, 78, 203-231.
Kuhfeld, M., Soland, J., Tarasawa, B., Johnson, A., Ruzek, E., & Liu, J. (2020).
Projecting the potential impacts of COVID-19 school closures on academic
achievement (EdWorkingPaper No. 20-226). Providence, RI: Annenberg
Institute.
König, N., Tomasik, M. J., Berger, S., Giesinger, L., Helbling, L. A., & Moser, U. (2020,
March). An adaptive computer-based tool for formative student assessment.
Practitioner demonstration at the 2020 Learning Analytics and Knowledge
Conference, Frankfurt am Main, Germany. https://video01.uni-
frankfurt.de/Mediasite/Showcase/lak2020/Presentation/efea148f6100411ea59e
daf6b13df5a81d
Lavy, V. (2015). Do differences in schools’ instruction time explain international
achievement gaps? Evidence from developed and developing countries.
Economic Journal, 125, F397-F424.
Lee, J. (2014). Universal factors of student achievement in high-performing Eastern
and Western countries. Journal of Educational Psychology, 106, 364-374.
Lee, J. (2020). Mental health effects of school closures during COVID-19. The
Lancet: Child and Adolescent Health, 4, P421.
Lerner, R. M., Lerner, J. V., Bowers, E. P., & Geldhof, G. J. (2015). Positive youth
development and relational-developmental-systems. In R. M. Lerner (Ed.),
Handbook of child psychology and developmental science: Theory and method
(7th ed., Vol. 1, pp. 1-45). Hoboken, NJ: Wiley.
Lerner, R. M., Wang, J., Chase, P. A., Gutierrez, A. S., Harris, E. M., Rubin, R. O., &
Yalin, C. (2014). Using relational developmental systems theory to link program
COVID-19 SCHOOL CLOSURES - 32
goals, activities, and outcomes: The sample case of the 4-H study of positive
youth development. New Directions for Youth Development, 144, 17-28.
Levesque, C., Zuehlke, A. N., Stanek, L. R., & Ryan, R. M. (2004). Autonomy and
competence in German and American university students: A comparative study
based on self-determination theory. Journal of Educational Psychology, 96, 68-
84.
Lieberman, M. (2020, April 17). Taking attendance during coronavirus closures: Is it
even worth it? Retrieved from
https://www.edweek.org/ew/articles/2020/04/17/taking-attendance-is-
trickyduring-coronavirus-closures.html
Liu, J., Lee, M., & Gershenson, S. (2020). The short- and long-run impacts of
secondary school absences (EdWorkingPaper No. 20-125). Providence, RI:
Annenberg Institute.
Lucas, M., Nelson, J., & Sims, D. (2020). Schools’ responses to COVID-19: Pupil
engagement in remote learning. Slough, United Kingdom: National Foundation
for Educational Research
Masonbrink, A. R., & Hurley, E. (2020). Advocating for children during the COVID-19
school closures. Pediatrics, 146, e20201440.
Masten, A. S., Roismann, G. I., Long, J. D., Burt, K. B., Obradović, J. Riley, J. R.,
Boelke-Stennes, K., & Tellegen, A. (2005). Developmental cascades: Linking
academic performance and externalizing and internalizing symptoms over 20
years. Developmental Psychology, 41, 733-746.
Moore, S. A., Faulkner, G., Rhodes, R. E., Brussoni, M., Chulak-Bozzer, T., Ferguson,
L. J., Mitra, R., O’Reilly, N., Spence, J. C., Vanderloo, L. M., & Tremblay, M. S.
COVID-19 SCHOOL CLOSURES - 33
(2020). Impact of the COVID-19 virus outbreak on movement and play
behaviours of Canadian children and youth: A national survey. International
Journal of Behavioral Nutrition and Physical Activity, 17, 85.
Pinquart, M., & Silbereisen, R. K. (2008). Coping with increased uncertainty in the
field of work and family life. International Journal of Stress Management, 15,
209-221.
Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and
reporting: The state of the art and future directions for psychological research.
Developmental Review, 41, 71-90.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of
intrinsic motivation, social development, and well-being. American Psychologist,
55, 68-78.
Schoon, I., & Bynner, J. (2019). Young people and the great recession: Variations in
the school-to-work transition in Europe and the United States. Longitudinal and
Life Course Studies, 10, 153-173.
Schoon, I., Bynner, J., Joshi, H., Parsons, S., Wiggins, R. D., & Sacker, A. (2002). The
influence of context, timing, and duration of risk experiences for the passage
from childhood to midadulthood. Child Development, 73, 14861504.
Serafini, G., Parmigiani, B., Amerio, A., Aguglia, A., Sher, L., & Amore, M. (2020). The
psychological impact of COVID-19 on the mental health in the general
population. QJM: An International Journal of Medicine, 113, 531-537.
Silbereisen, R. K., & Tomasik, M. J. (2011). Psychosocial functioning in the context of
social, economic, and political change. In X. Chen & K. H. Rubin (Eds.),
Socioemotional development in cultural context (pp. 305-331). New York:
COVID-19 SCHOOL CLOSURES - 34
Guilford Press.
Tomasik, M. J., Berger, S., & Moser, U. (2018). On the development of a computer-
based tool for formative student assessment: Epistemological, methodological,
and practical issues. Frontiers in Psychology: Educational Psychology, 9, 2245.
Tomasik, M. J., Napolitano, C. M., & Moser, U. (2019). Trajectories of academic
performance across compulsory schooling and thriving in young aduthood.
Child Development, 90, e745-e762.
Tomasik, M. J., & Silbereisen, R. K. (2009). Demands of social change as a function
of the political context, institutional filters, and psychosocial resources. Social
Indicators Research, 94, 13-28.
Tomasik, M. J., & Silbereisen, R. K. (2016). Demands of social change across
multiple domains of life and across time at the advent of the global financial
crisis. Research in Human Development, 13, 312-327.
Trautwein, U., & Lüdtke, O. (2007). Epistemological beliefs, school achievement, and
college major: A large-scale longitudinal study on the impact of certainty beliefs.
Contemporary Educational Psychology, 32, 348-366.
United Nations Sustainable Development Group (2020). Policy brief: Education
during COVID-19 and beyond. New York: United Nattions.
van Lancker, W., & Parolin, Z. (2020). COVID-19, school closures, and cild poverty: A
social crisis. The Lancet: Public Health, 5, E242-E244.
von Hippel, P. T., & Hamrock, C. (2019). Do test score gaps grow before, during, or
between school years? Measurement artifacts and what we can know in spite of
them? Sociological Science, 6, 43-80.
COVID-19 SCHOOL CLOSURES - 35
Willse, J. T., Goodman, J. T., Allen, N., & Klaric, J. (2008). Using structural equation
modelling to examine group differences in assessment booklet designs with
sparse data. Applied Measurement in Education, 21, 253-272.
Xiao, J., & Bulut, O. (in press). Evaluating the performance of missing data handling
methods in ability estimation from sparse data. Educational and Psychological
Measurement.
Zar, H. J., Dawa, J., Fischer, G. B., & Castro-Rodriguez, J. A. (2020). Challenges of
COVID-19 in children in low- and middle-income countries. Paediatric
Respiratory Reviews, 35, 70-74.
Zax, J. S., & Rees, D. I. (2002). IQ, academic performance, environment, and
earnings. Review of Economics and Statistics, 84, 600-616.
COVID-19 SCHOOL CLOSURES - 36
Table 1. Number of Assessments Completed and Number of Pupils Having
Completed these Assessments.
Assessments
Before
School Closures
During
School Closures
Pupils
Only Before
Before and
During
Only During
Mathematics
13,816
(3,898/9,918)
50,760
(25,044/26,716)
3,225
(983/2,242)
2,928
(1,197/1,731)
13,536
(7,337/6,199)
Reading
5,811
(1,893/3,918)
22,600
(12,401/10,199)
2,350
(620/1,730)
1,340
(577/763)
9,364
(5,268/4,096)
Grammar
10,773
(3,451/7,322)
39,405
(18,931/20,747)
3,036
(1,011/2,025)
2,984
(1,162/1,822)
12,997
(7,149/5,848)
Note: Total number of assessments and total number of pupils in the upper rows;
respective numbers, divided into primary and secondary schools, in the bottom rows
in brackets.
... In Bezug auf den Lernerfolg kamen die meisten hierzu bisher vorliegenden nationalen und internationalen Studien zu dem Ergebnis, dass die Leistungsentwicklung der Schü-ler_innen insgesamt in den durch pandemieeinschränkende Maßnahmen geprägten Schuljahren beeinträchtigt war (z. B. Amplify 2020; Blainey und Hannay 2020;Brzyska et al. 2021;Department for Education 2021;Domingue et al. 2021;Engzell et al. 2021;Juniper Education 2021;Kogan und Lavertu 2021;Kuhfeld et al. 2020;Maldonado und de Witte 2020;Pier et al. 2021;Rose et al. 2021;Schult et al. 2022;Tomasik et al. 2020). Derartige Nachteile zeigten sich aber nicht in allen Studien bzw. ...
... nicht für alle der in den Studien untersuchten Kompetenzdomänen (Depping et al. 2021;Förster et al. 2021;Kuhfeld et al. 2020). Einige Studien deuteten auf Entwicklungsnachteile insbesondere bei jüngeren Schüler_innen hin (Amplify 2020;Juniper Education 2021;Pier et al. 2021;Tomasik et al. 2020). Engzell et al. (2021) berichteten in einer großangelegten niederländischen Studie sogar von Lernverlusten in Mathematik, Lesen und Rechtschreibung in der Primarstufe. ...
Article
Full-text available
In current empirical educational research, one important question is how the societal measures taken to contain the COVID-19 pandemic affected students’ cognitive development. This also raises the question whether disadvantaged student groups suffered particular disadvantages in their cognitive development depending on their individual characteristics or family background. We analyzed a data set of 104 students of three Austrian elementary schools (Grade 3). Students’ basic cognitive abilities were assessed by two subscales (inductive reasoning, numerical thinking) of the Cognitive Abilities Test 1–3 (KFT 1–3; Heller and Geisler 1983). We also analyzed effects of students’ individual characteristics and indicators of their home learning environment. Basic cognitive abilities were assessed at the beginning and end of the 2020/21 school year and thus before and after societal and school-related measures to contain the second and third pandemic waves in Austria (including contact restrictions, school closures, distance learning and quarantine measures). According to intelligence research (Rindermann 2011), an age-appropriate development in students’ mean test performance of about d = 0.40 would have been expected over the nine-months study period. Empirically, however, there were no significant changes in the average and heterogeneity of students’ test performances. At the same time, children from homes with a higher parental educational background and those with more digital devices and internet access showed a significantly higher development. The findings thus indicate that, on average, the children did not develop as positively in the school year characterized by pandemic containment measures as would have been expected. Beyond this, our results indicate cognitive losses among disadvantaged student groups.
... Several studies have shown the impact of COVID-19 closures on education and mental health. It has been estimated that the already existing learning loss for students due to COVID-19 closure could be several months and the achievement gap with students in the same grade whose schools did not close or reopened sooner could increase up to 1.5 years [7][8][9][10]. Schools are attempting to provide education through online courses; however, the effectiveness of online courses depends on the level of students and their Internet access, parental support, and school support [11][12][13]. ...
Article
Full-text available
Even as the COVID-19 pandemic raged worldwide, schools strived to provide consistent education to their students. In such situations, schools require customized schedules that can address the health concerns and safety of the students to safely reopen and remain open. School schedules can be customized in many ways, and different approaches’ impact on education and effectiveness in reducing infectious risks are different. To address this issue, we developed the School Virus Infection Simulation-Model (SVISM) for teachers and education policymakers. By taking into account the students’ lesson schedules, classroom volume, air circulation rates in the classrooms, and infectability of the students, SVISM simulates the spread of infection at a school. We demonstrate the impact of several school schedules in self-contained and departmentalized classrooms and evaluate them in terms of the maximum number of students infected simultaneously, and the percentage of face-to-face lessons. The results show that the impact of increasing the classroom ventilation rate is not as stable as that of customizing school schedules. In addition, school schedules can differently impact the maximum number of students infected simultaneously, depending on whether classrooms are self-contained or departmentalized. We found that the maximum number of students infected simultaneously under a certain schedule with 50 percentage of face-to-face lessons in self-contained classrooms is higher than the maximum number of students infected simultaneously having schedules with a higher percentage of face-to-face lessons; this phenomenon was not found in departmentalized classrooms. These results show that the SVISM can help teachers and education policymakers plan school schedules appropriately to reduce the maximum number of students infected simultaneously, while also maintaining a certain rate of face-to-face lessons.
... During the last decade, the number of students who use online learning environments to study mathematics increased steadily until the outbreak of the COVID-19 pandemic. Then, the use of online learning environments increased steeply due to COVID-19-related school closures (Meeter, 2021;Spitzer & Musslick, 2021;Tomasik et al., 2020;Velde et al., 2021). For instance, the usage (in terms of students enrolled and problem sets computed per month) of the curriculum-based online learning environment Bettermarks increased by about 300% within March 2020. ...
Article
Full-text available
Background Mastering fractions seems among the most critical mathematical skills for students to acquire in school as fraction understanding significantly predicts later mathematic achievements, but also broader academic and vocational prospects. As such, identifying longitudinal predictors of fraction understanding (e.g., mastery of numbers and operations) is highly relevant. However, almost all existing studies identifying more basic numerical skills as predictors of fraction understanding rest on data acquired in face‐to‐face testing—mostly in classrooms. Objectives In this article, we evaluated whether results obtained in these previous studies generalized to data from the curriculum‐based online learning environment Bettermarks for mathematics used in schools in the Netherlands. Methods We considered data from more than 5000 students who solved over 1 million mathematical problem sets on basic mathematical skills, fractions, but also algebra. Results and Conclusions In line with previous findings, we found that fraction understanding was predicted significantly by more basic mathematical skills. Our analyses also indicated that algebra achievement was predicted significantly by fraction understanding beyond influences of more basic mathematical skills. Implications Together, these findings generalized previous results based on face‐to‐face testing to the context of data from online learning environments and thus, indicate that data from such large‐scale online learning environments may well qualify to provide significant insights into the hierarchical development of mathematical skills.
... Most of the distance learning measures introduced in the spring of 2020, during the first school closures, were ineffective in terms of student learning (Hammerstein et al. 2021) and were also a burden for parents (Amuedo-Dorantes et al. 2020). The younger age groups (Andrew et al. 2020;Blainey et al. 2020;Blaskó and Schnepf 2020;Tomasik et al. 2020) and children from low SES families (Bacher-Hicks et al. 2021;Blainey et al. 2020;Engzell et al. 2021;Blaskó and Schnepf 2020;Maldonado and De Witte 2021) were the most negatively affected by school closures associated with COVID-19. This finding is also in line with predictions of widening learning gaps and cumulative learning losses in later school years (Grewenig et al. 2020;Haeck and Lefebvre 2020;Kaffenberger 2021;Pensiero et al. 2020). ...
Article
Full-text available
Previous research has shown that COVID-19-induced school closures and the subsequent transition to online/digital distance education have had a negative effect on student achievement, and this is a negative effect particularly pronounced for students with low socioeconomic status, which foreshadows an increase in educational inequalities. In this study, we examined how students in schools at risk of dropout have adapted to this changed educational situation and what the individual, family and school-related characteristics are that differentiate their adaptation strategy. Our analysis is based on the responses of 3222 Hungarian seventh-grade students to an online survey. Cluster analysis was used to create four groups that illustrate differing perceptions of online/digital distance learning. Descriptive statistical methods were used to analyse and compare the learning patterns of these student groups. Our results show that students have not responded in the same way to changes brought about by COVID-19. There are fundamental differences between the two groups facing difficulties and the two groups experiencing fewer difficulties, but the former and the latter two groups differ on several other factors too. Students with unfavourable individual and family factors were more likely to have learning difficulties. In their case, the extent of support provided by the school is very important.
Chapter
As a consequence of the COVID-19 pandemic, the Swiss Federal Government proclaimed an ‘extraordinary situation’ in March 2020, offering it full power to fight the COVID-19 disease. Part of the economy switched to distance-working with domestic office activities, and so did the entire educational sector. Face-to-face teaching was replaced by distance teaching, computers replaced spontaneous contacts with teachers, and social interaction in school gave way to isolation at home. Parents suddenly had to assume the role of supervisors and teachers. Most restrictions were lifted in May and hopes were high in summer that the nightmare was over. However, further waves of the pandemic arrived in autumn and continued through 2021 and 2022, complicated by the arrival of the Omicron variant. This time, however, schools stayed open and infections were dealt with in a pragmatic way. Since December 2021, children could be vaccinated. The consequences of the pandemic so far were not only technical but also social, particularly inequality and marginality. This chapter, written during the pandemic, reflects on how in our decentralised school system the major actors coped with this new way of learning: authorities, teachers, pupils and parents.KeywordsPandemicCOVID-19EducationSchool systemsSwitzerland
Article
We study the effects of the COVID-19 pandemic on children’s academic performance in Denmark 14 months into the pandemic using nationwide and exceptionally rich data on reading test scores and family background (N ≈ 200,000 per year). We find no evidence of a major learning loss. While pupils in grade 8 experienced a three percentile points loss in reading performance, pupils in grades 2 and 4 experienced a learning gain of about five percentile points, possibly resulting from school closures being significantly longer among older (22 weeks) than younger children (eight weeks). Importantly and in contrast to pre-registered expectations, we find little evidence of widening learning gaps by family background. Further analyses point to that all of these patterns were already in place a few months into pandemic, suggesting that learning gaps did not widen during subsequent, longer school closures. We also find some indication that boys and low-performing pupils suffered more from school closures than girls and high-performing pupils, but these differences are minor. We discuss which political measures may have been instrumental for overcoming the COVID-19 learning slide in Denmark.
Article
In this paper, we consider the response of the education system of the Republic of Serbia to the COVID-19 epidemiological crisis in relation to the teaching process and distance learning activities in primary schools in Serbia. We examine learning through the televised programs of Serbia’s public broadcaster and other related internet services. The aim of this research is to estimate the quality of the teaching process and distance learning in primary schools. In accordance with the purpose of the research, a questionnaire about the educational experience and practices in primary schools during the spring and autumn of 2020 was created. The questionnaire is designed for school principals, teachers, students and parents/legal guardians of students. In total, 28186 people participated in this study conducted in March and April 2021. The results show that most schools used online resources efficiently despite the initially low digital competencies of teachers and students. Limited access to the internet and digital resources, as well as difficulties in engaging students, were gradually overcome during the fall of 2020. Based on the results of this research, we formulated the most important recommendations for the organization of distance learning and its future integration in the hybrid model of teaching and learning in school practice.
Chapter
Im Zentrum des vierten Beiheftes der DDS zum Thema „Schule und Corona“ steht die Frage, wie das von Bund und Ländern beschlossene Aktionsprogramm „Aufholen nach Corona“ in den Ländern bisher umgesetzt wurde. Auf der Grundlage von umfassenden Dokumentenanalysen und Expert*inneninterviews werden die von den Ländern geplanten Förder- und Unterstützungsmaßnahmen beschrieben und vergleichend ausgewertet. Der Aufbau flächendeckender Unterstützungsstrukturen ist für alle Länder mit konzeptionellen und praktischen Herausforderungen verbunden, die zudem noch durch einen chronischen Mangel an pädagogischem Personal massiv verschärft werden. Der Ländervergleich zeigt, wie unterschiedlich die Länder mit diesen Herausforderungen umgehen. Während einige Länder gut an vorhandenen Strukturen anknüpfen konnten, fehlt es in anderen Ländern weitgehend an einem den Zielsetzungen des Aktionsprogramms entsprechenden und in sich stimmigen Konzept. Besonders deutlich wird dies bei der Verteilung der Mittel an die Schulen. Hier steht eine zielgruppenorientierte Mittelzuweisung dem altbekannten Gießkannenprinzip gegenüber. Nichtsdestotrotz zeichnen sich auch wegweisende Ansätze für die kompensatorische Förderung von Schüler*innen ab, die künftig für den Abbau von Bildungsungleichheit auch jenseits von Corona genutzt werden können. Ergänzend werden in einem Beitrag von Armin Himmelrath die Aufholprogramme der EU-Länder in den Blick genommen und deren finanzielles Engagement für den Abbau von Lernrückständen verglichen.
Article
Full-text available
This paper combines novel data on the time use, home‐learning practices and economic circumstances of families with children during the COVID‐19 lockdown with pre‐lockdown data from the UK Time Use Survey to characterise the time use of children and how it changed during lockdown, and to gauge the extent to which changes in time use and learning practices during this period are likely to reinforce the already large gaps in educational attainment between children from poorer and better‐off families. We find considerable heterogeneity in children's learning experiences – amount of time spent learning, activities undertaken during this time and availability of resources to support learning. Concerningly, but perhaps unsurprisingly, this heterogeneity is strongly associated with family income and in some instances more so than before lockdown. Furthermore, our analysis suggests that any impacts of inequalities in time spent learning between poorer and richer children are likely to be compounded by inequalities not only in learning resources available at home, but also in those provided by schools.
Article
Full-text available
Across the globe, students have been away from schools and their teachers, but literacy learning has continued. In many countries, students' literacy proficiency is often measured via high-stakes assessment tests. However, such tests do not make visible students' literacy lives away from formal learning settings, so students are positioned as task responders, rather than as agentive readers and writers. The authors explore the fluidity and diversity of literacy events and practices for students and their teachers observed during the recent period of COVID-19 lockdown restrictions.
Article
Full-text available
As the COVID-19 pandemic upended the 2019–2020 school year, education systems scrambled to meet the needs of students and families with little available data on how school closures may impact learning. In this study, we produced a series of projections of COVID-19-related learning loss based on (a) estimates from absenteeism literature and (b) analyses of summer learning patterns of 5 million students. Under our projections, returning students are expected to start fall 2020 with approximately 63 to 68% of the learning gains in reading and 37 to 50% of the learning gains in mathematics relative to a typical school year. However, we project that losing ground during the school closures was not universal, with the top third of students potentially making gains in reading.
Article
Full-text available
Background: Healthy childhood development is fostered through sufficient physical activity (PA; including time outdoors), limiting sedentary behaviours (SB), and adequate sleep; collectively known as movement behaviours. Though the COVID-19 virus outbreak has changed the daily lives of children and youth, it is unknown to what extent related restrictions may compromise the ability to play and meet movement behaviour recommendations. This secondary data analysis examined the immediate impacts of COVID-19 restrictions on movement and play behaviours in children and youth. Methods: A national sample of Canadian parents (n = 1472) of children (5-11 years) or youth (12-17 years) (54% girls) completed an online survey that assessed immediate changes in child movement and play behaviours during the COVID-19 outbreak. Behaviours included PA and play, SB, and sleep. Family demographics and parental factors that may influence movement behaviours were assessed. Correlations between behaviours and demographic and parental factors were determined. For open-ended questions, word frequency distributions were reported. Results: Only 4.8% (2.8% girls, 6.5% boys) of children and 0.6% (0.8% girls, 0.5% boys) of youth were meeting combined movement behaviour guidelines during COVID-19 restrictions. Children and youth had lower PA levels, less outside time, higher SB (including leisure screen time), and more sleep during the outbreak. Parental encouragement and support, parental engagement in PA, and family dog ownership were positively associated with healthy movement behaviours. Although families spent less time in PA and more time in SB, several parents reported adopting new hobbies or accessing new resources. Conclusions: This study provides evidence of immediate collateral consequences of the COVID-19 outbreak, demonstrating an adverse impact on the movement and play behaviours of Canadian children and youth. These findings can guide efforts to preserve and promote child health during the COVID-19 outbreak and crisis recovery period, and to inform strategies to mitigate potential harm during future pandemics.
Article
Full-text available
Children of today have been surrounded by digital technology since their birth. However, children of today are not equally equipped for their technology rich future: various kinds of digital divides still prevail in the society and affect the young generation and their digital futures. Schools and education of children should undergo an extensive digital transformation to be able to meet the needs of the young generation and their digitalized future. The COVID-19 pandemic has suddenly and abruptly forced schools and education indeed to engage in such a transformation. In this study we examine the digital transformation initiated by the COVID-19 pandemic in the basic education of the young generation, the variety of digital divides emerging and reinforced, and the possible barriers reported along the way. We argue that information management research should better acknowledge children, their digitalized everyday life and their basic education as significant areas of concern. We should understand them as well as allow them to shape the education we offer in the context of higher education, but we should also aim at influencing the basic education of the young generation – for the purpose of equipping them with important skills and competencies for their digital futures but also for the purpose of arousing their interest in this important field, maybe even as a career option.
Article
We provide novel evidence on the causal impacts of student absences in middle and high school on state test scores, course grades, and educational attainment using a rich administrative dataset that tracks the date and class period of each absence. We use two similar but distinct identification strategies that address potential endogeneity due to time-varying student-level shocks by exploiting within-student, between-subject variation in class-specific absences. We also leverage information on the timing of absences to show that absences that occur after the annual window for state standardized testing do not affect test scores, providing a further check of our identification strategy. Both approaches yield similar results. We find that absences in middle and high school harm contemporaneous student achievement and longer-term educational attainment: On average, missing 10 classes reduces math or English Language Arts test scores by 3–4% of a standard deviation and course grades by 17–18% of a standard deviation. 10 total absences across all subjects in 9th grade reduce both the probability of on-time graduation and ever enrolling in college by 2%. Learning loss due to school absences can have profound economic and social consequences.
Article
This paper presents simulations of the potential effect of COVID-19-related school closures on schooling and learning outcomes. It considers four scenarios - varying in both the duration of school closures and the effectiveness of any mitigation strategies being deployed by governments. Using data on 174 countries, the analysis finds that the global level of schooling and learning will fall substantially. School closures could result in a loss of between 0.3 and 1.1 years of schooling adjusted for quality, bringing down the effective years of basic schooling that students achieve during their lifetime from 7.8 years to between 6.7 and 7.5 years. Close to 11 million students from primary up to secondary education could drop out due to the income shock of the pandemic alone. Exclusion and inequality will likely be exacerbated if already marginalized and vulnerable groups, such as girls, ethnic minorities, and persons with disabilities, are more adversely affected by school closures. Students from the current cohort could, on average, face a reduction of $366 to $1,776 in yearly earnings. In present value terms, this amounts to between $6,680 and $32,397 dollars in lost earnings over a typical student's lifetime. Globally, a school shutdown of 5 months could generate learning losses that have a present value of $10 trillion. By this measure, the world could stand to lose as much as 16 percent of the investments that governments make in the basic education of this cohort of students. In the pessimistic and very pessimistic scenarios, cumulative losses could add up to between $16 and $20 trillion in present value terms. Unless drastic remedial action is taken, the world could face a substantial setback in achieving the goal of halving the percentage of learning poor by 2030. © 2021 The Author(s). Published by Oxford University Press on behalf of the International Bank for Reconstruction and Development / THE WORLD BANK.
Article
As the coronavirus pandemic extends to low and middle income countries (LMICs), there are growing concerns about the risk of coronavirus disease (COVID-19) in populations with high prevalence of comorbidities, the impact on health and economies more broadly and the capacity of existing health systems to manage the additional burden of COVID-19. The direct effects of COVID are less of a concern in children, who seem to be largely asymptomatic or to develop mild illness as occurs in high income countries; however children in LMICs constitute a high proportion of the population and may have a high prevalence of risk factors for severe lower respiratory infection such as HIV or malnutrition. Further diversion of resources from child health to address the pandemic among adults may further impact on care for children. Poor living conditions in LMICs including lack of sanitation, running water and overcrowding may facilitate transmission of SARS-CoV-2. The indirect effects of the pandemic on child health are of considerable concern, including increasing poverty levels, disrupted schooling, lack of access to school feeding schemes, reduced access to health facilities and interruptions in vaccination and other child health programs. Further challenges in LMICs include the inability to implement effective public health measures such as social distancing, hand hygiene, timely identification of infected people with self-isolation and universal use of masks. Lack of adequate personal protective equipment, especially N95 masks is a key concern for health care worker protection. While continued schooling is crucial for children in LMICs, provision of safe environments is especially challenging in overcrowded resource constrained schools. The current crisis is a harsh reminder of the global inequity in health in LMICs. The pandemic highlights key challenges to the provision of health in LMICs, but also provides opportunities to strengthen child health broadly in such settings.
Article
As a result of the emergence of coronavirus disease 2019 (COVID-19) outbreak caused by acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection in the Chinese city of Wuhan, a situation of socio-economic crisis and profound psychological distress rapidly occurred worldwide. Various psychological problems and important consequences in terms of mental health including stress, anxiety, depression, frustration, uncertainty during COVID-19 outbreak emerged progressively. This work aimed to comprehensively review the current literature about the impact of COVID-19 infection on the mental health in the general population. The psychological impact of quarantine related to COVID-19 infection has been additionally documented together with the most relevant psychological reactions in the general population related to COVID-19 outbreak. The role of risk and protective factors against the potential to develop psychiatric disorders in vulnerable individuals has been addressed as well. The main implications of the present findings have been discussed.