Learning from errors: students' and instructors' practices, attitudes, and beliefs

Abstract

In some educational contexts, such as during assessments, it is essential to avoid errors. In other contexts, however, generating an error can foster valuable learning opportunities. For instance, generating errors can improve memory for correct answers. In two surveys conducted at three large public universities in North America, we investigated undergraduate students' and instructors' awareness of the pedagogical benefits of generating errors, as well as related practices, attitudes, and beliefs. Surveyed topics included the incorporation of errors into learning activities, opinions about the consequences of studying errors, and approaches to feedback. Many students had an aversion towards making errors during learning and did not use opportunities to engage in errorful generation, yet studied or analysed errors when they occurred. Many instructors had a welcoming attitude towards errors that occur during learning, yet varied in providing students with resources that facilitate errorful generation. Overall, these findings reveal the prevalence of an ambivalent approach to errors: Students and instructors avoid generating errors but prioritise learning from them when they occur. These results have important implications for the implementation of pretesting, productive failure, and other error-focused learning techniques in educational contexts.

Full text

Learning from errors: students’and instructors’practices, attitudes, and beliefs
Steven C. Pan
a
, Faria Sana
b,c
, Joshua Samani
d
, James Cooke
e
and Joseph A. Kim
c
a
Department of Psychology, University of California, Los Angeles, CA, USA;
b
Centre for Social Sciences, Athabasca University, Athabasca, AB,
Canada;
c
Department of Psychology, Neuroscience, & Behaviour, McMaster University, Hamilton, ON, Canada;
d
Department of Physics and
Astronomy, University of California, Los Angeles, CA, USA;
e
Division of Biological Sciences, University of California, San Diego, CA, USA
ABSTRACT
In some educational contexts, such as during assessments, it is essential to avoid errors. In other
contexts, however, generating an error can foster valuable learning opportunities. For instance,
generating errors can improve memory for correct answers. In two surveys conducted at three
large public universities in North America, we investigated undergraduate students’and
instructors’awareness of the pedagogical benefits of generating errors, as well as related
practices, attitudes, and beliefs. Surveyed topics included the incorporation of errors into
learning activities, opinions about the consequences of studying errors, and approaches to
feedback. Many students had an aversion towards making errors during learning and did not
use opportunities to engage in errorful generation, yet studied or analysed errors when they
occurred. Many instructors had a welcoming attitude towards errors that occur during
learning, yet varied in providing students with resources that facilitate errorful generation.
Overall, these findings reveal the prevalence of an ambivalent approach to errors: Students
and instructors avoid generating errors but prioritise learning from them when they occur.
These results have important implications for the implementation of pretesting, productive
failure, and other error-focused learning techniques in educational contexts.
ARTICLE HISTORY
Received 3 July 2020
Accepted 22 August 2020
KEYWORDS
Errorful generation; learning
from errors; pretesting;
prequestions; productive
failure; survey
Making errors and mistakes in assessments and other high-
stakes situations often results in unwanted consequences,
and accordingly most human beings have an aversion
towards doing so. However, a frequently overlooked
benefit of errors is that they can lead to valuable learning
opportunities. In the mid-to-late twentieth century,
Skinner (1953), Bandura (1986), and others (e.g., Ausubel
et al., 1968), believing that generating errors increases
the likelihood of their recurrence (a premise that was, iro-
nically, erroneous), advocated for errorless learning—that
is, entirely eliminating or minimising errors from education
and training situations. In contrast, recent laboratory and
classroom research shows that errorful learning—that is,
generating errors and subsequently receiving correct
answer feedback—can lead to better memory for correct
information than errorless learning (e.g., Bjork et al., 2015;
Kornell et al., 2009). Whether learners and educators
appreciate this updated perspective on the pedagogical
benefits of errors remains unclear. The present manuscript
examines the degree to which undergraduate students
and university instructors embrace learning from errors,
as well as related practices and beliefs.
Generating errors benefits learning
Errors can be defined as facts or processes that do not
match given norms (Oser & Spychiger, 2005; for a
taxonomy, see Reason, 1995), and a growing body of
research indicates that generating them (and then proces-
sing correct answer feedback) can yield substantial learn-
ing benefits (for a review, see Metcalfe, 2017). For
example, generating and/or studying errors can help lear-
ners acquire negative knowledge (Gartmeier et al., 2008;
Minsky, 1997), which is an understanding of incorrect
facts and processes and how they differ from correct
counterparts. That knowledge can be useful in determining
correct information or actions in the future. In some cases,
generating an error can enhance learning relative to not
generating one at all. This rather counterintuitive finding
is supported by studies of pretesting and productive
failure, which are described in turn next.
Studies of pretesting typically have two conditions: pre-
testing and reading. In the pretesting condition, learners
take pretests on information that they have yet to learn,
a process that commonly involves generating numerous
errors (e.g., generating “Sydney”in response to “The
capital city of Australia is ______”). After pretesting, the
correct answers (e.g., “Canberra”) are shown. In the
reading condition, participants simply study correct infor-
mation from the outset (e.g., “The capital city of Australia
is Canberra”) and do not answer any questions or generate
any errors. On a subsequent criterial test, the typical finding
is that pretesting yields better memory for the correct
answers than reading. This pretesting effect or errorful
© 2020 Informa UK Limited, trading as Taylor & Francis Group
CONTACT Steven C. Pan stevencpan@psych.ucla.edu Department of Psychology, UCLA Life Sciences, University of California, 502 Portola Plaza, 1285
Pritzker Hall, Box 951563, Los Angeles, CA 90095-1563, USA
MEMORY
https://doi.org/10.1080/09658211.2020.1815790

generation effect has been replicated with a wide range of
educationally-relevant stimuli, including video-recorded
lectures (e.g., Carpenter & Toftness, 2017), scientific texts
(e.g., Richland et al., 2009), foreign language vocabulary
(e.g., Potts & Shanks, 2014), and facts (e.g., Kornell et al.,
2009), and has also been demonstrated in classrooms
(e.g., Bjork et al., 2015), across different retention intervals
(e.g., Little & Bjork, 2015), and when the correct answer is
shown immediately or up to 24 hrs later (Kornell, 2014).
Although more research on pretesting is needed to fully
establish its pedagogical potential (including to address
the degree to which guesses need to be somewhat
informed (e.g., Kang et al., 2011), the role of associative
strength between pretested cues and targets (e.g., Grimaldi
& Karpicke, 2012; Knight et al., 2012; cf. Metcalfe & Huelser,
2020), as well as the finding that high confidence errors fol-
lowed by feedback yields more learning (Butterfield & Met-
calfe, 2001), which is also known as hypercorrection), and
not all studies have shown benefits of pretesting (e.g.,
Geller et al., 2017), this body of research suggests that stu-
dents often stand to benefit from taking pretests or
attempting practice questions before new course content
is presented (Pan et al., 2020). Such pretesting could
occur before relevant readings, lectures, or discussion sec-
tions are completed, during which the correct answers
could be learned.
In studies of productive failure, learners attempt to
produce solutions to novel problems before receiving
instruction on the correct solution (e.g., Holmes et al.,
2014; Kapur, 2008; Kapur & Rummel, 2012; for a review,
see Kapur, 2015). In these studies, initially attempting and
failing to solve an unfamiliar problem (e.g., a problem
that requires the application of principles from Newtonian
physics), which frequently involves generating erroneous
solutions, often enhances learning from subsequent
instruction and practice relative to being instructed from
the outset. Thus, attempting and failing to solve a
problem can be helpful for learning. Benefits of productive
failure have been demonstrated in such domains as
physics (e.g., Kapur, 2008), statistics (e.g., Loibl & Rummel,
2014), and engineering (e.g., Lai et al., 2017), and also in
classrooms and across extended retention intervals (e.g.,
Trueman, 2014). Although more research into productive
failure is also needed to fully establish its pedagogical
potential, the findings to date suggest that if students
are given the opportunity to solve new problem types
before the correct solutions are presented, then they
may be able to learn those solutions more effectively.
As indicated by the growing literature showing benefits
of pretesting and productive failure, researchers are
increasingly affirming the pedagogical value of generating
errors in educational contexts. More broadly, the finding
that generating errors enhances learning aligns with the
observation that learning techniques that are more error-
prone or challenging, at least initially, can be ultimately
more effective than comparatively error-free and easier
techniques, although more effortful processing may not
always facilitate learning (e.g., Geller et al., 2020; Pan,
Tajran, et al., 2019; Taylor et al., 2020). Bjork (1994)
described such learning techniques (e.g., retrieval practice
and distributing out learning over time) as “desirable
difficulties”(see also Pan & Bjork, in press; Schmidt &
Bjork, 1992). It remains to be investigated, however,
whether learners are open towards more error-prone learn-
ing techniques or instead regard errors as a sign of an
ineffective learning technique.
Prior research on practices and beliefs involving
learning from errors
Several popular learning strategy surveys, including the
Motivated Strategies for Learning Questionnaire (MSLQ;
Pintrich et al., 1991) and the Learning and Study Strategies
Inventory (LASSI; Weinstein & Palmer, 2002), include indi-
vidual items that address learning from errors (e.g., in the
MSLQ, learners rate their level of agreement with the state-
ment: “Even when I do poorly on a test I try to learn from my
mistakes”). However, these surveys do not specifically con-
centrate on learning from errors. Of the studies that have
done so, most target K-12 instructors’teaching practices
and attitudes (for a review, see Matteucci et al., 2015).
Such studies have often used observational methods (i.e.,
video recordings) and have commonly addressed cross-
cultural differences (e.g., Dalehefte et al., 2012; Santagata,
2005; Stigler et al., 1999) or whether instructors promote
a positive error climate wherein errors are accepted and
well-integrated into the social environment (e.g., Steuer
et al., 2013; Tulis, 2013). These studies provide compelling
evidence of cultural variation in discussions of errors, the
frequency of such discussions, and attitudes towards
errors. For instance, among middle school math instructors,
American teachers tend to minimise or deemphasise stu-
dents’errors, Italian teachers tend to be overtly critical of
errors, and Japanese and Chinese teachers often have a
positive attitude towards errors and devote substantial
amounts of time to discussing them with their students
(Santagata, 2005; Stevenson & Stigler, 1994; Stigler et al.,
1999).
A few studies have examined relationships between stu-
dents’or instructors’approaches to errors and academic
outcomes (e.g., Steuer et al., 2013). These studies provide
some evidence that positive error climates are associated
with academic achievement (e.g., Steuer & Dresel, 2015).
For example, Leighton et al. (2018) found that undergradu-
ate students’academic achievement goals were predictive
of their willingness to publicly disclose and discuss the
errors that they had made in their classes.
Recent empirical research suggests that most learners
lack an appreciation for the pedagogical benefits of gen-
erating errors. Huelser and Metcalfe (2012)hadunder-
graduate students learn a series of weakly-associated
word pairs (e.g., bagel - breakfast) via pretesting (e.g.,
attempting to generate the answer to bagel -?,after
which the correct answer, breakfast,wasshownas
2S. C. PAN ET AL.

feedback) or reading (i.e., viewing intact word pairs for an
equivalent period of time), take a recall test, and then rank
therelativeefficacy of the methods that they had used. In
both experiments, there was strong evidence of a pretest-
ing effect, yet most students ranked pretesting as less
effective than reading. The authors proffered two poten-
tial explanations for this metacognitive illusion. First, stu-
dents may have considered the occurrence of errors
during pretesting as evidence of a poor learning tech-
nique (Bjork, 1994). Alternatively, students may have
had a preexisting bias against believing that generating
errors is helpful for learning.
Similarly, Yang et al. (2017) found that adult participants
did not appreciate the benefits of pretesting for learning
word pairs. That pattern was observed when participants
judged the efficacy of pretesting versus reading for learn-
ing individual word pairs (i.e., judgments were solicited at
the item level) and when they made global judgments
(i.e., across all pretested and all read pairs). Directly inform-
ing participants of the pretesting effect, however, did make
them more appreciative of the benefits of errorful gener-
ation, resulting in item-level judgments of learning that
were higher for pretested than read word pairs. Yang
et al. also conducted a brief online survey wherein partici-
pants were asked to imagine using reading and pretesting
to learn word pairs and predict the relative efficacy of the
two methods; reading was judged to be more effective by
a 65-to-35% margin. Building on Huelser and Metcalfe’s
(2012) explanations, Yang et al. suggested that preexisting
beliefs about the pedagogical utility of generating errors
and reading may be a source of these inaccurate assess-
ments. Overall, both studies reveal a disconnect between
the amount of learning that results from generating
errors and learners’beliefs in the pedagogical benefits of
doing so.
The present study
The foregoing work by Huelser and Metcalfe (2012) and
Yang et al. (2017) suggests that many learners are
unaware of and do not appreciate the benefits of learning
from errors, at least in the context of the pretesting effect. If
so, then many students might not prioritise error gener-
ation, studying errors, and/or learning from feedback on
errors in their course preparation and associated activities.
Indeed, some instructors report such patterns in their
courses (e.g., Mason & Singh, 2010), but their prevalence
has yet to be widely investigated. Further, observational
research by Stigler et al. (1999), Santagata (2005), and
others highlights the existence of multiple instructional
approaches to errors in K-12 classes, including differences
in the frequency and manner of relevant discussions. The
relative popularity of these approaches at the university
level, however, remains unexplored. Finally, any explora-
tion of learning from errors occurs against the backdrop
of the historically influential errorless learning approach.
In discussions that informed the development of this
research, some instructors speculated that errorless learn-
ing remains fairly prevalent.
To address these issues, we employed a survey
approach akin to that used by Geller et al. (2018), Kornell
and Bjork (2007), McCabe (2011), Wissman et al. (2012)
and others wherein we directly questioned respondents
about their practices, attitudes, and beliefs. We fielded
two surveys, one for undergraduate students and
another for instructors (similar to Morehead et al., 2016).
Our primary goal was to measure beliefs and attitudes
about learning from errors at the undergraduate level,
including any aversion towards errors or bias against the
belief that generating errors improves learning. We also
investigated related topics, including reactions to errors
when they occur, beliefs about the frequency and
sources of errors, experiences and attitudes towards feed-
back, and the extent to which students and instructors
incorporate errorful learning into their own learning
activities.
Method
Participants
Both the student and instructor surveys were administered
at three large public research universities in Canada and
the United States (McMaster University in Hamilton,
Ontario; University of California, Los Angeles (UCLA) in
Los Angeles, California; and University of California, San
Diego (UCSD) in La Jolla, California) between March and
June 2020. The surveys were approved by each university’s
Institutional Review Board (IRB) and administered online.
Participation in each survey was completely voluntary.
Demographic information for the respondents is listed in
Tables 1 and 2. Combined across sites, the sample sizes
for the student and instructor surveys were 1,052 and
141 respondents, respectively.
Student survey
At McMaster University, the student survey was adminis-
tered as an extra credit opportunity for students in an
introductory psychology class, PSYCH 1X03 (“Introduction
to Psychology, Neuroscience, & Behaviour”), which is a
large-enrollment lower-division course that attracts stu-
dents from a variety of different majors. At UCLA, the
student survey was administered as an extra credit oppor-
tunity in two lower-division courses that are prerequisites
for natural science majors: Physics 1B (“Physics for Scien-
tists and Engineers: Oscillations, Waves, Electric and Mag-
netic Fields”)andPhysics5C(“Physics for Life Sciences
Majors: Electricity, Magnetism, and Modern Physics”). At
UCSD, the student survey was offered as a participation
credit opportunity (worth up to 1% of the course grade)
in an upper-division physiology course, BIPN 134
(“Human Reproduction”). The UCLA and UCSD courses,
which cover foundational materials in their subject
areas, are large-enrollment courses that attract
MEMORY 3

undergraduate students from a wide range of majors and
are regularly taught by the third and fourth authors,
respectively.
The number of respondents per sample was between
161 and 363 (total n=1,052 respondents). As illustrated
in Table 1, the combined sample across all sites was
Table 1. Student survey respondent demographics.
Demographic category Characteristics Combined sample McMaster UCLA (1B) UCLA (5C) UCSD
Sample size
Total (n) 1,052 191 161 337 363
Response rate 96% n/a 85% 97% 100%
Gender
Female 70% 79% 77% n/a 42%
Male 30% 20% 23% n/a 57%
Other 0% 0% 0% n/a <1%
Decline to state <1% 1% 0% n/a 1%
Age
Mean, in yrs 20.0 18.7 18.8 n/a 21.1
Ethnic background
Aboriginal <1% 1% 0% n/a 0%
African American or Black 2% 2% 1% n/a 2%
Asian or Pacific Islander 45% 39% 52% n/a 45%
Caucasian or White 29% 46% 31% n/a 20%
Latinx 11% 1% 6% n/a 19%
Other 12% 12% 9% n/a 13%
Decline to state 1% 1% 2% n/a 1%
Academic major
Business or finance <1% 3% 0% 0% 0%
Clinical sciences 5% 21% 0% 0% 3%
Engineering 8% 4% 46% 0% 0%
Humanities or liberal arts 1% 3% <1% 1% <1%
Mathematics or computing 5% 4% 28% <1% 0%
Natural sciences 67% 35% 20% 78% 95%
Social sciences 11% 23% 1% 20% 2%
Undeclared or decline to state 3% 8% 5% 1% 0%
Note: 1B = Physics 1B, 5C = Physics 5C, and n/a = data not collected or not applicable.
Table 2. Instructor survey respondent demographics.
Demographic category Characteristics Combined sample McMaster UCLA UCSD
Sample size
Total (n) 141 40 59 42
Years of teaching experience
0–5 yrs 23% 5% 31% 31%
6–10 yrs 20% 25% 14% 24%
11–15 yrs 16% 15% 10% 24%
16–20 yrs 11% 18% 5% 12%
Over 21 yrs 30% 38% 41% 10%
Level primarily taught
Undergraduate 74% 68% 71% 83%
Undergraduate and graduate equally 14% 23% 19% 0%
Graduate 11% 10% 8% 17%
Post-graduate 1% 0% 2% 0%
Current position
Professor 27% 25% 44% 5%
Associate professor 11% 33% 2% 2%
Assistant professor 16% 23% 15% 10%
Teaching professor or lecturer 23% 8% 14% 52%
Adjunct professor 4% 3% 7% 2%
Emeritus professor 1% 3% 2% 0%
Postdoctoral scholar 1% 3% 0% 2%
Graduate teaching assistant 16% 5% 15% 26%
Academic advisor or administrator 1% 0% 2% 0%
Subject area
Business or finance 2% 3% 2% 2%
Clinical sciences 2% 8% 0% 0%
Engineering 13% 13% 3% 29%
Humanities or liberal arts 7% 15% 0% 10%
Mathematics or computing 10% 5% 17% 5%
Natural sciences 48% 13% 75% 43%
Social sciences 15% 35% 3% 12%
Decline to state 3% 10% 0% 0%
4S. C. PAN ET AL.

fairly diverse in terms of ethnic background and academic
major.
Instructor survey
At all three institutions, the instructor survey was adver-
tised via university-wide faculty email listservs (sent with
the assistance of a local campus teaching centre), as well
as via departmental email listservs and directories. The
number of respondents per sample was between 40 and
59 instructors (total n=141 respondents). As illustrated in
Table 2, respondents included instructors with a consider-
able range of teaching experience, with most focused on
undergraduate teaching and the most common area of
expertise in the natural sciences. Variation between
samples can be attributed in part to the composition of
the mailing lists and the willingness of different depart-
ments to assist with publicising the survey.
Materials
Both surveys were designed to address three main cat-
egories of interest: (1) learning and teaching practices
regarding errors and mistakes; (2) practices involving the
presentation and use of feedback on errors and mistakes;
and (3) attitudes, beliefs, reactions, and other opinions
regarding errors, mistakes, and feedback. The survey ques-
tions originated from a list of 54 items that the first author
drafted in consultation with the second author. Drawing on
their teaching expertise in their respective subject areas,
the other authors then helped select, refine, and/or add
to those items, resulting in 31 and 16 questions appearing
on the student and instructor surveys, respectively (the
latter kept relatively short per teaching centre requests).
Many of the same or similar questions appeared on both
surveys.
The survey questions were multiple-choice and largely
identical across the different survey sites. We sought to
present questions in a neutral context, including in the
wording and selection of answer options (cf. Tversky & Kah-
neman, 1981). Questions addressing learning practices pri-
marily featured four answer options addressing frequency
(often, sometimes, not very often, and never), whereas ques-
tions on attitudes and beliefs primarily featured four-
option scales of importance or helpfulness (e.g., very
helpful, moderately helpful, minimally helpful, and not at
all helpful)orfive-option scales of positivity or agreement
(including a neutral option). In cases where respondents
could choose between a list of possible actions or
responses, a fill-in-the-blank “Other”option was provided.
Per IRB request, the McMaster University instructor survey
allowed respondents to decline answering any question;
such instances were rare and are not discussed further.
Additionally, at the end of the student survey, participants
were asked to provide responses to the 9-item Multidimen-
sional Perfectionism Scale (Frost et al., 1990) and the 7-item
Attitudinal Cognition Subscale (Leighton et al., 2018); both
scales were added as exploratory measures and the results
are accessible via the Open Science Framework at: https://
osf.io/uycre/.
In line with recommendations by Krosnick and Presser
(2010) and others, questions on both surveys were
grouped by topic and were largely ordered from general
to specific. Further, questions involving learning practices
generally preceded questions involving opinions and
beliefs. A series of demographic questions appeared
before or after the survey questions. To ensure that respon-
dents understood the questions being asked (Kalton &
Schuman, 1982), we defined “errors and mistakes”at the
outset of both surveys using concrete examples (“calculat-
ing an answer incorrectly, recalling incorrect information,
misunderstanding a concept or idea, among other possibili-
ties”). Other jargon terms (e.g., “error rate”) were also
defined or replaced using plain language. The surveys
were further reviewed for comprehensibility by under-
graduate students and instructors prior to their
administration.
Student survey
The student survey consisted of: (a) 7 questions about self-
regulated learning activities, (b) 7 questions involving how
instructors approach errors or feedback, (c) 4 questions
involving hypothetical learning scenarios, and (d) 13 ques-
tions involving attitudes and beliefs. The questions on (a)
addressed how often errors are made during learning;
time spent studying, correcting, or analysing errors and/
or feedback; methods of learning from errors; and time
engaged in specific activities that involve error generation
or pretesting. The questions on (b) drew on prior research
into instructors’approaches to errors in the classroom (e.g.,
Santagata, 2005; Stevenson & Stigler, 1994) and addressed
the frequency of discussions involving errors and mistakes,
the manner in which errors and mistakes are discussed,
and students’perceptions of instructors’attitudes and
reactions. The four scenarios addressed in (c) included pre-
testing (e.g., Kornell et al., 2009), techniques that yield
more or less errors (e.g., Schmidt & Bjork, 1992), changes
in error rates (e.g., Bjork, 1994), and productive failure
(e.g., Kapur, 2008). Questions on (d) addressed attributions
for errors; beliefs about generating, correcting, and study-
ing errors; the importance and optimal timing of feedback;
and degree of endorsement in each of six statements
about the role of errors for learning (cf. Leighton et al.,
2018).
The student survey usually took respondents 15–20 min
to complete. One exception involved Physics 5C students
at UCLA, for which the student survey was the final part
of a larger questionnaire that addressed experiences
specific to the course itself (e.g., whether students had
studied for that course on their own or with a partner,
how much of the assigned readings they had completed,
and what prior relevant courses they had taken); that
entire questionnaire took up to 30 min to complete.
MEMORY 5

Instructor survey
The instructor survey consisted of: (a) 7 questions about
teaching activities and (b) 9 questions focused on attitudes
and beliefs. All but two questions corresponded to those
on the student survey. The exceptions were a question
on discussing the value of learning from errors and
another question on the amount of errors that successful
students tend to make. Further, at the end of the survey,
an optional open-ended question gave instructors the
opportunity to elaborate on their responses to any of the
earlier questions and provide additional comments. The
instructor survey usually took respondents 5–10 min to
complete.
Procedure
Both surveys were accessed online. The instructions
directed respondents to answer each question as honestly
as possible. Students were further told to answer on the
basis of their entire undergraduate experience and instruc-
tors were told to answer on the basis of their entire teach-
ing experience (that instruction may have been especially
pertinent given that classroom instruction at all three insti-
tutions shifted online during the survey period due to the
global coronavirus pandemic). Students were also assured
that their credit was not contingent on any of their answers
and all respondents were told that their answers would not
be disclosed in any publicly identifiable way. Each survey
was completed within a 1-hour time window. The
surveys automatically ended once respondents had
finished responding to all the questions.
Results and discussion
Descriptive statistics for the student and instructor surveys
are presented in Tables 3–8, respectively. For simplicity,
questions have been organised into the tables by type
(e.g., practices, beliefs). Across the samples from McMaster
University, UCLA, and UCSD, the general response patterns
to most questions were similar for the student survey and
for the instructor survey. Accordingly, in our interpretation
and reporting of the data, we focused on results for com-
bined datasets—that is, results for the student survey
that were combined from all sampled sites, as well as
data for the instructor survey that were combined in the
same manner. These results can be found in each table
under the column labelled “Combined sample.”All data-
sets are archived at the Open Science Framework and
accessible at: https://osf.io/uycre/.
Our presentation of the results begins with our findings
for students (from the student survey) followed by our
findings for instructors (drawing on both instructor and
student survey data, as there were relevant questions in
both surveys). For brevity, the order in which the survey
questions are discussed does not exactly match the order
in the tables.
Students’learning practices
How errors during learning are addressed
As detailed in Table 3, the vast majority (83%) of students
report sometimes or often making errors during their own
learning. Efforts to learn from those errors are common:
90% report sometimes or often spending time studying
or analysing their errors. To do so, students most often
use the following techniques: (a) determining the correct
method and contrasting it with what led to the error
(75%), (b) studying the error itself (73%), and (c) studying
feedback on the error (72%). Additionally, 60% report
often going back to correct their errors on their own, and
in a follow-up question addressing the frequency of enga-
ging in the study of feedback when it is provided, 92% indi-
cate sometimes or often doing so. Thus, students
commonly make errors during learning, and when they
do, report often making attempts to correct or study the
errors and/or feedback.
Use of opportunities for errorful learning
Despite often making efforts to learn from errors when
they occur, most students do not engage in errorful gener-
ation as a means of enhancing learning. If and when prac-
tice questions are provided, just 14% often attempt them
before completing relevant readings, lectures, or discus-
sion sections. Afterwards, 69% often attempt them; at
this point, the correct responses are likely to be known,
and although learning from errors could still occur during
such practice (e.g., when retrieval failures occur), the full
benefits of pretesting likely cannot be realised. A similar
pattern is evident for the case of practice questions
found in textbooks: Most students (52%) report never
attempting such questions before doing the relevant
reading, but many report sometimes or often attempting
them during (59%) or after (74%) the reading has already
been performed. These patterns indicate that many stu-
dents do not often use practice questions to engage in
errorful generation and pretesting, possibly because of
an unawareness of the pedagogical benefits of doing so.
Students’attitudes and beliefs
Beliefs regarding errors during learning
As detailed in Table 4, the vast majority of students express
an aversion towards committing errors during learning.
Ninety-one percent believe that it is moderately important
or very important to avoid such errors. Just 2% believe that
avoiding them is not at all important. Fewer students,
however, endorse going to extreme measures to avoid
errors—that is, avoiding them “as much as possible”
(43% somewhat or strongly agree). Further, most students
believe that errors should be considered as somewhat
positive or very positive (79%) from the standpoint of
“being a successful learner,”that making errors is a
normal part of the learning process (78% strongly agree),
that studying errors is moderately helpful or very helpful
6S. C. PAN ET AL.

Table 3. Students’self-regulated learning activities involving errors and feedback.
No. Questions
Choices Combined
sample McMaster
UCLA
(1B)
UCLA
(5C) UCSD
1. When studying or practising for the academic subjects
that you are trying to master, how often do you make
errors or mistakes (such as calculating an answer
incorrectly, misunderstanding a concept or idea, among
other possibilities)?
Often 50% 41% 54% 52% 51%
Sometimes 43% 48% 42% 42% 42%
Not very often 7% 11% 4% 6% 7%
Never 0% 0% 1% 0% 1%
2. In your own learning, how often do you spend time
studying or analysing the errors that you make?
Often 46% 38% 45% 48% 48%
Sometimes 44% 52% 43% 41% 42%
Not very often 10% 10% 9% 11% 9%
Never 1% 1% 2% 0% 1%
3. If you study or analyse the errors or mistakes that you
make, which of the following methods do you use (you
may choose more than one)?
I study my errors or mistakes 73% 72% 76% 70% 75%
I study feedback on my errors or
mistakes
72% 72% 71% 70% 82%
I determine the correct method
and contrast it with what I did
that led to my error
75% 71% 81% 61% 72%
I try to connect my errors or
mistakes with prior mistakes
to find patterns
27% 25% 25% 76% 27%
I try to correct my errors or
mistakes on my own
53% 54% 70% 29% 58%
I go back and study the topics/
skills that I made an error or
mistake in
72% 69% 75% 41% 77%
I seek out instructors or tutors
for help
32% 23% 37% 67% 38%
I seek out peers for help 58% 69% 62% 29% 57%
I try similar exercises or
assignments
65% 65% 70% 52% 66%
I do not specifically try to learn
from my errors or mistakes
1% 1% 1% 62% 1%
4. In your own learning, how often do you go back and
correct the errors or mistakes that you have made?
Often 60% 59% 58% 62% 58%
Sometimes 33% 34% 34% 31% 35%
Not very often 6% 6% 6% 7% 7%
Never 1% 1% 2% 0% 1%
5. If you receive feedback on the errors or mistakes that you
make (that is, are told or find out specifically how many
and which errors you have made), how often do you
spend time studying or analysing that feedback?
Often 51% 40% 55% 52% 53%
Sometimes 41% 49% 37% 39% 40%
Not very often 7% 10% 7% 6% 7%
Never 1% 1% 1% 1% 0%
I do not receive feedback 0% 0% 0% 1% 0%
6. In your undergraduate courses, if and when practice
questions are provided:
a. How often do you attempt to solve them before doing
the relevant assigned reading or attending the relevant
lecture/discussion section?
Often 14% 16% 12% 12% 17%
Sometimes 28% 35% 37% 16% 31%
Not very often 40% 34% 40% 47% 36%
Never 18% 15% 11% 25% 15%
b. How often do you attempt to solve them after doing
the relevant assigned reading or attending the relevant
lecture/discussion section?
Often 69% 54% 68% 80% 67%
Sometimes 24% 32% 25% 17% 28%
Not very often 5% 11% 6% 3% 4%
Never 1% 3% 1% 0% 1%
7. Many textbook chapters have practice questions
associated with them, either interspersed throughout
the chapter or at the end of the chapter. Regarding
those questions:
a. How often do you attempt to answer those questions
before doing the assigned reading?
Often 2% 2% 3% 1% 1%
Sometimes 11% 14% 18% 4% 13%
Not very often 35% 39% 48% 20% 41%
Never 52% 45% 30% 74% 45%
b. How often do you attempt to answer those questions
as you are doing the assigned reading?
Often 20% 16% 22% 21% 22%
Sometimes 39% 42% 42% 31% 43%
Not very often 23% 25% 24% 21% 24%
Never 17% 16% 12% 27% 11%
c. How often do you attempt to answer those questions
after doing the assigned reading?
Often 34% 40% 40% 28% 33%
Sometimes 40% 36% 42% 39% 41%
Not very often 17% 17% 14% 18% 18%
Never 10% 6% 4% 15% 8%
Note: 1B = Physics 1B and 5C = Physics 5C.
MEMORY 7

(96%), and that one learns more from errors than correct
responses (76% somewhat agree or strongly agree).
Eighty percent of students somewhat or strongly disagree
with the possibility that making errors during learning
increases the likelihood of the same errors being com-
mitted again in the future—contrary to the views of learn-
ing theorists that championed the errorless learning
approach. Thus, although the belief in avoiding errors
during learning is widespread among students, that
belief is often accompanied by an awareness of the peda-
gogical value of such errors and, in particular, the benefits
of studying them.
Students also commonly endorse the value of error cor-
rection: 87% consider it very important to go back and
correct errors. Further, if feedback on errors is provided,
56% believe that such feedback would be the most ben-
eficial for learning if it is provided immediately. That prac-
tice has received mixed support in the feedback literature,
however (e.g., Kornell, 2014; Metcalfe et al., 2009; Mullet
et al., 2014; for review see Bangert-Drowns et al., 1991),
with one account suggesting that students pay closer
attention to immediate feedback due to higher levels of
interest (Kulik & Kulik, 1988).
Scenarios involving errorful learning
As detailed in Table 5, when presented with a hypothetical
scenario wherein pretesting or studying could be used to
memorise information, students’opinions on the relative
effectiveness of both methods are somewhat split: 56%
believe that pretesting would be more effective and 44%
believe the reverse. This pattern provides further evidence
that awareness of the benefits of errorful generation is not
widespread among undergraduate students, which is
broadly consistent with results reported by Huelser and
Metcalfe (2012) and Yang et al. (2017) but without as
strong of a bias against the technique.
When asked to estimate the relative efficacy of learning
techniques that yield some errors versus few or no errors at
all, opinions are also split: 53% believe that the former is
more effective whereas 47% believe the reverse. These pat-
terns suggest that awareness of “desirable difficulties”
(Bjork, 1994; wherein better learning techniques are often
more error-prone) is not particularly strong among stu-
dents. Further, when learning something for the first
time, most students prefer a gradual drop (54%) or rapid
drop (39%) in error rate. That finding is consistent with a
desire to reduce errors during learning.
Finally, in a scenario involving learning to solve a chal-
lenging physics problem, students are somewhat split in
their preference for methods that involve some form of
instructional support (i.e., scaffolding), productive failure,
or neither: 29% prefer being instructed on how to solve
the challenging problem from the outset, 45% prefer prac-
tising with simpler problems and transitioning gradually
towards the more challenging version, and 27% prefer
attempting the challenging problem on one’s own before
any instruction is provided. Notably, the latter option is
the most error-prone and, in some instances, possibly the
most beneficial (e.g., Kapur, 2008).
Attributions and reactions to errors during learning
Students commonly attribute errors during learning to a
lack of practice (38%), carelessness (22%), or misconcep-
tions with target materials (19%). Emotional reactions
vary, with the most common including frustration (30%),
disappointment (16%), and motivation to try harder
(12%). Most students also somewhat agree or strongly
agree (65%) that making mistakes makes them feel less
intelligent, whereas opinions are split regarding whether
errors would be reduced if an instructor is “doing a good
job”(29% each agree and disagree). These results
suggest that negative reactions to errors are common,
but not ubiquitous.
Instructors’learning practices
How errors during learning are addressed
As detailed in Table 7, there is substantial evidence that
instructors discuss errors in their courses. Seventy-five
percent of instructors report that they sometimes or often
discuss errors during lectures or discussion sections; 95%
sometimes or often do so during office hours; and 45%
and 66% sometimes or often do so via announcements on
course websites and via online messaging systems, respect-
ively. When discussing errors, instructors state that they
most frequently focus on the misconceptions that lead to
those errors (94%). How to correct errors is the next most
commonly used approach (76%). In addition, 74% of instruc-
tors report that they sometimes or often discuss the benefits
of learning from errors with their students.
The timing of instructor-provided feedback on errors
and mistakes varies substantially. Such feedback most
often occurs later in the same week (33%) or one week
after (40%) an exam or assignment. The three most
common feedback methods are: (a) marking specific
answers as correct or incorrect (84%), (b) giving an
overall score such as percent correct (82%), and (c) provid-
ing correct answers to individual questions (75%). From the
perspective of the feedback literature, however, it is
notable that (a) and (b) are potentially ineffective
whereas (c) is more likely to facilitate learning (e.g.,
Pashler et al., 2005; see also Bangert-Drowns et al., 1991).
Additionally, as detailed in Table 6, student survey data
indicate that 62% of instructors sometimes or often
provide at least some feedback on errors and mistakes,
but a substantial portion, 38%, do not often do so.
Providing opportunities for errorful learning
Instructors could potentially facilitate or encourage stu-
dents to engage in errorful learning by furnishing relevant
resources. As indicated in Table 7, 50% of instructors some-
times or often provide practice questions before relevant
readings, lectures, or discussion sections are completed,
whereas 75% do so after relevant course content has
8S. C. PAN ET AL.

Table 4. Students’attitudes and beliefs towards errors and feedback.
No. Questions
Choices Combined
sample McMaster
UCLA
(1B)
UCLA
(5C) UCSD
1. In your own learning, what is the most common reason for
the errors or mistakes that you make?
Carelessness 22% 25% 27% 26% 14%
Fatigue 8% 12% 4% 10% 5%
Misconceptions about the
materials
19% 15% 12% 22% 20%
Inherent difficulty of the
materials
9% 4% 16% 9% 9%
Lack of practice with the
materials
38% 41% 35% 30% 46%
Information
communicated
ineffectively to me
3% 3% 4% 1% 4%
Other 2% 1% 1% 2% 2%
2. When you are learning something and make an error or
mistake, what is your most common emotional reaction?
Anger 2% 4% 2% 3% 1%
Anxiety 11% 6% 11% 12% 15%
Curiosity 10% 6% 8% 14% 10%
Disappointment 16% 15% 14% 17% 17%
Disgust 0% 0% 1% 1% 0%
Embarrassment 3% 4% 1% 1% 4%
Enthusiasm 0% 0% 0% 0% 0%
Frustration 30% 38% 40% 24% 27%
Happiness 0% 1% 0% 0% 0%
Irritation 9% 7% 8% 12% 6%
Motivation (to try harder) 12% 14% 9% 12% 13%
Sadness 2% 2% 2% 1% 3%
Surprise 1% 2% 2% 1% 1%
Other 2% 2% 3% 0% 3%
3. During the learning of an academic subject (e.g., biology,
chemistry, or physics), how important is it for you to avoid
making errors or mistakes?
Very important 35% 39% 31% 32% 38%
Moderately important 46% 47% 52% 46% 42%
Minimally important 17% 12% 16% 21% 18%
Not at all important 2% 2% 2% 1% 2%
4. From the standpoint of being a successful learner, how
positive (“it’s a good thing”)ornegative (“it’s a bad thing”)
do you believe the making of errors or mistakes should be
regarded?
Very positive 28% 25% 21% 32% 29%
Somewhat positive 51% 55% 52% 45% 54%
Neither positive nor
negative
12% 13% 16% 14% 9%
Somewhat negative 8% 7% 11% 9% 7%
Very negative 1% 1% 1% 1% 1%
5. When learning an academic subject, how helpful do you
believe it is to spend time studying the errors or mistakes
that you have made?
Very helpful 69% 66% 64% 66% 77%
Moderately helpful 27% 29% 32% 30% 21%
Minimally helpful 3% 4% 2% 4% 2%
Not at all helpful 0% 0% 1% 0% 0%
6. When learning an academic subject, how important do you
believe it is to correct the errors or mistakes that you have
made (that is, to go back and modify your responses)?
Very important 87% 83% 82% 88% 91%
Moderately important 12% 14% 17% 11% 9%
Minimally important 1% 3% 1% 1% 0%
Not at all important 0% 1% 1% 0% 0%
7. When do you believe is the best time for feedback to be given
on the errors or mistakes that one has made?
Immediately 56% 61% 52% 49% 61%
Later in the same day 24% 21% 27% 25% 24%
Later in the same week 18% 15% 20% 23% 14%
A week later 2% 3% 0% 3% 1%
Two or more weeks later 0% 0% 0% 0% 0%
8. Rate this statement: “During learning, one should work to
avoid making errors or mistakes as much as possible.”
Strongly agree 10% 8% 17% 13% 7%
Somewhat agree 33% 30% 41% 31% 32%
Neither agree nor disagree 19% 22% 18% 19% 19%
Somewhat disagree 30% 29% 20% 31% 32%
Strongly disagree 8% 10% 4% 5% 10%
9. Rate this statement: “Making errors or mistakes is a normal
part of the learning process.”
Strongly agree 78% 86% 81% 71% 80%
Somewhat agree 19% 11% 18% 25% 17%
Neither agree nor disagree 2% 2% 0% 2% 2%
Somewhat disagree 0% 0% 1% 1% 0%
Strongly disagree 1% 1% 0% 1% 1%
10. Rate this statement: “We learn more from an error or mistake
than we do from a correct response or success.”
Strongly agree 30% 31% 24% 28% 35%
Somewhat agree 46% 46% 43% 46% 48%
Neither agree nor disagree 14% 16% 19% 16% 9%
Somewhat disagree 7% 5% 12% 8% 6%
Strongly disagree 2% 2% 2% 2% 2%
11. Rate this statement: “When an instructor is doing a good job,
students tend to not make errors or mistakes.”
Strongly agree 8% 7% 9% 8% 8%
Somewhat agree 29% 31% 33% 30% 25%
Neither agree nor disagree 28% 34% 20% 26% 30%
Somewhat disagree 29% 24% 29% 29% 31%
Strongly disagree 7% 5% 9% 7% 6%
(Continued)
MEMORY 9

been covered. Thus, instructors provide practice questions
more commonly for practising recall of materials that have
already been learned (at least partially), rather than specifi-
cally for errorful generation. However, practice assign-
ments, assignments that are graded for completion only,
and other activities wherein performance does not
impact the course grade are sometimes or often (74%) pro-
vided. These resources could be used for errorful learning,
but are not necessarily designed specifically for that
purpose.
Instructors’attitudes and beliefs
Beliefs regarding errors during learning
As detailed in Table 8, many instructors strongly agree
(79%) that errors are a normal part of the learning
process, that it is very helpful (67%) for students to
spend time studying the errors that they make on exams
or assignments, and that one learns more from errors
than correct responses (62% somewhat agree or strongly
agree). Further, very few instructors strongly agree that it
is important to avoid errors during learning “as much as
possible”(9%), that students make fewer errors when an
instructor “is doing a good job”(1%), or that successful stu-
dents make fewer mistakes during learning (5%). In
addition, most instructors do not endorse the belief that
making errors during learning increases the likelihood of
the same errors being committed again in the future
(77% somewhat disagree or strongly disagree), which
mirrors patterns observed in the student data and further
suggests that errorless learning has fallen out of favour.
Overall, these results indicate that most instructors are
open to students making errors during learning, believe
that such errors are not necessarily a sign of poor instruc-
tion, and do not regard such errors as detrimental for
future performance.
With respect to how feedback on students’errors and
mistakes should be timed to help learning, many
Table 4. Continued.
No. Questions
Choices Combined
sample McMaster
UCLA
(1B)
UCLA
(5C) UCSD
12. Rate this statement: “Making errors or mistakes during
learning increases the likelihood that one will make the
same errors at a later point.”
Strongly agree 2% 3% 2% 2% 2%
Somewhat agree 9% 7% 12% 11% 8%
Neither agree nor disagree 8% 11% 8% 6% 9%
Somewhat disagree 47% 51% 44% 44% 48%
Strongly disagree 33% 28% 34% 37% 32%
13. Rate this statement: “When I make a mistake it makes me feel
less intelligent.”
Strongly agree 17% 17% 16% 18% 17%
Somewhat agree 48% 51% 49% 45% 48%
Neither agree nor disagree 18% 19% 15% 19% 18%
Somewhat disagree 12% 9% 17% 12% 12%
Strongly disagree 5% 3% 4% 5% 5%
Note: 1B = Physics 1B and 5C = Physics 5C.
Table 5. Students’views of learning scenarios involving errors and feedback.
No. Questions
Choices Combined
sample McMaster
UCLA
(1B)
UCLA
(5C) UCSD
1. If your goal is to memorise the answers to a set of
questions on an academic subject (e.g., biology,
chemistry, or physics), which method would be
more effective?
First trying to guess the answers (and
possibly making many incorrect
guesses), then studying the correct
answers
56% 51% 54% 63% 53%
Studying the correct answers from the
outset
44% 49% 46% 37% 47%
2. When used for studying or practising, some
learning techniques result in more errors and
mistakes than others. Which is more effective for
learning?
Learning techniques that yield some
errors during studying and practising
53% 53% 49% 60% 49%
Learning techniques that yield few or
no errors during studying and
practising
47% 47% 51% 40% 51%
3. When you are learning for the first time,
sometimes errors or mistakes are unavoidable.
Which of the following error rates (i.e., the
fraction of problems that I make an error on) is
better for your learning?
Error rate rapidly drops 39% 28% 43% 43% 43%
Error rate gradually drops 54% 63% 52% 50% 50%
Error rate gradually rises 3% 3% 2% 3% 3%
Error rate quickly rises 0% 0% 0% 0% 0%
Error rate remains stable 4% 6% 2% 4% 4%
4. When you are learning a difficult skill, such as how
to solve a challenging physics problem, which of
the following learning methods would you
prefer?
From the outset, having the instructor
walk you through how to solve the
problem correctly
29% 32% 24% 31% 27%
First practising with simpler versions of
a problem and then working
gradually up to the challenging
version
45% 39% 47% 45% 47%
First trying to solve challenging
problems on your own, and then
having the instructor show you how
to do so
27% 29% 29% 24% 26%
Note: 1B = Physics 1B and 5C = Physics 5C.
10 S. C. PAN ET AL.

instructors believe that feedback on assignments should
occur either immediately (37%) or later in the same week
(36%), whereas feedback on exams is most commonly
thought to be beneficial when it occurs later in the same
week (45%). As previously noted, research on the optimal
timing of feedback is mixed.
Attitudes and reactions to errors during learning
Instructors commonly regard errors that occur during
learning as somewhat positive or very positive (75%),
with far fewer expressing a somewhat negative or very
negative evaluation (4%). These results are largely substan-
tiated by the student survey data, with students reporting
Table 6. Students’instructional experiences involving errors and feedback.
No. Questions
Choices Combined
sample McMaster
UCLA
(1B)
UCLA
(5C) UCSD
1. In your courses, how often do your instructors (i.e.,
professors, TAs) spend time discussing the errors or
mistakes that students make:
a. During lectures? Often 4% 3% 1% 5% 4%
Sometimes 33% 37% 23% 38% 31%
Not very often 54% 51% 61% 47% 58%
Never 10% 9% 15% 10% 7%
b. During discussion sections? Often 14% 6% 6% 19% 16%
Sometimes 50% 47% 50% 52% 51%
Not very often 31% 41% 36% 25% 29%
Never 5% 6% 8% 4% 4%
c. During office hours? Often 34% 40% 29% 36% 31%
Sometimes 45% 39% 55% 45% 44%
Not very often 16% 15% 14% 13% 21%
Never 5% 7% 2% 6% 4%
2. If and when your instructors discuss errors or
mistakes that students make, which of the
following do they commonly focus on?
How to correct those errors 23% 33% 23% 18% 22%
How to avoid those errors 15% 17% 17% 13% 14%
The misconceptions that lead to
those errors
60% 48% 55% 67% 61%
Other 1% 1% 1% 0% 1%
My instructors do not discuss errors
or mistakes
3% 2% 4% 2% 3%
3. Which of the following best describes the typical
approach that your instructors take towards errors
or mistakes?
At the moment when a student
makes an error or mistake, they
will discuss it
18% 16% 24% 19% 15%
At the moment when a student
makes an error or mistake, they
will tend to ignore it
2% 2% 2% 2% 1%
They will specifically bring up
potential errors or mistakes that
students may make
30% 30% 39% 33% 23%
They will specifically bring up errors
or mistakes that students have
made
49% 51% 34% 46% 58%
Other 1% 1% 2% 0% 3%
4. In your courses, how often do you receive feedback
from your instructors on the errors or mistakes that
you have made?
Often 19% 21% 17% 21% 18%
Sometimes 43% 43% 44% 44% 42%
Not very often 35% 32% 34% 33% 38%
Never 3% 3% 5% 2% 2%
5. When does that feedback, if any, usually occur? Immediately 14% 22% 10% 12% 13%
Later in the same day 13% 16% 14% 12% 11%
Later in the same week 26% 19% 36% 24% 28%
A week later 25% 20% 20% 27% 27%
Two or more weeks later 13% 12% 9% 17% 12%
I usually do not receive feedback 9% 11% 11% 8% 9%
6. In general, how positive (“it’s a good thing”)or
negative (“it’s a bad thing”) would you describe
your instructors’attitudes towards the errors or
mistakes that students make during the learning of
new materials, skills or topics?
Very positive 20% 24% 14% 22% 17%
Somewhat positive 38% 34% 41% 35% 42%
Neither positive nor negative 32% 33% 34% 32% 30%
Somewhat negative 10% 9% 9% 9% 11%
Very negative 1% 1% 1% 1% 1%
7. What would you describe is your instructors’most
common emotional reaction towards the errors or
mistakes that students make?
Anger 0% 0% 0% 0% 1%
Curiosity 33% 31% 34% 34% 33%
Disappointment 14% 17% 8% 11% 17%
Enthusiasm 21% 21% 25% 26% 13%
Frustration 5% 8% 3% 4% 5%
Happiness 2% 2% 0% 3% 2%
Irritation 5% 4% 4% 4% 8%
Sadness 1% 3% 1% 1% 1%
Surprise 8% 6% 6% 6% 13%
Other 11% 8% 19% 0% 8%
Note: 1B = Physics 1B and 5C = Physics 5C.
MEMORY 11

that their instructors commonly have somewhat positive or
very positive attitudes to errors that occur during learning
(58%), with the most common emotional reactions includ-
ing curiosity (33%) and enthusiasm (21%), but also disap-
pointment (14%). Overall, these results are consistent
with the finding that many instructors have a generally
welcoming approach to errors that occur during learning.
Instructors’open-ended comments
Twenty-seven instructors answered the optional open-
ended question at the end of the instructor survey. The
most common comments reflected individual beliefs
about learning from errors (e.g., “Making mistakes is never
the goal, but when it happens, at least make use of it”;
“Obviously you should try to avoid mistakes. But when mis-
takes happen, they can be really helpful to study”). Several
respondents commented on logistics (e.g., the resources
that would be required to provide immediate feedback).
Other comments focused on individual teaching practices
(e.g., varying feedback methods depending on assignment
type) and limitations of the survey questions (e.g., pointing
out cases wherein there is not one correct answer).
General discussion
We investigated undergraduate students’and instructors’
practices, attitudes, and beliefs in regard to learning from
errors. Across both surveys, a host of intriguing findings
emerged, two of which are especially salient. First, students
and instructors often avoid opportunities for errorful gen-
eration. That is, most students do not use pretests and
half of surveyed instructors do not provide practice ques-
tions in advance of relevant course content. These
findings contrast with the greater adoption of other evi-
dence-based learning techniques such as retrieval practice
(Kornell & Bjork, 2007) and distributed practice (e.g., More-
head et al., 2016), although the pedagogical benefits of
those techniques are often not fully recognised by their
users either (e.g., Hartwig & Dunlosky, 2012). Second,
both students and instructors acknowledge the value of
errors when they are committed. Students often attempt
to correct their errors and make efforts to learn from
them in several ways (e.g., comparing erroneous and
correct methods, analysing errors, and studying feedback),
whereas instructors commonly discuss students’errors in
lectures, discussion sections, office hours, and other
venues. When errors do occur, they are usually not ignored.
Interestingly, both students and instructors believe that
committing an error does not irrevocably increase the like-
lihood of its recurrence, which is a critical but flawed
assumption on the part of prominent twentieth century
learning theorists (e.g., Bandura, 1986; Skinner, 1953) that
informed the errorless learning approach. That result
suggests that errorless learning is not as influential as it
once was, at least at the undergraduate level. Further, it
appears that the avoidance of errorful generation is
unrelated to a fear of remembering misinformation.
Rather, errors may simply make the learning process
disfluent (for related discussion see Bjork et al., 2013). Stu-
dents’associations of errors with undesirable outcomes,
including negative emotional states and reduced apprai-
sals of their own intelligence, may also contribute to their
avoidance of errorful generation (and may contribute to
a preference for techniques that do not involve making
errors, as shown in the Huelser & Metcalfe, 2012; and
Yang et al., 2017 studies).
Together, the present findings reveal the prevalence of
an arguably ambivalent or conditional approach to learn-
ing from errors among undergraduate students and
instructors. Under this approach, the deliberate generation
of errors is rare. However, if and when errors do occur,
efforts are made to learn from them. As discussed next,
there are compelling reasons to expect that this approach
is popular.
Accounting for students’and instructors’
approaches to learning from errors
The student survey results appear to stem from the fact
that instructors commonly evaluate learning via course
grades (McMorran et al., 2017). Accordingly, most students’
primary objective is to learn course content to a level that
will allow them to obtain a desired grade. To achieve that
objective, errors must be avoided, and especially on high-
stakes exams and graded assignments. Consequently, stu-
dents quickly develop an aversion to errors. Crucially, our
data indicate that this aversion to errors is pervasive—
that is, it extends beyond situations wherein errors are
costliest and encompasses the learning process itself.
By this account, students consider errors that occur
during learning, which they often attribute to insufficient
or poor preparation and have negative emotional reactions
towards, as indicators of suboptimal performance. Accord-
ingly, errors are undesirable. However, learning from errors
when they occur is valued insofar as such learning may
help prevent the recurrence of errors in the future. Thus,
the study or analysis of errors and feedback on errors, as
well as error correction, is prioritised because those prac-
tices serve as preventative measures. Importantly, that
prioritisation can manifest without any awareness of the
capacity of errorful generation to enhance learning and
memory (i.e., that deliberately making errors can have ped-
agogical benefits). All that is required is an understanding
that errors can serve as a reference for actions or responses
to avoid in the future (Gartmeier et al., 2008).
A similar account can be applied to the instructor survey
results. Instructors aim to impart accurate knowledge and
commonly make efforts to help their students perform
well in their courses. Accordingly, instructors treat errors
that occur during learning as welcome developments
(i.e., they often foster a positive error climate) insofar as
those errors provide opportunities for error correction,
enable students to acquire negative knowledge, help
12 S. C. PAN ET AL.

Table 7. Instructors’teaching activities involving errors and feedback.
No. Questions
Choices Combined
sample McMaster UCLA UCSD
1. On average, how often do you spend instructional time
discussing, with your students, the errors and mistakes that
they make:
a. During lectures or discussion sections? Often 25% 23% 15% 40%
Sometimes 50% 53% 53% 43%
Not very often 25% 25% 31% 17%
Never 1% 0% 2% 0%
b. During office hours? Often 70% 55% 76% 74%
Sometimes 25% 40% 17% 21%
Not very often 4% 3% 5% 2%
Never 2% 3% 2% 2%
c. Via online postings on a course website? Often 12% 13% 3% 24%
Sometimes 33% 38% 27% 38%
Not very often 31% 23% 46% 19%
Never 23% 28% 24% 19%
d. Via online messaging such as email, chat, or a discussion
forum?
Often 23% 15% 22% 33%
Sometimes 43% 45% 41% 43%
Not very often 25% 33% 27% 14%
Never 9% 5% 10% 10%
2. If and when you discuss with students the errors or mistakes that
they make, which of the following do you do (please select all
that apply)?
How to correct those errors 76% 78% 70% 84%
How to avoid those errors 63% 56% 58% 72%
The misconceptions that lead to
those errors
94% 98% 90% 93%
Other 7% 7% 5% 9%
I do not discuss errors or mistakes 0% 0% 0% 0%
3. In your teaching, how often do you provide practice questions
that students can attempt:
a. Before relevant assigned readings or lectures? Often 27% 30% 22% 31%
Sometimes 23% 30% 15% 29%
Not very often 28% 25% 32% 24%
Never 22% 15% 31% 17%
b. After relevant assigned readings or lectures? Often 46% 40% 51% 45%
Sometimes 29% 35% 24% 31%
Not very often 16% 18% 15% 14%
Never 9% 8% 10% 10%
4. Besides assignments and exams wherein student performance
counts towards course grades, how often do you provide
students with the opportunity to make errors (or study them)
without their performance impacting their course grade?
Examples include practice assignments, assignments that are
graded for completion only, and others.
Often 41% 38% 42% 43%
Sometimes 33% 30% 31% 38%
Not very often 20% 23% 20% 17%
Never 6% 10% 7% 2%
5. On exams and assignments, feedback (that is, how many errors
were made, which errors were made, and/or what the correct
answers were) can be given. Regarding feedback and its
placement after an exam or an assignment:
a. Do you provide feedback for assignments, and if so, when? Immediately 13% 10% 12% 19%
Later in the same day 3% 0% 0% 10%
Later in the same week 33% 28% 34% 36%
A week later 40% 38% 46% 36%
Two or more weeks later 8% 20% 5% 0%
No feedback at all 2% 3% 3% 0%
b. Do you provide feedback for exams, and if so, when? Immediately 4% 8% 0% 7%
Later in the same day 6% 0% 5% 14%
Later in the same week 34% 25% 36% 40%
A week later 40% 30% 54% 31%
Two or more weeks later 6% 13% 5% 2%
No feedback at all 6% 18% 0% 5%
6. If and when you provide feedback, what forms do you commonly
provide (please select all that apply)?
Providing an overall score, such as
percent correct
82% 80% 76% 88%
Marking specific answers as correct
or incorrect
84% 76% 83% 88%
Providing correct answers to specific
questions, (e.g., via an answer key)
75% 54% 82% 84%
Providing explanations of correct or
incorrect answers
75% 80% 72% 70%
Other 9% 2% 8% 16%
7. In your teaching, how often, if at all, do you discuss the potential
value of learning from one’s errors and mistakes?
Often 31% 33% 24% 40%
Sometimes 43% 40% 42% 45%
Not very often 21% 25% 25% 12%
Never 5% 3% 8% 2%
MEMORY 13

Table 8. Instructors’attitudes and beliefs towards errors and feedback.
No. Questions
Choices Combined
sample McMaster UCLA UCSD
1. From the standpoint of being a successful learner, how positive (“it’s a good
thing”)ornegative (“it’s a bad thing”) do you believe that students’making of
errors or mistakes (as they are learning new materials or skills) should be
regarded?
Very positive 32% 23% 36% 36%
Somewhat positive 43% 43% 36% 52%
Neither positive nor
negative
22% 35% 22% 10%
Somewhat negative 3% 0% 7% 0%
Very negative 1% 0% 0% 2%
2. How helpful do you believe it is for your students to spend time studying the
errors or mistakes that they make on exams and/or assignments?
Very helpful 67% 45% 76% 76%
Moderately helpful 28% 48% 22% 19%
Minimally helpful 2% 3% 2% 2%
Not at all helpful 2% 5% 0% 2%
3. Regarding feedback on students’errors and mistakes and their placement after
an exam or an assignment:
a. From the standpoint of helping students’learning, when is the best time to
provide feedback for assignments?
Immediately 37% 38% 39% 33%
Later in the same
day
16% 18% 14% 17%
Later in the same
week
36% 20% 41% 45%
A week later 11% 25% 7% 5%
Two or more weeks
later
0% 0% 0% 0%
No feedback at all 0% 0% 0% 0%
b. From the standpoint of helping students’learning, when is the best time to
provide feedback for exams?
Immediately 23% 28% 20% 21%
Later in the same
day
16% 18% 14% 17%
Later in the same
week
45% 23% 58% 50%
A week later 11% 15% 8% 10%
Two or more weeks
later
0% 0% 0% 0%
No feedback at all 2% 5% 0% 2%
4. Rate this statement: “Making errors or mistakes is a normal part of the learning
process.”
Strongly agree 79% 73% 75% 90%
Somewhat agree 16% 23% 19% 5%
Neither agree nor
disagree
1% 0% 2% 0%
Somewhat disagree 0% 0% 0% 0%
Strongly disagree 5% 5% 5% 5%
5. Rate this statement: “We learn more from an error or mistake than we do from a
correct response or success.”
Strongly agree 29% 20% 29% 38%
Somewhat agree 33% 35% 31% 36%
Neither agree nor
disagree
28% 28% 32% 24%
Somewhat disagree 6% 13% 7% 0%
Strongly disagree 2% 3% 2% 2%
6. Rate this statement: “Making errors or mistakes during learning increases the
likelihood that one will make the same errors at a later point.”
Strongly agree 4% 5% 0% 7%
Somewhat agree 9% 8% 14% 5%
Neither agree nor
disagree
10% 8% 12% 10%
Somewhat disagree 37% 43% 36% 33%
Strongly disagree 40% 38% 39% 45%
7. Rate this statement: “During learning, one should work to avoid making errors
or mistakes as much as possible.”
Strongly agree 9% 10% 10% 7%
Somewhat agree 22% 13% 31% 19%
Neither agree nor
disagree
20% 23% 22% 14%
Somewhat disagree 26% 35% 17% 29%
Strongly disagree 23% 20% 20% 31%
8. Rate this statement: “When an instructor is doing a good job, students tend to
not make errors or mistakes.”
Strongly agree 1% 0% 0% 2%
Somewhat agree 11% 15% 12% 7%
Neither agree nor
disagree
25% 33% 25% 17%
Somewhat disagree 37% 38% 37% 36%
Strongly disagree 26% 15% 25% 38%
9. Rate this statement: “Successful students make fewer mistakes during learning
than others.”
Strongly agree 5% 3% 7% 5%
Somewhat agree 24% 20% 34% 14%
Neither agree nor
disagree
23% 30% 25% 14%
Somewhat disagree 31% 30% 27% 38%
Strongly disagree 16% 18% 7% 29%
14 S. C. PAN ET AL.

identify content that students are struggling with in the
course, and serve as opportunities to obtain feedback on
their own teaching. As with students, this approach can
manifest without any awareness of the benefits of errorful
generation for learning and memory.
More broadly, students’and instructors’typical
approach to learning from errors is analogous to that
which is commonly observed for retrieval practice. Stu-
dents and instructors often use practice tests and value
them for their assessment purposes, but overlook their
ability to enhance learning (Hartwig & Dunlosky, 2012;
Kornell & Bjork, 2007). Many learning scientists, however,
argue that the capacity of retrieval practice to enhance
learning and memory is the technique’s most important
benefit (e.g., Dunlosky et al., 2013; Pan & Rickard, 2018).
The benefits of errorful generation are
unappreciated
The present results reveal that many students have little
awareness of the fact that making errors, followed by
correct answer feedback, improves memory (e.g., Kornell
et al., 2009; Pan & Bjork, in press). When predicting the rela-
tive effectiveness of learning techniques in a hypothetical
scenario, students only modestly favoured errorful learning
over the study of correct answers. Similarly, many students
did not express a preference for learning techniques that
are more error-prone, which is a hallmark of “desirable
difficulties”(Bjork, 1994), or prefer learning to solve pro-
blems via techniques that would be likely to induce pro-
ductive failure (Kapur, 2015). However, it does not appear
that many students have a strong bias against believing
that generating errors is helpful for learning; rather, their
baseline beliefs appear to be relatively agnostic on the
issue. Accordingly, when making experience-based meta-
cognitive judgments in experimental paradigms (e.g.,
Huelser & Metcalfe, 2012; Yang et al., 2017), learners poss-
ibly rely on other cues to inform their judgments (which
may lead to being swayed by feelings of fluency and
other characteristics), with those cues commonly leading
to a stated preference for reading and studying over error-
ful generation. However, when errors do occur, they are
commonly treated as learning opportunities.
Our findings for instructor- or textbook-provided prac-
tice questions provide further evidence that the benefits
of errorful generation are unappreciated. Most students
never or rarely use such questions to engage in pretesting,
yet commonly use them to engage in retrieval practice (for
related findings, see Hartwig & Dunlosky, 2012; Kornell &
Bjork, 2007; Pan & Sana, 2020). Further, instructors
provide relevant resources (e.g., practice questions in
advance of relevant course content) on an inconsistent
basis. That pattern can be interpreted as another indication
that the benefits of errorful generation are unappreciated,
and it may also contribute to students’infrequent use of
pretesting.
Preferred versus actual learning practices
In several instances, students’and instructors’practices fell
short of stated preferences. For instance, the rate at which
students engage in error correction (60%) is substantially
less than their endorsement of its importance (87%). That
result suggests a disparity between intended and actual
learning behaviours (see Blasiman et al., 2017 for analo-
gous findings involving distributed practice). Additionally,
92% of students report spending time studying or analys-
ing feedback when it is provided, which implies a strong
positive evaluation of such feedback, yet 38% of their
instructors reportedly seldom or never provide it (higher
rates of feedback were however reported in the instructor
survey). Further, 80% of students prefer immediate or
same-day feedback, yet such feedback is reportedly pro-
vided only 27% of the time (analogous patterns were
observed in the instructor survey data). All of these pat-
terns may reflect logistical and other challenges that
impede the implementation of desired learning practices.
Limitations and future research
Limitations of the present study could be addressed in
future research. Although the survey results are likely gener-
alisable to students and instructors at other universities, par-
ticularly in North America, additional studies involving non-
Western cultures are advisable to address potential cultural
differences (cf. Santagata, 2005; Stigler et al., 1999). Random
samples could be used to reduce any effects of selection
bias. Potential moderating influences of academic achieve-
ment level (e.g., Geller et al., 2017) and academic mindset
(e.g., Rattan et al., 2015) could also be investigated.
Notably, our data relied entirely on self-report measures
that asked respondents to make judgments on issues and
topics that were, in some cases, fairly abstract; independent
verification of learning behaviours where feasible (e.g., Blasi-
man et al., 2017) could be used to test the accuracy and val-
idity of those measures. In addition, future surveys could
field a greater variety of questions to further probe students’
and instructors’approaches to learning from errors. Such
questions might include more fine-grained answer options
to explore different methods of analysing errors, various
types of feedback, and more diverse learning contexts.
The types of errors that are made (which could range
from somewhat plausible to completely offthe mark)
could also be explored.
Relatedly, further research on the efficacy of pretesting
and productive failure is needed before either technique
can be endorsed for widespread use. Such research
might occur in authentic educational environments (e.g.,
Geller et al., 2017), and for the case of pretesting, address
the specificity of learning that has repeatedly been
observed in some experiments (e.g., James & Storm,
2019; cf. Pan, Lovelett, et al., 2019), the role of surprise (But-
terfield & Metcalfe, 2001), the finding that generating
errors that are semantically related to the correct answers
MEMORY 15

yields more potent learning (Cyr & Anderson, 2018;
Zawadzka & Hanczakowski, 2019), various types of
pretest and criterial test questions (e.g., St. Hilaire et al.,
2019), and the absence of pretesting effects for materials
that lack strong cue-target associations (e.g., Grimaldi &
Karpicke, 2012; cf. Seabrooke et al., 2019). These studies
could help clarify potential benefits and limitations of gen-
erating errors for learning.
Practical implications
In terms of application, our most important finding is that
many undergraduate students and instructors currently
undervalue and underutilise errorful generation.
Although research on the benefits of pretesting and pro-
ductive failure is still ongoing, compelling evidence
already exists regarding the efficacy of such techniques
across a variety of pedagogical circumstances (and for
pretesting especially). Accordingly, when choosing how
information should be processed and how study time
should be allocated (Dunlosky & Ariel, 2011;Nelson&
Narens, 1990), students and instructors should be cogni-
sant of the benefits that errorful generation can provide.
Even a brief discussion of the benefits of making errors
can be impactful (e.g., Yang et al., 2017). Moreover, facil-
itating errorful generation need not be highly complex;
for instance, instructors could simply provide practice
questions—which are already often implemented—
earlier or after a minimal amount of prerequisite instruc-
tionhasoccurred.Wesubmitthatitisnotenoughto
simply value learning from errors; the deliberate gener-
ation of errors, followed by feedback, should be con-
sidered as a viable learning technique. A growing
awareness of the benefits of errorful learning among
instructors and students has the potential to augment
learning with minimal costs: Pretests can be used to intro-
duce and practice materials and productive failure can be
embraced to help consolidate learning.
Acknowledgements
The authors thank the Paul R. MacPherson Institute for Leadership,
Innovation and Excellence in Teaching at McMaster University, the
Center for Education Innovation and Learning in the Sciences at
UCLA, and the Center for Advancing Multidisciplinary Scholarship for
Excellence in Education at UCSD for their assistance with publicising
the survey. Thanks to Shanna Shaked and other university personnel
for logistical support, Michelle Rivers for helpful comments on an
earlier draft of this manuscript, and Yunning Qiu for assistance with
data verification.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Funding
This work was partially funded by a Social Sciences and Humanities
Research Council Insight Development grant #430-2020-00925 to F.S.
ORCID
Steven C. Pan http://orcid.org/0000-0001-9080-5651
Faria Sana http://orcid.org/0000-0002-2202-7592
Joshua Samani http://orcid.org/0000-0001-8774-6646
Joseph A. Kim http://orcid.org/0000-0001-5671-8693
References
Ausubel, D., Novak, J., & Hanesian, H. (1968). Educational psychology: A
cognitive view. Holt, Rinehart and Winston.
Bandura, A. (1986). Social foundations of thought and action. Prentice
Hall.
Bangert-Drowns, R. L., Kulik, C. L. C., Kulik, J. A., & Morgan, M. (1991).
The instructional effect of feedback in test-like events. Review of
Educational Research,61(2), 213–238. https://doi.org/10.3102/
00346543061002213
Bjork, E. L., Soderstrom, N. C., & Little, J. L. (2015). Can multiple-choice
testing induce desirable difficulties? Evidence from the laboratory
and the classroom. The American Journal of Psychology,128(2),
229–239. https://doi.org/10.5406/amerjpsyc.128.2.0229
Bjork, R. A. (1994). Memory and metamemory considerations in
the training of human beings. In J. Metcalfe & A. Shimamura
(Eds.), Metacognition: Knowing about knowing (pp. 185–205).
MIT Press.
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning:
Beliefs, techniques, and illusions. Annual Review of Psychology,64
(1), 417–444. https://doi.org/10.1146/annurev-psych-113011-
143823
Blasiman, R. N., Dunlosky, J., & Rawson, K. A. (2017). The what, how
much, and when of study strategies: comparing intended versus
actual study behaviour. Memory,25(6), 784–792. https://doi.org/
10.1080/09658211.2016.1221974
Butterfield, B., & Metcalfe, J. (2001). Errors committed with high confi-
dence are hypercorrected. Journal of Experimental Psychology:
Learning, Memory, and Cognition,27(6), 1491–1494. https://doi.
org/10.1037/0278-7393.27.6.1491
Carpenter, S. K., & Toftness, A. R. (2017). The effect of prequestions on
learning from video presentations. Journal of Applied Research in
Memory and Cognition,6(1), 104–109. https://doi.org/10.1016/j.
jarmac.2016.07.014
Cyr, A. A., & Anderson, N. D. (2018). Learning from your mistakes: Does
it matter if you’re out in left foot, I mean field? Memory,26(9), 1281–
1290. https://doi.org/10.1080/09658211.2018.1464189
Dalehefte, I. M., Seidel, T., & Prenzel, M. (2012). Reflecting on learning
from errors in school instruction: Findings and suggestions from a
Swiss-German video study. In J. Bauer & C. Harteis (Eds.), Human fal-
libility (pp. 197–213). Springer. https://doi.org/10.1007/978-90-481-
3941-5_12
Dunlosky, J., & Ariel, R. (2011). Self-regulated learning and the allo-
cation of study time. In B. H. Ross (Ed.), Psychology of learning and
motivation (Vol. 54, pp. 103–140). Academic Press. https://doi.org/
10.1016/B978-0-12-385527-5.00004-8
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T.
(2013). Improving students’learning with effective learning tech-
niques: Promising directions from cognitive and educational psy-
chology. Psychological Science in the Public Interest,14(1), 4–58.
https://doi.org/10.1177/1529100612453266
Frost, R. O., Marten, P., Lahart, C., & Rosenblate, R. (1990). The dimen-
sions of perfectionism. Cognitive Therapy and Research,14(5),
449–468. https://doi.org/10.1007/BF01172967
Gartmeier, M., Bauer, J., Gruber, H., & Heid, H. (2008). Negative knowl-
edge: Understanding professional learning and expertise. Vocations
and Learning,1(2), 87–103. https://doi.org/10.1007/s12186-008-
9006-1
Geller, J., Carpenter, S. K., Lamm, M. H., Rahman, S., Armstrong, P. I., &
Coffman, C. R. (2017). Prequestions do not enhance the benefits of
16 S. C. PAN ET AL.

retrieval in a STEM classroom. Cognitive Research: Principles and
Implications,2(1), 42. https://doi.org/10.1186/s41235-017-0078-z
Geller, J., Davis, S. D., & Peterson, D. (2020). Sans forgetica is not desir-
able for learning. PsyArXiv,https://doi.org/10.31234/osf.io/ku5bz
Geller, J., Toftness, A. R., Armstrong, P. I., Carpenter, S. K., Manz, C. L.,
Coffman, C. R., & Lamm, M. H. (2018). Study strategies and beliefs
about learning as a function of academic achievement and achieve-
ment goals. Memory (Hove, England),26(5), 683–690. https://doi.
org/10.1080/09658211.2017.1397175
Grimaldi, P. J., & Karpicke, J. D. (2012). When and why do retrieval
attempts enhance subsequent encoding? Memory & Cognition,40
(4), 505–513. https://doi.org/10.3758/s13421-011-0174-0
Hartwig, M. K., & Dunlosky, J. (2012). Study strategies of college stu-
dents: Are self-testing and scheduling related to achievement?
Psychonomic Bulletin & Review,19(1), 126–134. https://doi.org/10.
3758/s13423-011-0181-y
Holmes, N. G., Day, J., Park, A. H., Bonn, D. A., & Roll, I. (2014). Making the
failure more productive: Scaffolding the invention process to
improve inquiry behaviors and outcomes in invention activities.
Instructional Science,42(4), 523–538. https://doi.org/10.1007/
s11251-013-9300-7
Huelser, B. J., & Metcalfe, J. (2012). Making related errors facilitates
learning, but learners do not know it. Memory & Cognition,40(4),
514–527. https://doi.org/10.3758/s13421-011-0167-z
James, K. K., & Storm, B. C. (2019). Beyond the pretesting effect: What
happens to the information that is not pretested? Journal of
Experimental Psychology: Applied,25(4), 576–587. https://doi.org/
10.1037/xap0000231
Kalton, G., & Schuman, H. (1982). The effect of the question on survey
responses: A review. Journal of the Royal Statistical Society: Series A
(General),145(1), 42–57. https://doi.org/10.2307/2981421
Kang, S. H., Pashler, H., Cepeda, N. J., Rohrer, D., Carpenter, S. K., &
Mozer, M. C. (2011). Does incorrect guessing impair fact learning?
Journal of Educational Psychology,103(1), 48–59. https://doi.org/
10.1037/a0021977
Kapur, M. (2008). Productive failure. Cognition and Instruction,26(3),
379–424. https://doi.org/10.1080/07370000802212669
Kapur, M. (2015). Learning from productive failure. Learning: Research
and Practice,1(1), 51–65. https://doi.org/10.1080/23735082.2015.
1002195
Kapur, M., & Rummel, N. (2012). Productive failure in learning from
generation and invention activities. Instructional Science,40(4),
645–650. https://doi.org/10.1007/s11251-012-9235-4
Knight, J. B., Ball, B. H., Brewer, G. A., DeWitt, M. R., & Marsh, R. L.
(2012). Testing unsuccessfully: A specification of the underlying
mechanisms supporting its influence on retention. Journal of
Memory and Language,66(4), 731–746. https://doi.org/10.1016/j.
jml.2011.12.008
Kornell, N. (2014). Attempting to answer a meaningful question
enhances subsequent learning even when feedback is delayed.
Journal of Experimental Psychology: Learning, Memory, and
Cognition,40(1), 106–114. https://doi.org/10.1037/a0033699
Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regu-
lated study. Psychonomic Bulletin & Review,14(2), 219–224. https://
doi.org/10.3758/BF03194055
Kornell, N., Hays, M. J., & Bjork, R. A. (2009). Unsuccessful retrieval
attempts enhance subsequent learning. Journal of Experimental
Psychology: Learning, Memory, and Cognition,35(4), 989–998.
https://doi.org/10.1037/a0015729
Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design.
In J. D. Wright & P. V. Marsden (Eds.), Handbook of survey research
(pp. 263–314). Elsevier.
Kulik, J. A., & Kulik, C. L. C. (1988). Timing of feedback and verbal learn-
ing. Review of Educational Research,58(1), 79–97. https://doi.org/10.
3102/00346543058001079
Lai, P. K., Portolese, A., & Jacobson, M. J. (2017). Does sequence matter?
Productive failure and designing online authentic learning for
process engineering. British Journal of Educational Technology,48
(6), 1217–1227. https://doi.org/10.1111/bjet.12492
Leighton, J. P., Tang, W., & Guo, Q. (2018). Undergraduate students’
attitudes towards mistakes in learning and academic achievement.
Assessment & Evaluation in Higher Education,43(4), 612–628. https://
doi.org/10.1080/02602938.2017.1387230
Little, J. L., & Bjork, E. L. (2015). Optimizing multiple-choice tests as tools
for learning. Memory & Cognition,43(1), 14–26. https://doi.org/10.
3758/s13421-014-0452-8
Loibl, K., & Rummel, N. (2014). Knowing what you don’t know makes
failure productive. Learning and Instruction,34,74–85. https://doi.
org/10.1016/j.learninstruc.2014.08.004
Mason, A., & Singh, C. (2010). Do advanced physics students learn from
their mistakes without explicit intervention? American Journal of
Physics,78(7), 760–767. https://doi.org/10.1119/1.3318805
Matteucci, M. C., Corazza, M., & Santagata, R. (2015). Learning from
errors, or not. An analysis of teachers’beliefs about errors and
error-handling strategies through questionnaire and video.
Progress in Education,37,33–54.
McCabe, J. (2011). Metacognitive awareness of learning strategies in
undergraduates. Memory & Cognition,39(3), 462–476. https://doi.
org/10.3758/s13421-010-0035-2
McMorran, C., Ragupathi, K., & Luo, S. (2017). Assessment and learning
without grades? Motivations and concerns with implementing gra-
deless learning in higher education. Assessment & Evaluation in
Higher Education,42(3), 361–377. https://doi.org/10.1080/
02602938.2015.1114584
Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology,
68(1), 465–489. https://doi.org/10.1146/annurev-psych-010416-
044022
Metcalfe, J., & Huelser, B. J. (2020). Learning from errors is attributable
to episodic recollection rather than semantic mediation.
Neuropsychologia,138, 107296. https://doi.org/10.1016/j.
neuropsychologia.2019.107296
Metcalfe, J., Kornell, N., & Finn, B. (2009). Delayed versus immediate
feedback in children’s and adults’vocabulary learning. Memory &
Cognition,37(8), 1077–1087. https://doi.org/10.3758/MC.37.8.1077
Minsky, M. (1997). Negative expertise. In P. J. Feltovich, K. M. Ford, & R. R.
Hoffman (Eds.), Expertise in context: Human and machine (pp. 515–
521). American Association for Artificial Intelligence; The MIT Press.
Morehead, K., Rhodes, M. G., & DeLozier, S. (2016). Instructor and
student knowledge of study strategies. Memory (Hove, England),
24(2), 257–271. https://doi.org/10.1080/09658211.2014.1001992
Mullet, H. G., Butler, A. C., Verdin, B., von Borries, R., & Marsh, E. J. (2014).
Delaying feedback promotes transfer of knowledge despite student
preferences to receive feedback immediately. Journal of Applied
Research in Memory and Cognition,3(3), 222–229. https://doi.org/
10.1016/j.jarmac.2014.05.001
Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical frame-
work and new findings. The Psychology of Learning and
Motivation,26, 125–173. https://doi.org/10.1016/S0079-7421
(08)60053-5
Oser, F., & Spychiger, M. (2005). Lernen ist schmerzhaft. Beltz.
Pan, S. C., & Bjork, R. A. (in press). Acquiring an accurate mental model
of learning: Towards an owner’s manual. In A. Wagner & M. Kahana
(Eds.), Oxford handbook of learning & memory: Foundations and
applications (pp. 1–68). Oxford University Press.
Pan, S. C., Lovelett, J., Stoeckenius, D., & Rickard, T. C. (2019). Conditions
of highly specific learning through cued recall. Psychonomic Bulletin
& Review,26(2), 634–640. https://doi.org/10.3758/s13423-019-
01593-x
Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning:
Meta-analytic review and synthesis. Psychological Bulletin,144(7),
710–756. https://doi.org/10.1037/bul0000151
Pan, S. C., & Sana, F. (2020). Pretesting versus posttesting: Comparing the
pedagogical benefits of errorful generation and retrieval practice.
Manuscript submitted for publication.
MEMORY 17

Pan, S. C., Sana, F., Schmitt, A., & Bjork, E. L. (2020). Pretesting reduces
mind wandering and enhances learning during online lectures.
Journal of Applied Research in Memory and Cognition.https://doi.
org/10.31234/osf.io/qsebx
Pan, S. C., Tajran, J., Lovelett, J., Osuna, J., & Rickard, T. C. (2019). Does
interleaved practice enhance foreign language learning? The
effects of training schedule on Spanish verb conjugation skills.
Journal of Educational Psychology,111(7), 1172–1188. https://doi.
org/10.1037/edu0000336
Pashler, H., Cepeda, N. J., Wixted, J. T., & Rohrer, D. (2005). When does
feedback facilitate learning of words? Journal of Experimental
Psychology: Learning, Memory, and Cognition,31(1), 3–8. https://
doi.org/10.1037/0278-7393.31.1.3
Pintrich, P. R., Smith, D. A., García, T., & McKeachie, W. J. (1991). A
manual for the use of the motivational strategies for learning ques-
tionnaire (MSLQ). University of Michigan, National Center for
Research to Improve Postsecondary Teaching and Learning.
Potts, R., & Shanks, D. R. (2014). The benefit of generating errors during
learning. Journal of Experimental Psychology: General,143(2), 644–
667. https://doi.org/10.1037/a0033194
Rattan, A., Savani, K., Chugh, D., & Dweck, C. S. (2015). Leveraging
mindsets to promote academic achievement: Policy recommen-
dations. Perspectives on Psychological Science,10(6), 721–726.
https://doi.org/10.1177/1745691615599383
Reason,J.(1995). A systems approach to organizational error. Ergonomics,
38(8), 1708–1721. https://doi.org/10.1080/00140139508925221
Richland, L. E., Kornell, N., & Kao, L. S. (2009). The pretesting effect: Do
unsuccessful retrieval attempts enhance learning? Journal of
Experimental Psychology: Applied,15(3), 243–257. https://doi.org/
10.1037/a0016496
Santagata, R. (2005). Practices and beliefs in mistake-handling activi-
ties: A video study of Italian and US mathematics lessons.
Teaching and Teacher Education,21(5), 491–508. https://doi.org/
10.1016/j.tate.2005.03.004
Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice:
Common principles in three paradigms suggest new concepts for
training. Psychological Science,3(4), 207–218. https://doi.org/10.
1111/j.1467-9280.1992.tb00029.x
Seabrooke, T., Hollins, T. J., Kent, C., Wills, A. J., & Mitchell, C. J. (2019).
Learning from failure: Errorful generation improves memory for
items, not associations. Journal of Memory and Language,104,70–
82. https://doi.org/10.1016/j.jml.2018.10.001
Skinner, B. F. (1953). Science and human behavior (No. 92904). Simon
and Schuster.
Steuer, G., & Dresel, M. (2015). A constructive error climate as an
element of effective learning environments. Psychological Test
and Assessment Modeling,57(2), 262.
Steuer,G.,Rosentritt-Brunn,G.,&Dresel,M.(2013). Dealing
with errors in mathematics classrooms: Structure and relevance
of perceived error climate. Contemporary Educational
Psychology,38(3), 196–210. https://doi.org/10.1016/j.cedpsych.
2013.03.002
Stevenson, H., & Stigler, J. W. (1994). The learning gap: Why our schools
are failing and what we can learn from Japanese and Chinese edu-
cation. Simon and Schuster.
St. Hilaire, K. J., Carpenter, S. K., & Jennings, J. M. (2019). Using pre-
questions to enhance learning from reading passages: The roles
of question type and structure building ability. Memory (Hove,
England),27(9), 1204–1213. https://doi.org/10.1080/09658211.
2019.1641209
Stigler, J., Gonzales, P., Kawanaka, T., Knoll, S., & Serrano, A. (1999). The
TIMSS 1995 videotape classroom study: Methods and findings from an
exploratory research project on eighth-grade mathematics instruction
in Germany, Japan, and the United States. US Dept of Education.
National Center for Education Statistics. http://nces.ed.gov/
pubs99/1999074.pdf
Taylor,A.,Sanson,M.,Burnell,R.,Wade,K.A.,&Garry,M.(2020).
Disfluent difficulties are not desirable difficulties: The (lack
of) effect of Sans Forgetica on memory. Memory (Hove,
England),28(7), 850–857. https://doi.org/10.1080/09658211.
2020.1758726
Trueman, R. J. (2014). Productive failure in STEM education. Journal of
Educational Technology Systems,42(3), 199–214. https://doi.org/10.
2190/ET.42.3.b
Tulis, M. (2013). Error management behavior in classrooms: Teachers’
responses to student mistakes. Teaching and Teacher Education,
33,56–68. https://doi.org/10.1016/j.tate.2013.02.003
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the
psychology of choice. Science,211(4481), 453–458. https://doi.org/
10.1126/science.7455683
Weinstein, C. E., & Palmer, D. R. (2002). Learning and study Strategies
Inventory (LASSI): User’s manual (2nd ed.). H & H Publishing.
Wissman, K. T., Rawson, K. A., & Pyc, M. A. (2012). How and when do
students use flashcards? Memory (Hove, England),20(6), 568–579.
https://doi.org/10.1080/09658211.2012.687052
Yang, C., Potts, R., & Shanks, D. R. (2017). Metacognitive unawareness of
the errorful generation benefit and its effects on self-regulated
learning. Journal of Experimental Psychology: Learning, Memory,
and Cognition,43(7), 1073–1092. https://doi.org/10.1037/
xlm0000363
Zawadzka, K., & Hanczakowski, M. (2019). Two routes to memory
benefits of guessing. Journal of Experimental Psychology: Learning,
Memory, and Cognition,45(10), 1748–1760. https://doi.org/10.
1037/xlm0000676
18 S. C. PAN ET AL.