The Critical Importance of Retrieval for Learning

Abstract

Learning is often considered complete when a student can produce the correct answer to a question. In our research, students in one condition learned foreign language vocabulary words in the standard paradigm of repeated study-test trials. In three other conditions, once a student had correctly produced the vocabulary item, it was repeatedly studied but dropped from further testing, repeatedly tested but dropped from further study, or dropped from both study and test. Repeated studying after learning had no effect on delayed recall, but repeated testing produced a large positive effect. In addition, students' predictions of their performance were uncorrelated with actual performance. The results demonstrate the critical role of retrieval practice in consolidating learning and show that even university students seem unaware of this fact.

Full text

DOI: 10.1126/science.1152408
, 966 (2008);319 Science
Jeffrey D. Karpicke and Henry L. Roediger III
The Critical Importance of Retrieval for Learning
This copy is for your personal, non-commercial use only.
clicking here.colleagues, clients, or customers by , you can order high-quality copies for yourIf you wish to distribute this article to others
here.following the guidelines can be obtained byPermission to republish or repurpose articles or portions of articles
): May 2, 2012 www.sciencemag.org (this information is current as of
The following resources related to this article are available online at
http://www.sciencemag.org/content/319/5865/966.full.html
version of this article at: including high-resolution figures, can be found in the onlineUpdated information and services,
http://www.sciencemag.org/content/suppl/2008/02/12/319.5865.966.DC1.html
can be found at: Supporting Online Material
http://www.sciencemag.org/content/319/5865/966.full.html#related
found at: can berelated to this article A list of selected additional articles on the Science Web sites
32 article(s) on the ISI Web of Sciencecited by This article has been
http://www.sciencemag.org/content/319/5865/966.full.html#related-urls
18 articles hosted by HighWire Press; see:cited by This article has been
http://www.sciencemag.org/cgi/collection/psychology
Psychology subject collections:This article appears in the following
registered trademark of AAAS. is aScience2008 by the American Association for the Advancement of Science; all rights reserved. The title CopyrightAmerican Association for the Advancement of Science, 1200 New York Avenue NW, Washington, DC 20005.
(print ISSN 0036-8075; online ISSN 1095-9203) is published weekly, except the last week in December, by theScience
on May 2, 2012www.sciencemag.orgDownloaded from

The Critical Importance
of Retrieval for Learning
Jeffrey D. Karpicke
1
*and Henry L. Roediger III
2
Learning is often considered complete when a student can produce the correct answer to a
question. In our research, students in one condition learned foreign language vocabulary words in
the standard paradigm of repeated study-test trials. In three other conditions, once a student had
correctly produced the vocabulary item, it was repeatedly studied but dropped from further testing,
repeatedly tested but dropped from further study, or dropped from both study and test. Repeated
studying after learning had no effect on delayed recall, but repeated testing produced a large
positive effect. In addition, students’predictions of their performance were uncorrelated with
actual performance. The results demonstrate the critical role of retrieval practice in consolidating
learning and show that even university students seem unaware of this fact.
Ever since the pioneering work of Ebbinghaus
(1), scientists have generally studied hu-
man learning and memory by presenting
people with information to be learned in a study
period and testing them on it in a test period to
see what they retained. When this procedure oc-
curs over many trials, an exponential learning
curve is produced. The standard assumption in
nearly all research is that learning occurs while
people study and encode material. Therefore, ad-
ditional study should increase learning. Retriev-
ing information on a test, however, is sometimes
considered a relatively neutral event that mea-
sures the learning that occurred during study but
does not by itself produce learning. Over the
years, researchers have occasionally argued that
learning can occur during testing (2–6). However,
the assumptions that repeated studying promotes
learning and that testing represents a neutral event
that merely measures learning still permeate con-
temporary memory research as well as contem-
porary educational practice, where tests are also
considered purely as assessments of knowledge.
Our goal in the present research was to ex-
amine these long-standing assumptions regard-
ing the effects of repeated studying and repeated
testing on learning. Specifically, once informa-
tion can be recalled from memory, what are the
effects of repeated encoding (during study trials)
or repeated retrieval (during test trials) on learn-
ing and long-term retention, assessed after a
week delay? A second purpose of this research
was to examine students’assessments of their
own learning. After learning a set of materials
under repeated study or repeated test conditions,
we asked students to predict their future recall
on the week-delayed final test. Our question
was, would students show any insight into their
own learning?
A final purpose of the experiment was to
address another venerable issue in learning and
memory, concerning the relation between the
speed with which something is learned and the
rate at which it is forgotten. Is speed of learning
correlated with long-term retention, and if so, is
the correlation positive (processes that promote
fast learning also slow forgetting and promote
good retention) or negative (quick learning may
be superficial and produce rapid forgetting)? Early
research led to the conclusion that quick learn-
ing reduced the rate of forgetting and improved
long-term retention (7), but later critics argued
that, when forgetting is assessed more properly
than in the early studies, no differences exist be-
tween forgetting rates for fast and slow learning
conditions (8,9). By any account, conditions that
exhibit equivalent learning curves should produce
equivalent retention after a delay (9).
Using foreign language vocabulary word pairs,
we examined the contributions of repeated study
and repeated testing to learning by comparing a
standard learning condition to three dropout condi-
tions. The standard method of measuring learning,
used since Ebbinghaus’s research (1), involves
presenting subjects with information in a study
period, then testing them on it in a test period,
then presenting it again, testing on it again, and so
on. The dropout learning conditions of the present
experiment differed from the standard learning
condition in that, once an item was successfully
recalled once on a test, it was either (i) dropped
from study periods but still tested in one con-
dition, (ii) dropped from test periods but still re-
peatedly studied in a second condition, or (iii)
dropped altogether from both study and test pe-
riods in a third condition (Table 1).
Surprisingly, standard learning conditions
and dropout conditions have seldom been com-
pared in memory research, despite their critical
importance to theories of learning and their prac-
tical importance to students (in using flash cards
and other study methods). Dropout conditions
were originally developed to remedy methodo-
logical problems that arise from repeated practice
in the standard learning condition (10), but they
can also be used to examine the effect of re-
peated practice in its own right, as we did in the
present experiment. If learning happens exclu-
sively during study periods and if tests are neutral
assessments, then additional study trials should
have a strong positive effect on learning, whereas
additional test trials should produce no effect.
Further, if repeated study or test practice after an
item has been learned does indeed benefit long-
term retention, this would contradict the conven-
tional wisdom that students should drop material
that they have learned from further practice in
order to focus their effort on material they have
not yet learned. Dropping learned facts may create
the same long-term retention as occurs in stan-
dard conditions but in a shorter amount of time,
or it may improve learning by allowing stu-
dents to focus on items they have not yet recalled.
This strategy is implicitly endorsed by contem-
porary theories of study-time allocation (11,12)
and is explicitly encouraged in many popular
study guides (13).
1
Department of Psychological Sciences, Purdue University,
West Lafayette, IN 47907, USA.
2
Department of Psychol-
ogy, Washington University in St. Louis, St. Louis, MO
63130, USA.
*To whom correspondence should be addressed. E-mail:
karpicke@purdue.edu
Table 1. Conditions used in the experiment, average number of trials within each study or test
period, and total number of trials in the learning phase in each condition. S
N
indicates that only
vocabulary pairs not recalled in the previous test period were studied in the current study period. T
N
indicates that only pairs not recalled in the previous test period were tested in the current test
period. Students in all conditions performed a 30-s distracter task that involved verifying multi-
plication problems after each study period.
Condition
Study (S) or test (T) period and number of trials per period Total
number
of trials
12 3 4 5 6 7 8
ST S T S T S T S T
40 40 40 40 40 40 40 40 320
S
N
TSTS
N
TS
N
TS
N
T
40 40 26.8 40 8.0 40 2.0 40 236.8
ST
N
ST S T
N
ST
N
ST
N
40 40 40 27.9 40 11.8 40 3.3 243.0
S
N
T
N
ST S
N
T
N
S
N
T
N
S
N
T
N
40 40 27.1 27.1 8.8 8.8 1.5 1.5 154.8
15 FEBRUARY 2008 VOL 319 SCIENCE www.sciencemag.org966
REPORTS
on May 2, 2012www.sciencemag.orgDownloaded from

In the experiment, we had college students
learn a list of foreign language vocabulary word
pairs and manipulated whether pairs remained in
the list (and were repeatedly practiced) or were
dropped after the first time they were recalled,
as shown in Table 1. All students began by study-
ing a list of 40 Swahili-English word pairs (e.g.,
mashua-boat) in a study period and then testing
over the entire list in a test period (e.g., mashua-?).
All conditions were treated the same in the ini-
tial study and test periods. Once a word pair was
recalled correctly, it was treated differently in the
four conditions. In the standard condition, sub-
jects studied and were tested over the entire list in
each study and test period (denoted ST). In a
second condition, once a pair was recalled, it was
dropped from further study but tested in each sub-
sequenttestperiod(denotedS
N
T, wh e re S
N
indi-
cates that only nonrecalled pairs were restudied).
In a third condition, recalled pairs were dropped
from further testing but studied in each subsequent
study period (denoted ST
N
,whereT
N
indicates
that only nonrecalled pairs were kept in the list
during test periods). In a fourth condition, recalled
pairs were dropped entirely from both study and
test periods (S
N
T
N
). The final condition repre-
sents what conventional wisdom and many edu-
cators instruct students to do: Study something
until it is learned (i.e., can be recalled) and then
drop it from further practice.
At the end of the learning phase, students in
all four conditions were asked to predict how
many of the 40 pairs they would recall on a final
test in 1 week. They were then dismissed and
returned for the final test a week later. Of key
importance were the effects of the four learning
conditions on the speed with which the vocabulary
words were learned, on students’predictions of
their future performance, and on long-term reten-
tion assessed after a week delay (14).
Figure 1 shows the cumulative proportion of
word pairs recalled during the learning phase,
which gives credit the first time a student recalled
a pair. We also analyzed traditional learning
curves (the proportion of the total list recalled
in each test period) for the two conditions that
required recall of the entire list (ST and S
N
T),
and the results by the two measurement meth-
ods were identical. Thus, we restrict our dis-
cussion to the cumulative learning curves on
which all four conditions can be compared.
Figure 1 shows that performance was virtually
perfect by the end of learning (i.e., all 40 English
target words were recalled by nearly all sub-
jects). More importantly, there were no differences
in the learning curves of the four conditions.
Given the similarity of acquisition perform-
ance, it is not too surprising that students in the
four conditions did not differ in their aggregate
judgments of learning (their predictions of their
future performance). On average, the students in
all conditions predicted they would recall about
50% of the pairs in 1 week. The mean number of
words predicted to be recalled in each condition
were as follows: ST = 20.8, S
N
T=20.4,ST
N
=
22.0, and S
N
T
N
= 20.3. An analysis of variance did
not reveal significant differences among the
conditions (F<1).
Although students’cumulative learning per-
formance was equivalent in the four conditions
and predicted final recall was also equivalent,
actual recall on the final delayed test differed
widely across conditions, as shown in Fig. 2.
The results show that testing (and not studying)
is the critical factor for promoting long-term re-
call. In fact, repeated study after one successful
recall did not produce any measurable learning
a week later. In the learning conditions that re-
quired repeated retrieval practice (ST and S
N
T),
students correctly recalled about 80% of the
pairs on the final test. In the other conditions in
which items were dropped from repeated test-
ing (ST
N
and S
N
T
N
), students recalled just
36% and 33% of the pairs. It is worth em-
phasizing that, despite the fact that students
repeatedly studied all of the word pairs in every
study period in the ST
N
condition, their long-
term recall was much worse than students who
were repeatedly tested on the entire list. Com-
bining the two conditions that involved repeated
testing (ST and S
N
T) and combining the two
conditions that involved dropping items from
testing after they were recalled once (ST
N
and
S
N
T
N
), repeated retrieval increased final recall
by 4 standard deviations (d= 4.03). The distri-
butions of scores in these two groups did not
overlap: Final recall in the drop-from-testing
conditions ranged from 10% to 60%, whereas
final recall in the repeated test conditions ranged
from 63% to 95%. Whether students repeatedly
studied the entire set or whether they restudied
only pairs they had not yet recalled produced
virtually no effect on long-term retention. The
dramatic difference shown in Fig. 2 was caused
by whether or not the pairs were repeatedly tested.
Even though cumulative learning perform-
ance was identical in the four conditions, the
total number of trials (study or test) in each con-
dition varied greatly. Table 1 shows the mean
number of trials in each study and test period
and the total number of trials in each condition.
The standard condition (ST) involved the most
trials (320) because all 40 items were presented
in each study and test period. The S
N
T
N
condi-
tion involved the fewest trials (154.8, on aver-
age) because the number of trials in each period
grew smaller as items were recalled and dropped
from further practice. The other two conditions
(S
N
TandST
N
) involved about the same number
of trials (236.8 and 243.0, respectively) but be-
cause they differed in terms of whether items
were dropped from study or test periods, they
produced dramatically different effects on long-
term retention. In other words, about 80 more
study trials occurred in the ST
N
condition than
in the S
N
T
N
condition, but this produced prac-
tically no gain in retention. Likewise, about 80
more study trials occurred in the ST condition
than in the S
N
T condition, and this produced no
gain whatsoever in retention. However, when
about 80 more test trials occurred in the learning
phase (in the ST condition versus the ST
N
con-
dition, and in the S
N
T condition versus the S
N
T
N
condition), repeated retrieval practice led to greater
than 150% improvements in long-term retention.
The present research shows the powerful ef-
fect of testing on learning: Repeated retrieval
practice enhanced long-term retention, whereas
repeated studying produced essentially no ben-
efit. Although educators and psychologists often
consider testing a neutral process that merely
assesses the contents of memory, practicing re-
trieval during tests produces more learning than
additional encoding or study once an item has
been recalled (15–17). Dropout methods such as
the ones used in the present experiment have
seldom been used to investigate effects of re-
peated practice in their own right, but compar-
ison of the dropout conditions to the repeated
practice conditions revealed dramatic effects of
retrieval practice on learning.
Fig. 1. Cumulative performance during the learn-
ing phase.
Fig. 2. Proportion recalled on the final test 1 week
after learning. Error bars represent standard errors
of the mean.
www.sciencemag.org SCIENCE VOL 319 15 FEBRUARY 2008 967
REPORTS
on May 2, 2012www.sciencemag.orgDownloaded from

The experiment also shows a striking ab-
sence of any benefit of repeated studying once
an item could be recalled from memory. A basic
tenet of human learning and memory research is
that repetition of material improves its retention.
This is often true in standard learning situations,
yet our research demonstrates a situation that
stands in stark contrast to this principle. The
benefits of repetition for learning and long-term
retention clearly depend on the processes learners
engage in during repetition. Once information
can be recalled, repeated encoding in study trials
produced no benefit, whereas repeated retrieval
in test trials generated large benefits for long-
term retention. Further research is necessary to
generalize these findings to other materials. How-
ever, the basic effects of testing on retention have
been shown with many kinds of materials (16),
so we have confidence that the present results
will generalize, too.
Our experiment also speaks to an old debate
in the science of memory, concerning the rela-
tion between speed of learning and rate of for-
getting (7–9). Our study shows that the forgetting
rate for information is not necessarily deter-
mined by speed of learning but, instead, is greatly
determined by the type of practice involved.
Even though the four conditions in the experi-
ment produced equivalent learning curves, re-
peated recall slowed forgetting relative to
recalling each word pair just one time.
Importantly, students exhibited no awareness
of the mnemonic effects of retrieval practice, as
evidenced by the fact that they did not predict
they would recall more if they had repeatedly
recalled the list of vocabulary words than if they
only recalled each word one time. Indeed, ques-
tionnaires asking students to report on the strat-
egies they use to study for exams in education
also indicate that practicing recall (or self-testing)
is a seldom-used strategy (18). If students do test
themselves while studying, they likely do it to
assess what they have or have not learned (19),
rather than to enhance their long-term retention
by practicing retrieval. In fact, the conventional
wisdom shared among students and educators is
that if information can be recalled from mem-
ory, it has been learned and can be dropped
from further practice, so students can focus their
effort on other material. Research on students’
use of self-testing as a learning strategy shows
that students do tend to drop facts from further
practice once they can recall them (20). However,
the present research shows that the conventional
wisdom existing in education and expressed in
many study guides is wrong. Even after items
can be recalled from memory, eliminating those
items from repeated retrieval practice greatly re-
duces long-term retention. Repeated retrieval in-
duced through testing (and not repeated encoding
during additional study) produces large positive
effects on long-term retention.
References and Notes
1. H. Ebbinghaus, Memory: A Contribution to Experimental
Psychology, H. A. Ruger, C. E. Bussenius, Transls. (Dover,
New York, 1964).
2. R. A. Bjork, in Information Processing and Cognition:
The Loyola Symposium, R. L. Solso, Ed. (Erlbaum,
Hillsdale, NJ, 1975), pp. 123–144.
3. M. Carrier, H. Pashler, Mem. Cognit. 20, 633 (1992).
4. A. I. Gates, Arch. Psychol. 6, 1 (1917).
5. C. Izawa, J. Math. Psychol. 8, 200 (1971).
6. E. Tulving, J. Verb. Learn. Verb. Behav. 6, 175 (1967).
7. J. A. McGeoch, The Psychology of Human Learning
(Longmans, Green, New York, 1942).
8. N. J. Slamecka, B. McElree, J. Exp. Psychol. Learn. Mem.
Cogn. 9, 384 (1983).
9. B. J. Underwood, J. Verb. Learn. Verb. Behav. 3, 112
(1964).
10. W. F. Battig, Psychon. Sci. Monogr. 1(suppl.), 1 (1965).
11. J. Metcalfe, N. Kornell, J. Exp. Psychol. Gen. 132, 530
(2003).
12. K. W. Thiede, J. Dunlosky, J. Exp. Psychol. Learn. Mem.
Cognit. 25, 1024 (1999).
13. S. Frank, The Everything Study Book (Adams, Avon, MA,
1996).
14. Materials and methods are available as supporting
material on Science Online.
15. J. D. Karpicke, H. L. Roediger, J. Mem. Lang. 57, 151
(2007).
16. H. L. Roediger, J. D. Karpicke, Perspect. Psychol. Sci. 1,
181 (2006).
17. H. L. Roediger, J. D. Karpicke, Psychol. Sci. 17, 249
(2006).
18. N. Kornell, R. A. Bjork, Psychon. Bull. Rev. 14, 219 (2007).
19. J. Dunlosky, K. Rawson, S. McDonald, in Applied
Metacognition, T. Perfect, B. Schwartz, Eds. (Cambridge
Univ. Press, Cambridge, 2002), pp. 68–92.
20. J. D. Karpicke, thesis, Washington University, St. Louis,
MO (2007).
21. We thank J. S. Nairne for helpful comments on the
manuscript. This research was supported by a
Collaborative Activity Grant of the James S. McDonnell
Foundation to the second author.
Supporting Online Material
www.sciencemag.org/cgi/content/full/319/5865/966/DC1
Materials and Methods
Table S1
References
31 October 2007; accepted 12 December 2007
10.1126/science.1152408
15 FEBRUARY 2008 VOL 319 SCIENCE www.sciencemag.org968
REPORTS
on May 2, 2012www.sciencemag.orgDownloaded from