Retrieval Practice Improves Memory in Survivors of
Severe Traumatic Brain Injury
James F. Sumowski, PhD,a,bJulia Coyne, PhD,a,cAmanda Cohen, BA,a
John DeLuca, PhDa,b
From theaNeuropsychology and Neuroscience, Kessler Foundation, West Orange, NJ;bDepartment of Physical Medicine and Rehabilitation,
Rutgers, New Jersey Medical School, Newark, NJ; andcChildren’s Specialized Hospital, Mountainside, NJ.
Objective: To investigate whether retrieval practice (RP) improves delayed recall after short and long delays in survivors of severe traumatic brain
injury (TBI) relative to massed restudy (MR) and spaced restudy (SR).
Design: 3(learning condition: MR, SR, RP)?2(delayed recall: 30min, 1wk) within-subject experiment.
Setting: Nonprofit medical rehabilitation research center.
Participants: Memory-impaired (<5th percentile) survivors of severe TBI (NZ10).
Intervention: During RP, patients are quizzed on to-be-learned information shortly after it is presented, such that patients practice retrieval. MR
consists of repeated restudy (ie, cramming). SR consists of restudy trials separated in time (ie, distributed learning).
Main Outcome Measures: Forty-eight verbal paired associates (VPAs) were equally divided across 3 learning conditions (16 per condition).
Delayed recall for one half of the VPAs was assessed after 30 minutes (8 per condition) and for the other half after 1 week (8 per condition).
Results: There was a large effect of learning condition after the short delay (P<.001, h2Z.72), with much better recall of VPAs studied through
RP (46.3%) relative to MR (12.5%) and SR (15.0%). This large effect of learning condition remained after the long delay (PZ.001, h2Z.56), as
patients recalled 11.3% of the VPAs studied through RP, but nothing through MR (0.0%) and only 1.3% through SR. That is, RP was essentially
the only learning condition to result in successful recall after 1 week, with most patients recalling at least 1 VPA.
Conclusions: The robust effect of RP among TBI survivors with severe memory impairment engenders confidence that this strategy would work
outside the laboratory to improve memory in real-life settings. Future randomized controlled trials of RP training are needed.
Archives of Physical Medicine and Rehabilitation 2014;95:397-400
ª 2014 by the American Congress of Rehabilitation Medicine
More than 200,000 survivors of moderate and severe traumatic
brain injury (TBI) are discharged from American hospitals
annually.1Many of these survivors suffer chronic memory
impairment,2leading to diminished quality of life. Unfortu-
nately, current memory rehabilitation interventions for persons
with brain injury lack efficacy,3thereby highlighting the urgent
need for new and effective treatments. Extensive research
within the cognitive psychology literature supports retrieval
practice (RP) (also known as the testing effect) as an effective
mnemonic strategy among healthy college undergraduates.4
Testing in educational and clinical settings is considered a
tool for evaluation, but RP research demonstrates that the act of
retrieving information also strengthens one’s memory trace.4
That is, when persons are quizzed on information during
learning (RP), they are better able to subsequently recall the
information than if they restudied the information multiple
times without testing. Translating this mnemonic effect to
clinical samples, RP has improved recall after a short delay
(45min) in cross-sectional experiments with memory-impaired
patients with multiple sclerosis5and survivors of severe TBI,6
and these memory benefits of RP are maintained after a long
delay (1wk) in memory-impaired patients with multiple scle-
rosis.7Here, we investigate whether RP leads to better memory
after short (30min) and long (1wk) delays among memory-
impaired survivors of severe TBI.
Supported by the Kessler Foundation and Children’s Specialized Hospital.
No commercial party having a direct financial interest in the results of the research supporting
this article has conferred or will confer a benefit on the authors or on any organization with which
the authors are associated.
0003-9993/14/$36 - see front matter ª 2014 by the American Congress of Rehabilitation Medicine
Archives of Physical Medicine and Rehabilitation
journal homepage: www.archives-pmr.org
Archives of Physical Medicine and Rehabilitation 2014;95:397-400
Our sample included 10 survivors of severe TBI with memory
impairment (?5th percentile on delayed recall of the Hopkins
Verbal Learning Test, Revised). See table 1 for sample charac-
terization. This study was approved by the Kessler Foundation
Institutional Review Board, and written informed consent was
obtained from all subjects.
In a within-subject design, subjects studied 48 verbal paired
associates (VPAs) (eg, Ground-Cold ) equally divided across 3
learning conditions: massed restudy (MR), spaced restudy (SR),
and RP. (To ensure against any possible systematic error asso-
ciated with differential VPA difficulty, we  used only those
VPAs that were previously classified as weakly associated, 
randomly assigned the 48 VPAs to 1 of 3 lists [A, B, C], and then
 counterbalanced lists [A, B, C] across learning cognitions
[MR, SR, RP] across subjects.) As illustrated in figure 1, MR is
tantamount to “cramming,” a ubiquitous memory strategy among
college students and neurologic patients alike. SR represents
distributed learning, recognized as superior to MR for over a
century.8For RP, VPAs were presented on the same schedule as
SR; however, after the VPA was presented in its complete form
initially (eg, GroundeCold ), the 2 subsequent reexposure trials
were framed as cued recall tests (eg, Grounde _____). A more
detailed description of learning trials is available in figure 1 and
elsewhere.5,6Dependent measures included delayed recall for
one half of the VPAs (8 in each condition: MR, SR, RP) after a
short delay (30min) and for the other half after a long delay
(1wk). Subjects were presented with the first word of each VPA
and were asked to recall the second word.
Repeated-measures analysis of variance assessed differences in
short delay recall across the 3 learning conditions: MR, SR, RP.
Next, pairwise comparisons investigated differences in recall
across pairs of learning conditions (eg, MR vs SR). These analyses
were repeated for long delay recall.
There was a large main effect of learning condition after the
short delay (F2,18Z23.41, PZ.00001, h2Z.72). Subjects recal-
led 46.3% of the VPAs studied through RP compared with only
12.5% through MR (P<.0001) and 15% through SR (PZ.002).
SR did not result in better memory than MR (PZ.555). The
beneficial effect of RP was enduring, as the large effect of learning
condition remained after the long delay (F2,18Z11.53, PZ.001,
h2Z.56). Patients recalled 11.3% of the VPAs studied through
RP compared with 0.0% through MR (PZ.004) and 1.3%
through SR (PZ.011). MR and SR did not reliably differ from
each other (PZ.343).
The magnitude of the RP effect is perhaps better captured by
examining the raw data on a case-by-case basis (see table 1). RP
was the best memory strategy for each and every patient after a
short delay. After 1 week, subjects could not recall a single VPA
learned through MR and only 1 subject recalled 1 VPA learned
through SR. In contrast, most subjects were able to recall at least 1
VPA learned through RP.
RP resulted in much better recall than restudy strategies in
memory-impaired survivors of severe TBI even after a weeklong
delay. Moreover, RP was the most effective memory strategy for
every patient after a short delay, and RP was essentially the only
strategy that supported recall after a long delay (1wk). These
findings highlight the strength of the RP strategy and engender
confidence that RP might result in improved real-life memory
functioning for survivors of TBI. Importantly, however, healthy
persons4and persons with TBI6identify MR (ie, cramming) as a
more effective memory strategy. As such, education, training, and
practice will be required for persons with TBI to replace MR with
the more effective RP technique.
Sample characteristics and memory performance
SubAge (y)SexEducation (y)
Injury (y)Cause of Injury
DR T-sc SDMRSDSRSDRPLDMRLDSR LDRP
Hit by train
Abbreviations: F, female; HVLT-R DR, Hopkins Verbal Learning Test, Revised Delayed Recall; LD, long delay of 1wk; M, male; MVA, motor vehicle
accident; SD, short delay of 30min; Sub, subject; T-sc, T score.
List of abbreviations:
MR massed restudy
RCT randomized controlled trial
RP retrieval practice
SR spaced restudy
TBI traumatic brain injury
VPA verbal paired associate
398J.F. Sumowski et al
As discussed elsewhere,7RP is an example of a compensatory
(vs restorative) approach to memory rehabilitation (for further
distinction, see Cicerone et al9). Similar to the use of a cane for
walking, RP supports memory without repairing/restoring the
neurophysiologic basis of memory. Although restoration of function
may seem optimal or preferable, restoration is often impossible
after normal neurophysiologic function is damaged. For this reason,
compensatory approaches to rehabilitation are often more effective
than attempts at restoration (rehabilitation of hemianopic dyslexia
represents a fine example10). Therefore, RP will be effective only
if persons with TBI learn to incorporate RP into their routines and
apply the strategy in new learning situations. For example, persons
wishing to learn information from a newspaper or textbook may
engage in intermittent self-quizzing throughout their reading (ie,
after each paragraph or page). This act of RP will result in greater
subsequent memory than rereading the information multiple times.
A randomized controlled trial (RCT) of RP training will be an
important next step in this line of research because such a study
will evaluate whether persons with TBI can learn and apply the RP
strategy. Importantly, given that RP represents a compensatory
(not a restorative) strategy, the primary outcome of an RP RCT
(or an RCT of any other compensatory memory intervention)
should not include standardized neuropsychological memory tests
that prevent subjects from using the very compensatory strategy
one wishes to evaluate. RCTs of compensatory memory strategies
should instead train subjects in the use of the strategy and then
investigate the strategy’s efficacy with less-standardized tasks
on which subjects are free to learn/study new information using
whichever strategy they choose. There are some drawbacks of
using less-standardized tasks (ie, unestablished reliability). Also,
there may be great variability in how different persons with TBI
apply the RP strategy. Alternate/secondary outcomes could include
self-report and observer-report of memory change in daily activ-
ities after receiving RP training (relative to controls). Regarding
variability in RP execution across subjects, single-subject designs
may be helpful in investigating the ways in which different persons
with TBI apply RP strategies to learn new information.
Our sample size was small, although this is somewhat mitigated
by the within-subject design and robust results.
RP represents a promising memory strategy for survivors of TBI
with memory impairment. In addition to the apparent effective-
ness of RP, this strategy appears simple/straightforward to apply
(quizzing oneself or someone else), cost-effective, safe, and
noninvasive. RCTs of RP training are needed.
Brain trauma; Memory; Rehabilitation; Traumatic brain injuries
condition was presented for 3 consecutive trials (no intervening trials of other VPAs). Each VPA within the SR or RP condition was presented in a
spaced fashion, with 3 intervening trials (of other VPAs) between the first presentation and first restudy (or test) trial and 6 intervening trials
(of other VPAs) between the first and second restudy (or test) trials. This basic presentation schedule was used for all 48 VPAs, totaling 144 six-
second trials during a single 14-minute 24-second learning phase (6-s slides ? 144 slides) viewed on a computer screen. Note that the 5-second
RP trials consisted of a cued-recall screen for 5 seconds (eg, CLOTHe _____) followed by a 1-second feedback screen (eg, CLOTHeSHEEP). Given
that VPAs in each condition were evenly distributed throughout the learning phase, any retroactive or proactive interference or practice effects
were also equally distributed across groups. Also, VPAs were counterbalanced across the 3 different conditions across subjects and across delayed
recall tests (30min, 1wk).
Sample presentation schedule for VPAs in each learning cognition: MR (white), SR (light gray), RP (dark gray). Each VPA within the MR
Retrieval practice improves memory in TBI survivors399
James F. Sumowski, PhD, Neuropsychology and Neuroscience,
Kessler Foundation Research Center, 300 Executive Dr, Ste 70,
West Orange, NJ 07052. E-mail address: jsumowski@
1. Hirtz D, Thurman DJ, Gwinn-Hardy K, Mohamed M, Chaudhuri AR,
Zalutsky R. How common are the “common” neurologic disorders?
2. van Zomeren AH, van den Burg W. Residual complaints of patients
two years after severe head injury. J Neurol Neurosurg Psychiatry
3. Rohling ML, Faust ME, Beverly B, Demakis G. Effectiveness of
cognitive rehabilitation following acquired brain injury: a meta-
analytic re-examination of Cicerone et al.’s (2000, 2005) systematic
reviews. Neuropsychology 2009;23:20-39.
4. Karpicke JD, Roediger HL III. The critical importance of retrieval for
learning. Science 2008;319:966-8.
5. Sumowski JF, Chiaravalloti N, Deluca J. Retrieval practice improves
memory in multiple sclerosis: clinical application of the testing effect.
6. Sumowski JF, Wood HG, Chiaravalloti N, Wylie GR, Lengenfelder J,
DeLuca J. Retrieval practice: a simple strategy for improving
memory after traumatic brain injury. J Int Neuropsychol Soc 2010;
7. Sumowski JF, Leavitt VM, Cohen A, Paxton J, Chiaravalloti ND,
Deluca J. Retrieval practice is a robust memory aid for memory-
impaired patients with MS. Mult Scler 2013;19:1943-6.
8. Cepeda NJ, Pashler H, Vul E, Wixted JT, Rohrer D. Distributed
practice in verbal recall tasks: a review and quantitative synthesis.
Psychol Bull 2006;132:354-80.
9. Cicerone KD, Langenbahn DM, Braden C, et al. Evidence-based
cognitive rehabilitation: updated review of the literature from 2003
through 2008. Arch Phys Med Rehabil 2011;92:519-30.
10. Schuett S. The rehabilitation of hemianopic dyslexia. Nat Rev Neurol
400 J.F. Sumowski et al