Imagery and retrieval of auditory and visual information: neural correlates of successful and unsuccessful performance.

Abstract

Remembering past events - or episodic retrieval - consists of several components. There is evidence that mental imagery plays an important role in retrieval and that the brain regions supporting imagery overlap with those supporting retrieval. An open issue is to what extent these regions support successful vs. unsuccessful imagery and retrieval processes. Previous studies that examined regional overlap between imagery and retrieval used uncontrolled memory conditions, such as autobiographical memory tasks, that cannot distinguish between successful and unsuccessful retrieval. A second issue is that fMRI studies that compared imagery and retrieval have used modality-aspecific cues that are likely to activate auditory and visual processing regions simultaneously. Thus, it is not clear to what extent identified brain regions support modality-specific or modality-independent imagery and retrieval processes. In the current fMRI study, we addressed this issue by comparing imagery to retrieval under controlled memory conditions in both auditory and visual modalities. We also obtained subjective measures of imagery quality allowing us to dissociate regions contributing to successful vs. unsuccessful imagery. Results indicated that auditory and visual regions contribute both to imagery and retrieval in a modality-specific fashion. In addition, we identified four sets of brain regions with distinct patterns of activity that contributed to imagery and retrieval in a modality-independent fashion. The first set of regions, including hippocampus, posterior cingulate cortex, medial prefrontal cortex and angular gyrus, showed a pattern common to imagery/retrieval and consistent with successful performance regardless of task. The second set of regions, including dorsal precuneus, anterior cingulate and dorsolateral prefrontal cortex, also showed a pattern common to imagery and retrieval, but consistent with unsuccessful performance during both tasks. Third, left ventrolateral prefrontal cortex showed an interaction between task and performance and was associated with successful imagery but unsuccessful retrieval. Finally, the fourth set of regions, including ventral precuneus, midcingulate cortex and supramarginal gyrus, showed the opposite interaction, supporting unsuccessful imagery, but successful retrieval performance. Results are discussed in relation to reconstructive, attentional, semantic memory, and working memory processes. This is the first study to separate the neural correlates of successful and unsuccessful performance for both imagery and retrieval and for both auditory and visual modalities.

Full text

Page 1

This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright

Page 2

Author's personal copy
Neuropsychologia 49 (2011) 1730–1740
Contents lists available at ScienceDirect
Neuropsychologia
journal homepage: www.elsevier.com/locate/neuropsychologia
Imagery and retrieval of auditory and visual information: Neural correlates of
successful and unsuccessful performance
Willem Huijbersa,b,∗, Cyriel M.A. Pennartza, David C. Rubinc,d, Sander M. Daselaara,c
aUniversity of Amsterdam, Swammerdam Institute for Life Sciences, Faculty of Science, Amsterdam, The Netherlands
bMartinos Center for Biomedical Imaging, Massachusetts General Hospital, Department of Neurology, Brigham and Women’s Hospital, Boston, MA, USA
cDuke University, Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
dCenter on Autobiographical Memory Research, Aarhus University, Aarhus, Denmark
a r t i c l ei n f o
Article history:
Received 3 June 2010
Received in revised form 21 February 2011
Accepted 26 February 2011
Available online 9 March 2011
Keywords:
Episodic memory
Imagery
fMRI
Visual
Auditory
a b s t r a c t
Remembering past events – or episodic retrieval – consists of several components. There is evidence
that mental imagery plays an important role in retrieval and that the brain regions supporting imagery
overlap with those supporting retrieval. An open issue is to what extent these regions support success-
ful vs. unsuccessful imagery and retrieval processes. Previous studies that examined regional overlap
between imagery and retrieval used uncontrolled memory conditions, such as autobiographical memory
tasks, that cannot distinguish between successful and unsuccessful retrieval. A second issue is that fMRI
studies that compared imagery and retrieval have used modality-aspecific cues that are likely to acti-
vate auditory and visual processing regions simultaneously. Thus, it is not clear to what extent identified
brain regions support modality-specific or modality-independent imagery and retrieval processes. In the
current fMRI study, we addressed this issue by comparing imagery to retrieval under controlled memory
conditionsinbothauditoryandvisualmodalities.Wealsoobtainedsubjectivemeasuresofimageryqual-
ityallowingustodissociateregionscontributingtosuccessfulvs.unsuccessfulimagery.Resultsindicated
that auditory and visual regions contribute both to imagery and retrieval in a modality-specific fashion.
In addition, we identified four sets of brain regions with distinct patterns of activity that contributed to
imagery and retrieval in a modality-independent fashion. The first set of regions, including hippocam-
pus, posterior cingulate cortex, medial prefrontal cortex and angular gyrus, showed a pattern common
to imagery/retrieval and consistent with successful performance regardless of task. The second set of
regions, including dorsal precuneus, anterior cingulate and dorsolateral prefrontal cortex, also showed
a pattern common to imagery and retrieval, but consistent with unsuccessful performance during both
tasks. Third, left ventrolateral prefrontal cortex showed an interaction between task and performance
and was associated with successful imagery but unsuccessful retrieval. Finally, the fourth set of regions,
including ventral precuneus, midcingulate cortex and supramarginal gyrus, showed the opposite inter-
action, supporting unsuccessful imagery, but successful retrieval performance. Results are discussed in
relation to reconstructive, attentional, semantic memory, and working memory processes. This is the
first study to separate the neural correlates of successful and unsuccessful performance for both imagery
and retrieval and for both auditory and visual modalities.
© 2011 Elsevier Ltd. All rights reserved.
1. Introduction
Mental imagery – perceiving with the mind’s eye and ear –
is considered to be a critical component of vivid remembering
of past events – or episodic retrieval (Baddeley, 1992; Greenberg
& Rubin, 2003). Theories of memory hold that episodic retrieval
does not only involve the activation of previously formed memory
traces, but also reconstructive processes that support the gener-
∗Corresponding author. Tel.: +1 617 726 5573.
E-mail address: huijbers@nmr.mgh.harvard.edu (W. Huijbers).
ation of mental representations (Bartlett, 1932; Daselaar et al.,
2008; Rubin, Schrauf, & Greenberg, 2003; Tulving, 1983). In line
withthesetheories,recentfunctionalMRI(fMRI)studieshaveindi-
cated that imagery and retrieval show overlapping activations in
several regions including the hippocampus, medial prefrontal cor-
tex (mPFC), posterior midline region (PMR) and the angular gyrus
of the ventral parietal cortex (Addis, Pan, Vu, Laiser, & Schacter,
2009; Addis, Wong, & Schacter, 2007; Buckner & Carroll, 2007;
Hassabis & Maguire, 2007). However, these studies used uncon-
trolled,open-endedautobiographicalmemorytaskstoexaminethe
neural correlates of memory, which have no objective measures
of successful vs. unsuccessful performance (Cabeza & St Jacques,
0028-3932/$ – see front matter © 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.neuropsychologia.2011.02.051

Page 3

Author's personal copy
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
1731
Table 1
Task setup and data used in the fMRI analyses.
ConditionDay 1Day 2
TimingImagery taskData used Timing Retrieval taskData used
Imagery1.0s
3.0s
1.5s
1.0s
3.0s
1.5s
Cue word
Imagine image or sound
Rate richness (1–4)
Cue word
Perceive image or sound
Rate quality (1–4)
Included4.0s Retrieve imagined
Event from day 1b
Rate confidence (1–2)
Retrieve perceived
Event from day 1b
Rate confidence (1–2)
Not included
1.5s
4.0s
Perception Not includeda
Included
1.5s
aPerceptual trials are used for the modality-specific analysis only.
bNote that both retrieval and confidence rating required a response.
2007; Cabeza et al., 2004; Cabeza & Nyberg, 2000). Moreover,
the subjective quality of the imagery experience (successful vs.
unsuccessful imagery) was generally not assessed (see however
Addis et al., 2007). Thus, the overlap between brain regions that
support successful vs. unsuccessful imagery and retrieval remains
unclear.
A second issue is that fMRI studies that compared imagery and
retrieval have used general cues that did not emphasize a spe-
cific modality (e.g. Addis et al., 2009; Addis et al., 2007; Hassabis,
Kumaran,&Maguire,2007;Szpunar,Watson,&McDermott,2007).
Such cues are likely to activate both auditory and visual processing
regions simultaneously. Thus, without word cues that are spe-
cific to the auditory or visual modality, it remains unclear to what
extent previously identified brain regions truly support modality-
independent imagery and retrieval processes.
To address these issues we conducted an fMRI experiment
that consisted of a perceptual task, an imagery task with sub-
jective performance ratings and a controlled retrieval task with
objective memory measures. Moreover, to assess the modality-
independence of regions, each task included both auditory and
visual conditions. As shown in Table 1, on Day 1, participants
imagined sounds and pictures based on imagery word cues. In
addition, they also perceived sounds and pictures following dif-
ferent word cues. Imagery-related activity was assessed by using
imagery quality ratings following each trial. Successful imagery
was defined as more activity for high- than for low-rated imagery
experiences and the opposite pattern as unsuccessful imagery.
On Day 2, participants retrieved the previously perceived and
imagined sounds and pictures based on retrieval word cues. To
keep our findings consistent with controlled studies of retrieval
(Buckner & Wheeler, 2001; Nyberg, Habib, Mcintosh, & Tulving,
2000), we only assessed the retrieval of externally presented
auditory and visual events (Table 1). Retrieval-related activity
was assessed by contrasting items that were remembered with
high confidence (successful retrieval) to those that were forgot-
ten (unsuccessful retrieval). Regional overlap between imagery
and retrieval was assessed by employing a factorial design in
which the factors task (imagery/retrieval) and performance (suc-
cessful/unsuccessful) were crossed for both auditory and visual
modalities.AlthoughmanyfMRIstudieshavecomparedactivations
during successful and unsuccessful retrieval conditions (Buckner &
Wheeler, 2001), to our knowledge this is the first retrieval study
to compare these activations with successful and unsuccessful
imagery.
In the present study, we tested two main predictions. First, we
predicted that in line with previous fMRI studies, imagery and
retrieval of auditory and visual information should activate audi-
tory and visual regions respectively (Buckner, Logan, Donaldson, &
Wheeler, 2000; Deacon, Kosslyn, & Scarry, 2001; Kraemer, Macrae,
Green, & Kelley, 2005). Second, we predicted that, in line with
studies that used uncontrolled memory tasks (Hassabis et al.,
2007a; Schacter, Addis, & Buckner, 2007; Szpunar et al., 2007), the
hippocampus, posterior cingulate cortex, medial prefrontal cor-
tex and angular gyrus will show a pattern associated both with
imagery success and retrieval success. We also expected this pat-
tern to occur regardless of whether the task had an auditory
or visual orientation. Although we did not have clear predic-
tions regarding brain regions, we also assessed the other cells of
the task×performance factorial design, shared by both modal-
ities: the opposite main effect (unsuccessful>successful), and
Table 2
overview of behavioral results.
aOnly black values are used in fMRI analysis, AVG: both imagery low/both retrieval miss trials combined.
bNote that given the 4 options during retrieval, chance-level is 25%.
cNote that MISS HIGH includes missing values.

Page 4

Author's personal copy
1732
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
Fig. 1. Modality-specific effects of imagery and retrieval: Light green: activation of auditory regions related to perception, imagery and retrieval. Dark green: activation of
visual regions related to perception, imagery and retrieval. Bar graphs (y-axis), indicate mean cluster activity (parameter estimates) for each regions, respectively from left
to right: (1) auditory perception>visual perception, (2) auditory perception<visual perception, (3) auditory imagery>visual imagery (4) auditory imagery<visual imagery,
(5) auditory retrieval>visual retrieval, and (6) auditory retrieval<visual retrieval. Lines reflect the standard error of the mean and asterisks annotate the significance of the
difference between the auditory and visual contrast. Number of asterisks denotes P-values (*P<0.05, **P<0.01, ***P<0.001).
interactions between task (imagery×retrieval) and performance
(successful×unsuccessful).
2. Results
2.1. Behavioral data
Table 2 gives a complete overview of reaction times (RTs) and
proportions of responses for each condition. Below, we only report
the outcome of the repeated measures ANOVAs concerning the RT
and accuracy differences between the trials that were included
in the successful (imagery quality rating of 4) and unsuccess-
ful (imagery quality ratings of 1 or 2) imagery, and successful
(high confidence correct) and unsuccessful (incorrect) retrieval
contrasts.
Regarding imagery performance, a repeated measures imagery
(unsuccessful/successful)×modality
showed a main effect of performance on RTs (P=0.0088) indicating
that the ratings for successful imagery trials were faster than those
for unsuccessful imagery trials. Although we found no effect of
modality (P=0.62), there was a significant modality×imagery
interaction(P=0.0098)indicatingagreaterperformancedifference
in RTs for the visual condition. Consistent with the RT results,
the percentage of successful items was also significantly lower
(P<0.0001) for auditory than for visual imagery suggesting that
the former condition was more difficult.
Retrieval trials involved two responses, one during the retrieval
period and one during the confidence rating period. During the
retrieval period, we found a main effect of memory performance
on RTs (P<0.001) reflecting faster RTs for successful than unsuc-
cessful retrieval responses. In addition, we found a significant main
effect of modality on RTs (P=0.023) indicating that retrieval of
auditory information was slower than retrieval of visual infor-
mation. However in terms of memory accuracy, there was no
effect of modality (P=0.092). During the rating period of the
retrieval task, we found a main effect of memory (P<0.001), but
notmodality(P=0.13),onRTs.Wealsofoundasignificantmodality
(auditory/visual)×memory (successful/unsuccessful) interaction
(P=0.025) indicating a greater confidence rating RT difference
between successful and unsuccessful responses for the visual than
the auditory modality. Overall the behavioral results suggest that
(auditory/visual)ANOVA
auditory imagery was more difficult than visual imagery, but, at
least based on the memory accuracy, there was no evidence for a
modality difference in difficulty during retrieval.
2.2. fMRI data
2.2.1. Modality-specific contributions to imagery and retrieval
Following fMRI studies that examined imagery and retrieval
separately(Buckneretal.,2000;Deaconetal.,2001;Kosslyn,Ganis,
& Thompson, 2001; Kraemer et al., 2005; Wheeler, Petersen, &
Buckner,2000),wepredictedthatimageryandretrievalofauditory
and visual information should activate auditory and visual sensory
regions, respectively. Confirming this prediction for the audi-
tory modality, we found common activity for auditory perception
(perceptionofsounds>images),auditoryimagery(successfulaudi-
tory imagery>successful visual imagery), and auditory memory
retrieval (successful auditory retrieval>successful visual retrieval)
within auditory processing regions including bilateral superior
temporal gyrus/secondary auditory cortex (Fig. 1 and Table 3).
Likewise, confirming our prediction for the visual modality, we
found that a region comprising bilateral V3/V4 and fusiform gyrus
(Fig. 1 and Table 3) was not only activated during visual per-
ception (perception of images>sounds), but also during visual
imagery (successful visual imagery>successful auditory imagery),
andvisualmemoryretrieval(successfulvisualretrieval>successful
auditory retrieval). These findings confirm the modality-specific
role of sensory processing regions in imagery and retrieval of audi-
tory and visual information (Buckner et al., 2000; Deacon et al.,
2001; Kosslyn et al., 2001; Kraemer et al., 2005; Wheeler et al.,
2000a; Wheeler et al., 2000b).
2.3. Overlap and interactions between imagery and retrieval in
modality-independent regions
Table 4 and Figs. 2–5 summarize the regions that displayed
consistent patterns of imagery and retrieval activity for both audi-
tory and visual modalities. Regions that did not show significant or
trending activity for one of the modalities were not included in the
table.

Page 5

Author's personal copy
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
1733
Table 3
Regions showing overlapping activations during perception, imagery and retrieval.
RegionSide BA
XYZ
T-PERT-IMGT-RET
Auditory>visual
Auditory cortex/superior temporal gyrusL
R
L
L
L
R
22/41
22/41
20
47
–
–
−60
48
−42
−54
−15
12
−42
−39
−6
1512.5
7.90
11.6
8.59
6.43
7.12
2.61
2.27
5.65
8.37
3.95
2.19
2.05
3.25
7.30
6.20
3.24
3.05
6
Inferior temporal gyrus
Inferior frontal gyrus
caudate
−15
189
6
6
9
−6
Visual>auditory
Fusiform gyrusL
R
L
R
L
R
R
R
L
R
36/37
36/37
19
19
19/39
19/39
19/40
19
30
31
−27
27
−42
42
−27
45
36
12
−18
−57
−45
−6
−78
−72
−45
−60
−48
−57
−33
−15
−15
−9
−15
24
51
42
15.0
23.2
11.8
10.1
8.55
3.83
3.61
5.11
3.08
4.29
5.78
4.45
7.91
7.88
9.40
6.74
4.11
4.62
2.92
3.22
6.12
5.54
4.04
3.15
2.98
3.18
2.84
5.01
3.08
4.90
Visual cortex (V3/V4)
Parieto-occipital cortex
Parahippocampus
Posterior cingulate cortex
9
12
363
T-PER=T-values perception, T-IMG=T-values imagery, T-RET=T-values retrieval.
2.3.1. Main positive effect: successful imagery and successful
retrieval
AsshowninFig.3andinlinewithourprediction,thehippocam-
pus, posterior cingulate cortex (PCC), medial prefrontal cortex
(mPFC) and angular gyrus were associated both with imagery
success (successful>unsuccessful imagery) and retrieval success
(successful>unsuccessful retrieval) in both auditory and visual
modalities (Table 4). These findings confirm results from previ-
ous studies showing overlapping activity in these regions during
imageryandretrieval(Addisetal.,2007;Addisetal.,2009;Buckner
& Carroll, 2007; Hassabis & Maguire, 2007). At the same time, the
currentresultsextendthesefindingsbyshowingthattheseregions
are specifically associated with successful, rather than unsuccess-
ful, imagery and retrieval performance, and also operate in a
modality-independent fashion. As outlined in the discussion sec-
tion, we interpret the activation in these regions as supporting the
(re)construction and (re)experiencing of mental representations.
Fig. 2. Main positive effect: successful imagery and successful retrieval. Regions
showing more activity for successful than unsuccessful trials regardless of task.
Bars reflect the mean activity (parameter estimates on the y-axis) for each region
contrasting the difference between successful (S) vs. unsuccessful (U) imagery and
successful (S) vs. unsuccessful (U) retrieval for both auditory (AUD) and visual (VIS)
conditions. From left to right bars, show activity of the angular gyrus (ANG), poste-
rior cingulate cortex (PCC) and hippocampus (HIP). Lines reflect the standard error
of the mean.
Fig. 3. Main negative effect: unsuccessful imagery and unsuccessful retrieval.
Regions showing less activity for successful than unsuccessful trials regardless
of task. Bars reflect the mean activity (parameter estimates on the y-axis) for
each region contrasting the difference between successful (S) vs. unsuccessful (U)
imagery and successful (S) vs. unsuccessful (U) retrieval for both auditory (AUD)
andvisual(VIS)conditions.Fromlefttorightbars,showactivityofdorsalprecuneus
(D.PREC),anteriorcingulatecortex(ACC)anddorsolateralprefrontalcortex(DLFPC).
Lines reflect the standard error of the mean.
Fig. 4. Interaction I: successful imagery and unsuccessful retrieval. The left ven-
trolateral prefrontal cortex (VLPFC) shows greater activity for unsuccessful than
successful retrieval trials, but greater activity for successful than unsuccessful
imagery trials. Bars reflect the mean activity (parameter estimates on the y-axis)
for the VLPFC contrasting the difference between successful (S) vs. unsuccessful (U)
imageryandunsuccessful(U)vs.successful(S)retrievalforbothauditory(AUD)and
visual (VIS) conditions. Lines reflect the standard error of the mean.

Page 6

Author's personal copy
1734
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
Table 4
Regions showing patterns of activity common to auditory and visual modalities.
Region
Side
BA
Mental imagery
Memory retrieval
Max
S>U
Interaction
Max
S>U
Interaction
X
Y
Z
T max
AUD
VIS
AUD>VIS
RT: F>S
X
Y
Z
T max
AUD
VIS
AUD>VIS
Positive main effect: (IMG: S>U) and (RET: S>U)
Posterior cingulate cortex
L
29/30
−6
−51
12
4.63
??
??
ns
na
−18
−57
3
5.94
???
??
ns
Medial prefrontal cortex
L
10
−3
66
9
3.54
?
??
ns
na
−3
63
6
3.41
??
?
ns
Angular gyrus
L
39
−48
−69
24
2.93
+
??
na
−48
−66
18
3.26
??
?
ns
Hippocampus
R
na
24
−18
−12
2.98
??
?
ns
na
24
−21
−18
4.80
???
?
+
Supplemental motor area
–
6
0
−12
54
4.32
?
?
ns
na
−3
−15
48
4.59
??
??
ns
Putamen
L
na
−18
9
−15
5.62
???
ns
?
na
−21
6
−15
5.40
???
??
ns
R
na
15
9
−12
4.98
???
+
ns
na
18
12
−12
5.47
???
??
ns
Negative main effect: (IMG: U>S) and (RET: U>S)
RT sensitive
Anterior cingulate cortex
R
32
9
30
30
−6.35
???
???
ns
??
9
30
36
−5.20
???
???
ns
Dorsal precuneus
R
7
12
−66
54
−6.35
??
???
ns
?
9
−63
48
−6.32
???
??
ns
Cuneus
L
17
−15
−90
−3
−4.02
???
??
ns
???
−15
−90
0
−3.10
??
?
ns
RT insensitive
Dorsolateral prefrontal cortex
R
46
33
51
18
−7.49
???
???
ns
ns
30
42
18
−6.48
???
???
ns
Superior prefrontal cortex
R
9
42
27
39
−5.33
???
???
ns
ns
42
30
33
−5.46
???
???
ns
Insula
L
13
−48
12
3
−3.73
???
−
ns
ns
−48
−18
0
−3.24
?
?
ns
R
13
36
−12
15
−3.81
??
???
ns
ns
36
−24
6
−5.52
?
???
ns
Interaction I: (IMG: S>U) and (RET: U>S)
Ventrolateral prefrontal cortex
L
45
−54
6
21
4.24
??
+
ns
na
−48
6
21
−3.94
−
???
ns
Interaction II: (IMG: U>S) and (RET: S>U)
Ventral precuneus
R
7
6
−72
30
−4.79
???
?
ns
na
9
−66
27
4.12
???
???
ns
Midcingulate cortex
R
24/31
9
−36
39
−6.25
?
?
ns
na
3
−21
39
4.21
?
???
ns
Supramarginal gyrus
R
40
48
−51
51
−3.67
???
?
?
na
48
−48
45
5.99
??
??
ns
AUD=auditory, VIS=visual, S=successful, U=unsuccessful, ?=positive, ?=negative, ns=not significant, na=not applicable.
+P<0.10 (trending).
−P<0.10 (trending).
?P<0.05.
??P<0.01.
???P<0.001.

Page 7

Author's personal copy
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
1735
Fig. 5. Interaction II: unsuccessful imagery and successful retrieval. Regions show-
ing greater activity for successful than unsuccessful retrieval trials, but greater
activity for unsuccessful than successful imagery trials. Bars reflect the mean activ-
ity (parameter estimates on the y-axis) for each region contrasting the difference
between unsuccessful (U) vs. successful (S) imagery and successful (S) vs. unsuc-
cessful (U) retrieval for both auditory (AUD) and visual (VIS) conditions. From left
to right bars, show activity of the supramarginal gyrus (SMG), midcingulate cortex
(MCC) and ventral precuneus (V.PREC). Lines reflect the standard error of the mean.
Lines reflect the standard error of the mean.
2.3.2. Main negative effect: unsuccessful imagery and
unsuccessful retrieval
As shown in Fig. 3 and in line with our second prediction, the
anterior cingulate cortex (ACC) and dorsolateral prefrontal cortex
(DLPFC) showed greater activity for unsuccessful than success-
ful imagery and retrieval, for both auditory and visual modalities.
ThesefindingsareconsistentwitharecentstudybyFleck,Daselaar,
Dobbins, and Cabeza (2006) that compared successful and unsuc-
cessful performance conditions during memory retrieval and a
perceptual decision task. Similarly, they found increased activity in
the ACC, DLPFC, as well as the dorsal precuneus, superior PFC and
insula for unsuccessful performance conditions. Here, we not only
confirmtheirpatternofresults,butextendthemtomentalimagery.
Outside of our predictions, but consistent with Fleck et al. (2006),
the dorsal part of the precuneus also showed a pattern associated
with unsuccessful performance regardless of modality. As outlined
in the Introduction and Discussion sections, we interpret the acti-
vation in these regions as supporting effortful attentional control
and monitoring processes.
2.4. Interactions: task×performance
Outside of our predictions, the left ventrolateral prefrontal cor-
tex (VLPFC), showed a significant task×performance interaction
showing greater activity for successful than unsuccessful imagery,
but less activity for successful than unsuccessful retrieval (Fig. 4).
Asdiscussedlater,weinterpretthisfindinginrelationtotheroleof
this region in semantic memory processes. We also found regions
that showed the opposite interaction, less activity for successful
than unsuccessful imagery, but greater activity for successful than
unsuccessful retrieval. As shown in Fig. 5 and Table 4, these regions
involvedtheventralprecuneus,themidcingulatecortex(MCC),and
right supramarginal gyrus. The finding of this pattern in ventral
precuneus is very interesting, because it is one of the regions most
consistently activated during tasks with a strong imagery compo-
nent (Cavanna & Trimble, 2006; Fletcher et al., 1995). Yet, as far as
we know, this region has not yet been linked to poor imagery per-
formance.AsexplainedintheIntroductionandDiscussionsections,
we interpret the activation in these regions as supporting mental
inspection and working memory processes.
2.5. Follow-up analyses
We conducted four follow-up analyses. First, we tested to what
extentbrainregionsthatshowsimilaractivitypatternsforauditory
and visual conditions are driven by a single modality (Follow-up
analysis 1, Supplemental Materials). As shown in Table 4 under
the heading “A>V interaction”, all regions – except for those
within the parietal regions and the putamen – showed modality-
independent activity patterns. Secondly, we tested the influence of
responsetimesduringtherecognitionjudgmentsonregionsshow-
ing a negative main effect (Follow-up analysis 2, Supplemental
Materials). As shown in Table 4, the ACC and dorsal precuneus
showed a significant effect of response time. These findings are
consistent with Fleck et al. (2006) and will be discussed in rela-
tion to monitoring performance and cognitive control. Third, we
assessed the influence of response times during the confidence
rating on regions showing a negative main effect (Follow-up anal-
ysis 3, Supplemental Materials). As shown in Supplemental Fig. 1
and Supplemental Table 1, our findings cannot easily be attributed
to response time differences in the rating procedures. Finally, we
assessed the influence of low-confidence hits on the retrieval suc-
cess effect (Follow-up analysis 4, Supplemental Materials). These
results demonstrate that including low-confidence hits has little
effect on the overall activity patterns, although it generally results
in a slightly decreased activity difference between successful and
unsuccessful performance, possibly due to guesses.
3. Discussion
As a first finding, we found that regions activated dur-
ing auditory and visual perception are also activated during
imagery and retrieval in a modality-specific fashion. As a sec-
ond finding, we identified four sets of regions showing task
(imagery/retrieval)×performance (successful/unsuccessful) main
effects as well as interactions, which are modality-independent.
A first set of regions, including hippocampus, posterior cingu-
late cortex (PCC), medial prefrontal cortex (mPFC) and angular
gyrus, was associated with successful performance during both
imagery and retrieval. A second set of regions, including the ante-
rior cingulate cortex (ACC), superior/dorsolateral prefrontal cortex
(DLPFC), dorsal precuneus, and insular cortex that is associated
with unsuccessful performance in both tasks. A third region, the
left ventrolateral prefrontal cortex (VLPFC) showed an interac-
tions between task and performance for both auditory and visual
modalities. A fourth set of regions, including ventral precuneus,
the midcingulate cortex (MCC) and supramarginal gyrus, showed
the opposite interaction between task and performance for both
auditory and visual modalities. Below, we first discuss the role of
the modality-specific regions in imagery and retrieval and then the
modality-independent regions.
3.1. Modality-specific effects: auditory and visual sensory regions
contribute similarly to imagery and retrieval
Leading models of imagery (Farah, 1984; Kosslyn et al., 1994)
and retrieval (Marr, 1971; McClelland, McNaughton, & O’Reilly,
1995; Tulving, 1983) assume that regions activated during real-
world perception are also involved in the (re)construction of the
sensory aspects of auditory and visual mental representations.
These models have been supported by studies that examined
imagery and retrieval or auditory and visual information sepa-
rately. Imagery studies of auditory (Bunzeck, Wuestenberg, Lutz,
Heinze,&Jancke,2005;Halpern&Zatorre,1999;Kosslyn,2003)and

Page 8

Author's personal copy
1736
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
visual information (Chen et al., 1998; Ishai, Ungerleider, & Haxby,
2000;Kosslyn,2003)foundmodality-specificactivationinauditory
and visual cortex respectively. Memory studies have found similar
modality-specific involvement of auditory- and visual-processing
regions (Nyberg et al., 2000; Wheeler et al., 2000a; Wheeler et al.,
2000b). For instance, Wheeler and colleagues found that words
accompaniedbyeitherauditoryorvisualinformationduringlearn-
ing also showed increased activity in auditory and visual cortices
respectively when presented with those same words again during
retrieval. In the current study, we confirmed the role auditory- and
visual-processing regions in the construction of mental represen-
tations during imagery and retrieval. Yet, this is the first study that
demonstrates the modality-specific role of auditory- and visual-
processing regions during both imagery and retrieval within the
same experiment and the same participants.
3.2. Modality-independent effects: Regions associated with
successful imagery and successful retrieval
The first set of regions included hippocampus, posterior cingu-
late cortex, medial prefrontal cortex and angular gyrus, and was
associated both with imagery success (successful>unsuccessful
imagery) and retrieval success (successful>unsuccessful retrieval;
Fig. 2). In line with several studies on episodic simulation of
novel, future and past events, we interpret the activation overlap
between imagery and retrieval as reflecting the (re)construction
and (re)experiencing of mental representations derived from
episodicmemory(Addisetal.,2007;Hassabisetal.,2007a;Szpunar
et al., 2007). The current study adds to these previous studies by
showing that these regions are specifically linked to processes sup-
porting successful imagery and retrieval. Below, we briefly discuss
the possible role of these regions in reconstructive mental pro-
cesses.
Mainmemorymodelsassumethatthehippocampusisinvolved
in the formation of episodic memory traces, and plays a critical
roleinreactivationofmemorytracesduringretrieval(Eichenbaum,
Yonelinas, & Ranganath, 2007; Squire, Stark, & Clark, 2004). At the
same time, models of mental imagery assume that information
derived from episodic memory plays an important role in success-
ful(re)constructionofmentalrepresentations(Hassabis&Maguire,
2007; Moulton & Kosslyn, 2009). Strong support for a role of the
hippocampus in imagery was provided by a recent study focus-
ing on patients with hippocampal damage. In addition to severe
episodic memory deficits, these patients show an impoverished
ability to imagine fictitious events, even though these events never
happened in their real lives (Hassabis, Kumaran, Vann, & Maguire,
2007). These findings clearly illustrate that episodic processes play
an important role in mental imagery regardless of whether events
truly occurred in the past. Extrapolating these findings to our cur-
rent results, we argue that the increase in hippocampal activity
during successful imagery reflects the activation of episodic infor-
mation, which contributes to the construction of a vivid and rich
experience of both past and imaginary events.
The role of PCC in imagery and retrieval also confirms previous
studies (Addis et al., 2007; Hassabis et al., 2007a; Szpunar et al.,
2007) and can be explained by the strong anatomical (Kobayashi
& Amaral, 2003) and functional connections (Greicius, Supekar,
Menon, & Dougherty, 2009) of this region with the hippocampus.
This region is among the areas most consistently activated during
episodic retrieval tasks (Cabeza & Nyberg, 2000) and lesions within
the PCC can result in amnesic symptoms similar to those resulting
from damage to the hippocampus (Valenstein et al., 1987). Here
we confirm these findings under controlled memory conditions
and show that imager- and retrieval-related activity in PCC occurs
independent of sensory modality.
MPFCwasalsoactivatedduringsuccessfulimageryandretrieval
of both auditory and visual information. This region tends to be
activated during vivid retrieval (Daselaar, Fleck, & Cabeza, 2006;
Yonelinas, Otten, Shaw, & Rugg, 2005) and during tasks that have a
strongself-referentialcomponent(Buckner&Carroll,2007;Cabeza
et al., 2004; Gusnard, Akbudak, Shulman, & Raichle, 2001; Kelley
et al., 2002). Although, our memory and imagery cues were not
necessarily relating to ones self, we also found mPFC activity asso-
ciated with successful imagery and retrieval in the current study.
Again,theoverlapbetweenimageryandretrievalinmPFCsuggests
asupportingroleofepisodicmemoryprocessesintheconstruction
ofmentalrepresentationsoffamiliarconcepts,evenwhentheseare
not necessarily referring to ones own personal past.
Despitethefactthattheangulargyrusisoneoftheregionsmost
consistently activated during successful retrieval, its role is still
a matter of current debate (Wagner, Shannon, Kahn, & Buckner,
2005). According to the episodic output buffer account, the angular
gyrus forms a specialized memory buffer for episodic-like infor-
mation possibly coming from the hippocampus (Vilberg & Rugg,
2008;Wagneretal.,2005).Thisideafitswellwithrecentviewsthat
activity in the angular gyrus is sensitive to the amount and qual-
ity of information that is being generated in one’s mind (Vilberg
& Rugg, 2008). Thus, in general the regional overlap between suc-
cessful imagery and retrieval fits well with previous findings and
currentviewsaboutthefunctionalroleoftheseregionsinmemory.
3.3. Modality-independent effects: Regions associated with
unsuccessful imagery and unsuccessful retrieval
The second set of regions including anterior cingulate cortex
(ACC), dorsal precuneus, superior/dorsolateral prefrontal cortex
(DLPFC) and bilateral insula showed activity associated with
unsuccessful performance during both imagery and retrieval inde-
pendent of sensory modality (Fig. 3). This pattern fits with a
role of these regions in effortful imagery and retrieval processes
associatedwithattentionandmonitoring.Intermsofretrieval,suc-
cessful retrieval conditions typically lead to faster responses made
with greater confidence, whereas unsuccessful retrieval conditions
lead to slower, low-confidence responses (e.g., Huijbers, Pennartz,
Cabeza, & Daselaar, 2009; Huijbers, Pennartz, & Daselaar, 2009).
Models of recognition memory assume that memory responses are
based on the accumulation of evidence for or against a memory
decision and the careful monitoring of retrieval output (Henson,
Rugg, Shallice, & Dolan, 2000; Henson, Shallice, & Dolan, 1999;
Ratcliff, 1978; Ratcliff & Starns, 2009; Rugg & Wilding, 2000;
Tulving, 1983). When the decision is more difficult more evidence
needs to be accumulated leading to slower response times. In
line with a general role in effortful decision making, the regions
activated during unsuccessful retrieval have also been linked to
effortful decisions outside the memory domain. For instance, Fleck
et al. (2006) found that ACC and DLPFC activity was not specific to
retrieval effort, but also correlated with effort during a perceptual
decision task (Fleck et al., 2006).
We also conducted a follow-up response time analysis focus-
ing only on the successful retrieval trials. This analysis revealed
that ACC and dorsal precuneus also showed greater activity during
slow as compared to fast correct memory trials, whereas the supe-
rior/dorsolateral PFC and the insula did not. These results generally
confirm previous findings by Fleck and colleagues who separated
the contributions of decision confidence and reaction times to ACC
and DLPFC activity during both a retrieval task and a perceptual
decision task (Fleck et al., 2006). Whereas the ACC was associated
more with slow response times, the DLPFC was associated more
with low levels of confidence. Based on these results, Fleck et al.
concluded that the ACC is involved in continuous conflict monitor-
ing,whereastheDLPFCisinvolvedintheevaluationofaccumulated

Page 9

Author's personal copy
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
1737
information and response selection. Although they focused their
analyses on the ACC and DLPFC, the dorsal precuneus and insula
were also associated with effortful processes during both memory
retrieval and perception. As noted, the current results extend these
memory and perception findings by suggesting a similar involve-
ment of these regions in mental imagery.
With respect to the specific role of the ACC, dorsal precuneus
and superior PFC/DLPFC, each of these regions have been linked to
attentionalprocessesinvolvedinthecarefulmonitoringandevalu-
ationofinformationactivatedduringthesearchforamemorytrace
(Fleck et al., 2006; Henson, Rugg, Shallice, Josephs, & Dolan, 1999;
Kim & Cabeza, 2009). Moreover, the perception task used in the
study by Fleck et al. involved a complex size judgment, which also
required careful evaluation of alternative options. Similar atten-
tional processes have been suggested to play an important role in
mental imagery by helping to initiate, inspect and terminate the
imagery process when a specific goal has been met (Moulton &
Kosslyn, 2009). The ACC and DLFPC have repeatedly been linked
to top-down attention mechanisms necessary for monitoring per-
formance and cognitive control (Egner & Hirsch, 2005; MacDonald,
Cohen, Stenger, & Carter, 2000). Also, the dorsal part of the pre-
cuneus is believed to support goal-directed imagery processes,
and has been linked to selection of relevant details during men-
tal imagery (Ganis, Thompson, & Kosslyn, 2004). Finally, superior
PFC has been strongly linked to a frontoparietal top-down atten-
tionsystem(Corbetta&Shulman,2002;Corbetta,Patel,&Shulman,
2008). Thus, the current findings linking these regions to general
effort fit well with previous findings. At the same time, we believe
that the insular cortex is not part of the attention/monitoring set
of regions. Activity in this region has been linked to performance
failure in various tasks (Daselaar, Prince, & Cabeza, 2004; Menon,
Adleman,White,Glover,&Reiss,2001),aswellastoelevatedlevels
of stress and arousal (Liotti et al., 2000; Paulus, Rogalsky, Simmons,
Feinstein,&Stein,2003).Thus,onepossibleexplanationforthislast
finding is that insular activity during imagery and retrieval reflects
high levels of arousal or stress associated with performance failure.
Finally, we should note that, even though our behavioral and fMRI
findings fit well with previous studies, we cannot exclude the pos-
sibility that some of the activations we found for the unsuccessful
vs. successful performance comparisons, reflect either inattention
to or distraction from the stimuli and task.
3.4. Modality-independent effects: Regions showing interaction
between task and performance
We also found interactions between mental imagery and mem-
ory retrieval. The left ventrolateral prefrontal cortex (VLPFC)
showed greater activity during successful imagery than success-
ful retrieval (Fig. 4). The left VLPFC is strongly activated during
semanticclassificationandproductiontasks(Grossmanetal.,2002;
Thompson, D’Esposito, Aguirre, & Farah, 1997; Vandenberghe,
Price, Wise, Josephs, & Frackowiak, 1996). One notable difference
between the tasks is that the imagery condition uses a novel cue
word describing a semantic entity, whereas the retrieval task uses
a previously studied cue. Thus compared to episodic retrieval,
successful mental imagery will rely more heavily on semantic
generation and elaboration processes to construct a mental repre-
sentation based on a novel imagery cue (Addis et al., 2007; Kosslyn
et al., 2001). A similar semantic generation strategy can be used
during retrieval. However, this will be particularly the case when
the memory cue does not automatically activate an appropriate
memory trace, and the retrieval search is extended and likely to
fail, which might explain the pattern we found in this region. Yet,
the current data do not allow any strong conclusions regarding the
role of VLPFC in imagery and retrieval.
Ventral precuneus, the midcingulate cortex and supramarginal
gyrus were associated with successful retrieval performance but
with unsuccessful imagery performance (Fig. 5). The finding that
ventral precuneus is associated with unsuccessful imagery is sur-
prising. Ventral precuneus is one of the regions most consistently
activatedduringtasksthatareheavilydependentonvisualimagery
(Fletcheretal.,1995;Kosslyn,2003).Forthisreason,thisregionhas
even been dubbed “the mind’s eye” (Fletcher et al., 1995; Kosslyn,
2003). Here, we confirm the involvement of ventral precuneus in
visual imagery, but our findings indicate that this region plays a
similar role in auditory imagery. Moreover, our data suggest that
activity in this region is not linked to successful imagery processes,
but rather to processes associated with unsuccessful imagery.
One interpretation for the interaction between unsuccessful
imagery and successful retrieval is provided by models of imagery
that include mental inspection and working memory processes
(Farah, 1984; Kosslyn et al., 1994; Moulton & Kosslyn, 2009).
These models state that during imagery, information first has
to become accessible within working memory, before it can be
mentally inspected. Similar to more semantic generation during
unsuccessful retrieval, relatively unsuccessful imagery may lead
to the generation of more alternative mental representations that
need to be selected and held in working memory as compared to a
successful imagery event.
A working memory interpretation fits well with evidence that
ventral precuneus, the midcingulate cortex and supramarginal
gyrus have all been linked to working memory maintenance pro-
cesses (Cohen et al., 1997; Petit, Courtney, Ungerleider, & Haxby,
1998; Schon, Quiroz, Hasselmo, & Stern, 2009). Moreover, these
regions are activated during tasks that tax working memory
demands, such as mental arithmetics, deductive reasoning and
visuospatial planning (Knauff, Mulack, Kassubek, Salih, & Greenlee,
2002; Menon, Rivera, White, Glover, & Reiss, 2000; van den Heuvel
et al., 2003). Damage to these regions results in a condition called
dyscalculia involving deficits in mental arithmetics (Jung et al.,
2001), as well as in impoverished autobiographical memories,
whichlackinexplicitperceptualdetails(Berryhill,Phuong,Picasso,
Cabeza, & Olson, 2007). These patient findings could easily be
accounted for by a working memory deficit. Thus, we speculate
that the supramarginal gyrus, midcingulate cortex and ventral
precuneus assist in the mental inspection of alternative mental
representations, and that this mental process is closely linked to
working memory. At the same time, we should acknowledge that
other interpretations of the imagery and retrieval activations in
these regions are possible. For instance, even though the follow-up
response time analysis does not seem to support this explanation
(see Supplemental Materials), the interaction between imagery
(unsuccessful>successful) and retrieval (successful> unsuccess-
ful) may still be related to the different types of ratings (quality
vs. confidence) in the two tasks. One explanation for the oppo-
site effect during retrieval – greater activity during successful vs.
unsuccessfulretrieval–isthatsuccessfulretrievalleadstothegen-
eration of episodic information, which needs to be maintained in
working memory, whereas unsuccessful retrieval may not. Future
researchwillbenecessarytofurtherseparatetheneuralandcogni-
tive components of unsuccessful memory processes, in particular,
the difference between multiple attempts vs. a failure to imag-
ine/retrieve information.
4. Conclusions
In this fMRI study, we contrasted the neural correlates of suc-
cessfulandunsuccessfulmentalimageryandmemoryretrieval.We
identified four distinct patterns of activity in several brain regions,
which were shared by auditory and visual modalities. The first
set of regions – including hippocampus posterior cingulate cortex,

Page 10

Author's personal copy
1738
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
medial prefrontal cortex and angular gyrus – was associated with
successful performance regardless of task. This pattern is consis-
tent with the conscious experiencing of past and imaginary events.
The second set of regions – including dorsal precuneus, anterior
cingulate and dorsolateral prefrontal cortex – showed the oppo-
site pattern, again common to imagery and retrieval and consistent
with attention/monitoring processes. Third, the left VLPFC showed
an interaction between task (imagery/retrieval) and performance
(successful/unsuccessful) This pattern is consistent with seman-
tic generation processes, that might assist mental imagery when
episodicretrievalisunsuccessful.Thefinalsetofregions–including
ventral precuneus, midcingulate cortex and supramarginal gyrus –
showed the opposite interaction between task and performance.
This pattern is consistent with components of working memory
that benefit retrieval and imagery. This is the first study to separate
the neural correlates of successful and unsuccessful performance
for both imagery and retrieval.
5. Methods
5.1.1. Participants
Twenty-oneparticipants(16female,meanage22)recruitedfromtheUniversity
ofAmsterdamcommunitytookpartintheexperiment.Allparticipantswereingood
health, and right-handed. Their native language was Dutch and they were paid 65
euroforparticipation.Allparticipantsgavetheirinformedconsentandthestudymet
thecriteriaforapprovaloftheAcademicMedicalCenterMedicalEthicalCommittee.
5.1.2. Stimuli
Thestimuliconsistedof456Dutchcue-words(nouns)specificallymatchedwith
456 corresponding images and 456 corresponding sounds. The matched sounds
were2-channelstereowithasamplerateof22kHz,16-bitsamplesize,WAV-format,
withadurationof3s.Thematchedimagesconsistedof640×480pixels,16-bitcolor,
BMP-format and were also presented for 3s. To mimic the dynamical characteristic
of sounds, the images faded-in (1s), stayed on the screen (1s) and then faded-out
(1s). Visual stimuli were projected on a screen at the foot of the fMRI scanner, and
seen via a mirror mounted on the head-coil. Auditory stimuli were presented via a
MR-compatible headphone with passive noise dampening (MR Confon). Behavioral
responses were collected by an MR-compatible four-button box (Lumitouch).
5.1.3. Procedure
Before the experiment, cue-words were randomly assigned to one of four con-
ditions for each individual subject: (1) imagine sound, (2) hear sound, (3) imagine
image or (4) observe image. The cue-words were carefully selected so they could
match each condition. Thus, for example, cue-words like “cat” or “airplane” can eas-
ilybepresentedorimaginedaseitherasoundorvisualimage.Beforethestartofthe
actual experiment, participants conducted a brief 20-trial practice session to check
the volume of the headphones and to habituate to the scanner noise. The presen-
tation order of the stimuli was pseudorandomized, to ensure that two consecutive
trials were never of the same condition. Because pilot studies had indicated that
the “imagine sound” condition was slightly more difficult and to ensure a sufficient
number of successful trials in each of the four conditions, we used 132 trials for the
“imagine sound” condition and 108 trials for the other conditions.
As shown in Table 1, the fMRI experiment consisted of an imagery task and a
retrieval task on consecutive days. Each task consisted of six runs, approximately
8min long and 76 trials each. During the imagery task, a single trial consisted of
a cue period, an imagery/perception period, and a rating period. During the cue
period, a 1000-ms cue-word was presented with an instruction to imagine or per-
ceiveasoundorimageassociatedwiththecueword.Duringtheimagery/perception
period, participants either imagined or perceived an image or sound according to
the instruction. During the 1500-ms rating period, participants rated on a four-
point scale (1=very low, 4=very high) the quality of their imagery experience or
the richness of their perceptual experience. Perceptual richness was defined as the
amount of subjectively experienced perceptual detail. Critically, participants were
notinformedthattheirmemoryfortheseeventswouldbetestedthenextdayduring
the retrieval task. A retrieval trial consisted of a retrieval period and a rating period.
During the 4000-ms retrieval period, the cue-words from the previous day were
presented again and participants indicated the condition in which they had seen
the word the day before (imagined sound/heard sound/imagined image/observed
image). During the 1500-ms rating period, participants rated the confidence of
theirjudgmentonatwo-pointscale(unsure/sure).Duringbothimagery/perception
and memory retrieval phases, the intertrial intervals were jittered between 100
and 2100ms. As illustrated in Table 1, analyses included both mental imagery
and perceptual data during the first day, but in keeping with previous fMRI stud-
ies that compared the neural correlates of imagery and retrieval (e.g. Buckner &
Wheeler, 2001; Cabeza, Locantore, & Anderson, 2003; Prince, Daselaar, & Cabeza,
2005; Spaniol et al., 2009), we only analyzed retrieval of perceived auditory and
visual events.
5.1.4. Scanning parameters
FunctionalMRIimageswerecollectedonaPhillipsIntera3.0Tusinga6-channel
SENSE head coil and a T2* sensitive gradient echo sequence (96×96 matrix, TR
2000ms, TE 30ms, FA 80◦, 34 slices, 2.3mm×2.3mm voxel size, 3-mm thick trans-
verseslices).Additionally,ahigh-resolutionT1-weightedstructuralscan(256×256
matrix, TR 12ms, TE 5ms, FOV 24cm, 68 slices, 1mm slice thickness) was collected
on the second day.
5.1.5. fMRI preprocessing and analysis
Statistical Parametric Mapping (SPM5; (http://www.fil.ion.ucl.ac.uk/spm) soft-
ware was used to preprocess and analyze the MR data. For each session, functional
images were slice-time corrected, motion-corrected, coregistered to the structural
scan acquired during the second day, and then normalized. First, individual nor-
malization parameters were obtained by normalizing the segmented structural
scan of each subject using the Montreal Neurological Institute (MNI) T1 tem-
plate image. These normalization parameters were then applied to the functional
images. Next, the normalized functional images were resliced to a resolution of
3mm×3mm×3mm and spatially smoothed using an 8-mm isotropic Gaussian
kernel. For each subject, trial related activity was modeled by convolving a vector
of trial onsets with a canonical hemodynamic response function (HRF). The General
Linear Model (GLM), as implemented in SPM5, was used to model effects of inter-
est and remove confounding effects. Statistical Parametrical Maps were identified
for each participant by applying linear contrasts to the parameter estimates (beta
weight) for the events of interest, resulting in a t-statistic for every voxel. Group
effects were assessed by applying random effects analyses.
We distinguished between 12 relevant trial types: low and high perception,
unsuccessful and successful imagery, and unsuccessful and successful retrieval of
perceived items, for both auditory and visual trials. We defined low perception as
rating 1 and 2 on the four-point quality/richness scale and high perception as rating
4.Similarly,wedefinedunsuccessfulimageryasrating1and2onthefour-pointqual-
ity/richness scale, and successful imagery as rating 4. We defined successful memory
asthecorrectresponsesthatwereratedasconfident,whileunsuccessfulmemorywas
defined as all incorrect responses regardless of confidence. We also modeled suc-
cessful and unsuccessful retrieval of previously imagined information separately,
but we do not report this data here (see Table 2). Other trials, including omitted
responses, were included as a separate trial type of no interest.
5.1.6. Modality-specific activation of sensory regions
Toassesswhether,inlinewithpreviousfMRIstudies,activityinsensoryregions
was associated with imagery and retrieval in a modality-specific fashion (Buckner
& Wheeler, 2001; Kosslyn et al., 2001; Kraemer et al., 2005; Shannon & Buckner,
2004), we used a four-step approach for each modality separately. For the auditory
modality, we first compared the externally presented items in the visual modality
to those presented in the auditory modality (high auditory perception>high visual
perception). Second, we identified regions related to auditory mental imagery (suc-
cessful auditory imagery>successful visual imagery). Third, we performed the same
step for auditory memory retrieval (successful auditory retrieval>successful visual
retrieval). Fourth, we inclusively masked the resulting T-maps to look for overlaps
betweenauditoryperception,auditoryimageryandauditoryretrieval.Foreachmap,
we used an uncorrected threshold of P<0.023 with a cluster size of 8 for each map.
Given that perception, imagery and retrieval were measured during independent
trials, the joint probability for three independent measures can be calculated using
the Fisher’s method, resulting in a joint probability of P<0.001 (Mosteller & Fisher,
1948). The same four steps were repeated for the visual modality.
5.1.7. Activation of modality-independent regions
To assess overlap in activations of modality-independent regions involved in
imagery and retrieval, we combined the auditory and visual trials. Next, we crossed
the factors task (memory/imagery) and performance (successful/unsuccessful) and
created maps using inclusive masking (also at P<0.01 uncorrected, c=8) for all four
combinations (1. [successful>unsuccessful retrieval]∩[successful>unsuccessful
imagery];2. [unsuccessful>successful
imagery]; 3. [successful>unsuccessful
imagery];4.[unsuccessful>successful
imagery]). Given that imagery and retrieval were measured during independent
trials, the joint probability can again be calculated using the Fisher’s method,
resulting also in a joint probability of P<0.001 (Mosteller & Fisher, 1948). To assess
whether the regions identified showed an activation pattern consistent with a
modality-independent role, we extracted their mean cluster activity and checked
whether the patterns were significant for both modalities at P<0.05, uncorrected.
retrieval]∩[unsuccessful>successful
retrieval]∩[unsuccessful>successful
retrieval]∩[successful>unsuccessful

Page 11

Author's personal copy
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
1739
Acknowledgements
The authors sincerely thank the reviewer for providing
constructive comments. Funding: WH. was supported by the Ams-
terdam Brain Imaging Platform (ABIP), CMP. by the Netherlands
Organization for Scientific Research (NWO) grant 918.46.609 and
SMD. by NWO grant 916.66.022 and ABIP.
Appendix A. Supplementary data
Supplementary data associated with this article can be found, in
theonlineversion,atdoi:10.1016/j.neuropsychologia.2011.02.051.
References
Addis, D. R., Pan, L., Vu, M. A., Laiser, N., & Schacter, D. L. (2009). Constructive
episodic simulation of the future and the past: Distinct subsystems of a core
brain network mediate imagining and remembering. Neuropsychologia, 47(11),
2222–2238.
Addis, D. R., Wong, A. T., & Schacter, D. L. (2007). Remembering the past and imagin-
ingthefuture:Commonanddistinctneuralsubstratesduringeventconstruction
and elaboration. Neuropsychologia, 45(7), 1363–1377.
Baddeley, A. (1992). What is autobiographical memory? In C. M. A. D. C. Rubin, H.
Spinnler, & W. A. Wagenaar (Eds.), Theoretical perspectives on autobiographical
memory (pp. 13–29). Dordrecht: Kluwer.
Bartlett, F. C. (1932). Remembering. Cambridge, UP: Cambridge.
Berryhill, M. E., Phuong, L., Picasso, L., Cabeza, R., & Olson, I. R. (2007). Parietal lobe
and episodic memory: Bilateral damage causes impaired free recall of autobio-
graphical memory. Journal of Neuroscience, 27(52), 14415–14423.
Buckner,R.L.,&Carroll,D.C.(2007).Self-projectionandthebrain.TrendsinCognitive
Sciences, 11(2), 49–57.
Buckner, R. L., Logan, J., Donaldson, D. I., & Wheeler, M. E. (2000). Cognitive neuro-
science of episodic memory encoding. Acta Psychologica, 105(2–3), 127–139.
Buckner, R. L., & Wheeler, M. E. (2001). The cognitive neuroscience of remembering.
Nature Reviews Neuroscience, 2(9), 624–634.
Bunzeck, N., Wuestenberg, T., Lutz, K., Heinze, H.-J., & Jancke, L. (2005). Scanning
silence: Mental imagery of complex sounds. Neuroimage, 26(4), 1119–1127.
Cabeza,R.,Locantore,J.K.,&Anderson,N.D.(2003).Lateralizationofprefrontalactiv-
ity during episodic memory retrieval: Evidence for the production-monitoring
hypothesis. Journal of Cognitive Neuroscience, 15(2), 249–259.
Cabeza, R., & Nyberg, L. (2000). Imaging cognition II: An empirical review of 275 PET
and fMRI studies. Journal of Cognitive Neuroscience, 12(1), 1–47.
Cabeza, R., Prince, S. E., Daselaar, S. M., Greenberg, D. L., Budde, M., Dolcos, F., et al.
(2004). Brain activity during episodic retrieval of autobiographical and labora-
tory events: An fMRI study using a novel photo paradigm. Journal of Cognitive
Neuroscience, 16(9), 1583–1594.
Cabeza, R., & St Jacques, P. (2007). Functional neuroimaging of autobiographical
memory. Trends in Cognitive Sciences, 11(5), 219–227.
Cavanna, A. E., & Trimble, M. R. (2006). The precuneus: A review of its functional
anatomy and behavioural correlates. Brain, 129(Pt 3), 564–583.
Chen, W., Kato, T., Zhu, X.-H., Ogawa, S., Tank, D. W., & Ugurbil, K. (1998). Human
primary visual cortex and lateral geniculate nucleus activation during visual
imagery. Neuroreport, 9(16), 3669–3674.
Cohen, J. D., Perlstein, W. M., Braver, T. S., Nystrom, L. E., Noll, D. C., Jonides, J., et al.
(1997). Temporal dynamics of brain activation during a working memory task.
Nature, 386(6625), 604–608.
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven
attention in the brain. Nature Reviews Neuroscience, 3(3), 201–215.
Corbetta, M., Patel, G., & Shulman, G. L. (2008). The reorienting system of the human
brain: from environment to theory of mind. Neuron, 58(3), 306–324.
Daselaar,S.M.,Fleck,M.S.,&Cabeza,R.(2006).Tripledissociationinthemedialtem-
poral lobes: Recollection, familiarity, and novelty. Journal of Neurophysiology,
96(4), 1902–1911.
Daselaar, S. M., Prince, S. E., & Cabeza, R. (2004). When less means more: Deacti-
vations during encoding that predict subsequent memory. Neuroimage, 23(3),
921–927.
Daselaar,S.M.,Rice,H.J.,Greenberg,D.L.,Cabeza,R.,LaBar,K.S.,&Rubin,D.C.(2008).
Thespatiotemporaldynamicsofautobiographicalmemory:Neuralcorrelatesof
recall, emotional intensity, and reliving. Cerebral Cortex, 18(1), 217–229.
Deacon, T., Kosslyn, S. M., & Scarry, E. (2001). Science, culture, meaning, values: A
dialogue. Annals of New York Academy Science, 935, 233–257.
Egner, T., & Hirsch, J. (2005). Cognitive control mechanisms resolve conflict through
cortical amplification of task-relevant information. Nature Neuroscience, 8(12),
1784–1790.
Eichenbaum, H., Yonelinas, A. P., & Ranganath, C. (2007). The medial temporal lobe
and recognition memory. Annual Reviews Neuroscience, 30, 123–152.
Farah, M. J. (1984). The neurological basis of mental imagery: A componential anal-
ysis. Cognition, 18(1–3), 245–272.
Fleck, M. S., Daselaar, S. M., Dobbins, I. G., & Cabeza, R. (2006). Role of prefrontal and
anterior cingulate regions in decision-making processes shared by memory and
nonmemory tasks. Cerebral Cortex,
Fletcher,P.C.,Frith,C.D.,Baker,S.C.,Shallice,T.,Frackowiak,R.J.,&Dolan,R.J.(1995).
The minds eye-precuneus activation in memory-related imagery. Neuroimage,
2(3), 195–200.
Ganis, G., Thompson, W. L., & Kosslyn, S. M. (2004). Brain areas underlying visual
mental imagery and visual perception: An fMRI study. Brain Research Cognitive
Brain Research, 20(2), 226–241.
Greenberg, D. L., & Rubin, D. C. (2003). The neuropsychology of autobiographical
memory. Cortex, 39(4–5), 687–728.
Greicius, M. D., Supekar, K., Menon, V., & Dougherty, R. F. (2009). Resting-state func-
tional connectivity reflects structural connectivity in the default mode network.
Cerebral Cortex, 19(1), 72–78.
Grossman, M., Koenig, P., DeVita, C., Glosser, G., Alsop, D., Detre, J., et al. (2002). The
neural basis for category-specific knowledge: An fMRI study. Neuroimage, 15(4),
936–948.
Gusnard, D. A., Akbudak, E., Shulman, G. L., & Raichle, M. E. (2001). Medial prefrontal
cortex and self-referential mental activity: Relation to a default mode of brain
function. Proceedings of the National Academy of Sciences of the United States of
America, 98(7), 4259–4264.
Halpern, A. R., & Zatorre, R. J. (1999). When that tune runs through your head: A PET
investigation of auditory imagery for familiar melodies. Cerebral Cortex, 9(7),
697–704.
Hassabis, D., Kumaran, D., & Maguire, E. A. (2007). Using imagination to under-
stand the neural basis of episodic memory. Journal of Neuroscience, 27(52),
14365–14374.
Hassabis, D., Kumaran, D., Vann, S. D., & Maguire, E. A. (2007). Patients with hip-
pocampal amnesia cannot imagine new experiences. Proceedings of the National
Academy of Sciences of the United States of America, 104(5), 1726–1731.
Hassabis, D., & Maguire, E. A. (2007). Deconstructing episodic memory with con-
struction. Trends in Cognitive Sciences, 11(7), 299–306.
Henson,R.,Rugg,M.,Shallice,T.,&Dolan,R.(2000).Confidenceinrecognitionmem-
ory for words. Journal of Cognitive Neuroscience, 23–123.
Henson, R. A., Rugg, M. D., Shallice, T., Josephs, O., & Dolan, R. J. (1999). Recollection
and familiarity in recognition memory: An event-related functional magnetic
resonance imaging study. Journal of Neuroscience, 19(10), 3962–3972.
Henson, R. A., Shallice, T., & Dolan, R. J. (1999). Right prefrontal cortex and episodic
memory retrieval: A functional MRI test of the monitoring hypothesis. Brain,
122(Pt 7), 1367–1381.
Huijbers, W., Pennartz, C. M., Cabeza, R., & Daselaar, S. M. (2009). When learning and
remembering compete: A functional MRI study. PLoS Biology, 7(1), e11.
Huijbers, W., Pennartz, C. M., & Daselaar, S. M. (2009). Dissociating the “retrieval
success” regions of the brain: Effects of retrieval delay. Neuropsychologia.
Ishai, A., Ungerleider, L. G., & Haxby, J. V. (2000). Distributed neural systems for the
generation of visual images. Neuron, 28(3), 979–990.
Jung, R. E., Yeo, R. A., Sibbitt, W. L., Jr., Ford, C. C., Hart, B. L., & Brooks, W. M. (2001).
Gerstmann syndrome in systemic lupus erythematosus: Neuropsychological,
neuroimaging and spectroscopic findings. Neurocase, 7(6), 515–521.
Kelley, W. M., Macrae, C. N., Wyland, C. L., Caglar, S., Inati, S., & Heatherton, T.
F. (2002). Finding the self? An event-related fMRI study. Journal of Cognitive
Neuroscience, 14(5), 785–794.
Kim, H., & Cabeza, R. (2009). Common and specific brain regions in high- versus
low-confidence recognition memory. Brain Research, 1282, 103–113.
Knauff, M., Mulack, T., Kassubek, J., Salih, H. R., & Greenlee, M. W. (2002). Spatial
imageryindeductivereasoning:AfunctionalMRIstudy.BrainResearchCognitive
Brain Research, 13(2), 203–212.
Kobayashi, Y., & Amaral, D. G. (2003). Macaque monkey retrosplenial cortex. II Cor-
tical afferents. Journal of Comparative Neurology, 466(1), 48–79.
Kosslyn, S. M. (2003). Understanding the mind’s eye and nose. Nature Neuroscience,
6(11), 1124–1125.
Kosslyn,S.M.,Alpert,N.M.,Thompson,W.L.,Chabris,C.F.,Rauch,S.L.,&Anderson,A.
K.(1994).Identifyingobjectsseenfromdifferentviewpoints,aPETinvestigation.
Brain, 117(Pt 5), 1055–1071.
Kosslyn, S. M., Ganis, G., & Thompson, W. L. (2001). Neural foundations of imagery.
Nature Reviews Neuroscience, 2(9), 635–642.
Kraemer, D. J., Macrae, C. N., Green, A. E., & Kelley, W. M. (2005). Musical imagery:
Sound of silence activates auditory cortex. Nature, 434(7030), 158.
Liotti, M., Mayberg, H. S., Brannan, S. K., McGinnis, S., Jerabek, P., & Fox, P. T. (2000).
Differentiallimbic–corticalcorrelatesofsadnessandanxietyinhealthysubjects:
Implications for affective disorders. Biological Psychiatry, 48(1), 30–42.
Marr, D. (1971). Simple memory: A theory for archicortex. Philosophical Trans-
actions of the Royal Society of London Series B: Biological Sciences, 262(841),
23–81.
McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are com-
plementary learning systems in the hippocampus and neocortex: Insights from
the successes and failures of connectionist models of learning and memory.
Psychological Review, 102(3), 419–457.
MacDonald, A. W., III, Cohen, J. D., Stenger, V. A., & Carter, C. S. (2000). Dissociating
the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive
control. Science, 288(5472), 1835–1838.
Menon, V., Adleman, N. E., White, C. D., Glover, G. H., & Reiss, A. L. (2001). Error-
relatedbrainactivationduringaGo/NoGoresponseinhibitiontask.HumanBrain
Mapping, 12(3), 131–143.
Menon, V., Rivera, S. M., White, C. D., Glover, G. H., & Reiss, A. L. (2000). Disso-
ciating prefrontal and parietal cortex activation during arithmetic processing.
Neuroimage, 12(4), 357–365.
Mosteller, F., & Fisher, R. A. (1948). Questions and answers. The American Statistician,
2(5), 30–31.

Page 12

Author's personal copy
1740
W. Huijbers et al. / Neuropsychologia 49 (2011) 1730–1740
Moulton, S. T., & Kosslyn, S. M. (2009). Imagining predictions: Mental imagery as
mental emulation. Philosophical Transactions of the Royal Society of London Series
B: Biological Sciences, 364(1521), 1273–1280.
Nyberg, L., Habib, R., Mcintosh, A. R., & Tulving, E. (2000). Reactivation of encoding-
related brain activity during memory retrieval. Proceedings of the National
Academy of Sciences of the United States of America, 97(20), 11120–11124.
Paulus,M.P.,Rogalsky,C.,Simmons,A.,Feinstein,J.S.,&Stein,M.B.(2003).Increased
activation in the right insula during risk-taking decision making is related to
harm avoidance and neuroticism. Neuroimage, 19(4), 1439–1448.
Petit, L., Courtney, S. M., Ungerleider, L. G., & Haxby, J. V. (1998). Sustained activity in
the medial wall during working memory delays. Journal of Neuroscience, 18(22),
9429–9437.
Prince,S.E.,Daselaar,S.M.,&Cabeza,R.(2005).Neuralcorrelatesofrelationalmem-
ory: Successful encoding and retrieval of semantic and perceptual associations.
Journal of Neuroscience, 25(5), 1203–1210.
Ratcliff, R. (1978). A theory of memory retrieval [review]. Psychological Review,
88(88), 552–572.
Ratcliff, R., & Starns, J. J. (2009). Modeling confidence and response time in recogni-
tion memory. Psychological Review, 116(1), 59–83.
Rubin, D. C., Schrauf, R. W., & Greenberg, D. L. (2003). Belief and recollection of
autobiographical memories. Memory & Cognition, 31(6), 887–901.
Rugg,M.D.,&Wilding,E.L.(2000).Retrievalprocessingandepisodicmemory.Trends
in Cognitive Sciences, 4(3), 108–115.
Schacter,D.L.,Addis,D.R.,&Buckner,R.L.(2007).Rememberingthepasttoimagine
the future: The prospective brain. Nature Reviews Neuroscience, 8(9), 657–661.
Schon, K., Quiroz, Y. T., Hasselmo, M. E., & Stern, C. E. (2009). Greater working mem-
ory load results in greater medial temporal activity at retrieval. Cerebral Cortex,
19(11), 2561–2571.
Shannon, B. J., & Buckner, R. L. (2004). Functional-anatomic correlates of mem-
ory retrieval that suggest nontraditional processing roles for multiple distinct
regions within posterior parietal cortex. Journal of Neuroscience, 24(45),
10084–10092.
Spaniol, J., Davidson, P. S., Kim, A. S., Han, H., Moscovitch, M., & Grady, C. L. (2009).
Event-related fMRI studies of episodic encoding and retrieval: Meta-analyses
using activation likelihood estimation. Neuropsychologia, 47(8–9), 1765–1779.
Squire, L. R., Stark, C. E., & Clark, R. E. (2004). The medial temporal lobe. Annual
Reviews Neuroscience, 27, 279–306.
Szpunar, K. K., Watson, J. M., & McDermott, K. B. (2007). Neural substrates of envi-
sioning the future. Proceedings of the National Academy of Sciences of the United
States of America, 104(2), 642–647.
Thompson, S. S. L., D’Esposito, M., Aguirre, G. K., & Farah, M. J. (1997). Role of left
inferior prefrontal cortex in retrieval of semantic knowledge: A reevaluation.
Proceedings of the National Academy of Sciences of the United States of America,
94(26), 14792–14797.
Tulving, E. (1983). Elements of episodic memory. Oxford: Clarendon Press.
Valenstein, E., Bowers, D., Verfaellie, M., Heilman, K. M., Day, A., & Watson, R. T.
(1987). Retrosplenial amnesia. Brain, 110(Pt 6), 1631–1646.
van den Heuvel, O. A., Groenewegen, H. J., Barkhof, F., Lazeron, R. H., van Dyck, R., &
Veltman,D.J.(2003).Frontostriatalsysteminplanningcomplexity:Aparametric
functional magnetic resonance version of Tower of London task. Neuroimage,
18(2), 367–374.
Vandenberghe, R., Price, C., Wise, R., Josephs, O., & Frackowiak, R. S. J. (1996). Func-
tional anatomy of a common semantic system for words and pictures. Nature,
383(6597), 254–256.
Vilberg, K. L., & Rugg, M. D. (2008). Memory retrieval and the parietal cortex: A
review of evidence from a dual-process perspective. Neuropsychologia, 46(7),
1787–1799.
Wagner, A. D., Shannon, B. J., Kahn, I., & Buckner, R. L. (2005). Parietal lobe con-
tributions to episodic memory retrieval. Trends in Cognitive Sciences, 9(9),
445–453.
Wheeler,M.E.,Petersen,S.E.,&Buckner,R.L.(2000a).Memory’secho:Vividremem-
bering reactivates sensory-specific cortex. Proceedings of the National Academy
of Sciences of the United States of America, 20, 11125–11129.
Wheeler,M.E.,Petersen,S.E.,&Buckner,R.L.(2000b).Memory’secho:Vividremem-
bering reactivates sensory-specific cortex. Proceedings of the National Academy
of Sciences of the United States of America, 97(20), 11125–11129.
Yonelinas, A. P., Otten, L. J., Shaw, K. N., & Rugg, M. D. (2005). Separating the brain
regions involved in recollection and familiarity in recognition memory. Journal
of Neuroscience, 25(11), 3002–3008.