Multisensory cortical processing of object shape and its relation to mental imagery.
ABSTRACT Here, we used functional magnetic resonance imaging to investigate the multisensory processing of object shape in the human cerebral cortex and explored the role of mental imagery in such processing. Regions active bilaterally during both visual and haptic shape perception, relative to texture perception in the respective modality, included parts of the superior parietal gyrus, the anterior intraparietal sulcus, and the lateral occipital complex. Of these bimodal regions, the lateral occipital complexes preferred visual over haptic stimuli, whereas the parietal areas preferred haptic over visual stimuli. Whereas most subjects reported little haptic imagery during visual shape perception, experiences of visual imagery during haptic shape perception were common. Across subjects, ratings of the vividness of visual imagery strongly predicted the amount of haptic shape-selective activity in the right, but not in the left, lateral occipital complex. Thus, visual imagery appears to contribute to activation of some, but not all, visual cortical areas during haptic perception.
- [show abstract] [hide abstract]
ABSTRACT: Agnosia is an abnormality of recognition that is not explained by sensory or cognitive disorders. We studied a patient who had combined visual-tactile agnosia without prosopagnosia after a left hemisphere infarct. Although he copied figures presented visually and he performed intramodal or crossmodal visual-tactile matches, he could not indicate recognition by either sight or touch. The lesion involved areas of the occipital and temporal lobes that may be important for human performance of tasks that require construction and application of meaning to percepts presented both visually and tactually.Neurology 07/1986; 36(6):864-7. · 8.25 Impact Factor
- [show abstract] [hide abstract]
ABSTRACT: Regional cerebral blood flow was measured with positron emission tomography during the performance of tasks that required cognitive spatial transformations of alphanumeric stimuli. In the mirror image task, the subjects were required to discriminate between the normal and the mirror images of alphanumeric stimuli presented in the upright orientation. In the mental rotation task, the same judgement was required, but now the stimuli were presented in various orientations other than the upright one. The subjects therefore had to rotate the stimuli, in mind, into the upright position before making their decision. In relation to the control task, which involved discrimination of these same stimuli but not any form of mental transformation, there were significant increases in the right postero-superior parietal cortex and the left inferior parietal cortex in both experimental tasks. For mental rotation, specific activity was seen only within the left inferior parietal region and the right head of the caudate nucleus. These results specified the parietal areas involved in a purely cognitive spatial process and demonstrated a close interaction between these areas and the anterior neostriatum and certain lateral frontal cortical areas in the discrimination of rotated forms of stimuli.Neuropsychologia 03/1997; 35(2):111-8. · 3.48 Impact Factor
- [show abstract] [hide abstract]
ABSTRACT: Explicit memory tests such as recognition typically access semantic, modality-independent representations, while perceptual implicit memory tests typically access presemantic, modality-specific representations. By demonstrating comparable cross- and within-modal priming using vision and haptics with verbal materials (Easton, Srinivas, & Greene, 1997), we recently questioned whether the representations underlying perceptual implicit tests were modality specific. Unlike vision and audition, with vision and haptics verbal information can be presented in geometric terms to both modalities. The present experiments extend this line of research by assessing implicit and explicit memory within and between vision and haptics in the nonverbal domain, using both 2-D patterns and 3-D objects. Implicit test results revealed robust cross-modal priming for both 2-D patterns and 3-D objects, indicating that vision and haptics shared abstract representations of object shape and structure. Explicit test results for 3-D objects revealed modality specificity, indicating that the recognition system keeps track of the modality through which an object is experienced.Psychonomic Bulletin & Review 01/1997; 4(3):403-410. · 2.61 Impact Factor
251Copyright 2004 Psychonomic Society, Inc.
Cognitive, Affective, & Behavioral Neuroscience
2004, 4 (2), 251-259
It is now well established that visual cortical activity reg-
ularly accompanies normal human tactile perception. This
was first shown in a functional neuroimaging study from
our laboratory (Sathian, Zangaladze, Hoffman, & Grafton,
1997). In this study, tactile discrimination of grating ori-
entation recruited an extrastriate region of the parieto-
occipital cortex (POC) that had previously been reported
to be active during visual discrimination of grating orien-
tation (Sergent, Ohta, & MacDonald, 1992). We later demon-
strated that transcranial magnetic stimulation over the
POC disrupts the tactile ability to perform grating orien-
tation discrimination (Zangaladze, Epstein, Grafton, &
Sathian, 1999), thus proving that the POC activity con-
tributes to optimal tactile perception. Many groups, in-
cluding ours, subsequently have found that various visual
cortical areas are engaged during tactile perception in a
task-specific manner, so that areas of the visual cortex are
active during particular types of processing, not only for
visual, but also for tactile stimuli (Sathian, Prather, &
Zhang, 2004). Thus, perception of both tactile and visual
motion recruits the human MT complex (Blake, Sobel, &
James, 2004; Hagen et al., 2002), and mental rotation of
both tactile (Prather, Votaw, & Sathian, 2004) and visual
(Alivisatos & Petrides, 1997; Cohen etal., 1996) forms ac-
tivates a common focus in the intraparietal sulcus (IPS).
Haptic object identification, which has been studied by a
number of workers (Amedi, Jacobson, Hendler, Malach, &
Zohary, 2002; Amedi, Malach, Hendler, Peled, & Zohary,
2001; James et al., 2002; Stoeckel et al., 2003), elicits ac-
tivity in a temporo-occipital region in the ventral visual path-
way known as the lateral occipital complex (LOC). LOC
activation also occurs during tactile perception of two-
dimensional (2-D) forms, as shown by recent studies from
our laboratory (Prather et al., 2004; Stoesz et al., 2003).
Although there is agreement that tactile perception re-
cruits visual cortical regions, the reasons for such recruit-
ment remain unclear. We have suggested that visual imagery
could be responsible, since macrospatial tasks, which show
a greater tendency to trigger visual imagery than do mi-
crospatial tasks, are also accompanied by more visual corti-
cal involvement (Sathian & Zangaladze, 2001; Sathian etal.,
1997; Stoesz etal., 2003; Zangaladze etal., 1999). However,
other groups do not agree with this view, favoring instead
the idea that both visual and haptic processing engage a
common multisensory representation (Amedi et al., 2002;
Amedi et al., 2001; James et al., 2002). The present study
used functional magnetic resonance imaging (fMRI) to in-
vestigate (1)which regions of the sensory cortex are shape se-
lective during both visual and haptic perception, (2)whether
responses in these multisensory regions are stronger for vi-
sual or haptic presentations, and (3) the relationship of ac-
tivity in multisensory regions to mental imagery.
This work was supported by Grant RO1 EY 12440 to K.S. from the
National Eye Institute at the NIH. We thank Xiaoping Hu for his help,
Rainer Goebel and Norihiro Sadato for their suggestions, and our sub-
jects for their willing participation. Correspondence concerning this ar-
ticle should be addressed to K. Sathian, Department of Neurology,
Emory University School of Medicine, WMRB 6000, 1639 Pierce Drive,
Atlanta, GA 30322 (e-mail: firstname.lastname@example.org).
Multisensory cortical processing of object shape
and its relation to mental imagery
Emory University School of Medicine, Atlanta, Georgia
and Zhejiang University Medical School, Hangzhou, China
VALERIE D. WEISSER, RANDALL STILLA, S. C. PRATHER,and K. SATHIAN
Emory University School of Medicine, Atlanta, Georgia
Here, we used functional magnetic resonance imaging to investigate the multisensory processing of
object shape in the human cerebral cortex and explored the role of mental imagery in such processing.
Regions active bilaterally during both visual and haptic shape perception, relative to texture perception
in the respective modality, included parts of the superior parietal gyrus, the anterior intraparietal sul-
cus, and the lateral occipital complex. Of these bimodal regions, the lateral occipital complexes preferred
visual over haptic stimuli, whereas the parietal areas preferred haptic over visual stimuli. Whereas most
subjects reported little haptic imagery during visual shape perception, experiences of visual imagery
during haptic shape perception were common. Across subjects, ratings of the vividness of visual im-
agery strongly predicted the amount of haptic shape-selective activity in the right, but not in the left,
lateral occipital complex. Thus, visual imagery appears to contribute to activation of some, but not all,
visual cortical areas during haptic perception.
252ZHANG, WEISSER, STILLA, PRATHER, AND SATHIAN
Eight subjects (4 females and 4 males) took part in this study, after
giving informed consent. Their ages ranged from 19 to 25 years
(mean, 21.6 years). Seven were right-handed, and 1 was left-handed,
as assessed by the high-validity subset of the Edinburgh handedness
inventory (Raczkowski, Kalat, & Nebes, 1974). All the subjects were
neurologically normal and had vision that was normal or corrected
to normal with contact lenses. Subjects with callused fingerpads or
a history of injury to the hands or their innervation were excluded,
as were those with a history of dyslexia, which is associated with
tactile impairments (Grant, Zangaladze, Thiagarajah, & Sathian,
1999; Sathian et al., 2003). All the procedures were approved by the
Institutional Review Board of Emory University.
Stimuli and Tasks
There were four experimental conditions, two haptic and two vi-
sual, with a shape and texture condition in each modality. During the
haptic conditions, the subjects actively explored various objects or
surfaces with the thumb and first two fingers of the right hand. Dur-
ing the visual conditions, the subjects viewed photographs of vari-
ous objects or surfaces displayed centrally on a black screen. In the
shape conditions, the subjects were instructed to covertly think of
words that were descriptive of the geometrical shape of each object
presented. In the texture conditions, they were instructed to covertly
think of words that described the texture of each surface encoun-
tered. The subjects were given examples of each kind of stimulus
prior to scanning. These examples were not repeated during subse-
In the haptic shape (HS) condition, the stimuli were objects cut
out of a stiff foam material. Object shape varied in two dimensions,
and object size was designed to allow easy exploration of contour.
In the haptic texture (HT) condition, different materials were affixed
to both sides of a square piece of cardboard. The shape stimuli were
all of the same texture, and the texture stimuli were all of the same
shape. The subjects’ eyes were closed during haptic exploration, and
subjects were never allowed to see the haptic stimuli. The images used
for visual presentations were scanned into a computer, grayscaled,
and resized to 5º square, using Adobe Photoshop. The visual shape
(VS) condition used photographs of objects or parts of objects, ob-
tained from Microsoft Clip Art or published collections (Blossfeldt,
1997a,1997b; Feinstein, 2001). The subjects could choose to describe
the entire visible shape or the shape of a particular component. For
the visual texture (VT) condition, commonly available materials or
photographs of common textures (Brodatz, 1966) were used to gen-
erate the stimuli.
The stimuli, which are listed in Table 1, were chosen on the basis
of pilot studies in other subjects. The criteria were that a given stim-
ulus evoked either geometric or textural descriptions, but not both,
and that similar descriptors were used by the subjects for both visual
and haptic presentations. Our assumption was that asking the sub-
jects to focus on generating descriptions of either shape or texture
would elicit processing of the relevant attributes, while keeping lin-
guistic demands constant.
Magnetic Resonance (MR) Scanning
MR scans were carried out on a 3 Tesla Siemens Trio whole-body
scanner, using a standard head coil. Blood oxygenation level-
dependent (BOLD) contrast (T2* weighted) functional images were
acquired using a single-shot gradient-recalled echoplanar imaging
sequence. Thirty horizontal slices of 4-mm thickness were acquired
with the following parameters: repetition time (TR), 3,000 msec;
echo time (TE), 31msec; matrix size, 64 ? 64 ?30; flip angle, 90º;
in-plane resolution, 3.4 ? 3.4 mm. High-resolution anatomic im-
ages were also acquired, using a 3-D magnetization-prepared rapid
gradient-echo sequence (Mugler & Brookman, 1990) comprising 176
sagittal slices of 1-mm thickness (inversion time [TI], 900msec; TR,
2,600 msec; TE, 3.9 msec; flip angle, 8º; matrix size, 176 ? 208 ?
256; in-plane resolution, 1 ? 1 mm).
The subject lay supine in the scanner with the right arm out-
stretched. The fourth and fifth fingers of the right hand were flexed
and immobilized with adhesive tape secured to the palm. Foam
padding under the body and beside the right arm was used to mini-
mize movement and transfer of vibration from the gradient coil,
while ensuring the subject’s comfort. A mirror positioned above the
subject’s eyes provided unobstructed visualization of images pro-
jected on a screen at the rear magnet aperture. Head restraint straps
and foam pads were utilized to minimize head movement. The sub-
jects used ear plugs to muffle scanner noise; additional noise attenua-
tion was provided by headphones that also served to convey verbal in-
structions. The subjects’ eyes remained open throughout the entire
functional sequence. The subjects were instructed to fixate on a fixa-
tion cross in the center of a black screen during the rest baseline and
haptic conditions and to keep the right hand still during the visual con-
ditions. During each haptic trial, an experimenter manually placed the
stimulus in the subject’s right hand, removing it at the end of the trial.
No overt response was required during fMRI scanning. We assumed
that activity related to hand movements (for the two haptic conditions)
and eye movements (for the two visual conditions) would be similar.
A block design paradigm was employed, comprising alternating
blocks with and without stimulation. Block duration was 21 sec. Im-
mediately preceding each stimulation block, a screen was presented
Stimuli Used During fMRI Scanning
and Typical Descriptions Obtained
part of guard rail
part of fire hydrant
part of hot air balloon
peas in a pod
Spanish roof tiles
part of a waffle
MULTISENSORY CORTICAL SHAPE PROCESSING253
for 1 sec to cue the subject as to which condition followed. The words
used to identify each condition were tactile texture, tactile shape, visual
texture, visual shape, and, for the rest condition, fixate. Each stimu-
lation block contained three stimulus presentations of 6 sec apiece,
with a 1-sec interstimulus interval. The timing of haptic stimulus
presentation was guided by cues displayed to the experimenter on a
screen. There were 12 stimulation blocks (3 per condition) and 13
rest blocks. This resulted in a total of nine trials per condition. A
new stimulus was used in each trial. The stimulation blocks were in-
terleaved in a predetermined pseudorandom sequence, using Pre-
sentation software (Neurobiobehavioral Systems, Albany, CA) hat
also controlled visual stimulus timing and the display of cues to the
subject and the experimenter.
Following MR scanning, the Vividness of Visual Imagery Ques-
tionnaire (VVIQ) (Marks, 1973) was orally administered to the sub-
jects with their eyes closed. This questionnaire uses a scale of 1–5
(1 being the most vivid) to rate the vividness with which common-
place items can be visually imaged. It thus provides a general index
of image vividness and has been used previously as a correlational
probe for mediation of haptic processes via visual imagery (e.g.,
Lederman, Klatzky, Chataway, & Summers, 1990). The subjects were
familiarized with the rating scale before administration of the ques-
tionnaire. In addition, the objects and surfaces used during scanning
were presented again, and the subjects were asked to report the
words they thought of during scanning. At this time, using the VVIQ
rating scale, each subject was also asked to provide one score for the
vividness of visual imagery experienced overall during HS percep-
tion (VI_HS) and another score for the vividness of haptic imagery
experienced overall during VS perception (HI_VS). These ratings
were intended to index cross-modal mental imagery during shape
perception and were deliberately administered after the scanning
session so as to avoid biasing the subjects toward the use of such im-
agery during fMRI scanning.
Image Processing and Analysis
Image processing and analysis was performed using BrainVoy-
ager 2000 Version 4.91 (Brain Innovation, Maastrict, The Nether-
lands). Each subject’s BOLD images were realigned to the middle
image of the series, using a rigid-body transformation procedure.
Functional 2-D images were preprocessed with trilinear interpola-
tion for motion correction, sinc-interpolation for slice scan time cor-
rection, and high-pass temporal filtering at 1Hz to remove slow time
drifts and then were coregistered with the anatomic images and trans-
formed into Talairach space (Talairach & Tournoux, 1988). For group
analysis, the transformed data were spatially smoothed with an
isotropic Gaussian kernel (full width half maximum ? 8 mm) and
were z-normalized. Statistical analysis of group data used fixed-
effects, general linear models. Activations were localized with respect
to 3-D cortical anatomy with the help of an MRI atlas (Duvernoy, 1999).
Multiple analyses were performed. First, we identified regions
that were bimodally shape selective, using the conjunction of contrasts
(HS ? HT) ? (VS ? VT). In this conjunction analysis, activations
were Bonferroni corrected for multiple comparisons ( p ? .01)
within the entire image space. Second, at the peak voxels of the re-
gions of interest (ROIs) thus identified, we compared activity for VS
and HS on a direct contrast between these two conditions. Since this
comparison was restricted to ROIs taken from the first analysis, a
statistical threshold of p ? .05 (uncorrected) was used. Both these
analyses were performed on the grouped data. Finally, we investi-
gated the relationship across subjects between the activation strengths
in selected ROIs and imagery ratings, by performing correlations
and regressions. This level of analysis employed data from individ-
ual subjects in a random-effects design, the individual fMRI data
being used without spatial smoothing or z-normalizing.
Six regions (Figure1, Table2) were bimodally shape se-
lective on the conjunction analysis of (HS ? HT)?(VS ?
VT). These regions were located bilaterally in the superior
parietal gyrus (SPG), the anterior IPS (aIPS), and the
LOC. Figure2 shows time course data from the activation
maxima of these ROIs, demonstrating clearly that the
evoked BOLD signal was higher in each shape condition
than in the texture condition of the corresponding modal-
ity. Furthermore, it is apparent that there was varying
modality preference in different regions. Statistical analy-
sis of such modality preferences at the maxima of these
ROIs, using the VS ?HS contrast (Table 2), revealed that
three of the four more dorsal ROIs (the left SPG and the
paired aIPS foci) significantly preferred haptic over visual
presentation. The right SPG also showed a preference in
the same direction, but this was not statistically signifi-
cant. In both ventral regions (the paired LOC foci), there
was a significant preference for visual over haptic stimuli.
Figure 1. Bimodal shape-selective activations (conjunction of HS ? HT and VS ? VT), displayed on horizontal MR slices
taken from one subject. Display threshold, t ? 4.4; Talairach z value is below and t scale to the right of each image. SPG: su-
perior parietal gyrus; aIPS, anterior intraparietal sulcus; LOC, lateral occipital complex.
z = 55
z = 40
z = –7
254 ZHANG, WEISSER, STILLA, PRATHER, AND SATHIAN
Table 1 lists the typical descriptions used by the sub-
jects for the various shape and texture stimuli. As in the
pilot studies, the descriptors were generally similar across
modalities, while differing for tasks (shape vs. texture).
Although both visual imagery ratings (VVIQ and VI_HS)
showed a good deal of intersubject variability, the haptic
imagery ratings (HI_VS) were much less variable and re-
flected little tendency to use haptic imagery during VS
(Table 3). Hence, the haptic imagery ratings will not be
Individual Subject Analyses
In order to address the potential relationship between
visual cortical activation and visual imagery during haptic
perception, we focused on the two regions with a signifi-
cant preference for visual over haptic stimuli (i.e., both
LOC ROIs). These ROIs were defined in individual sub-
jects, using the VS ? VT contrast, which is an accepted
method of defining the LOC (Malach etal., 1995). A con-
sistent statistical threshold of t ? 2.8 (p ? .006, uncor-
rected) was used in all the subjects. The ROIs were con-
strained to be a maximum of 729 mm3in volume (i.e., a
cube of 9-mm side). As Table 3 shows, the ROIs were all
smaller than this arbitrary maximum. In each subject, over
the ROIs thus defined, the beta weights for the VS ? VT
and HS ? HT contrasts were determined (Table 3). These
beta weights were taken to index the strength of shape-
selective activity in the LOC. As Table 3 shows, the beta
weights and the visual imagery ratings exhibited substan-
tial intersubject variability. Hence, these beta weights and
the visual imagery ratings were used to perform the cor-
relations listed in Table 4.
A few points emerge from these correlations. First, the
two ratings of visual imagery, VVIQ and VI_HS, were es-
sentially uncorrelated. Second, the beta weights in the left
and the right LOC were significantly correlated for visual,
but not haptic shape-selective activity. Third, the beta
weights in each LOC were uncorrelated between visual
and haptic shape-selective activity. Fourth, the beta weight
for visual shape-selective activity in the right, but not the
left, LOC correlated significantly with VVIQ, whereas
neither of these beta weights correlated with VI_HS. Fi-
nally, the beta weights for haptic shape-selective activity
were not significantly correlated with either imagery rat-
ing, although the correlations for the right LOC were
fairly high and approached significance. Note that, owing
to the nature of the rating scales used (1 being the most
vivid and 5 the least vivid), a negative correlation with an
imagery rating score indicates that more vivid imagery is
associated with stronger activations.
Given that the VVIQ and the VI_HS scores were un-
correlated, we next asked whether the combination of
these two scores could predict the strength of LOC activ-
ity. This possibility seemed especially attractive in the
case of the right LOC for the haptic shape-selective beta
weights, where each imagery score yielded a relatively
high correlation, albeit short of significance. To test this
idea, a multiple regression analysis was performed for
each LOC, using the haptic shape-selective beta weight as
the dependent variable and the two visual imagery scores
as independent variables. The results are shown in Table5.
(The single regressions on VVIQ and VI_HS do not add
much information to the correlations already considered
and are included in Table 5 merely for emphasis.) The
main point is that the multiple regression has no predictive
value for the left LOC, whereas for the right LOC, it is not
only highly significant but also accounts for 90% of the
variance in the haptic shape-selective beta weight.
Although one of our subjects (No. 8) was left-handed,
we do not believe that this materially affects our results. The
imagery scores and beta weights did not stand out in any
way from the other subjects’. Furthermore, we repeated the
regression analysis of LOC beta weights on imagery rat-
ings with this subject’s data excluded and found that the
results were essentially similar (data not shown).
The main conclusions of the present study are that (1)a
set of bilateral regions in the SPG, aIPS, and LOC is shape
selective during both visual and haptic perception, (2)these
bimodal regions can be segregated into a dorsal, parietal
group (bilateral SPG and aIPS) showing a relative prefer-
Regions Active During Both Visual and Haptic Shape Perception
(VS ? VT) ? (HS ? HT)
(V ? H)
Note—See the text for details and for abbreviations; x, y, z, Talairach coordinates of ac-
tivation maxima in grouped data for the conjunction of visual (VS ? VT) and haptic
(HS ? HT) shape effects; tmax values are listed for these sites (p ? .01, Bonferroni-
corrected for multiple comparisons within the entire brain). Rightmost column lists
t values (at maxima of the conjunction ROIs) for visual preference (V ? H; negative
values represent H ? V).*Significant at p ? .05, uncorrected.
MULTISENSORY CORTICAL SHAPE PROCESSING255
ence for haptic stimuli and a ventral, occipito-temporal
group (bilateral LOC) showing a relative preference for
visual stimuli, and (3) across subjects, the strength of ac-
tivation by HS perception in the right, but not the left, LOC
is predicted by ratings of the vividness of visual imagery.
A few words on experimental design are in order. Our
design relied on contrasting shape and texture perception
in each modality. The contrast between shape and texture
conditions is a standard method of investigating shape-
selective processing (Amedi et al., 2002; Amedi et al.,
2001; Malach etal., 1995). Although the subjects engaged
only in covert description of shapes and textures, their
subsequent recall of the descriptors used during scanning
affords some measure of confidence that they did, indeed,
focus on the relevant stimulus attributes in each condition.
Moreover, there was statistically significant bimodal shape
selectivity in a number of regions, at a conservative sta-
tistical threshold, despite the fact that the shape and tex-
ture stimuli were not strictly balanced for their detailed
stimulus characteristics. Since it is difficult to achieve such
balance for haptic stimuli, we also chose not to attempt it
for visual stimuli.
Cortical Areas Common to VS
and HS perception
The LOC was originally described as a visual-object–
selective region in the occipito-temporal cortex (Malach
et al., 1995) and is thought to be homologous to macaque
Figure2. Time course of BOLD signal change (mean ?SEM) at maxima of activations shown in
Figure 1; each scan lasted 3 sec. See text for details. R, right; L, left; SPG, superior parietal gyrus;
aIPS, anterior intraparietal sulcus; LOC, lateral occipital complex.
BOLD signal change (%)
BOLD signal change (%)
BOLD signal change (%)
BOLD signal change (%)
BOLD signal change (%)
BOLD signal change (%)
256ZHANG, WEISSER, STILLA, PRATHER, AND SATHIAN
inferotemporal cortex (Grill-Spector etal., 1998). Not sur-
prisingly, we found a relative preference for visual over
haptic stimuli in the LOC in both hemispheres. LOC ac-
tivity in the present study was also selective for object
geometry, relative to texture, whether stimuli were pre-
sented visually or haptically, as was described previously
(Amedi et al., 2002; Amedi et al., 2001). Prior studies
showed that parts of the LOC in both hemispheres are ac-
tive during both visual and haptic object exploration
(Amedi et al., 2002; Amedi et al., 2001; James et al.,
2002). The LOC has been found, in other studies from our
laboratory (Prather et al., 2004; Stoesz et al., 2003), to be
recruited by tactile perception of 2-D forms presented to
the passive right hand. One of these studies showed LOC
activity only on the right (Prather et al., 2004), whereas the
other showed bilateral activity (Stoesz et al., 2003). In the
present study, the activations were correlated in strength
between the two sides for VS, which was to be expected
Correlations Across Subjects Between Beta Weights for Visual (VS ?
VT) and Haptic (HS ? HT) Shape Effects and Visual Imagery Scores
Variable 1 Variable 2
Correlations Between Visual Imagery Scores
Correlations Between Activity in Left and Right LOC
L LOC VS ? VTR LOC VS ? VT
L LOC HS ? HT R LOC HS ? HT
Correlations Between Visual and Haptic Shape-Selective Activity
L LOC VS ? VTL LOC HS ? HT
R LOC VS ? VT R LOC HS ? HT
Correlations Between Visual Shape-Selective Activity and Visual Imagery Scores
L LOC VS ? VT VVIQ
L LOC VS ? VTVI_HS
R LOC VS ? VTVVIQ
R LOC VS ? VTVI_HS
Correlations Between Haptic Shape-Selective Activity and Visual Imagery Scores
L LOC HS ? HTVVIQ
L LOC HS ? HTVI_HS
R LOC HS ? HTVVIQ
R LOC HS ? HTVI_HS
Note—L, left; R, right; LOC, lateral occipital complex.
tion (p ? .05).
Individual Subject Data, Including Imagery Scores and Details
of Lateral Occipital Complex (LOC) Regions of Interest (ROIs;
Volume, Talairach Coordinates of Activation Maxima in Each Subject,
and Beta Weights for Visual (VS ? VT) and Haptic (HS ? HT) Shape Effects)
Left LOC ROI
VS ? VT
HS ? HT
Right LOC ROI
VS ? VT
HS ? HT
Note—See the text for further details and for abbreviations.
MULTISENSORY CORTICAL SHAPE PROCESSING 257
given that our visual stimuli spanned the midline. The LOC
is engaged bilaterally by a stimulus in either visual field
(Grill-Spector etal., 1998); it would be interesting to know
whether activity levels between the two sides are correlated
for unilateral stimuli. For HS with stimuli presented to the
right hand, left and right LOC beta weights were uncorre-
lated in the present study, suggesting that different mech-
anisms might be involved in the two hemispheres. This is
supported by the correlational analysis, as will be discussed
below. Furthermore, the lack of correlation on either side
between VS and HS activation strengths implies that in-
terindividual variations in activity levels may not depend
on the same factors in both sensory modalities.
The cortex around the IPS was reported previously to
be active during both visual and haptic object perception
(Amedi etal., 2001), but the exact parts of the IPS involved
were not specified. The present study localizes common
visual and haptic processing of shapes to foci in the aIPS
bilaterally. The left aIPS site was termed IPA by Bodegård,
Geyer, Grefkes, Zilles, & Roland (2001), who character-
ized this focus as unimodal, processing somatosensory,
but not visual, information. However, this characterization
was based on comparisons across different studies, whereas
the present study, in which stimuli were presented in both
modalities, demonstrates that this is actually a multisen-
sory focus. This is not surprising, given its proximity to
other, more laterally located multisensory foci that are ac-
tive during various tasks, including visual or haptic object
recognition and visuo-haptic matching of objects (Gre-
fkes, Weiss, Zilles, & Fink, 2002), attending to contralat-
eral visual or tactile stimuli (Macaluso, Frith, & Driver,
2002), mentally rotating stimuli in visual (Cohen et al.,
1996) or tactile (Prather et al., 2004) space, and motion
processing of visual, auditory, or tactile stimuli (Bremmer
et al., 2001). The right aIPS focus has not, to our knowl-
edge, previously been reported to be multisensory. The
preference, in the present study, of both aIPS sites for hap-
tic over visual stimuli suggests that somatosensory infor-
mation is likely to be dominant at these foci.
In the present study, the SPG was also recruited bilat-
erally during HS, as well as VS, perception, with stronger
activation for haptic than for visual stimuli (although the
difference between modalities was significant only for the
left SPG). Other studies have implicated these foci in both
the spatial deployment of visual attention (Leonards,
Sunaert, Van Hecke, & Orban, 2000; Vandenberghe, Gitel-
man, Parrish, & Mesulam, 2001) and kinesthesis (Binkof-
ski et al., 1999; Lloyd, Shore, Spence, & Calvert, 2003),
so that there is a precedent for considering these regions
to be multisensory. How contributions to shape perception
might differ between the various parietal foci and between
these foci and the LOCs is presently unclear.
Relation of Multisensory Activity
to Visual Imagery
One question that has vexed the field is whether corti-
cal regions associated primarily with one modality are re-
cruited cross-modally by bottom-up (feedforward) inputs,
top-down (feedback) inputs, or both. We have previously
suggested, on the basis of observations that macrospatial
tasks tend to evoke both more visual imagery and more vi-
sual cortical involvement than do microspatial tasks
(Sathian & Zangaladze, 2001; Sathian et al., 1997; Stoesz
et al., 2003; Zangaladze et al., 1999), that visual imagery
could mediate cross-modal engagement of visual cortical
areas during tactile tasks. According to this idea, visual
cortical areas are recruited by tactile tasks via top-down
mechanisms, perhaps because these areas are best suited
to or most frequently used for the analysis of object geom-
etry. This can be viewed as a specific instantiation of a gen-
eral principle that information is translated into the format
appropriate for the most adept modality, as was proposed 3
decades ago (Freides, 1974). Clear activity has, in fact, been
found in the left LOC during mental imagery of object
shape based on prior visual exposure in sighted subjects
and haptic exposure in blind subjects (De Volder etal., 2001).
On the other hand, some have argued against visual im-
agery as the basis for visual cortical engagement (Amedi
et al., 2002; Amedi et al., 2001; James et al., 2002), since
auditory objects (Amedi et al., 2002) and visual imagery
(Amedi etal., 2001) fail to evoke substantial activity in the
LOC, as compared with haptic or visual object perception.
The results of the present study do not definitively re-
solve this controversy but do offer further support for the
notion that visual imagery is associated with recruitment
of visual processing areas in nonvisual tasks. We found a
novel and interesting dissociation between right and left
LOC: Left LOC shape-selective activity during haptic per-
ception appeared to be unrelated to visual imagery, since its
strength was not predicted by VVIQ ratings, VI_HS scores,
or a multiple regression on both. In contrast, in the right
LOC, shape-selective activity during haptic perception
was predicted very well by a multiple regression on VVIQ
and VI_HS scores together. At present, it is uncertain
whether these differences between the right and the left
LOC should be interpreted as exemplifying hemispheric
specialization or merely as characteristics of the LOC in
the hemisphere ipsilateral or contralateral to the exploring
hand. To distinguish between these alternatives requires
repeating our study with the left hand being used for hap-
tic perception. However, since right, but not left, LOC ac-
tivity was correlated with VVIQ for central visual stimuli,
the right LOC may indeed be preferentially involved in
Results of Regression Analysis of Beta Weights for Haptic
Shape-Selective Activity on Imagery Scores
VVIQ and VI_HS
(multiple regression) VVIQVI_HS
Note—LOC, lateral occipital complex.*Significant effect (p ? .05).
258ZHANG, WEISSER, STILLA, PRATHER, AND SATHIAN
form imagery in both vision and touch. Further work is
necessary to investigate this possibility.
Our findings strongly implicate visual imagery in re-
cruitment of the right LOC. The results cannot be explained
by a nonspecific tendency for right LOC activity to scale
up and down between individuals regardless of task, for
the following reasons. (1) Right LOC shape-selective ac-
tivity during visual perception correlated significantly
with VVIQ alone, whereas that during haptic perception
did not. (2) Visually and haptically evoked activities were
not correlated with each other. It could be argued that vi-
sual imagery was simply an epiphenomenon, a by-product
that had nothing to do with perceptual processing of ob-
ject shape, and we cannot absolutely exclude this possi-
bility. Nevertheless, our results fit with the concept that
visual imagery underlies visual cortical recruitment dur-
ing haptic perception. The converse, however, does not ap-
pear to be true, since haptic imagery was hardly used dur-
ing visual shape perception. This asymmetry is consistent
with the dominance of vision over haptics in shape per-
ception (Klatzky, Lederman, & Reed, 1987; Rock & Vic-
If visual imagery is a possible trigger for right LOC en-
gagement during HS perception, what mechanism could
recruit the left LOC? One possibility is that verbal factors
might be involved, although this seems unlikely given the
similar linguistic demands of the shape and texture tasks.
Alternatively, left LOC activity might reflect a common
multisensory representation of shape that can be accessed
via either visual or haptic inputs. If this is the case, how-
ever, the lack of correlation between VS- and HS-selective
activity implies that inter-individual variability in the
strength of visual inputs into the LOC is independent of
that for haptic inputs. Multiple lines of evidence point to
representations of object geometry that are shared between
vision and haptics. First, psychophysical work shows that
visuo-haptic priming is as effective as within-modality
priming (Easton, Greene, & Srinivas, 1997; Easton, Srini-
vas, & Greene, 1997; Reales & Ballesteros, 1999). Second,
imaging data reveal that prior visual and haptic explo-
ration produces equivalent priming effects on fMRI re-
sponses in the LOC to subsequently viewed objects (James
et al., 2002). Third, a patient with visual agnosia due to a
left occipito-temporal lesion also had tactile agnosia de-
spite intact basic somatosensory function (Feinberg, Rothi,
& Heilman, 1986), and a prosopagnosic patient demon-
strated similar abnormalities of face recognition for visual
and haptic presentations (Kilgour, de Gelder, & Leder-
man, 2004). It is worth noting that multisensory represen-
tations might be accessible either via bottom-up or top-
down connections. Recent studies of multisensory inputs
into early sensory cortical areas that are traditionally con-
sidered to be unimodal have suggested that some inputs
are probably top down (Falchier, Clavagnier, Barone, &
Kennedy, 2002; Rockland & Ojima, 2003; Schroeder
etal., 2003), whereas others might be bottom up (Schroeder
et al., 2003). An important issue for future research to ad-
dress is the relative role of bottom-up and top-down mech-
anisms in multisensory processes.
Alivisatos, B., & Petrides, M. (1997). Functional activation of the
human brain during mental rotation. Neuropsychologia, 36, 111-118.
Amedi, A., Jacobson, G., Hendler, T., Malach, R., & Zohary, E.
(2002). Convergence of visual and tactile shape processing in the
human lateral occipital complex. Cerebral Cortex, 12, 1202-1212.
Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001).
Visuo-haptic object-related activation in the ventral visual pathway.
Nature Neuroscience, 4, 324-330.
Binkofski, F., Buccino, G., Posse, S., Seitz, R. J., Rizzolatti, G., &
Freund, H.-J. (1999). A fronto-parietal circuit for object manipula-
tion in man: Evidence from an fMRI-study. European Journal of Neu-
roscience, 11, 3276-3286.
Blake, R., Sobel, K. V., & James, T. W. (2004). Neural synergy be-
tween kinetic vision and touch. Psychological Science, 15, 397-402.
Blossfeldt, K. (1997a). The alphabet of plants (A. Wilde & J. Wilde,
Eds.). New York: te Neues Publishing.
Blossfeldt, K. (1997b). Photography (A. Wilde & J. Wilde, Eds.). New
York: Distributed Art Publishers.
Bodegård, A., Geyer, S., Grefkes, C., Zilles, K., & Roland, P. E.
(2001). Hierarchical processing of tactile shape in the human brain.
Neuron, 31, 317-328.
Bremmer, F., Schlack, A., Shah, N. J., Zafiris, O., Kubischik, M.,
Hoffmann, K., Zilles, K., & Fink, G. R. (2001). Polymodal motion
processing in posterior parietal and premotor cortex: A human fMRI
study strongly implies equivalencies between humans and monkeys.
Neuron, 29, 287-296.
Brodatz, P. (1966). Textures: A photographic album for artists and de-
signers. Mineola, NY: Dover.
Cohen, M. S., Kosslyn, S. M., Breiter, H. C., DiGirolamo, G. J.,
Thompson, W. L., Anderson, A. K., Bookheimer, S. Y., Rosen, B. R.,
& Belliveau, J. W. (1996). Changes in cortical activity during men-
tal rotation: A mapping study using functional MRI. Brain, 119, 89-
De Volder, A. G., Toyama, H., Kimura, Y., Kiyosawa, M., Nakano, H.,
Vanlierde, A., Wanet-Defalque, M. C., Mishina, M., Oda, K., Ishi-
wata, K., & Senda, M. (2001). Auditory triggered mental imagery
of shape involves visual association areas in early blind humans. Neuro-
Image, 14, 129-139.
Duvernoy, H. M. (1999). The human brain: Surface, blood supply and
three-dimensional sectional anatomy (2nd ed.). New York: Springer-
Easton, R. D., Greene, A. J., & Srinivas, K. (1997). Transfer between
vision and haptics: Memory for 2-D patterns and 3-D objects. Psy-
chonomic Bulletin & Review, 4, 403-410.
Easton, R. D., Srinivas, K., & Greene, A. J. (1997). Do vision and
haptics share common representations? Implicit and explicit memory
within and between modalities. Journal of Experimental Psycholol-
ogy: Learning, Memory, & Cognition, 23, 153-163.
Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002).
Anatomical evidence of multimodal integration in primate striate cor-
tex. Journal of Neuroscience, 22, 5749-5759.
Feinberg, T. E., Rothi, L. J., & Heilman, K. M. (1986). Multimodal ag-
nosia after unilateral left hemisphere lesion. Neurology, 36, 864-867.
Feinstein, H. (2001). Foliage. Boston: Bulfinch Press.
Freides, D. (1974). Human information processing and sensory modal-
ity: Cross-modal functions, information complexity, memory and
deficit. Psychological Bulletin, 81, 284-310.
Grant, A. C., Zangaladze, A., Thiagarajah, M. C., & Sathian, K.
(1999). Tactile perception in developmental dyslexia: A psychophys-
ical study using gratings. Neuropsychologia, 37, 1201-1211.
Grefkes, C., Weiss, P. H., Zilles, K., & Fink, G. R. (2002). Cross-
modal processing of object features in human anterior intraparietal
cortex: An fMRI study implies equivalencies between humans and
monkeys. Neuron, 35, 173-184.
Grill-Spector, K., Kushnir, T., Hendler, T., Edelman, S.,
Itzchak, Y., & Malach, R. (1998). A sequence of object-processing
stages revealed by fMRI in the human occipital lobe. Human Brain
Mapping, 6, 316-328.
Hagen, M. C., Franzen, O., McGlone, F., Essick, G., Dancer, C., &
Pardo, J. V. (2002). Tactile motion activates the human middle
MULTISENSORY CORTICAL SHAPE PROCESSING259
temporal/V5 (MT/V5) complex. European Journal of Neuroscience,
James, T. W., Humphrey, G. K., Gati, J. S., Servos, P., Menon, R. S.,
& Goodale, M. A. (2002). Haptic study of three-dimensional objects
activates extrastriate visual areas. Neuropsychologia, 40, 1706-1714.
Kilgour, A. R., de Gelder, B., & Lederman, S. J. (2004). Haptic face
recognition and prosopagnosia. Neuropsychologia, 42, 707-712.
Klatzky, R. L., Lederman, S. J., & Reed, C. (1987). There’s more to
touch than meets the eye: The salience of object attributes for haptics
with and without vision. Journal of Experimental Psychology: Gen-
eral, 116, 356-369.
Lederman, S. J., Klatzky, R. L., Chataway, C., & Summers, C. D.
(1990). Visual mediation and the haptic recognition of two-dimensional
pictures of common objects. Perception & Psychophysics, 47, 54-64.
Leonards, U., Sunaert, S., Van Hecke, P., & Orban, G. A. (2000).
Attention mechanisms in visual search: An fMRI study. Journal of
Cognitive Neuroscience, 12, 61-75.
Lloyd, D. M., Shore, D. I., Spence, C., & Calvert, G. A. (2003). Multi-
sensory representation of limb position in human premotor cortex.
Nature Neuroscience, 6, 17-18.
Macaluso, E., Frith, C. D., & Driver, J. (2002). Directing attention
to locations and to sensory modalities: Multiple levels of selective
processing revealed with PET. Cerebral Cortex, 12, 357-368.
Malach, R., Reppas, J. B., Benson, R. R., Kwong, K. K., Jiang, H.,
Kennedy, W. A., Ledden, P. J., Brady, T. J. Rosen, B. R., &
Tootell, R. B. H. (1995). Object-related activity revealed by func-
tional magnetic resonance imaging in human occipital cortex. Pro-
ceedings of the National Academy of the Sciences, 92, 8135-8139.
Marks, D. F. (1973). Visual imagery differences in the recall of pictures.
British Journal of Psychology, 64, 17-24.
Mugler, J. P., & Brookman, J. R. (1990). Three dimensional
magnetization-prepared rapid gradient-echo imaging (3D MPRAGE).
Magnetic Resonance Medicine, 15, 152-157.
Prather, S. C., Votaw, J. R., & Sathian, K. (2004). Task-specific re-
cruitment of dorsal and ventral visual areas during tactile perception.
Neuropsychologia, 42, 1079-1087.
Raczkowski, D., Kalat, J. W., & Nebes, R. (1974). Reliability and va-
lidity of some handedness questionnaire items. Neuropsychologia,12,
Reales, J. M., & Ballesteros, S. (1999). Implicit and explicit mem-
ory for visual and haptic objects: Cross-modal priming depends on
structural descriptions. Journal of Experimental Psychology: Learn-
ing, Memory, & Cognition, 25, 644-663.
Rock, I., & Victor, J. (1964). Vision and touch: An experimentally cre-
ated conflict between the two senses. Science, 143, 594-596.
Rockland, K. S., & Ojima, H. (2003). Multisensory convergence in cal-
carine visual areas in macaque monkey. International Journal of Psy-
chophysiology, 50, 19-26.
Sathian, K., Cascio, C., Rice, D., Morris, M., Dancer, C., & Mc-
Glone, F. (2003). Is tactile temporal processing impaired in develop-
mental dyslexia? Cognitive Neuroscience Society Abstracts, E323.
Sathian, K., Prather, S. C., & Zhang, M. (2004). Visual cortical in-
volvement in normal tactile perception. In G. Calvert, C. Spence, &
B. Stein (Eds.), The handbook of multisensory processes (pp. 703-
709). Cambridge, MA: MIT Press.
Sathian, K., & Zangaladze, A. (2001). Feeling with the mind’s eye:
The role of visual imagery in tactile perception. Optometry & Vision
Science, 78, 276-281.
Sathian, K., Zangaladze, A., Hoffman, J. M., & Grafton, S. T.
(1997). Feeling with the mind’s eye. NeuroReport, 8, 3877-3881.
Schroeder, C. E., Smiley, J., Fu, K. G., McGinnis, T., O’Connell,
M. N., & Hackett, T. A. (2003). Anatomical mechanisms and func-
tional implications of multisensory convergence in early cortical pro-
cessing. International Journal of Psychophysiology, 50, 5-17.
Sergent, J., Ohta, S., & MacDonald, B. (1992). Functional neuro-
anatomy of face and object processing. A positron emission tomogra-
phy study. Brain, 115, 15-36.
Stoeckel, M. C., Weder, B., Binkofski, F., Buccino, G., Shah, N. J.,
& Seitz, R. J. (2003). A fronto-parietal circuit for tactile object dis-
crimination: An event-related fMRI study. NeuroImage, 19, 1103-
Stoesz, M., Zhang, M., Weisser, V. D., Prather, S. C., Mao, H., &
Sathian, K. (2003). Neural networks active during tactile form per-
ception: Common and differential activity during macrospatial and mi-
crospatial tasks. International Journal of Psychophysiology,50, 41-49.
Talairach, J., & Tournoux, P. (1988). Co-planar stereotaxic atlas of
the brain. New York: Thieme Medical.
Vandenberghe, R., Gitelman, D. R., Parrish, T. B., & Mesulam, M. M.
(2001). Functional specificity of superior parietal mediation of spatial
shifting. NeuroImage, 14, 661-673.
Zangaladze, A., Epstein, C. M., Grafton, S. T., & Sathian, K.
(1999). Involvement of visual cortex in tactile discrimination of ori-
entation. Nature, 401, 587-590.
(Manuscript received October 11, 2003;
revision accepted for publication April 14, 2004.)