ArticlePDF Available

From Shape to Meaning: Evidence for Multiple Fast Feedforward Hierarchies of Concept Processing in the Human Brain


Abstract and Figures

A number of fMRI studies have provided support for the existence of multiple concept representations in areas of the brain such as the anterior temporal lobe (ATL) and inferior parietal lobule (IPL). However, the interaction among different conceptual representations remains unclear. To better understand the dynamics of how the brain extracts meaning from sensory stimuli, we conducted a human high-density electroencephalography (EEG) study in which we first trained participants to associate pseudowords with various animal and tool concepts. After training, multivariate pattern classification of EEG signals in sensor and source space revealed the representation of both animal and tool concepts in the left ATL and tool concepts within the left IPL within 250ms. Finally, we used Granger Causality analyses to show that orthography-selective sensors directly modulated activity in the parietal-tool selective cluster. Together, our results provide evidence for distinct but parallel “perceptual-to-conceptual” feedforward hierarchies in the brain.
Content may be subject to copyright.
From shape to meaning: Evidence for multiple fast feedforward hierarchies
of concept processing in the human brain
Srikanth R. Damera , Jacob G. Martin , Clara Scholl , Judy S. Kim , Laurie Glezer ,
Patrick S. Malone , Maximilian Riesenhuber
Department of Neuroscience, Georgetown University Medical Center, Washington, DC, USA
A number of fMRI studies have provided support for the existence of multiple concept representations in areas of the brain such as the anterior temporal lobe (ATL)
and inferior parietal lobule (IPL). However, the interaction among different conceptual representations remains unclear. To better understand the dynamics of how the
brain extracts meaning from sensory stimuli, we conducted a human high-density electroencephalography (EEG) study in which we rst trained participants to
associate pseudowords with various animal and tool concepts. After training, multivariate pattern classication of EEG signals in sensor and source space revealed the
representation of both animal and tool concepts in the left ATL and tool concepts within the left IPL within 250 ms. Finally, we used Granger Causality analyses to
show that orthography-selective sensors directly modulated activity in the parietal-tool selective cluster. Together, our results provide evidence for distinct but parallel
perceptual-to-conceptualfeedforward hierarchies in the brain.
1. Introduction
Humans can rapidly and efciently assign meaning to visual objects.
For visual stimuli, this process is traditionally thought to be mediated by
the visual ventral stream, along which information is processed by a
simple-to-complex hierarchy, up to neurons in ventral temporal cortex
that are selective for complex objects such as faces, objects and words
(Kravitz et al., 2013). According to computational models (Ashby and
Spiering, 2004;Freedman et al., 2003;Nosofsky, 1986;Riesenhuber and
Poggio, 2000;Thomas et al., 2006) as well as human functional magnetic
resonance imaging (fMRI) and electroencephalography (EEG) studies
(Jiang et al., 2007;Scholl et al., 2013), these object-selective neurons in
high-level visual cortex can then provide input to task modules located in
downstream cortical areas, such as prefrontal cortex (PFC) and the
anterior temporal lobe (ATL), to mediate the identication, discrimina-
tion, or categorization of stimuli. It is at this level where these theories of
object categorization in the brain connect with inuential theories of
semantic cognition that have proposed that the ATL may act as a se-
mantic hub(Ralph et al., 2016), based on neuropsychological ndings
(Hodges et al., 2000;Jefferies, 2013;Mion et al., 2010) and studies that
have used fMRI (Coutanche and Thompson-Schill, 2015;Malone et al.,
2016;Vandenberghe et al., 1996) or intracranial EEG (iEEG) (Chan et al.,
2011) to decode category representations in the anteroventral temporal
Yet, the processing of some object classes has been shown to involve
additional pathways. A prominent example is the object class of tools,
which, in addition to the ventral stream, engages the dorsal, vision for
action(Goodale, 2011;Goodale and Milner, 1992;Kravitz et al., 2011)
pathway, with areas in parietal cortex being selectively activated by tool
stimuli, and lesions in parietal cortex affecting tool praxis (Buxbaum
et al., 2014;Vingerhoets et al., 2009). A key, but unresolved, question in
models of semantic processing is how the brain accesses these different
types of concept knowledge. In particular, are there separate pathways
that link sensory representations to distinct domain-specic concept
representations, compatible with distributed models of semantic pro-
cessing (Chen and Rogers, 2015), or does knowledge from one domain
access that from another indirectly through representations in a
domain-general hub (Almeida et al., 2013;Hodges et al., 2000)?
Resolving the question of how different types of concept knowledge
are processed and interact requires the ability to not only delineate the
underlying network of brain areas, but also, and crucially, the informa-
tion ow between them. Recent fMRI research has uncovered a complex
network of brain areas underlying semantic processing (Chen et al.,
2017;Pulvermüller, 2013). Yet, due to its limited temporal resolution,
fMRI is unable to directly probe the information ow within these net-
works, considering that numerous prior studies have shown that the
brain is able to extract meaning from sensory stimuli within about 200 ms
(Chan et al., 2011;Scholl et al., 2013;Thorpe et al., 1996), and, equally
* Corresponding author. Department of Neuroscience, Georgetown University Medical Center Research, Building Room WP-12, 3970 Reservoir Rd. NW Washington,
DC, 20007, USA.
E-mail address: (M. Riesenhuber).
Contents lists available at ScienceDirect
journal homepage:
Received 30 March 2020; Received in revised form 10 June 2020; Accepted 6 July 2020
Available online 11 July 2020
1053-8119/©2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (
NeuroImage 221 (2020) 117148
important, that neural processing dynamics do not just include a feed-
forwardow of information, but also, starting within 150 ms and
continuing for several hundred milliseconds, several re-entrant waves of
activation, e.g., associated with stimulus awareness (Cul et al., 2007;
Fahrenfort and Lamme, 2012), whereas typical fMRI scan paradigms
have temporal resolutions that are ten times slower. In contrast, EEG with
its millisecond resolution is better matched to the speed of visual pro-
cessing in the brain. We used a combination of multivariate pattern
analysis (MVPA) and EEG to address these challenges and gain a better
understanding of the dynamics of the neural networks underlying
concept processing. We trained participants to associate a vocabulary of
pseudowords (PWs) with various animal and tool categories (Fig. 1A).
Next, we collected high-density EEG data while subjects performed a
delayed-match-to-sample task. We applied searchlight MVPA in sensor as
well as source space to temporally and spatially localize concept infor-
mation in the brain. In addition, Granger Causality analyses were used to
elucidate the dynamics of signal ow within the semantic network.
2. Materials and Methods
2.1. Participants
A total of 11 right-handed healthy adults who were native English
speakers were enrolled in the experiment (ages 2029, 5 females).
Georgetown Universitys Institutional Review Board approved all
experimental procedures, and written informed consent was obtained
from all subjects before the experiment.
2.2. Behavioral training
PWs (all four letters in length) matched for bigram and trigram fre-
quency, and orthographic neighborhood were generated using MCWord
(Medler and Binder, 2005). Subjects were trained to learn a vocabulary of
60 PWs, with each PW assigned to one of six categories: monkey, donkey,
elephant, hammer, wrench, and screwdriver (i.e., 10 PWs were dened
as monkey,10PW as donkey,etc.). To learn the PW denitions, each
subject performed 0.51 h of training per session for a total of 8 sessions
in which a PW was presented on a screen followed by 6 pictures: one
picture for each of the animal and tool categories (randomly selected
from a large database of images for each category to prevent subjects
from associating particular words with particular images). Subjects
indicated their category choice using a numeric keypad. When subjects
answered incorrectly, an auditory beep and the correct answer were
presented. No feedback was given for correct answers. A single training
session consisted of 5 blocks, and a unique set of 2 PWs (within the 10 per
category) was used for each block to facilitate learning.
2.3. EEG paradigm
Following training, we probed the neural bases of the learned concept
memberships using an EEG paradigm. Each trial of the EEG experiment
consisted of a xation cross for 500 ms, a blank screen for 200 ms with
jitter, followed by the rst PW for 300 ms, blank screen for 400 m, and
second PW for 300 ms. Following the second PW, subjects were
instructed to indicate with a button click if both words referred to the
same or different superordinate categories (Fig. 1B). Trials were broken
into four different conditions depending on the relationship between the
rst and second PWs: same word/same basic-level category (SWSC),
different word/same basic-level category (DWSC, e.g., rst: monkey PW,
second: a different monkey PW), different word/different basic-level
category (DWDC, e.g., rst: monkey PW, second: a donkey PW), and
different word/different superordinate category (DWDSC, e.g., rst: an-
imal PW, second: a tool PW). Each subject participated in a single session
with two runs each. Each run had three blocks, each of which were 136
trials long, for a total of 408 trials in a run. Counts for the four trial types
were matched in each session and different conditions were presented in
a random manner. Trials with incorrect responses were removed from the
analyses. In this study, only data from the rst PW presentation were
analyzed for the decoding analyses.
2.4. EEG data acquisition and preprocessing
Scalp voltages were measured using an Electrical Geodesics (EGI,
Eugene, OR) 128-channel Hydrocel Geodesic Sensor Net and Net Amps
300 amplier. Incoming data were digitally low-pass ltered at 200 Hz
and sampled at 500 Hz using common mode rejection with vertex
reference. Impedances were set below 40 kΩbefore recording began and
maintained below this threshold throughout the recording session with
an impedance check during each break between blocks.
Pre-Processing: Data processing and statistical analyses were per-
formed using EEGLAB (Delorme and Makeig, 2004). Data were rst
high-pass ltered at 0.2 Hz and then low-pass ltered at 30 Hz. A
minimum-phase causal lter ensured that information did not get
smeared backwards and was used for decoding analyses to accurately
determine the onset latency of classication (Widmann and Schr
2012). Bad channel identication was automated using the clean_raw-
data EEGLab plugin that cleaned continuous data using artifact subspace
reconstruction. Bad channels identied with this method were then
reconstructed using spherical interpolation. Continuous data were
epoched on the interval of [-200 300] ms relative to the onset of the rst
word. The mean of the baseline period was then subtracted from the
signal. Bad trials were marked if the voltage exceeded 50
V on any
EEG channel. Finally, the data were re-referenced to average reference
since this has been shown to improve the quality of the source recon-
struction (Dien, 1998).
2.5. EEG source estimation
Source reconstruction was conducted using common forward and
inverse models implemented in Brainstorm (Tadel et al., 2011). In
Brainstorm (version 11-13- 2019), the EEG forward problem was solved
in the ICBM152 template anatomy. Realistically-shaped volume meshes
of the brain, skull and scalp were extracted from the provided template
MR image using the default number of 1922 vertices per layer for a total
of 15765 sources. The forward model (lead eld) from these source lo-
cations to the 128 channels was calculated using the symmetric boundary
Element method as implemented in the Open-MEEG package. Inverse
estimation of sources was carried out using the LCMV Beamformer. The
noise covariance matrix for this method was built using the 200 ms of
baseline data, and the data covariance matrix was constructed using the
300 ms of word presentation.
2.6. MVPA decoding
MVPA analyses were performed using The Decoding Toolbox (Hebart
et al., 2014) and custom MATLAB code (R2017b, The Mathworks, MA).
All classications implemented a linear support vector machine (SVM)
classier with a xed cost parameter (c ¼1). For each subject, all trials of
a particular PW were averaged over the entire session so that a single
pseudo-trial was created for each PW. This procedure has been shown to
increase the signal-to-noise-ratio (Grootswagers et al., 2016). 100-fold
Monte Carlo cross-validation was performed with an approximate
50/50 training to testing split. Training and testing sets were carefully
partitioned so that no PW was included in both the training and testing
set within a fold. Accuracies were averaged across folds to compute an
unbiased measure of classier performance.
Sensor Space Spatiotemporal Searchlight Decoding: Time-
resolved spatial decoding analyses were then performed in sensor
space. For these decoding analyses, a 3 cm searchlight was constructed
around each sensor (searchlight size ranged from 1 to 7 sensors, but
similar results were obtained with a 4 cm searchlight which had a range
of 38 sensors in a searchlight). Temporal data for each sensor in a given
S.R. Damera et al. NeuroImage 221 (2020) 117148
searchlight were averaged in order to create an N-channel-by-1 feature
vector per searchlight. This was done in 20 ms windows and advanced 2
ms over the duration of the word (Fig. 1C).
Source Space Spatial Decoding: To spatially localize sensor level
classication, voltage data were projected to source space (see above)
and a searchlight MVPA was performed using a 10 mm searchlight. The
value at a given source was taken as the average over a time window of
interest dened by sensor level decoding results (Fig. 1D).
2.7. MVPA analysis 1 (superordinate classication)
For each subject, an animal versus tool classication was rst per-
formed in sensor space to identify the onset latency and time window of
superordinate decoding. Within each cross-validation fold, the classier
was trained on 40 PWs (20 animal and 20 tool PW) and tested on the
remaining 20 PWs. Once a signicant time-window of decoding was
identied at the sensor level, this time window was then submitted to the
classication procedure in source space in order to estimate a neuroan-
atomical source of the classication.
2.8. MVPA analysis 2 (basic-level classication)
Next, a within-category classication was performed in sensor space
to identify the onset latency and time window of basic-level decoding.
There was a total of 6 (3 within-animal and 3 within-tool) unique clas-
sications. One subject was not shown any elephant or wrench PWs and
was excluded from the basic-level classication analyses. Within each
cross-validation fold, the classier was trained on 10 PWs (e.g., 5
hammer and 5 wrench PWs) and tested on the remaining 10 PWs. The
resulting spatiotemporal classication maps for all within-animal com-
parisons were averaged to produce a single accuracy map for basic-level
animal concepts. This was repeated for all within-tool comparisons as
well. Once a signicant time-window of decoding was identied at the
sensor level this was then submitted to the classication procedure in
source space in order to estimate a neuroanatomical source of the
2.9. Statistical analysis of classication results
Each MVPA analysis produced an accuracy value at each sensor or
source. At the group-level, multiple comparisons across sensors and time-
points or sources was done using cluster-mass correction (Maris and
Oostenveld, 2007). For this method, an empirical null distribution was
constructed for each subject at every sensor and time-point by permuting
labels across the entire data set, setting up cross-validation folds, and
then performing the classication procedure as before. This procedure
was repeated 1000 times, and then the observed and null distributions
were averaged across subjects at every sensor and time point. The p-value
associated with the accuracy observed at a single electrode time-point
combination was calculated as the proportion of values in the null dis-
tribution greater than the observed value. Next, an initial cluster level
, was used to generate either spatiotemporal or spatial clus-
ters corresponding to sensor space and source space analyses, respec-
was set to .002, for both sensor and source space analyses.
Previous fMRI studies have noted that a liberal
(i.e., >0.001) increases
the rate of false positives (cf. Fig. 1 in Eklund et al., 2016). However, this
is specic to parametric approaches using random eld theory and not
non-parametric permutation-based approaches (Eklund et al., 2016). It
Fig. 1. Task and analysis paradigms. (A) Subjects were trained on a set of 60 pseudowords (PW) belonging to one of three animal or tool categories (two PWs are
shown as examples with each category), as in Malone et al. (2016). (B) After training, subjects participated in an EEG experiment, in which they performed a
delayed-match-to-superordinate category task in which two PWs were sequentially presented on each trial and subjects had to indicate after the presentation of the
second word if both words belonged to the same or different superordinate categories (i.e., animals or tools). (C) At each sensor and for different time bins (the blue
shaded region illustrates one example time bin), a set of time-lagged support vector machine (SVM) classiers was constructed on half the trained pseudowords (PWs)
from animal and tool categories and then tested on the other half, resulting in dynamic classication accuracy maps for each subject in sensor space. (D) In separate
analyses, EEG sensor data were projected into source space (see Materials and Methods), followed by averaging over a time period of interest. Searchlight MVPA was
then performed, resulting in a spatial accuracy map per subject.
S.R. Damera et al. NeuroImage 221 (2020) 117148
has been shown that permutation-based methods control the false posi-
tive rate at the desired level independent of the
chosen (Maris and
Oostenveld, 2007;Sassenhagen and Draschkow, 2019;Woo et al., 2014).
The mass of clusters was computed by summing the accuracies of all the
sensors or sources in the cluster. The mass of each cluster in the actual
data was compared to an empirical distribution of maximal clusters of the
permuted data and was marked as signicant if it was greater than 95%
of the values.
2.10. Identication of clusters selective for orthography
Orthographically-selective space-time clusters were identied by
contrasting responses to the second words in each trial in the SWSC and
DWSC conditions, and signicance was assessed using cluster-based
permutation testing. Due to strong a priori hypotheses about the la-
tency of the N170 response to words (Brem et al., 2010;Dehaene et al.,
2015;Maurer et al., 2005) this analysis was constrained to left lateralized
sensors between 100 and 200 ms after the onset of the second word.
Using fMRI and EEG it has been shown that neuronal populations that are
selective for a feature of interest (e.g., orthography) have a reduced
(adapted) response for rapidly presented pairs of stimuli that are
consistent versus inconsistent for that feature (Glezer et al., 2009;Jiang
et al., 2007;Scholl et al., 2013). In the current study, we hypothesized
that sensors recording from populations of neurons sensitive to orthog-
raphy would exhibit an adapted N170 response to sequential presenta-
tion of the same PW versus presentation of two different PWs. Crucially,
to control for differences due to semantics, we used SWSC and DWSC
trials in which both the rst and the second word referred to the same
basic-level concept. As for the classication results, spatiotemporal
clusters were identied by setting the cluster-dening
to 0.002 and
then their signicance was controlled at the two-tailed p <.05 level.
2.11. Granger Causality
Granger Causality (Granger, 1969) (GC) has been used to investigate
causal relationships between prefrontal and early visual neural recording
sites in monkey and human electrophysiology recordings (Cui et al., 2008;
Gregoriouet al., 2009;Seth et al., 2015).In this work, we computedGranger
Causality within each subject at the single trial level between all pairs of
channels in the left anterior temporal superordinate-classication, parietal
basic-level tool, and left posterior orthographic sensors (see Fig. 5A). We
used the BSMART toolbox (Cui et al., 2008) with a model order of 15 and a
sliding temporal window of 60 ms. We only used artifact-free trials for
which correct responses were given. Before computing the Granger Cau-
sality, we performed the standard preprocessing step of subtracting the
temporal mean of each trial (in each channel) from all data points in that
trial. Next, we calculated the Granger causalities for each time point and
subject separately for each channel pair. Finally, we averaged the
frequency-specic Granger Causality values for all channel pairs into the
theta (47 Hz), alpha (812 Hz), and beta (1330 Hz) frequency bands for
each condition and subject to obtain a time-based version of Granger Cau-
sality within that subject, and frequency band.
2.12. Statistical analysis of Granger Causality results
A mixed effects modelling approach was taken to test the signicance
of Granger Causality effects. Models were built to test when the GC of
each of the proposed pathways (e.g., orthography to basic-level tool se-
lective sensors) deviated from baseline. The model included a xed-effect
of frequency band, and a random-effect of subjects. Since orthographic
processing indexed by the N170 ERP begins at ~150 ms, this model was
constructed at each time point between 150 and 300 ms, and two-tailed
p-values for the xed-effects of interest were FDR-corrected at the .05
level to control for multiple comparisons across time and frequency
3. Results
3.1. Behavior
Each subject (n ¼11) performed 8 training sessions over an average
of 13.8 days (SD 3.2). Subjects performed only a single training session
per day. Both accuracy and RT for identication of PW category
improved across training sessions (Fig. 2A). On average, subjects reached
an accuracy of 98.7% by their eighth training session. During the
delayed-match-to-basic-level category task, subjects reached 93.7% ac-
curacy on average (Fig. 2B). A two-way ANOVA with the factors condi-
tion (same word/same basic category, SWSC; different word/same basic
category, DWSC; different word/different basic category, DWDC;
different word/different superordinate category, DWDSC) and superor-
dinate category (animals and tools) of the rst word in each trial was
conducted in order to test for any effect of condition or superordinate
category on task performance. This revealed a signicant main effect of
condition on accuracy (F
¼6.88, two-tailed p ¼0.004). Impor-
tantly, there was no signicant main effect of superordinate category on
task accuracy (F
¼1.78, two-tailed p ¼0.18). In order to investigate
the source of the signicant effect of condition, post-hoc t-tests
comparing accuracy between all pairs of conditions showed that there
was a signicant difference in accuracy for SWSC vs. DWSC (two-tailed p
¼0.002) contrast as well as the DWDSC vs. DWSC (two-tailed p ¼0.008)
contrast. There was no signicant interaction between superordinate
category and condition on accuracy (F
¼1.03, two-tailed p ¼0.39).
3.2. EEG single-trial analysis reveals fast superordinate concept selectivity
over temporal sensors, localized to the ATL
We conducted a series of single-trial classication analyses of the EEG
signal in response to the rst word in each trial to analyze how the brain
maps sensory stimuli to concepts. We rst explored the onset of
superordinate-level (animals vs. tools) information at the sensor level
within the 100300 ms interval. This revealed one spatiotemporal cluster
with signicantly above-chance classication performance (Fig. 3A and
B). The left-anterior temporal cluster was signicantly above chance
from 204 to 268 ms post-word presentation with peak accuracy at 236 ms
(Fig. 3B; cluster-dening
¼0.002, one-tailed p ¼.0240, see Material
and Methods). We then projected sensor data to source space using an
LCMV (Linearly Constrained Minimum Variance) beamformer (Grech
et al., 2008) to localize the cortical source(s) of this classication.
Furthermore, to better characterize the timing of concept processing in
our broad temporal cluster (204268 ms) (Sassenhagen and Draschkow,
2019), we separately averaged source maps in the early (204236 ms)
and late (236268 ms) halves of the cluster (see Materials and Methods).
We hypothesized that concept information in the ATL should localize to
the early rather than late time window, given our overarching hypothesis
that activation of concept information in ATL would be the next step in
the ventral stream processing hierarchy following the orthographic rep-
resentation (which has been associated with the N170 in the literature
(Brem et al., 2010;Dehaene et al., 2015;Maurer et al., 2005), with a
temporal extent from about 170 to 200 ms), taking into account estimates
in the literature of each cortical processing stage taking on the order of
30 ms (Thorpe and Fabre-Thorpe, 2001). Searchlight source space
analysis indeed localized superordinate category information in the early
time window to the LATL (Fig. 3C; cluster-dening
¼0.002, one-tailed
3.3. EEG single-trial analysis reveals fast basic-level decoding of tools in
left parietal sensors
We next examined the latency and location of basic-level (within-
tool and within-animal) category decoding. We identied sensors that
distinguished among the different tools: Sensor-level searchlight
analysis between 100 and 300 ms after stimulus onset revealed a
S.R. Damera et al. NeuroImage 221 (2020) 117148
cluster of left posterior sensors that show signicantly above-chance
classication from 208 to 228 ms post-stimulus onset with peak ac-
curacy at 220 ms (Fig. 4A and B; cluster-dening
¼0.002, one-tailed
p¼.031). Searchlight source space analysis localized the classication
from this time period to the left inferior parietal lobe (Fig. 4C; cluster-
¼0.002, one-tailed p ¼.025. We also conducted basic-level
animal classication, but no signicant spatiotemporal clusters were
3.4. Granger Causality analysis supports feedforward simple-to-concept
The latency of concept information in the preceding analyses pro-
vides evidence for fast, feedforward concept processing areas in the
brain. The regions identied are compatible with the ventral and dorsal
concept processing streams discussed in the Introduction. We next used
Granger Causality analyses (Granger, 1969) to test if a common
perceptual representation could be used to access these different
Fig. 2. Pseudoword training performance and delayed-match-to-superordinate category task accuracy. (A) Mean accuracies for identication of pseudoword category
across training sessions during a 6 alternative-forced choice task (n ¼10). (B) Accuracy in the delayed-match-to-superordinate category task during which EEG data
was collected for each condition: SWSC, rst and second word were identical; DWSC, second word was different word, same basic-level category as the rst word;
DWDC, second word was different word, belonging to different basic-level category than the rst word; DWDSC, second word was different word, different super-
ordinate category than the rst word. Brackets indicate signicant differences across conditions: *p <0.01; n ¼11. Errors bars indicate SEM across subjects.
Fig. 3. Timing and location of superordinate classication of PW concept membership in sensor and source spaces and comparison to previous, fMRI results. (A) The
spatial topography of the sensors that showed signicant decoding of animal and tool PWs. Green sensors are referred to as left anteriorcluster. (B) Time course of
classication accuracy averaged across all sensors in the left anterior cluster. Red asterisks show time-window of signicant classication, which extends from 204 to
268 ms post-word presentation (n ¼11;
<0.002; one-tailed p ¼.024). Shading shows SEM across subjects. (C) Source estimation of Animals vs. Tools classication.
The locus of above-chance classication was investigated in an early (204236 ms) and late (236268 ms) time window. Activity in the early, but not the late time-
window was estimated in source space to the LATL (n ¼11;
<0.002; one-tailed p ¼.025).
S.R. Damera et al. NeuroImage 221 (2020) 117148
conceptual representations a key prediction of two-stage models of
category learning. Specically, we investigated whether activity at
orthographically-selective sensors directly modulated activity in dorsal
tool-selective sensors. To do so, we rst identied a high-level
perceptual (orthography) selective cluster (Fig. 5A; see methods), pu-
tatively identifying the visual word form area,VWFA(Brem et al.,
Fig. 4. Timing and location of tool classication of PW concept membership in sensor and source space. (A) The spatial topography of the sensors (shown in purple)
that show signicant decoding of tool PWs. (B) Time course of classication accuracy averaged across all sensors in the cluster. Red asterisks show time-window of
signicant classication, which extends from 208 to 228 ms post-word presentation (n ¼10;
<0.002; one-tailed p ¼.031), shading shows SEM across subjects. (C)
Source estimation of within-tools classication shown in panels A and B. The locus of above-chance classication during the time window of classication was
estimated in source space to the left parietal lobe (n ¼10;
<0.002; one-tailed p ¼.025).
Fig. 5. Neural dynamics in the concept processing network. (A) The three seeds are marked orthographic (red), superordinate (green, see Fig. 3), and basic-level tool-
selective (purple, see Fig. 4) sensor groups. (B) Average ERP of all sensors that show an orthographic response (shown in red in (A)) shows signicant adaptation of the
N170 response between 148 and 178 ms. Red asterisks mark the time window of the signicant cluster (n ¼11;
<0.002; one-tailed p ¼.034). Average ERP of all
sensors that show an orthographic response (shown in (A)) shows an N170 response during the rst word (inset). (C) Baseline-normalized Granger Causality (GC) from
orthographic (red) and anterior temporal (green) to parietal basic-level tool selective sensors in the theta frequency band. Red asterisks mark time points exhibiting
signicant increase in GC relative to baseline between orthographic and basic-level parietal tool selective sensors from 204 to 214 ms and from 246 to 284 ms (n ¼11;
two-tailed p <.05, FDR-corrected).
S.R. Damera et al. NeuroImage 221 (2020) 117148
2010;Dehaene-Lambertz et al., 2018;Maurer et al., 2005)the highest
orthographically selective stage in the ventral visual pathway. This
cluster showed a signicant adaptation effect between 148 and 178 ms
with a peak difference at 162 ms (Fig. 5B; cluster-dening
two-tailed p ¼.034). This orthographic N170 cluster overlapped in both
space and time with those reported in previous EEG studies (Maurer
et al., 2005;Scholl et al., 2013). The ERP for these sensors showed an
N170 response during the rst word with the negative deection
starting at 150 ms post-stimulus onset (Fig. 5B inset). The sensors in this
cluster were used as our orthographic seed.
We next calculated the change in Granger Causality (GC) relative to
a 200 ms pre-stimulus baseline among the orthographically-selective
sensors and the sensors representing concept information in the
ventral and dorsal pathways that were identied in the previous sec-
tions. To examine if orthographic information directly accessed parietal
tool-selective representations, we tested the change in GC between
orthography and parietal-tool selective sensors from baseline in the
theta (47 Hz) frequency band, which has been implicated in feedfor-
ward information ow (Bastos et al., 2014), as well as the alpha (812
Hz) and beta (1330 Hz) frequency bands. After FDR correction across
the three frequency bands and time points (see Materials and Methods),
we found evidence that theta-frequency (but not alpha or beta) activity
in orthography-selective sensors signicantly modulated activity in
parietal tool-selective sensors from 204 to 214 ms and 244284 ms
post-stimulus onset just prior to the onset of basic-level tool repre-
sentations (Fig. 5C; two-tailed p <.05 FDR corrected).
4. Discussion
The current study applied multivariate searchlight decoding of EEG
data from a word learning paradigm to cleanly dissociate perceptual and
conceptual information in the neural signals at high temporal resolution.
This approach allowed us to address two key unanswered questions:
when do different conceptual representations come online, and how are
they accessed by sensory input? We provide evidence for superordinate
category selectivity in left anterior sensors within 250 ms after word
presentation. Using source estimation techniques, this selectivity was
localized to the left anterior temporal lobe (LATL). Basic-level tool
category membership was also decoded in a set of left parietal sensors
between 208 and 228 ms after word presentation. The cortical generator
of this signal was source estimated to the left parietal lobe. Importantly,
these source-estimated ROI matched well with our previous fMRI study
that used the same training paradigm (Malone et al., 2016). Finally, we
used Granger Causality to investigate how high-level perceptual infor-
mation feeds into different conceptual representations in the brain. This
analysis revealed that orthography-selective sensors Granger-caused ac-
tivity in tool-selective sensors in the dorsal stream between 204 and 214
Our results show that concept-selective circuits receive rapid, feed-
forward input from perceptual (orthographic) representations. This
nding is consistent with the aforementioned two-stage models of cate-
gory learning (Ashby and Spiering, 2004;Riesenhuber and Poggio, 2002,
2000), which predict that learned perceptual representations can be
exibly recruited by distinct higher-order circuits in order to accomplish
different tasks. These two-stage models make clear predictions regarding
the latency of concept-selective circuits, predicting that, due to their
putative location in the cortical hierarchy following object-selective
representations which have been associated with the N170 EEG signal
component (Gauthier et al., 2003;Pegado et al., 2014;Rossion et al.,
2002;Tanaka and Curran, 2001), they should be activated around 200
ms (Thorpe and Fabre-Thorpe, 2001). Yet, the temporal onset of
concept-level processing is still debated (Carreiras et al., 2014), with
many studies claiming that concept representations (frequently localized
to the ATL (Coutanche and Thompson-Schill, 2015;Jackson et al., 2015;
Lau et al., 2008;Ralph et al., 2016) are activated around 400 ms
post-stimulus onset (Kutas and Federmeier, 2011;Jackson et al., 2015).
Our results add to a growing body of literature (Bankson et al., 2018;
Chan et al., 2011;Clarke et al., 2011) showing that concept information
comes online prior to 250 ms post-stimulus onset, indicating that concept
processing can be understood as a straightforward extension of Hubel
and Wieselssimple-to-complexfeedforward model of visual process-
ing from visual to concept processing. The qualitatively sustained rise in
classication accuracy above chance starting at ~160 ms post-stimulus
onset (Fig. 3B) suggests that the true onset of concept information may
be even earlier than identied in our study. Our results are in line with a
recent study that used a combination of behavioral modeling, deep
neural networks, and MEG to estimate the lower-bound of semantic
processing in the visual ventral stream at ~150 ms and a peak processing
at ~230 ms (Bankson et al., 2018).
Of note, we nd the rapid onset of concept representations not only
in left anterior temporal sensors, but also in left parietal sensors. Pre-
vious work (Garcea et al., 2018;Ishibashi et al., 2011;Pobric et al.,
2010) showed evidence that left parietal tool representations encode
tool manipulation knowledge, and LATL representations store tool
function knowledge. The interplay between these representations, and
to what degree access to one relies on access to the other is unclear. In
certain theories of semantic cognition, the ATL serves as a semantic
hubthat integrates over different attributes (shape, manipulability,
sound, etc.) of a concept and helps coordinate interactions between
them (Patterson et al., 2007;Ralph et al., 2016). Thus, some have
suggested that degeneration of the hub in semantic dementia precludes
access to manipulation knowledge in the visual dorsal stream via shape
representations in the visual ventral stream (Almeida et al., 2013;
Hodges et al., 2000). However, other lesion and TMS studies show that
tool manipulation knowledge can be accessed even when tool function
knowledge is impaired and vice versa (Buxbaum et al., 2000;Buxbaum
and Saffran, 2002;Garcea and Mahon, 2012;Ishibashi et al., 2011). Our
study supports the latter view by showing a direct feedforward func-
tional pathway between orthographic representations, likely in poste-
rior fusiform cortex, and left parietal tool representations that is
independent of the LATL. This is supported by DTI studies (Binder and
Desai, 2011;Wakana et al., 2004) that show direct anatomical con-
nections between the VWFA and LIPL, which could underlie the direct
Granger-causal connectivity seen in our data. An important limitation of
the current work is that we were unable to identify a parallel pathway
between orthography-selective sensors and anterior concept-selective
sensors in our data. However, the existence of feedforward anatom-
ical connections from posterior fusiform cortex to anterior temporal
areas has been well established (Bouhali et al., 2014;Kravitz et al.,
2013;Papinutto et al., 2016). Furthermore, in the context of concept
processing, Clarke et al. (2011) in a MEG study identied a cluster of
signicant phase-locking between posterior shape selective and ante-
rior concept selective sensors in the gamma frequency between 120 and
220 ms. Thus, the literature provides strong existing evidence for a
ventral pathway from orthography to concepts, and the novelty of our
study lies in showing the existence of a parallel ventral-to-dorsal stream
pathway from orthography-selective ventral stream areas to parietal
A notable aspect of our results is that the direct pathway between
orthography and parietal tool concepts exists after training despite the
fact that tool PW orthography, unlike tool images, does not contain any
visual features that could be leveraged to automatically access dorsal
stream manipulation information. Furthermore, our training paradigm
did not explicitly train participants on the association between orthog-
raphy and manipulation. This raises the intriguing question of how this
dorsal word-to-manipulation knowledge network is learned. Findings
from a recent computational investigation (Chen and Rogers, 2015)
suggest that feedback from other conceptual representations can help set
up this shape-to-manipulation pathway. In that study, the authors
implemented a connectionist model in which units encoding shape rep-
resentations in the visual ventral stream were linked to concept units
representing a semantic hub and to a separate set of concept units
S.R. Damera et al. NeuroImage 221 (2020) 117148
representing manipulation knowledge. In this model, as validated by our
results, information from shape units directly engaged the correct in-
formation in the manipulation units after training. In addition, they
found that hub units exerted a feedback inuence on shape units, which
in turn inuenced correct access to manipulation information. This
feedback from the semantic hub units could suggest a mechanism by
which the proper connections between mid-fusiform and parietal regions
can be set up. Alternatively, given its extensive connections with parietal
cortex, regions in prefrontal cortex (Friederici, 2009;Ruschel et al., 2014;
de Schotten et al., 2011) (e.g., in the inferior frontal gyrus, known to be
involved in semantic processing) could be involved in coordinating the
learning of these multiple perceptual-to-conceptual hierarchies in the
In summary, our results indicate that multiple perceptual-to-
conceptual hierarchies, once learned, exist independently of each other
in the brain, allowing fast, feedforward computation of function and
manipulation knowledge, respectively. Further work needs to be done to
understand how different types of representation interact with each other
to set up these hierarchies in the brain.
Declaration of competing interest
The authors declare no competing nancial interests.
CRediT authorship contribution statement
Srikanth R. Damera: Formal analysis, Investigation, Writing - orig-
inal draft, Writing - review &editing. Jacob G. Martin: Formal analysis,
Writing - review &editing. Clara Scholl: Conceptualization, Methodol-
ogy, Investigation. Judy S. Kim: Conceptualization, Methodology,
Investigation. Laurie Glezer: Conceptualization, Methodology, Investi-
gation. Patrick S. Malone: Methodology. Maximilian Riesenhuber:
Supervision, Conceptualization, Methodology, Writing - original draft,
Writing - review &editing, Funding acquisition.
This work was supported by National Science Foundation Grant
1026934 to M.R., National Science Foundation Grant PIRE OISE-
0730255, and NIH Intellectual and Development Disorders Research
Center Grant 5P30HD040677.
Almeida, J., Fintzi, A.R., Mahon, B.Z., 2013. Tool manipulation knowledge is retrieved by
way of the ventral visual object processing pathway. Cortex 49, 23342344. https://
Ashby, G.F., Spiering, B.J., 2004. The neurobiology of category learning. Behav. Cognit.
Neurosci. Rev. 3, 101113.
Bankson, B.B., Hebart, M.N., Groen, I.I.A., Baker, C.I., 2018. The temporal evolution of
conceptual object representations revealed through models of behavior, semantics
and deep neural networks. Neuroimage 178, 172182.
Bastos, A.M., Vezoli, J., Bosman, C.A., Schoffelen, J.-M., Oostenveld, R., Dowdall, J.R.,
Weerd, P.D., Kennedy, H., Fries, P., 2014. Visual areas exert feedforward and
feedback inuences through distinct frequency channels. Neuron 85, 390401.
Binder, J.R., Desai, R.H., 2011. The neurobiology of semantic memory. Trends Cognit.
Sci. 15, 527536.
Bouhali, F., de Schotten, M., Pinel, P., Poupon, C., Mangin, J.-F., Dehaene, S., Cohen, L.,
2014. Anatomical connections of the visual word form area. J. Neurosci. 34,
Brem, S., Bach, S., Kucian, K., Kujala, J.V., Guttorm, T.K., Martin, E., Lyytinen, H.,
Brandeis, D., Richardson, U., 2010. Brain sensitivity to print emerges when children
learn letterspeech sound correspondences. Proc. Natl. Acad. Sci. Unit. States Am.
107, 79397944.
Buxbaum, L.J., Saffran, E., 2002. Knowledge of object manipulation and object function:
dissociations in apraxic and nonapraxic subjects. Brain Lang. 82, 179199. https://
Buxbaum, L.J., Shapiro, A.D., Coslett, B.H., 2014. Critical brain regions for tool-related
and imitative actions: a componential analysis. Brain 137, 19711985. https://
Buxbaum, L.J., Veramontil, T., Schwartz, M.F., 2000. Function and manipulation tool
knowledge in apraxia: knowing what forbut not how. Neurocase 6, 8397. https://
Carreiras, M., Armstrong, B.C., Perea, M., Frost, R., 2014. The what, when, where, and
how of visual word recognition. Trends Cognit. Sci. 18, 9098.
Chan, A.M., Baker, J.M., Eskandar, E., Schomer, D., Ulbert, I., Marinkovic, K., Cash, S.S.,
Halgren, E., 2011. First-pass selectivity for semantic categories in human
anteroventral temporal lobe. J. Neurosci. 31, 1811918129.
Chen, L., Ralph, M.A., Rogers, T.T., 2017. A unied model of human semantic knowledge
and its disorders. Nat. Human Behav. 1,
Chen, L., Rogers, T.T., 2015. A model of emergent category-specic activation in the
posterior fusiform gyrus of sighted and congenitally blind populations. J. Cognit.
Neurosci. 27, 19811999.
Clarke, A., Taylor, K.I., Tyler, L.K., 2011. The evolution of meaning: spatio-temporal
dynamics of visual object recognition. J. Cognit. Neurosci. 23, 18871899. https://
Coutanche, M.N., Thompson-Schill, S.L., 2015. Creating concepts from converging
features in human cortex. Cerebr. Cortex 25, 25842593.
Cui, J., Xu, L., Bressler, S.L., Ding, M., Liang, H., 2008. BSMART: a Matlab/C toolbox for
analysis of multichannel neural time series. Neural Network. 21, 10941104. https://
Cul, A., Baillet, S., Dehaene, S., 2007. Brain dynamics underlying the nonlinear threshold
for access to consciousness. PLoS Biol. 5, e260
de Schotten, M., DellAcqua, F., Forkel, S.J., Simmons, A., Vergani, F., Murphy, D.G.,
Catani, M., 2011. A lateralized brain network for visuospatial attention. Nat.
Neurosci. 14, 1245.
Dehaene, S., Cohen, L., Morais, J., Kolinsky, R., 2015. Illiterate to literate: behavioural
and cerebral changes induced by reading acquisition. Nat. Rev. Neurosci. 16,
Dehaene-Lambertz, G., Monzalvo, K., Dehaene, S., 2018. The emergence of the visual
word form: longitudinal evolution of category-specic ventral visual areas during
reading acquisition. PLoS Biol. 16, e2004103
Delorme, A., Makeig, S., 2004. EEGLAB: an open source toolbox for analysis of single-trial
EEG dynamics including independent component analysis. J. Neurosci. Methods 134,
Dien, J., 1998. Issues in the application of the average reference: review, critiques, and
recommendations. Behav. Res. Methods Instrum. Comput. 30, 3443. https://
Eklund, A., Nichols, T.E., Knutsson, H., 2016. Cluster failure: why fMRI inferences for
spatial extent have inated false-positive rates. Proc. Natl. Acad. Sci. 113,
Fahrenfort, J.J., Lamme, V., 2012. A true science of consciousness explains
phenomenology: comment on Cohen and Dennett. Trends Cognit. Sci. 16, 138139.
Freedman, D.J., Riesenhuber, M., Poggio, T., Miller, E.K., 2003. A comparison of primate
prefrontal and inferior temporal cortices during visual categorization. J. Neurosci. 23,
Friederici, A.D., 2009. Pathways to language: ber tracts in the human brain. Trends
Cognit. Sci. 13, 175181.
Garcea, F.E., Chen, Q., Vargas, R., Narayan, D.A., Mahon, B.Z., 2018. Task- and domain-
specic modulation of functional connectivity in the ventral and dorsal object-
processing pathways. Brain Struct. Funct. 223, 25892607.
Garcea, F.E., Mahon, B.Z., 2012. What is in a tool concept? Dissociating manipulation
knowledge from function knowledge. Mem. Cognit. 40, 13031313.
Gauthier, I., Curran, T., Curby, K.M., Collins, D., 2003. Perceptual interference supports a
non-modular account of face processing. Nat. Neurosci. 6, nn1029.
Glezer, L.S., Jiang, X., Riesenhuber, M., 2009. Evidence for highly selective neuronal
tuning to whole words in the visual word form area.. Neuron 62, 199204. https://
Goodale, M.A., 2011. Transforming vision into action. Vis. Res. 51, 15671587. https://
Goodale, M.A., Milner, A.D., 1992. Separate visual pathways for perception and action.
Trends Neurosci. 15, 2025.
Granger, C., 1969. Investigating causal relations by econometric models and cross-
spectral methods. Econometrica 37, 424.
Grech, R., Cassar, T., Muscat, J., Camilleri, K.P., Fabri, S.G., Zervakis, M.,
Xanthopoulos, P., Sakkalis, V., Vanrumste, B., 2008. Review on solving the inverse
problem in EEG source analysis. J. NeuroEng. Rehabil. 5, 25.
Gregoriou, G.G., Gotts, S.J., Zhou, H., Desimone, R., 2009. High-frequency, long-range
coupling between prefrontal and visual cortex during attention. Science 324,
Grootswagers, T., Wardle, S.G., Carlson, T.A., 2016. Decoding dynamic brain patterns
from evoked responses: a tutorial on multivariate pattern analysis applied to time
series neuroimaging data. J. Cognit. Neurosci. 121.
S.R. Damera et al. NeuroImage 221 (2020) 117148
Hebart, M.N., G
orgen, K., Haynes, J.-D.D., 2014. The Decoding Toolbox (TDT): a versatile
software package for multivariate analyses of functional imaging data. Front.
Neuroinf. 8, 88.
Hodges, J.R., Bozeat, S., Ralph, M.A., Patterson, K., Spatt, J., 2000. The role of conceptual
knowledge in object use Evidence from semantic dementia. Brain 123, 19131925.
Ishibashi, R., Ralph, M.A., Saito, S., Pobric, G., 2011. Different roles of lateral anterior
temporal lobe and inferior parietal lobule in coding function and manipulation tool
knowledge: evidence from an rTMS study. Neuropsychologia 49, 11281135.
Jackson, R.L., Ralph, M.A., Pobric, G., 2015. The timing of anterior temporal lobe
involvement in semantic processing. J. Cognit. Neurosci. 27, 13881396. https://
Jefferies, E., 2013. The neural basis of semantic cognition: converging evidence from
neuropsychology, neuroimaging and TMS. Cortex 49, 611625.
Jiang, X., Bradley, E., Rini, R.A., Zefro, T., VanMeter, J., Riesenhuber, M., 2007.
Categorization training results in shape- and category-selective human neural
plasticity. Neuron 53, 891903.
Kravitz, D.J., Saleem, K.S., Baker, C.I., Mishkin, M., 2011. A new neural framework for
visuospatial processing. Nat. Rev. Neurosci. 12, 217.
Kravitz, D.J., Saleem, K.S., Baker, C.I., Ungerleider, L.G., Mishkin, M., 2013. The ventral
visual pathway: an expanded neural framework for the processing of object quality.
Trends Cognit. Sci. 17, 2649.
Kutas, M., Federmeier, K.D., 2011. Thirty years and counting: nding meaning in the
N400 component of the event-related brain potential (ERP). Ann. Rev. Psychol. 62,
Lau, E.F., Phillips, C., Poeppel, D., 2008. A cortical network for semantics: (de)
constructing the N400. Nat. Rev. Neurosci. 9, 920933.
Malone, P.S., Glezer, L.S., Kim, J., Jiang, X., Riesenhuber, M., 2016. Multivariate pattern
analysis reveals category-related organization of semantic representations in anterior
temporal cortex. J. Neurosci. 36, 1008910096.
Maris, E., Oostenveld, R., 2007. Nonparametric statistical testing of EEG- and MEG-data.
J. Neurosci. Methods 164, 177190.
Maurer, U., Brandeis, D., McCandliss, B.D., 2005. Fast, visual specialization for reading in
English revealed by the topography of the N170 ERP response. Behav. Brain Funct. 1,
Medler, D.A., Binder, J.R., 2005. An on-line orthographic database of the English
language. Available from. Accessed Sept. 10,
Mion, M., Patterson, K., Acosta-Cabronero, J., Pengas, G., Izquierdo-Garcia, D.,
Hong, Y.T., Fryer, T.D., Williams, G.B., Hodges, J.R., Nestor, P.J., 2010. What the left
and right anterior fusiform gyri tell us about semantic memory. Brain 133,
Nosofsky, R.M., 1986. Attention, similarity, and the identicationcategorization
relationship. J. Exp. Psychol. Gen. 115, 39.
Papinutto, N., Galantucci, S., Mandelli, M., Gesierich, B., Jovicich, J., Caverzasi, E.,
Henry, R.G., Seeley, W.W., Miller, B.L., Shapiro, K.A., Gorno -Tempini, M., 2016.
Structural connectivity of the human anterior temporal lobe: a diffusion magnetic
resonance imaging study. Hum. Brain Mapp. 37, 22102222.
Patterson, K., Nestor, P.J., Rogers, T.T., 2007. Where do you know what you know? The
representation of semantic knowledge in the human brain. Nat. Rev. Neurosci. 8,
Pegado, F., Comerlato, E., Ventura, F., Jobert, A., Nakamura, K., Buiatti, M., Ventura, P.,
Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Braga, L.W., Cohen, L., Dehaene, S.,
2014. Timing the impact of literacy on visual processing. Proc. Natl. Acad. Sci. U.S.A.
111, E5233E5242.
Pobric, G., Jefferies, E., Ralph, M.A., 2010. Category-specic versus category-general
semantic impairment induced by transcranial magnetic stimulation. Curr. Biol. 20,
Pulvermüller, F., 2013. How neurons make meaning: brain mechanisms for embodied and
abstract-symbolic semantics. Trends Cognit. Sci. 17, 458470.
Ralph, M.A., Jefferies, E., Patterson, K., Rogers, T.T., 2016. The neural and computational
bases of semantic cognition. Nat. Rev. Neurosci. 18, 4255.
Riesenhuber, M., Poggio, T., 2002. Neural mechanisms of object recognition. Curr. Opin.
Neurobiol. 12, 162168.
Riesenhuber, M., Poggio, T., 2000. Models of object recognition. Nat. Neurosci. 3,
Rossion, B., Gauthier, I., Goffaux, V., Tarr, M., Crommelinck, M., 2002. Expertise training
with novel objects leads to left-lateralized facelike electrophysiological responses.
Psychol. Sci. 13, 250257.
Ruschel, M., Kn
osche, T.R., Friederici, A.D., Turner, R., Geyer, S., Anwander, A., 2014.
Connectivity architecture and subdivision of the human inferior parietal cortex
revealed by diffusion MRI. Cerebr. Cortex 24, 24362448.
Sassenhagen, J., Draschkow, D., 2019. Cluster -based permutation tests of MEG/EEG data
do not establish signicance of effect latency or location. Psychophysiology e13335.
Scholl, C.A., Jiang, X., Martin, J.G., Riesenhuber, M., 2013. Time course of shape and
category selectivity revealed by EEG rapid adaptation. J. Cognit. Neurosci. 26,
Seth, A.K., Barrett, A.B., Barnett, L., 2015. Granger causality analysis in neuroscience and
neuroimaging. J. Neurosci. 35, 32933297.
Tadel, F., Baillet, S., Mosher, J.C., Pantazis, D., Leahy, R.M., 2011. Brainstorm: a user-
friendly application for MEG/EEG analysis. Comput. Intell. Neurosci. 2011, 879716.
Tanaka, J., Curran, T., 2001. A neural basis for expert object recognition. Psychol. Sci. 12,
Thomas, E., Hulle, M.M., Vogel, R., 2006. Encoding of categories by noncategory-specic
neurons in the inferior temporal cortex. J. Cognit. Neurosci. 13, 190200. https://
Thorpe, S.J., Fabre-Thorpe, M., 2001. NEUROSCIENCE: seeking categories in the brain.
Science 291, 260263.
Thorpe, S., Fize, D., Marlot, C., 1996. Speed of processing in the human visual system. Am.
J. Ophthalmol. 122, 608609.
Vandenberghe, R., Price, C., Wise, R., Josephs, O., Frackowiak, R., 1996. Functional
anatomy of a common semantic system for words and pictures. Nature 383, 254256.
Vingerhoets, G., Acke, F., Vandemaele, P., Achten, E., 2009. Tool responsive regions in
the posterior parietal cortex: effect of differences in motor goal and target object
during imagined transitive movements. Neuroimage 47, 18321843.
Wakana, S., Jiang, H., Nagae-Poetscher, L.M., van Zijl, P.C., Mori, S., 2004. Fiber
tractbased atlas of human white matter anatomy 1. Radiology 230, 7787. https://
Widmann, A., Schr
oger, E., 2012. Filter effects and lter artifacts in the analysis of
electrophysiological data. Front. Psychol. 3, 233.
Woo, C.-W., Krishnan, A., Wager, T.D., 2014. Cluster-extent based thresholding in fMRI
analyses: pitfalls and recommendations. Neuroimage 91, 412419.
S.R. Damera et al. NeuroImage 221 (2020) 117148
Full-text available
The existence of a neural representation for whole words (i.e., a lexicon) is a common feature of many models of speech processing. Prior studies have provided evidence for a visual lexicon containing representations of whole written words in an area of the ventral visual stream known as the “Visual Word Form Area” (VWFA). Similar experimental support for an auditory lexicon containing representations of spoken words has yet to be shown. Using fMRI rapid adaptation techniques, we provide evidence for an auditory lexicon in the “Auditory Word Form Area” (AWFA) in the human left anterior superior temporal gyrus that contains representations highly selective for individual spoken words. Furthermore, we show that familiarization with novel auditory words sharpens the selectivity of their representations in the AWFA. These findings reveal strong parallels in how the brain represents written and spoken words, showing convergent processing strategies across modalities in the visual and auditory ventral streams.
Full-text available
The existence of a neural representation for whole words (i.e., a lexicon) is a common feature of many models of speech processing. Prior studies have provided evidence for a visual lexicon containing representations of whole written words in an area of the ventral visual stream known as the “Visual Word Form Area” (VWFA). Similar experimental support for an auditory lexicon containing representations of spoken words has yet to be shown. Using fMRI rapid adaptation techniques, we provide evidence for an auditory lexicon in the “Auditory Word Form Area” (AWFA) in the human left anterior superior temporal gyrus that contains representations highly selective for individual spoken words. Furthermore, we show that familiarization with novel auditory words sharpens the selectivity of their representations in the AWFA. These findings reveal strong parallels in how the brain represents written and spoken words, showing convergent processing strategies across modalities in the visual and auditory ventral streams. Highlights Individual auditory word form areas (AWFA) were defined via an auditory localizer The AWFA shows tuning for individual real words but not untrained pseudowords The AWFA develops tuning for individual pseudowords after training
Full-text available
Visual object representations are commonly thought to emerge rapidly, yet it has remained unclear to what extent early brain responses reflect purely low-level visual features of these objects and how strongly those features contribute to later categorical or conceptual representations. Here, we aimed to estimate a lower temporal bound for the emergence of conceptual representations by defining two criteria that characterize such representations: 1) conceptual object representations should generalize across different exemplars of the same object, and 2) these representations should reflect high-level behavioral judgments. To test these criteria, we compared magnetoencephalography (MEG) recordings between two groups of participants (n = 16 per group) exposed to different exemplar images of the same object concepts. Further, we disentangled low-level from high-level MEG responses by estimating the unique and shared contribution of models of behavioral judgments, semantics, and different layers of deep neural networks of visual object processing. We find that 1) both generalization across exemplars as well as generalization of object-related signals across time increase after 150 ms, peaking around 230 ms; 2) representations specific to behavioral judgments emerged rapidly, peaking around 160 ms. Collectively, these results suggest a lower bound for the emergence of conceptual object representations around 150 ms following stimulus onset.
Full-text available
A whole-brain network of regions collectively supports the ability to recognize and use objects—the Tool Processing Network. Little is known about how functional interactions within the Tool Processing Network are modulated in a task-dependent manner. We designed an fMRI experiment in which participants were required to either generate object pantomimes or to carry out a picture matching task over the same images of tools, while holding all aspects of stimulus presentation constant across the tasks. The Tool Processing Network was defined with an independent functional localizer, and functional connectivity within the network was measured during the pantomime and picture matching tasks. Relative to tool picture matching, tool pantomiming led to an increase in functional connectivity between ventral stream regions and left parietal and frontal-motor areas; in contrast, the matching task was associated with an increase in functional connectivity among regions in ventral temporo-occipital cortex, and between ventral temporal regions and the left inferior parietal lobule. Graph-theory analyses over the functional connectivity data indicated that the left premotor cortex and left lateral occipital complex were hub-like (exhibited high betweenness centrality) during tool pantomiming, while ventral stream regions (left medial fusiform gyrus and left posterior middle temporal gyrus) were hub-like during the picture matching task. These results demonstrate task-specific modulation of functional interactions among a common set of regions, and indicate dynamic coupling of anatomically remote regions in task-dependent manner.
Full-text available
How does education affect cortical organization? All literate adults possess a region specialized for letter strings, the visual word form area (VWFA), within the mosaic of ventral regions involved in processing other visual categories such as objects, places, faces, or body parts. Therefore, the acquisition of literacy may induce a reorientation of cortical maps towards letters at the expense of other categories such as faces. To test this cortical recycling hypothesis, we studied how the visual cortex of individual children changes during the first months of reading acquisition. Ten 6-year-old children were scanned longitudinally 6 or 7 times with functional magnetic resonance imaging (fMRI) before and throughout the first year of school. Subjects were exposed to a variety of pictures (words, numbers, tools, houses, faces, and bodies) while performing an unrelated target-detection task. Behavioral assessment indicated a sharp rise in grapheme–phoneme knowledge and reading speed in the first trimester of school. Concurrently, voxels specific to written words and digits emerged at the VWFA location. The responses to other categories remained largely stable, although right-hemispheric face-related activity increased in proportion to reading scores. Retrospective examination of the VWFA voxels prior to reading acquisition showed that reading encroaches on voxels that are initially weakly specialized for tools and close to but distinct from those responsive to faces. Remarkably, those voxels appear to keep their initial category selectivity while acquiring an additional and stronger responsivity to words. We propose a revised model of the neuronal recycling process in which new visual categories invade weakly specified cortex while leaving previously stabilized cortical responses unchanged.
Full-text available
It has been reported that patients with semantic dementia semantic matching tasks. Object use was markedly impaired and, most importantly, correlated strongly with function well in everyday life and sometimes show striking preservation of the ability to use objects, even those naming and semantic knowledge. In a small number of instances, there was appropriate use of an object for specific objects for which the patient has degraded conceptual information. To explore this phenomenon in which the patient's knowledge on the semantic matching tasks was no better than chance; but this typically applied nine cases of semantic dementia, we designed a set of semantic tests regarding 20 everyday objects and to objects with a rather obvious relationship between appearance and use, or was achieved by trial and error. compared performance on these with the patients' ability to demonstrate the correct use of the same items. We The results suggest that object use is heavily dependent upon object-specific conceptual knowledge, supplemented also administered a test of mechanical problem solving utilizing novel tools, on which the patients had completely to some degree by a combination of visual affordances and mechanical problem solving. normal ability. All but the mildest affected patient showed significant deficits of naming and on the visually based
Full-text available
How is knowledge about the meanings of words and objects represented in the human brain? Current theories embrace two radically different proposals: either distinct cortical systems have evolved to represent different kinds of things, or knowledge for all kinds is encoded within a single domain-general network. Neither view explains the full scope of relevant evidence from neuroimaging and neuropsychology. Here we propose that graded category-specificity emerges in some components of the semantic network through joint effects of learning and network connectivity. We test the proposal by measuring connectivity amongst cortical regions implicated in semantic representation, then simulating healthy and disordered semantic processing in a deep neural network whose architecture mirrors this structure. The resulting neuro-computational model explains the full complement of neuroimaging and patient evidence adduced in support of both domain-specific and domain-general approaches, reconciling long-standing disputes about the nature and origins of this uniquely human cognitive faculty.
Full-text available
Semantic cognition refers to our ability to use, manipulate and generalize knowledge that is acquired over the lifespan to support innumerable verbal and non-verbal behaviours. This Review summarizes key findings and issues arising from a decade of research into the neurocognitive and neurocomputational underpinnings of this ability, leading to a new framework that we term controlled semantic cognition (CSC). CSC offers solutions to long-standing queries in philosophy and cognitive science, and yields a convergent framework for understanding the neural and computational bases of healthy semantic cognition and its dysfunction in brain disorders.
Full-text available
Semantic cognition refers to our ability to use, manipulate and generalize knowledge that is acquired over the lifespan to support innumerable verbal and non-verbal behaviours. This Review summarizes key findings and issues arising from a decade of research into the neurocognitive and neurocomputational underpinnings of this ability, leading to a new framework that we term controlled semantic cognition (CSC). CSC offers solutions to long-standing queries in philosophy and cognitive science, and yields a convergent framework for understanding the neural and computational bases of healthy semantic cognition and its dysfunction in brain disorders.
Cluster‐based permutation tests are gaining an almost universal acceptance as inferential procedures in cognitive neuroscience. They elegantly handle the multiple comparisons problem in high‐dimensional magnetoencephalographic and EEG data. Unfortunately, the power of this procedure comes hand in hand with the allure for unwarranted interpretations of the inferential output, the most prominent of which is the overestimation of the temporal, spatial, and frequency precision of statistical claims. This leads researchers to statements about the onset or offset of a certain effect that is not supported by the permutation test. In this article, we outline problems and common pitfalls of using and interpreting cluster‐based permutation tests. We illustrate these with simulated data in order to promote a more intuitive understanding of the method. We hope that raising awareness about these issues will be beneficial to common scientific practices, while at the same time increasing the popularity of cluster‐based permutation procedures. Cluster‐based permutation tests are a powerful solution to the multiple comparisons problem in EEG and MEG data. We report on extremely common, yet inapplicable interpretations of this procedure, suggesting unwarranted precision of the actual underlying test statistic and leading to strong, but unsubstantiated claims. In this article, we outline problems and common pitfalls of using and interpreting cluster‐based permutation tests. Accurate interpretations of cluster‐based permutation tests will contribute to the adequate utilization, as well as the popularity, of this powerful method.
The study of patients with semantic dementia, a variant of frontotemporal lobar degeneration, has emerged over the last two decades as an important lesion model for studying human semantic memory. Although it is well-known that semantic dementia is associated with temporal lobe degeneration, controversy remains over whether the semantic deficit is due to diffuse temporal lobe damage, damage to only a sub-region of the temporal lobe or even less severe damage elsewhere in the brain. The manner in which the right and left temporal lobes contribute to semantic knowledge is also not fully elucidated. In this study we used unbiased imaging analyses to correlate resting cerebral glucose metabolism and behavioural scores in tests of verbal and non-verbal semantic memory. In addition, a region of interest analysis was performed to evaluate the role of severely hypometabolic areas. The best, indeed the only, strong predictor of semantic scores across a set of 21 patients with frontotemporal lobar degeneration with semantic impairment was degree of hypometabolism in the anterior fusiform region subjacent to the head and body of the hippocampus. As hypometabolism in the patients' rostral fusiform was even more extreme than the abnormality in other regions with putative semantic relevance, such as the temporal poles, the significant fusiform correlations cannot be attributed to floor-level function in these other regions. More detailed analysis demonstrated more selective correlations: left anterior fusiform function predicted performance on two expressive verbal tasks, whereas right anterior fusiform metabolism predicted performance on a non-verbal test of associative semantic knowledge. This pattern was further supported by an additional behavioural study performed on a wider cohort of patients with semantic dementia, in which the patients with more extensive right-temporal atrophy (when matched on degree of naming deficit to a set of cases with more extensive left temporal atrophy) were significantly more impaired on the test of non-verbal semantics. Our preferred interpretation of this laterality effect involves differential strength of connectivity between different regions of a widespread semantic network in the human brain.