ArticlePDF Available

Recursive hierarchical embedding in vision is impaired by posterior middle temporal gyrus lesions

Authors:
  • Berufsbildungswerk für Leipzig für Hör- und Sprachgeschädigte

Abstract and Figures

The generation of hierarchical structures is central to language, music and complex action. Understanding this capacity and its potential impairments requires mapping its underlying cognitive processes to the respective neuronal underpinnings. In language, left inferior frontal gyrus and left posterior temporal cortex (superior temporal sulcus/middle temporal gyrus) are considered hubs for syntactic processing. However, it is unclear whether these regions support computations specific to language or more generally support analyses of hierarchical structure. Here, we address this issue by investigating hierarchical processing in a non-linguistic task. We test the ability to represent recursive hierarchical embedding in the visual domain by contrasting a recursion task with an iteration task. The recursion task requires participants to correctly identify continuations of a hierarchy generating procedure, while the iteration task applies a serial procedure that does not generate new hierarchical levels. In a lesion-based approach, we asked 44 patients with left hemispheric chronic brain lesion to perform recursion and iteration tasks. We modelled accuracies and response times with a drift diffusion model and for each participant obtained parametric estimates for the velocity of information accumulation (drift rates) and for the amount of information accumulated before a decision (boundary separation). We then used these estimates in lesion-behaviour analyses to investigate how brain lesions affect specific aspects of recursive hierarchical embedding. We found that lesions in the posterior temporal cortex decreased drift rate in recursive hierarchical embedding, suggesting an impaired process of rule extraction from recursive structures. Moreover, lesions in inferior temporal gyrus decreased boundary separation. The latter finding does not survive conservative correction but suggests a shift in the decision criterion. As patients also participated in a grammar comprehension experiment, we performed explorative correlation-analyses and found that visual and linguistic recursive hierarchical embedding accuracies are correlated when the latter is instantiated as sentences with two nested embedding levels. While the roles of the inferior temporal gyrus and posterior temporal cortex in linguistic processes are well established, here we show that posterior temporal cortex lesions slow information accumulation (drift rate) in the visual domain. This suggests that posterior temporal cortex is essential to acquire the (knowledge) representations necessary to parse recursive hierarchical embedding in visual structures, a finding mimicking language acquisition in young children. On the contrary, inferior frontal gyrus lesions seem to affect recursive hierarchical embedding processing by interfering with more general cognitive control (boundary separation). This interesting separation of roles, rooted on a domain-general taxonomy, raises the question of whether such cognitive framing is also applicable to other domains.
Content may be subject to copyright.
Recursive hierarchical embedding in vision is
impaired by posterior middle temporal gyrus
lesions
Mauricio J.D. Martins,
1,2,3
Carina Krause,
2,3,4
David A. Neville,
1,5
Daniele Pino,
2,3
Arno Villringer
1,2,3
and Hellmuth Obrig
1,2,3
The generation of hierarchical structures is central to language, music and complex action. Understanding this capacity and its
potential impairments requires mapping its underlying cognitive processes to the respective neuronal underpinnings. In language,
left inferior frontal gyrus and left posterior temporal cortex (superior temporal sulcus/middle temporal gyrus) are considered hubs
for syntactic processing. However, it is unclear whether these regions support computations specific to language or more generally
support analyses of hierarchical structure. Here, we address this issue by investigating hierarchical processing in a non-linguistic
task. We test the ability to represent recursive hierarchical embedding in the visual domain by contrasting a recursion task with an
iteration task. The recursion task requires participants to correctly identify continuations of a hierarchy generating procedure,
while the iteration task applies a serial procedure that does not generate new hierarchical levels. In a lesion-based approach, we
asked 44 patients with left hemispheric chronic brain lesion to perform recursion and iteration tasks. We modelled accuracies and
response times with a drift diffusion model and for each participant obtained parametric estimates for the velocity of information
accumulation (drift rates) and for the amount of information accumulated before a decision (boundary separation). We then used
these estimates in lesion-behaviour analyses to investigate how brain lesions affect specific aspects of recursive hierarchical embed-
ding. We found that lesions in the posterior temporal cortex decreased drift rate in recursive hierarchical embedding, suggesting an
impaired process of rule extraction from recursive structures. Moreover, lesions in inferior temporal gyrus decreased boundary
separation. The latter finding does not survive conservative correction but suggests a shift in the decision criterion. As patients also
participated in a grammar comprehension experiment, we performed explorative correlation-analyses and found that visual and
linguistic recursive hierarchical embedding accuracies are correlated when the latter is instantiated as sentences with two nested
embedding levels. While the roles of the inferior temporal gyrus and posterior temporal cortex in linguistic processes are well
established, here we show that posterior temporal cortex lesions slow information accumulation (drift rate) in the visual domain.
This suggests that posterior temporal cortex is essential to acquire the (knowledge) representations necessary to parse recursive
hierarchical embedding in visual structures, a finding mimicking language acquisition in young children. On the contrary, inferior
frontal gyrus lesions seem to affect recursive hierarchical embedding processing by interfering with more general cognitive control
(boundary separation). This interesting separation of roles, rooted on a domain-general taxonomy, raises the question of whether
such cognitive framing is also applicable to other domains.
1 Berlin School of Mind and Brain, Humboldt-Universita
¨t zu Berlin, Berlin, Germany
2 Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
3 Department of Cognitive Neurology, University Hospital Leipzig, Leipzig, Germany
4 Erziehungswissenschaftliche Fakulta
¨tPa
¨dagogik im Fo
¨rderschwerpunkt Sprache und Kommunikation, Leipzig University, Leipzig,
Germany
5 Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, The Netherlands
doi:10.1093/brain/awz242 BRAIN 2019: Page 3217 of 3229 |3217
Received November 26, 2018. Revised June 11, 2019. Accepted June 16, 2019
ßThe Author(s) (2019). Published by Oxford University Press on behalf of the Guarantors of Brain.
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits
non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com
Correspondence to: Mauricio Dias Martins
Max Planck Institute for Human Cognitive and Brain Sciences
Stephanstrasse 1a, 04103, Leipzig, Germany
E-mail: mmartins@cbs.mpg.de
Keywords: hierarchy; visuospatial; lesion; syntax; inferior frontal gyrus
Abbreviations: DDM = drift-diffusion model; IFG = inferior frontal gyrus; ITE = Visual Iteration Task; MTG = middle temporal
gyrus; pTC = posterior temporal cortex; REC = Visual Recursion Task; RHE = recursive hierarchical embedding
Introduction
Humans have the ability to process hierarchical structures
(Fitch and Martins, 2014;Wilson et al., 2017), and this
capacity has been mostly studied in the domain of lan-
guage. For instance, in the sentence ‘instinctively, birds
that fly swim’, the adverb ‘instinctively’ modifies the verb
‘swim’ and not the verb ‘fly’, despite being more distant to
the former in the linear structure. This is because, within
the underlying syntactic structure, ‘instinctively’ is closer to
‘swim’ than to ‘fly’ regarding hierarchical depth: {instinct-
ively, {birds {that fly} swim}} (Berwick and Chomsky,
2015). Such hierarchy in language is thought to result
from an innate recursive procedure (Berwick et al., 2013;
Berwick and Chomsky, 2015;Everaert et al., 2015), which
when applied stepwise, generates multiple nested hierarch-
ical levels. Although ‘infinite recursion’ is considered a core
feature of human language, it is rare to find sentences with
more than two levels of hierarchical embedding. One im-
portant limitation is that, in spoken language, working
memory capacity is strongly taxed due to an increasing
number of elements that have to be held active before sen-
tence meaning integration. In other domains, these memory
limitations might not be as pronounced (e.g. the visuospa-
tial), even though the theoretical limit of recursion is never
achieved.
The mechanisms supporting hierarchy in language are
thought to be implemented by dedicated neural systems
(Berwick et al., 2013). Evidence for this view comes from
neuroimaging (EEG and functional MRI) and lesion stu-
dies. The patholinguistic condition of agrammatism, for in-
stance, interferes with the processing of complex syntax,
while other linguistic abilities are largely preserved
(Grodzinsky and Santi, 2008). It remains unclear, however,
whether areas putatively supporting hierarchical linguistic
processes could also support recursive hierarchical embed-
ding (RHE) in other domains. Here, we aim to investigate
whether lesion patterns associated with agrammatism also
interfere with RHE in the visual domain, by testing patients
with an acquired chronic brain lesion in the left
hemisphere.
While studies on the neural correlates of RHE are scarce,
there is converging evidence that syntactic processing relies
on two major hubs: the inferior frontal gyrus (IFG) and the
posterior temporal cortex (pTC) (Friederici, 2011;Hagoort
and Indefrey, 2014;Matchin et al., 2017). Within the pTC,
some studies implicate the posterior superior temporal
sulcus and others implicate the posterior middle temporal
gyrus (MTG) (see Hagoort and Indefrey, 2014, for a meta-
analysis). The role of these areas has been discussed along
two rationales: computations within IFG may be necessary
to implement recursive generation of linguistic hierarchies
(Friederici et al., 2011;Zaccarella et al., 2017), a finding
that would explain central symptoms of agrammatism
(Matchin and Rogalsky, 2018). Alternatively, functional
MRI results are compatible with the interpretation that
IFG implements domain-general ‘computations’ (e.g. relat-
ing to working memory or cognitive control), which oper-
ate on domain-specific ‘representations’ supported by pTC
(Rogalsky et al., 2011;Matchin, 2018;Matchin et al.,
2017). These representations might be both simple lexical
units or complex hierarchical templates containing a set of
features that dictate how basic units can be further com-
bined (Matchin, 2018). These two models may be tested by
extending research on recursion to other domains.
Here we focus on the visual domain. We study recursive
abilities in individuals with a chronic circumscribed lesion
in the left hemisphere. As most of them participated in
another study on linguistic syntax processing, we are in
the unique position to gain a first insight into potential
correlations between RHE across different modalities.
RHE has been hypothesized to play a role in the process-
ing of visuo-spatial structures (Pinker and Jackendoff,
2005). To supply experimental evidence, a visual recursion
task (REC) was recently developed in which participants
generate novel hierarchical levels using a recursive embed-
ding rule (Martins et al., 2016). In a control iteration task
(ITE) items are added sequentially to existing hierarchies
without generating new levels. As recursive competence in
this task correlates with similar abilities in music and action
(Martins et al., 2017), we consider it ideal to tap into
shared RHE resources across domains.
In the visual domain, the contrast REC versus ITE acti-
vates anterior regions within superior temporal sulcus,
along with several regions within the default mode network
(DMN) and medial temporal lobe (MTL) (Martins et al.,
2014a;Fischmeister et al., 2017). These regions fit the
classical view that the visual ventral stream and the par-
ieto-medial temporal pathway (PMT) integrate items in
contextual frames (Kravitz et al., 2011). Recently, this re-
search has been extended to the domains of music and
action (Martins, 2017;Martins et al., 2017). While we
3218 |BRAIN 2019: Page 3218 of 3229 M. J. D. Martins et al.
found no evidence for the involvement of IFG or pTC in
these domains, we found areas involved in tonal sequence
representation (anterior STG) in music (Martins, 2017),
and areas involved in motor imagery for the motor
domain (premotor cortex, basal ganglia, cerebellum;
Martins et al., 2019). In favour of Matchin’s hypothesis
(Matchin, 2018;Matchin et al., 2017), this suggests that
domain-specific ‘representations’ might be involved in the
processing of RHE.
If domain-specific representations are recruited for highly
trained behaviour as applies to trained musicians, this does
not preclude the possibility that de novo analysis of recur-
sive structure may rely on more domain-general capacities,
therefore tapping into a similar neuronal network across
tasks. In fact, previous behavioural research with untrained
participants during the acquisition of RHE suggests com-
munalities across visual, music and action domains
(Martins et al., 2017).
Importantly, the general capacity to instantiate RHE is
thought to result from the interaction of a core RHE ma-
chinery with two peripheral systems: sensory-motor and
conceptual-intentional (Everaert et al., 2015). Crucially,
the interaction of the core capacity with sensorial systems
of different domains requires specialized interfaces. For in-
stance, RHE in music and oral language hinges on auditory
working memory system, while in the visual domain RHE
is dependent on a visual working memory system. While
these interfaces might have constraints specific to each
domain, it is possible that similar neural networks are ne-
cessary to instantiate the core capacity across domains.
To elucidate the processes involved in the acquisition of
RHE in the visuo-spatial domain vis a vis with the pro-
cesses involved in syntactic processing, we tested indivi-
duals with a chronic acquired left hemisphere lesion
resulting from various aetiologies. If a distinct network is
required for the inference of the recursive process in a novel
task, we expect that variance across participants will vary
depending on lesion location. Furthermore, we can test this
relationship by correlating performance in the visual recur-
sion task with some aspects of a linguistic task involving
syntactic embedding, and performed in a largely overlap-
ping cohort of participants.
As behavioural measures of response accuracy and la-
tency are expected to largely vary in clinical populations,
we extend our behavioural analysis to a more comprehen-
sive analytical framework. The drift-diffusion model
(DDM; Smith and Ratcliff, 2004) is a sequential sampling
model for the analysis of choice-reaction time data, which
is the combination of reaction time and accuracy measures.
The DDM model yields estimates of: (i) the velocity at
which a decision is made (v0, drift); (ii) the amount of
information that is required to make the decision (a0,
boundary); and (iii) the amount of time required to com-
plete non-decision processes (t0, non-decision time). Here,
we used a hierarchical version of the DDM model (Wiecki
et al., 2013) to obtain estimates that take into account
inter-subject variability.
As an example of how the DDM can explain non-linear
dynamics of the decision process, consider the scenario
where impairment may lead to slower responses in a spe-
cific task condition. Simple analyses of behavioural per-
formance would not be able to compare the alternative
explanations of whether the slow responses are due to
more information being needed to make the decision
(larger boundary separation) or due to a reduction in the
rate at which information accumulates (drift rate). Changes
in either of these two factors could produce responses with
larger reaction times. Therefore a sequential sampling
model is needed to correctly estimate the dynamics of the
underlying decision process.
For the detection of brain regions involved in parsing the
RHE properties of our stimuli, the v0(drift) parameter is
particularly informative since it has been shown to account
for information accumulation in both behaviour and neural
(spike) data (Gerstein and Mandelbrot, 1964;Srinivasan
and Sampath, 2013;Durstewitz et al., 2016).
Here, we investigate the mechanisms supporting the ac-
quisition of RHE in the visual domain, by assessing per-
formance in a population with acquired focal chronic
lesions of the left hemisphere. We target the effects of
lesion site on interindividual variance in behavioural per-
formance and hypothesize that lesions in brain areas central
for recursive analysis in the linguistic domain (IFG and
pTC) also affect RHE in vision. Regions within the right
hemisphere might be relevant for rule acquisition in general
and for the acquisition of linguistic competence (Dehaene-
Lambertz et al., 2002). However, since linguistic impair-
ment in adults is largely caused by left hemispheric lesions,
these will be the focus of our analysis. Thus, we predict
that patients with lesions in these areas of the left hemi-
sphere will be slower in the accumulation of information
necessary to process RHE (modelled as drift rate).
Comparing accuracy in the novel visual recursion task
and accuracy in a syntax task (reported in detail else-
where), we hypothesize that performance in REC (rather
than ITE) correlates with performance in understanding
multiple embedded sentences.
Materials and methods
Participants
Forty-four participants (n=19 female) with an acquired
chronic left hemispheric brain lesion were included. They
were recruited at the Clinic for Cognitive Neurology,
University Hospital Leipzig. Mean age SD (range) was 50
years 10.6 (24–74), and time since event 23.1
months 22.3 (3–115). Participants were in the chronic
stage after a vascular lesion (n=37; 25 ischaemic stroke, five
subarachnoid haemorrhage, seven intracerebral haemorrhage)
traumatic brain injury (n=3), encephalitis (n=1) or suffered
from brain tumours in the stage of a ‘stable disease’ (n=3).
We deliberately chose patients with lesions related to different
aetiologies as this reduces the bias for specific lesion patterns
Temporal lesions cause recursion deficits BRAIN 2019: Page 3219 of 3229 |3219
due to the nature of the disease (e.g. vascular territories in
ischaemic stroke). All patients were in the chronic stage of
the disease and showed no clinical or radiological sign of rele-
vant affection of brain areas distant to the focal lesion which
entered the analysis (Supplementary Table 1 for details, includ-
ing some additional comments on the rationale of patient se-
lection). Participants gave informed consent; data were
collected according to the Declaration of Helsinki after ap-
proval of the local ethics committee.
Patients underwent an extensive clinical testing during their
therapeutic stay at the clinic, which was used to judge eligibil-
ity for participation. Severe cognitive impairment was an ex-
clusion criterion. A left hemispheric lesion in a structure
considered to be part of the extended language network was
present in all patients. However, manifest aphasia at the time
of participation was not mandatory. Overall, 12 patients
showed no aphasia at the time of testing, of whom seven
had shown a clinically relevant aphasia in the acute stage of
their disease. At the time of testing 13 participants had an
aphasia that was classified according to the Aachen Aphasia
Test (AAT) (Huber et al., 1984), the standard diagnostic tool
for aphasia in German (seven amnestic, three Broca, two
Wernicke, one non-classifiable). Nineteen participants had clin-
ically manifest language impairment termed ‘residual aphasia’
according to AAT logics. In sum, the cohort was mildly im-
paired, but only four patients had never shown aphasic symp-
toms, while 32 showed a clinically-relevant language
impairment at the time of testing. For details see
Supplementary Table 1.
Imaging
For all participants (n=44), structural imaging was available.
Thirty-nine scans were performed at in-house scanners (3T
Siemens MRI system TrioÕor VerioÕsystem, Siemens
Medical Systems) including 3D T
1
-weighted (1 mm isovoxel),
and fluid-attenuated inversion recovery (FLAIR) images. In
four patients clinical MRI at a lower resolution [3–5 mm
slice thickness, including FLAIR or turbo inversion recovery
magnitude (TIRM) and T
1
images] was available; in one pa-
tient a cranial CT was used for lesion delineation. For the
lesion-behaviour analyses, lesions were manually delineated
in all three planes on each slice of the T
1
or cranial CT
images using MRIcron (Rorden and Brett, 2000), for MRI
FLAIR/TIRM-images served as a reference. Images were then
transformed into standard stereotactic space (MNI) using
SPM8 (www.fil.ion.ucl.ac.uk/spm) and the ‘clinical toolbox’
(nitrc.org/projects/clinicaltbx/), which allows for normalizing
images from different modalities into the same space. The uni-
fied segmentation approach was applied (Ashburner and
Friston, 2005) and estimation of normalization parameters
was restricted to healthy tissue using predefined lesion masks
(Brett et al., 2001). The resulting normalized binary lesion
maps were next analysed in NiiStat (https://github.com/neuro-
labusc/NiiStat), including a ‘traditional’ voxel-wise analysis but
also providing options for a region-based analysis. While the
former can be considered more sensitive to smaller lesion foci
correlating with behaviour, the latter is less susceptible to false
negatives, since the issue of multiple comparison correction is
greatly reduced.
Experimental tasks
Visual Recursion Task
For the Visual Recursion Task (REC), participants were shown
a sequence of three images (steps 1–3), which depicted a pro-
cess generating a visual fractal. After the first three images,
participants were asked to discriminate, from two choices,
the image corresponding to the correct continuation of the
previous sequence of three (i.e. the fourth step). One of the
choices was the correct image, and the other was a foil. The
task is an adaptation of the one used and described in detail
elsewhere (Martins, 2012;Martins et al., 2014a,b). REC was
composed of 27 trials, nine of each foil category. Variability
was achieved by varying the number of constituents compos-
ing the visual fractal, as described in (Martins et al., 2014b).
Each trial began with the presentation of three images of a
fractal generation in the top half of the screen, sequentially
from left to right (Fig. 1A) at a rate of 2 s between image
onsets. After the presentation of the first three steps, two
new images were presented simultaneously in the bottom
half of the screen. One image corresponded to the correct con-
tinuation of the recursive process that generated the first three
fractals, and the other corresponded to a foil (or ‘incorrect’
continuation). Participants were asked to select the image that
continued the recursive process. The position of the ‘correct’
image (left or right) was randomized. After the initial instruc-
tions, each trial had a maximum duration of 30 s before a
timeout. No feedback was given regarding the correctness of
choice. Total duration of the task (27 trials) was 12 min.
To control for effects of information processing demands,
we included stimuli with different degrees of visual complexity
(complexity ‘3’, ‘4’, and ‘5’). Furthermore, to control for the
usage of simple visual heuristic strategies in REC performance,
we included several categories of foils (‘Odd’, ‘Position’ and
‘Repetition’; Fig 1B). Complexity and foil-type allow for nine
types of stimuli (three complexity levels three foil types).
Three examples of each type of stimuli were generated using
the programming language Python, resulting in the total set of
27 stimuli.
Visual Iteration Task
The second task was iterative but non-recursive (Martins et al.,
2016). The principle underlying ITE is similar to REC in that
it involves a stepwise procedure applied to hierarchical struc-
tures. However, ITE lacks recursive embedding. Instead, in
ITE, additional elements are added to one pre-existing hier-
archical structure, without producing new hierarchical levels
(Fig. 1A, bottom right). As for REC, an understanding of
this stepwise procedure is necessary to correctly predict the
next step. Number of trials, visual complexity and foil cate-
gories and distributions were equivalent to REC.
Procedure
The visual recursion/iteration task and a task assessing com-
prehension of multiple degrees of sentential embedding
(Fig. 4A) were part of a larger assessment battery. The linguis-
tic task was performed one day prior to the visual tasks. For
the visual task half of the participants started the procedure
with ITE [order: Iteration (I)!Recursion (R)] and half started
the procedure with REC (order: R!I).
3220 |BRAIN 2019: Page 3220 of 3229 M. J. D. Martins et al.
Statistical analyses
Visual task data allows the analysis of two factors: Task (REC/
ITE) and Order (I!R/R!I) and their interaction. We ana-
lysed the base parameters of the behavioural experiment
(reaction time and accuracy) and then fed these into a drift-
diffusion-model yielding a measure for the velocity of informa-
tion accumulation (drift, v0) and the amount of information
needed to make a decision (boundary, a0). The third parameter
of the latter analysis, non-decision time (t0), is not analysed
further here. Posterior group-level distributions for all of the
parameters can be inspected in Supplementary Fig. 8.
Reaction time and accuracy
Statistical analyses were performed in R studio (1.1.453). We
ran linear mixed models with the function lmer() with package
lme4 (Bates et al., 2014), with participants as random factor.
Best lambda transformation was found using boxcox() with
package MASS (Venables and Ripley, 2002). All residual dis-
tributions reported in this manuscript were normal, calculated
using Shapiro-Wilk test (all P’s 40.2). Models are reported
using ANOVA (type = II) and the R package Anova() for P-
values. When main effects were found, we tested for pairwise
differences with emmeans() (Russell, 2018), using Kenward-
Roger methods to calculate the degrees of freedom, and
Tukey P-value adjustment when comparing three parameters.
Finally, we ran spearman correlations since variables were not
normally distributed, and one-tailed tests since we expected
grammar comprehension to positively correlate with visual re-
cursion accuracy and negatively with response time.
Drift diffusion analysis
Choice reaction time data were fitted to a hierarchical version
of drift diffusion model using customized scripts implemented
in the HDDM toolbox (Wiecki et al., 2013) for Python. Since
the dataset presented very long responses (410 s) we scaled all
of the reaction times by a constant factor (10) so that all of the
reaction times would be in the range of 0–10 s. The analyses
proceeded as follows. First, we fitted to the data models with
different combinations of free parameters over the two factors
of experiment (Order and Task). Each of the models was fitted
to the data using standard MCMC minimization routines with
50 000 iterations, a burn-in period of 5000 and thinning of 1.
For all of the chains, the results converged to stable estimates
Figure 1 Experimental paradigm. (A) The presentation of the visual recursion/iteration task (REC, ITE) comprised four steps including a
successive presentation of the steps 1–3 at the top of the screen to then present the two options for a forced choice at the bottom. Examples for
REC and ITE screen shots are provided for step 4; note that the final choice images are identical for both tasks. Location of correct image was
randomized (e.g. left in the ITE and right in the REC example provided). (B) Examples of fractals used in REC. There were different categories of
‘visual complexity’—3, 4 and 5—and different categories of foils. In ‘odd constituent’ foils, two elements within the whole hierarchy were
misplaced; in ‘positional error’ foils, all elements within new hierarchical levels were internally consistent, but inconsistent with the previous
iterations; in ‘repetition’ foils, no additional iterative step was performed after the third iteration.
Temporal lesions cause recursion deficits BRAIN 2019: Page 3221 of 3229 |3221
as indicated by the relative diagnostic plots and by the Rhat
statistics (51) which is a measure of chain convergence. For
each of the models we then computed the relative deviance
information criterion (DIC) which is a measure of the good-
ness of fit of the model to the data penalized by the complexity
of the model (i.e. functional form of the free parameters). A
model with a lower DIC score is to be preferred as the most
parsimonious account of the data. Comparisons of the DIC
scores for all models indicated a model with drift rate (v0)
and non-decision time (t0) free to vary over both experimental
conditions and boundary separation (a0) fixed over task-order
conditions as the most parsimonious account of the data. For
brevity, results of the model comparison are reported in the
supplementary information and only results from the best-fit-
ting model are reported in the main text. Estimated parameters
for the best fitting model were finally tested for significant
differences using linear regression analyses with permutation
based-calculation of significance levels implemented in RÕwith
the package lmPerm (Wheeler and Torchiano, 2016).
Lesion–behaviour analyses
Analyses were conducted with NiiStat (https://github.com/neu-
rolabusc/NiiStat). Anatomical correlates of REC and ITE ac-
curacy (with Task-Order as nuisance variable) were assessed
using both a region- and a voxel-based approach (Bates et al.,
2003;Rorden et al., 2007).
For the statistical region of interest approach two atlases
were used: the Atlas of Intrinsic Connectivity of Homotopic
Areas (AICHA), which contains 384 grey matter regions of
interest (Joliot et al., 2015), and the Brodmann atlas, contain-
ing 82 grey matter areas. To find out about the relevance of a
specific region of interest for task performance the proportion
of damage to a given region was computed in each participant
and entered into a general linear model (GLM). This statistic-
ally assesses a correlation between the amount of damage in a
given region and the behaviour in question. The result was
converted to a z-score for each region. To control for
family-wise error, the data were permuted 5000 times to es-
tablish a significance threshold. Only those regions with z-
scores above the permutation threshold P50.05 are reported
(Rorden et al., 2007). Relative to voxelwise approaches (see
below), this method increases statistical power by both aver-
aging data and limiting the number of statistical comparisons.
An initial statistical region of interest analysis was conducted
to examine each parameter individually. A second analysis
used each parameter as a nuisance factor for the other vari-
ables using the Freedman–Lane method (Freedman and Lane,
1983), this is a natural extension of GLM that allows us to
compute permutation (Winkler et al., 2014). We were espe-
cially interested in the correlates of REC when taking ITE and
order as nuisance regressors.
Additionally, we performed the ‘traditional’ voxel-wise ap-
proach, conducting independent statistical tests for each voxel
that is covered by a lesion overlap of at least four participants.
To control for multiple comparisons only voxels surviving
5000 permutation correction are reported in the statistical
maps. Using NiiStat, t-tests were computed for every voxel
to see if those individuals with a lesion at that location ex-
hibited reliably different behavioural performance (using our
continuous indices) than those without a lesion. The t-tests
were confined to voxels that sustained damage in at least
10% (n= 4 subjects) of the sample, defining the areas for
which the analysis can provide statistical inference.
Grammar task
All participants of the visual recursion/iteration-task also per-
formed another experiment targeting the comprehension of
complex syntax (Krause et al., submitted for publication). In
that experiment participants listened to sentences containing
three propositions regarding two animals interacting with
each other. Propositions were: (i) the action (e.g. X washes
Y); (ii) the colour of one animal (e.g. X is brown); and
(iii) the mood of one animal (e.g. X laughs). After the auditory
presentation of the sentence (overall set: 132 sentences), par-
ticipants had to choose the correct image from a set of four
images (one correct, three incorrect containing a distractor for
each proposition). Syntactic complexity of the sentences was
manipulated by the embedding depth in that the three prop-
ositions were serially linked by a conjunction or embedded
using embedded relative clauses. An additional manipulation
was introduced by varying argument order (i.e. subject first or
object first relative clause). Here, we only use the differences in
embedding depth, for which we supply an example in Fig. 4A.
Note that only the difference between EMB1 and EMB2 enters
the analysis. EMB0 sentences contain crossed-dependencies,
meaning that the personal pronoun ‘er’ [he] in the last prop-
osition can relate to the animal mentioned first and the one
mentioned second because they share grammatical gender. The
difference in argument order is illustrated in an example in the
Supplementary material including an example for the four-
image choice.
Data availability
Anonymized data are available on request.
Results
Reaction time and accuracy
Accuracy showed a main effect of Order and an interaction
between Task Order. The main effect for Task did not
reach statistical significance. For reaction time neither main
nor interaction effects were significant.
Results for REC and ITE are presented in Fig. 2. To test
for differences between ITE and REC, we performed two
independent analyses, one with accuracy and another with
response time as dependent variables. As predictors, we
included Task (REC/ITE), Order (R!I versus I!R,
balanced across participants), and the interaction
Task Order.
For accuracy, we found a significant effect of Order
[F(1,85) = 6.4, P= 0.01] and Task Order [F(1,85) = 7.4,
P= 0.007]. The main effect of Task was not significant
(P= 0.7). The proportion of correct responses in REC
was lower in the Order when this task was performed
first (R!I versus I!R) [t(61.7) = 3.5; P=0.0009].
Performance in ITE did not differ between the two orders
[t(61.7) = 1.0; P=0.3]. We repeated the procedure for
3222 |BRAIN 2019: Page 3222 of 3229 M. J. D. Martins et al.
reaction time and the best fit was the intercept-only model
[with restricted maximum likelihood (REML) = 180],
meaning that Task, Order, and the interaction were not
significant (all P’s 40.2).
Hierarchical drift diffusion model
results
As reaction time and accuracy interact in a complex
manner even in binary decision tasks, we performed an
additional analysis based on the values that were derived
from a DDM.
We ran a model selection procedure (Supplementary ma-
terial) and found that a model with drift and non-decision
time free to vary over Order and Task factors and bound-
ary distance over Task provided the most parsimonious
account of the data (best model). We then used the poster-
ior estimates for each Task (REC and ITE) while control-
ling for order effects. We thus obtained measures of
performance for REC and ITE independent of the (arbitrar-
ily assigned) order and the overall performance, the latter
depending on a large number of individual differences be-
tween patients which are of no specific interest here. As
illustrated in Fig. 2B the analysis showed a large variance
across participants and no significant differences between
Tasks. The fact that nearly equal numbers of participants
showed REC4ITE and vice versa for both orders supports
the effective cancellation of the order effect for this ana-
lysis, and thereby allows for the lesion–behaviour analysis
across all participants.
Lesion-behaviour correlations:
statistical region of interest and
voxelwise analyses
Lesion-behaviour analyses performed on the ‘base-param-
eters’ reaction time and accuracy yielded no statistically
robust results.
Figure 2 Behavioural data. (A) Percentage of correct answers (acc [%], left) and response time (RT [s], right) in the ITE (blue), and REC (red).
The order of visual tasks was either ‘I!R’ or ‘R!I’ as indicated by the light or darker shading. (B) We combined these data into a hierarchical
DDM (text for details) and obtained posterior estimates for drift rate (drift v0) and boundary separation (boundary a0). Note that order no longer
influences the performance since roughly equal numbers of patients showed values REC4ITE and ITE4REC for these measures (colour coding of
individual data points as in A). The variability can be used across participants in the lesion-behaviour analysis (main text and Fig. 3). For detailed
descriptive statistics, see Supplementary Table 2.
Temporal lesions cause recursion deficits BRAIN 2019: Page 3223 of 3229 |3223
On the contrary, for the drift rate (v0) and boundary
separation (a0) both statistical region of interest-based and
voxelwise analyses yielded lesion patterns which correlated
significantly with the variability of the parameters across
participants.
For the statistical region of interest analysis, temporal
cortex areas correlated with the variation of v0when fac-
toring out ITE as nuisance factor. Drift rate v0(speed of
information integration during REC decision-making
processes) was lower when the participant’s lesion included:
the posterior MTG (z= 3.0), according to the AICHA atlas,
and BA 21 according to the Brodmann altas (z= 3.3). The
results are depicted in Fig. 3C.
Using the voxelwise approach (Fig. 3B), lesions in the left
MTG correlated with decreased drift rate v0. Interestingly,
lesions in parts of the IFG were associated with a decrease
in the boundary separation a0, indicating that participants
with lesions in these areas acquire less information before
they make a decision. Only 39 voxels in the posterior MTG
survived thresholding in the drift results (z43.7, 5000 per-
mutations, P= 0.05), converging with the statistical region
of interest approach. No voxels survived thresholding for
the boundary separation.
Additional analysis regarding the
correlation with a Grammar task
As most of the participants (n=41) also performed a task
on complex grammar comprehension, which is reported
in detail elsewhere (Krause et al., submitted for publica-
tion), we performed a correlation analysis between the
performance in the visual tasks (REC and ITE) and an
aspect of the grammar experiment which can be con-
sidered a linguistic counterpart of the REC/ITE-learning
task reported here. As detailed in the ‘Materials and
methods’ section, the linguistic task requires that partici-
pants judge the meaning of sentences with increasing
levels of embedding. Here, we use the performance for
single- and double-embedded sentences (EMB1 and
EMB2, for an example see Fig. 4A). The percentage of
correct responses in the Grammar task are provided in
Supplementary Table 3.
To test for communalities between our visual task and
the performance in the different embedding levels of the
grammar tasks, we ran Spearman correlations between
accuracyinRECandITE,andaccuracyinEMB1and
EMB2. For this behavioural analysis we chose accuracy
and not the DDM parameters v0and a0in order to com-
pare similar constructs. The DDM is a model for 2-forced
choice tasks and therefore cannot be directly applied to
the 4-choice grammar tasks. Therefore, since we cannot
obtain a ‘grammar drift’ to compare with the ‘visual drift’,
we decided to compare grammar accuracy with visual
accuracy.
Scatterplots are depicted in Fig. 4B. P-values are given for
one-tailed tests, as we expected a positive correlation be-
tween the visual and grammar tasks. We found that, for
accuracy, the correlation with EMB1 was marginal for ITE
(rs = 0.24, P= 0.066) and significant for REC (rs = 0.38,
P= 0.008); EMB2 correlated only with REC (rs = 0.27,
P= 0.042) but not with ITE (rs = 0.01, P= 0.5). The full
correlation matrix (including EMB0) is depicted in
Supplementary Fig. 3.
To test whether these differences between REC and ITE
were consistent, i.e. to test if EMB2 was more correlated
Figure 3 Lesion-behaviour analyses. (A) The area covered.
Left: Coloured areas show a lesion in at least one patient, in the
lighter area at least four lesions overlap representing the area in
which the analysis was performed; right: area of maximal overlap
(n=15) projecting to the insular cortex as is typically seen in stroke
dominated lesion studies. (B) Voxel-wise approach: Uncorrected
(unc.) maps are shown for boundary separation (a0, red) and drift
rate (v0, purple) for the REC, with ITE as nuisance variable. IFG
lesions were associated with lower a0,meaning that participants
collected less information before reaching a decision. On the other
hand, lesions in the MTG and STG were associated with lower drift
rate, meaning that these patients collected information slower. Only
39 voxels in the MTG (blue area circled for illustration purpose)
survived correction for REC v0.(C) Statistical region of interest-
symptom mapping, shows significant correlations between REC v0
and MTG for the AICHA (purple) and the Brodmann atlas (BA21,
blue).
3224 |BRAIN 2019: Page 3224 of 3229 M. J. D. Martins et al.
with REC than with ITE, we ran linear mixed models with
accuracy (%) in the visual task as the dependent variable
and predicted by Task (ITE versus REC), and by the cov-
ariate ‘accuracy’ in grammar comprehension EMB(x) (for
x= 1 and 2), and the two-way interactions EMB(x)Task.
We ran two similar models, one for each grammar embed-
ding depth (EMB1 and EMB2) (Table 1).
For EMB1, the best model (REML = 27.2) included
grammar comprehension only, but no effect of Task, and
no interaction. This means that comprehension of sentences
with one level of embedding predicted equally well both
REC and ITE accuracy, as suggested by the correlation
analyses (a similar result was found for EMB0, as reported
in Supplementary Table 4). On the other hand, the best
model for EMB2 was the full model, including the inter-
action EMB2: Task. This means that the correlation be-
tween EMB2 and visual task accuracy differed between
REC and ITE, with this relationship being stronger for
REC (REC:EMB2, B= 0.004, SD = 0.002, t= 1.9). To in-
vestigate whether this effect could be caused by outliers, we
calculated Cook’s distances (Cook and Weisberg, 1982)
and found all were lower than 1.06. We repeated the analysis
removing the data points with highest Cook’s distances
(threshold of four times the mean) and obtained the same
results (Supplementary Fig. 4 and Supplementary Table 5).
Finally, considering that general cognitive abilities, such
as attention, could potentially account for these differences
between REC and ITE, we inspected the correlation matrix
between our visual tasks and a number of standard cogni-
tive measures, including verbal and spatial working
memory and measure of alertness as a basic function of
attention (see Supplementary Fig. 5 for details). Including
these variables in our model had no influence on the spe-
cific relationship between REC and EMB2 (Supplementary
Table 6).
Together, these results suggest that while grammar com-
prehension correlates with both REC and ITE, as previ-
ously shown in Martins et al, 2014b), for higher-levels of
sentence centre-embedding this correlation becomes specific
for REC and not for ITE.
Discussion
The ability to represent hierarchies with multiple levels of
embedding is an essential component of human cognition.
In language, this capacity has been extensively investigated
with both functional MRI and lesion studies highlighting
the importance of a network comprising an anterior and a
posterior ‘hub’ (in IFG and posterior temporal lobe, re-
spectively) (Friederici, 2009,2011;Hagoort and Indefrey,
2014;Matchin et al., 2017). The exact function of these
hubs remains ambiguous, and it is unclear whether they
support mechanisms specific to language or more generally
the processing of hierarchical structures (Matchin et al.,
2017;Matchin and Rogalsky, 2018).
Here, we report the first study investigating the acquisi-
tion of RHE in the visual domain in patients with an
acquired chronic brain lesion in the language network.
We find that, despite the brain lesion, participants were
able to acquire a recursive regularity in a sequence of
four steps. The generation of new hierarchical levels in
visuo-spatial structures was compared to the ability to ac-
quire an iterative rule that sequentially added visual elem-
ents within a fixed hierarchy, without generation of new
levels. The presence of a circumscribed chronic left hemi-
spheric brain lesion in all participants enabled us to per-
form lesion-behaviour analyses probing into whether left
hemispheric neuronal structures support the ability to
infer recursive visual processes, as they do for language.
Participants performed the task rather well with substan-
tial inter-individual variance. With regard to the neuronal
underpinnings, lesions in the left (posterior) middle tem-
poral lobe correlated with lower performance in the detec-
tion of the recursive process. As this applies to the drift
parameter of our analysis, the critical deficits affected in-
formation accumulation and integration during decision-
making. Less robustly, lesions in IFG decreased boundary
separation, in other words, patients with a lesion in this
area tended to acquire less information before deciding
how the hierarchy generating rule continued. The latter
can be conceived as a lower threshold at which participants
are confident to respond correctly. Regarding the issue of
supramodality of RHE, we compared the visual task to a
linguistic task performed by most participants. We found
that the ability to process single embedded sentences corre-
lated equally well with REC and ITE performance.
Conversely, patients performing worse on the comprehen-
sion of double embedded sentences performed worse spe-
cifically in REC.
These data, together with previous literature, suggest that
pTC is important for the formation of RHE representations
across different domains. In the next sections, we will dis-
cuss these findings in the broader context of hierarchical
cognition.
Table 1 LMM Dependent variable: ITE and REC accur-
acy (%)
Df SS F P
Model 1: EMB1
Task 1 0.02 0.6 0.45
EMB1 accuracy (%) 1 0.17 5.4 0.02*
EMB1* Task 1 0.06 2.0 0.15
Model 2: EMB2
Task 1 0.02 0.6 0.45
EMB2 accuracy (%) 1 0.06 1.9 0.17
EMB2* Task 1 0.12 3.9 0.05*
We ran two Linear Mixed Models (LMM), one for EMB1 and another for EMB2, to test
whether the visual tasks differed in how much they are predicted by EMB1 and EMB2.
We found that EMB1 predicted both REC and ITE, with no significant difference between
tasks (top). Conversely, EMB2 predicted bette r REC than ITE (bottom, see main text
for details).
SS = sum of squares.
*P50.05.
Temporal lesions cause recursion deficits BRAIN 2019: Page 3225 of 3229 |3225
The contrast REC–ITE isolates
recursive hierarchical embedding
The essential difference between REC and ITE is that only
for REC a self-similarity between the global structure and
subordinate elements evolves along increasingly complex
images. On the contrary, for ITE, the addition of more
elements follows a simple sequential rule. This parallels
differences in language when multiple embedded propos-
itions are compared to a serial presentation of the same
propositions. [As an example: ‘The mouse the cat the dog
bit chased escaped’ versus ‘the dog bit the cat, the cat
chased the mouse, the mouse escaped’.]
Previous research has demonstrated that performance for
REC is correlated with the ability to represent recursive
embedding in tonal hierarchies and in action planning,
when factoring out overall performance in both tasks
(Martins et al., 2017). This suggests that the cognitive re-
sources used in the acquisition of RHE representations are
shared across domains. This hypothesis would be consistent
with our behavioural results showing that accuracy in REC
correlates (more strongly than ITE) with the ability to ad-
equately parse sentences with two centre-embedded clauses.
Moreover, the areas which correlate with the derived par-
ameters of drift and boundary (Fig. 4 and Table 1)—IFG
and posterior temporal—are considered hubs for the pro-
cessing of complex grammar (Friederici, 2011;Hagoort
and Indefrey, 2014;Zaccarella et al., 2017).
Additional support for the fundamental difference be-
tween ITE and REC comes from the analysis of order ef-
fects in the current experiment. The alternating order of the
two tasks across patients was introduced to counterbalance
mere learning effects; however, we find a strong task-order
effect for REC only. Performance in REC, when performed
prior to ITE, was significantly worse than for the inverse
task order. On the contrary, ITE performance did not
change depending on task-order. These findings replicate
previous results with children (Martins et al., 2014b). The
fact that task-order effects are not present for ITE suggests
that there is a ‘natural sequence’ in the acquisition of RHE
representations in that acquisition of recursion requires pre-
vious acquisition of more simple iterative representations.
This has also been shown for the domain of language in
which children need to acquire conjunctive representations
before they are able to acquire the construction of subor-
dinate clauses (Roeper, 2011). This effect is influenced by
exposure and inductive processes and not only by natural
ontogenetic development (Dewar and Xu, 2010;Perfors
et al., 2011a,b). Our data showing an asymmetric effect
of Order for REC and ITE is additional evidence that REC
Figure 4 Relationship between REC, ITE and grammar task.(A) Example of the linguistic task of a different study in which the majority
of the participants took part. Regarding the tasks reported here (ITE/REC) performance for one aspect of syntactic complexity, namely
embedding, was correlated with results in the visual task. The three syntactic propositions were presented either sequentially (E0, i.e. no
embedding) or with an embedded relative clause (EMB1, one embedding containing two propositions) or with a 2-fold centre-embedded
structure (EMB2). Note that for the three example sentences (out of n=132) the same image would be correct in the successive picture
selection task (see screenshot in Supplementary Fig. 2). The task required selection of the correct picture from a set of four (one correct and
three distractors for each proposition). (B) Scatterplots depicting the relationship between accuracy in the visual tasks (REC and ITE) and in the
grammar tasks EMB1 (left) and EMB2 (right). Correlation coefficients are presented in the text. See also Table 1.
3226 |BRAIN 2019: Page 3226 of 3229 M. J. D. Martins et al.
builds on shared resources with ITE while also adding a
specific representation layer. The current results and previ-
ous research suggest that this layer pertains to the simul-
taneous representation of multiple hierarchical levels.
The temporal cortex supports
recursive hierarchical embedding
representations
Lesions in posterior temporal cortex were associated with
slower integration of information during processing of
RHE. The velocity of information integration during deci-
sion making (drift rate) has been proposed to reflect the
availability of memory representations that guide the accu-
mulation of perceptual information in a top-down manner,
as shown for instance in lexical priming (Voss et al., 2013;
Mulder et al., 2014). As our participants were naı
¨ve to the
visual stimuli and rules, this suggests that the formation of
RHE representations is crucial to process multiple hierarch-
ical levels simultaneously. A central neuronal hub for this
capacity seems to reside in posterior temporal cortex.
Our findings are partially consistent with language re-
search showing an involvement of the posterior temporal
cortex in top-down syntactic prediction and lexical access
(Matchin et al., 2017). A recent study also demonstrated
that the posterior MTG is a common area of activity
during syntactic processing in language and music (Musso
et al., 2015). Overall, different parts of the temporal cortex
have been shown to support processing of RHE in well-
trained participants in different domains: the representation
of tonal hierarchies mostly activates the anterior STG
(Martins, 2017), language the left posterior STG
(Friederici, 2011;Hagoort and Indefrey, 2014;Matchin
et al., 2017), and the visual domain more anterior portions
of the MTG (Martins et al., 2014a).
As the MTG is associated with semantic memory, these
findings invite the speculation that the bottleneck for the
capacity to acquire RHE is less dependent on general multi-
demand fronto-parietal systems (as also shown in Duncan,
2010;Fedorenko et al., 2012,2013), or on specialized
areas instantiating RHE ‘computations’ such as those pro-
posed for language (e.g. BA44; Friederici et al., 2011;
Zaccarella et al., 2017), but rather on the formation of
RHE ‘representations’. In support to this interpretation,
posterior temporal cortex is found to be active in both
semantics and syntax comprehension in children younger
than 7 years of age, while IFG becomes active only at the
age of 10 (Skeide et al., 2014). Thus, while posterior tem-
poral gyrus plays a significant role in the acquisition of
RHE, IFG is active during automatic and expert processing
(Jeon and Friederici, 2013).
Finally, we found IFG to be associated with boundary
separation, which could be more related with controlled
retrieval of existing representations. Interestingly, our re-
sults suggest a division of labour in the processing of
RHE between the anterior and the posterior hubs, which
is somewhat consistent with the language model in which
domain-general cognitive control systems operate with
domain-specific representations (Matchin et al., 2017).
Limitations and perspectives
First, in this study we have included only patients with
lesions in cortical areas in the left hemisphere. Therefore,
we are not able to determine whether the right MTG is
equally important in the acquisition of RHE representa-
tions. In our previous functional MRI results with similar
visual recursive tasks we found bilateral brain activity, but
no hemisphere-specific regions. While we cannot make
claims about the uniqueness of left hemisphere in the pro-
cessing of RHE, we can conclude that the left pTC is cru-
cial to instantiate RHE representations in both vision and
language. Future studies should address potential differ-
ences between hemispheres by our results with performance
in a comparable sample of participants with an acquired
right hemispheric lesion.
Second, while the MTG is important in the acquisition of
RHE representations, we cannot determine whether learn-
ing mechanisms supported by subcortical structures are
involved in this process. It is possible that the episodic
memory (supported by hippocampus) or procedural sys-
tems (supported by basal ganglia) are also fundamental to
build RHE representations. Future research including pa-
tients with lesions or functional impairment in these sys-
tems will be crucial to evaluate these hypotheses.
Finally, the current theory-driven investigation in mildly
affected patients does not serve an apparent clinical goal.
Nonetheless, the demonstration of supramodality of a cog-
nitive process such as RHE supports integrative interdiscip-
linary cognitive rehabilitation to be a promising and exciting
avenue in research (Cahana-amitay and Albert, 2015).
Conclusion
In this study, we hypothesized that the acquisition of RHE rep-
resentations in the visual domain were supported by neural areas
known to be involved in the processing of hierarchical structures
in language (IFG and pTC). We tested a group of patients with
chronic acquired left hemisphere lesions with a set of tasks de-
signed to isolate the ability to acquire RHE in vision. Crucially,
these patients had not been exposed to these tasks prior to this
study. We found that lesions in posterior MTG specifically im-
paired the ability to adequately integrate information about
RHE during task decision making. This area might be funda-
mental for the acquisition of RHE representations in vision and
across domains.
Acknowledgement
We thank Paolo Baldi for his valuable support and
suggestions.
Temporal lesions cause recursion deficits BRAIN 2019: Page 3227 of 3229 |3227
Funding
This study was supported by the Max Planck Society.
Competing interests
The authors report no competing interests.
Supplementary material
Supplementary material is available at Brain online.
References
Ashburner J, Friston KJ. Unified segmentation. Neuroimage 2005; 26:
839–51.
Bates D, Martin M, Bolker B, Walker S. lme4: Linear Mixed-Effects
Models Using Eigen and S4. R Package Version 1.0–6 [Internet].
2014. http://cran.r-project.org/package= lme4.
Bates E, Wilson SM, Saygin AP, Dick F, Sereno MI, Knight RT, et al.
Voxel-based lesion-symptom mapping. Nat Neurosci 2003; 6: 448.
Berwick RC, Chomsky N. Why only us: language and evolution.
Cambridge: The MIT Press; 2015.
Berwick RC, Friederici AD, Chomsky N, Bolhuis JJ. Evolution, brain,
and the nature of language. Trends Cogn Sci 2013; 17: 98.
Brett M, Leff AP, Rorden C, Ashburner J. Spatial normalization of
brain images with focal lesions using cost function masking.
Neuroimage 2001; 14: 486–500.
Cahana-Amitay D, Albert ML. Neuroscience of aphasia recovery: the
concept of neural multifunctionality. Curr Neurol Neurosci Rep
2015; 15: 41.
Cook RD, Weisberg S. Residuals and influence in regression. New
York: Chapman and Hall; 1982.
Dehaene-Lambertz G, Dehaene S, Hertz-Pannier L. Functional neuroi-
maging of speech perception in infants. Science 2002; 298: 2013–15.
Dewar KM, Xu F. Induction, overhypothesis, and the origin of ab-
stract knowledge: evidence from 9-month-old infants. Psychol Sci
2010; 21: 1871–7.
Duncan J. The multiple-demand (MD) system of the primate brain:
mental programs for intelligent behaviour. Trends Cogn Sci 2010;
14: 172–9.
Durstewitz D, Koppe G, Toutounji H. Computational models as stat-
istical tools. Curr Opin Behav Sci 2016; 11: 93–9.
Everaert MBH, Huybregts MAC, Chomsky N, Berwick RC, Bolhuis
JJ. Structures, not strings: linguistics as part of the cognitive sciences.
Trends Cogn Sci 2015; 19: 729–43.
Fedorenko E, Duncan J, Kanwisher N. Report language-selective and
domain-general regions lie side by side within Broca’s area. Curr
Biol 2012; 22: 2059–62.
Fedorenko E, Duncan J, Kanwisher N. Broad domain generality in
focal regions of frontal and parietal cortex. Proc Natl Acad Sci
2013; 110: 16616–21.
Fischmeister FP, Martins MD, Beisteiner R, Fitch WT. Self-similarity
and recursion as default modes in human cognition. Cortex 2017;
97: 183–201.
Fitch WT, Martins MD. Hierarchical processing in music, language,
and action: Lashley revisited. Ann N Y Acad Sci 2014; 1316: 87–
104.
Freedman D, Lane D. A nonstochastic interpretation of reported sig-
nificance levels. J Bus Econ Stat 1983; 1: 292–8.
Friederici AD. Pathways to language: fiber tracts in the human brain.
Trends Cogn Sci 2009; 13: 175–81.
Friederici AD. The brain basis of language processing: from structure
to function. Physiol Rev 2011; 91: 1357–92.
Friederici AD, Bahlmann J, Friedrich R, Makuuchi M. The neural
basis of recursion and complex syntactic hierarchy. Biolinguistics
2011; 5: 87–104.
Gerstein GL, Mandelbrot B. Random walk models for the spike ac-
tivity of a single neuron. Biophys J 1964; 4: 41–68.
Grodzinsky Y, Santi A. The battle for Broca’s region. Trends Cogn Sci
2008; 12: 474–80.
Hagoort P, Indefrey P. The neurobiology of language beyond single
words. Annu Rev Neurosci 2014; 37: 347–62.
Huber W, Poeck K, Willmes K. The Aachen aphasia test. Adv Neurol
1984; 42: 291–303.
Jeon H-A, Friederici AD. Two principles of organization in the pre-
frontal cortex are cognitive hierarchy and degree of automaticity.
Nat Commun 2013; 4: 2041.
Joliot M, Jobard G, Naveau M, Delcroix N, Petit L, Zago L, et al.
AICHA: an atlas of intrinsic connectivity of homotopic areas.
J Neurosci Methods 2015; 254: 46–59.
Kravitz DJ, Saleem KS, Baker CI, Mishkin M. A new neural
framework for visuospatial processing. Nat Rev Neurosci 2011;
12: 217–30.
Martins MD. Distinctive signatures of recursion. Philos Trans R Soc
Lond B Biol Sci 2012; 367: 2055–64.
Martins MD. The cognitive architecture of recursion: behavioral and
fMRI evidence from the visual, musical and motor domains. In:
Proceedings of the 39th Annual Conference of the Cognitive
Science Society. London, 2017.
Martins MJ, Bianco R, Sammler D, Villringer A. Recursion in action:
An fMRI study on the generation of new hierarchical levels in motor
sequences. Hum Brain Mapp 2019; 40: 2623–38.
Martins MD, Fischmeister FP, Puig-Waldmu
¨ller E, Oh J, Geißler A,
Robinson S, et al. Fractal image perception provides novel insights
into hierarchical cognition. Neuroimage 2014a; 96: 300–8.
Martins MD, Gingras B, Puig-Waldmueller E, Fitch WT. Cognitive
representation of “musical fractals”: Processing hierarchy and recur-
sion in the auditory domain. Cognition 2017; 161: 31–45.
Martins MD, Laaha S, Freiberger EMEM, Choi S, Fitch WT. How
children perceive fractals: hierarchical self-similarity and cognitive
development. Cognition 2014b; 133: 10–24.
Martins MD, Martins IP, Fitch WT. A novel approach to investigate
recursion and iteration in visual hierarchical processing. Behav Res
Methods 2016; 48: 1421–42.
Matchin WG. A neuronal retuning hypothesis of sentence-specificity in
Broca’s area. Psychon Bull Rev 2018; 25: 1682–94.
Matchin WG, Hammerly C, Lau E. The role of the IFG and pSTS in
syntactic prediction: evidence from a parametric study of hierarch-
ical structure in fMRI. Cortex 2017; 88: 106–123.
Matchin WG, Rogalsky C. Aphasia and Syntax. In: The Handbook of
Experimental Syntax; 2018. p. 1–54.
Mulder MJ, van Maanen L, Forstmann BU. Perceptual decision neu-
rosciences-a model-based review. Neuroscience 2014; 277: 872–84.
Musso M, Weiller C, Horn A, Glauche V, Umarova R, Hennig J, et al.
A single dual-stream framework for syntactic computations in music
and language. Neuroimage 2015; 117: 267–83.
Perfors A, Tenenbaum JB, Griffiths TL. A tutorial introduction to
Bayesian models of cognitive development. Cognition 2011a; 120:
1–61.
Perfors A, Tenenbaum JB, Regier T. The learnability of abstract syn-
tactic principles. Cognition 2011b; 118: 306–38.
Pinker S, Jackendoff R. The faculty of language: what’s special about
it? Cognition 2005; 95: 201–36.
Ratcliff R, McKoon G. The diffusion decision model: Theory and data
for two-choice decision tasks. Neural Comput 2008; 20: 873–922.
Roeper T. The acquisition of recursion: how formalism articulates the
child’s path. Biolinguistics 2011; 5: 57–86.
Rorden C, Brett M. Stereotaxic display of brain lesions. Behav Neurol
2000; 12: 191–200.
3228 |BRAIN 2019: Page 3228 of 3229 M. J. D. Martins et al.
Rogalsky C, Rong F, Saberi K, Hickok G. Functional anatomy of
language and music perception: temporal and structural factors
investigated using functional magnetic resonance imaging.
J Neurosci 2011; 31: 3843–52.
Rorden C, Karnath HO, Bonilha L. Improving lesion-symptom map-
ping. J Cogn Neurosci 2007; 19: 1081–8.
Russell L. emmeans: Estimated Marginal Means, Aka Least-Squares
Means. R Package Version 1.2.1 [Internet]. 2018. https://cran.r-pro-
ject.org/package= emmeans.
Skeide MA, Brauer J, Friederici AD. Syntax gradually segregates from
semantics in the developing brain. Neuroimage 2014; 100: 106–111.
Smith PL, Ratcliff R. Psychology and neurobiology of simple decisions.
Trends Neurosci 2004; 27: 161–8.
Srinivasan SK, Sampath G. Stochastic models for spike trains of single
neurons (Vol. 16). Berlin: Springer Science & Business Media; 2013.
Venables WN, Ripley BD. Modern Applied Statistics with S [Internet].
2002. http://www.stats.ox.ac.uk/pub/MASS4.
Voss A, Rothermund K, Jena F, Gast A, Wentura D, Saarlands U.
Cognitive processes in associative and categorical priming: a diffu-
sion model analysis. 2013; 142: 536–59.
Wheeler B, Torchiano M. lmPerm: Permutation Tests For Linear
Models [Internet]. 2016. https://github.com/mtorchiano/lmPerm.
Wiecki T V., Sofer I, Frank MJ. HDDM: hierarchical Bayesian esti-
mation of the drift-diffusion model in python. Front Neuroinform
2013; 7: 1–10.
Wilson B, Marslen-Wilson WD, Petkov CI. Conserved sequence
processing in primate frontal cortex. Trends Neurosci 2017; 40:
72–82.
Winkler AM, Ridgway GR, Webster MA, Smith SM, Nichols TE.
Permutation inference for the general linear model. Neuroimage
2014; 92: 381–97.
Zaccarella E, Meyer L, Makuuchi M, Friederici AD. Building by
syntax: the neural basis of minimal linguistic structures. Cereb
Cortex 2017; 27: 411–21.
Temporal lesions cause recursion deficits BRAIN 2019: Page 3229 of 3229 |3229
... The roles of the pMTG in semantic retrieval, verbal memory, and multiple advanced network systems have been well documented in imaging (Davey et al., 2016;Karnath, 2001;Mesgarani, Cheung, Johnson, & Chang, 2014;Xu et al., 2015;Xu et al., 2019) and lesion (Leff et al., 2009;Martins et al., 2019) studies. Verbal learning and semantic retrieval are abstract cognitive functions in humans. ...
Article
Full-text available
Even clinically "asymptomatic" carotid stenosis is associated with multidomain cogni-tive impairment, gray matter (GM) atrophy, and silent lesion. However, the links between them remain unclear. Using structural MRI data, we examined GM asymmetry index (AI) and white matter hyperintensity (WMH) in 24 patients with severe asymptomatic carotid stenosis (SACS), 24 comorbidity-matched controls, and independent samples of 84 elderly controls and 22 young adults. As compared to controls , SACS patients showed worse verbal memories, higher WMH burden, and right-lateralized GM in posterior middle temporal and mouth-somatomotor regions. These clusters extended to pars triangularis, lateral temporal, and cerebellar regions, when compared with young adults. Further, a full-path of WMH burden (X), GM volume (atrophy, M1), AI (asymmetry, M2), and neuropsychological variables (Y) through a serial mediation model was analyzed. This analysis identified that left-dominated GM atrophy and right-lateralized asymmetry in the posterior middle temporal cortex mediated the relationship between WMH burden and recall memory in SACS patients. These results suggest that the unbalanced hemispheric atrophy in the posterior middle temporal cortex is crucial to mediating relationship between WMH burden and verbal recall memories, which may underlie accelerated aging and cognitive deterioration in patients with SACS and other vascular cognitive impairment. K E Y W O R D S cortical organization, gradients, meta-analysis, recall, vascular cognitive impairment, verbal memory
Preprint
Full-text available
In this article, we explore the extraction of recursive nested structure in the processing of binary sequences. Our aim was to determine whether the brain learns the higher order regularities of a highly simplified input where only sequential order information marks the hierarchical structure. To this end, we implemented sequence generated by the Fibonacci grammar in a serial reaction time task. This deterministic grammar generates aperiodic but self-similar sequences. The combination of these two properties allowed us to evaluate hierarchical learning while controlling for the use of low-level strategies like detecting recurring patterns. The deterministic aspect of the grammar allowed us to predict precisely which points in the sequence should be subject to anticipation. Results showed that participants' pattern of anticipation could not be accounted for by "flat" statistical learning processes and was consistent with them anticipating upcoming points based on hierarchical assumptions. We also found that participants were sensitive to the structure constituency, suggesting that they organized the signal into embedded constituents. We hypothesized that the participants built this structure by merging recursively deterministic transitions.
Cover Page
Full-text available
Cover image: Structural and functional connectivity profiles seeded from volume of tissue activated predict deep brain stimulation-induced improvement in patients with essential tremor. From Al-Fatly et al. Connectivity profile of thalamic deep brain stimulation to effectively treat essential tremor. Pp. 3086–3098.
Article
Full-text available
In many domains of human cognition, hierarchically structured representations are thought to play a key role. In this paper, we start with some foundational definitions of key phenomena like “sequence” and “hierarchy," and then outline potential signatures of hierarchical structure that can be observed in behavioral and neuroimaging data. Appropriate behavioral methods include classic ones from psycholinguistics along with some from the more recent artificial grammar learning and sentence processing literature. We then turn to neuroimaging evidence for hierarchical structure with a focus on the functional MRI literature. We conclude that, although a broad consensus exists about a role for a neural circuit incorporating the inferior frontal gyrus, the superior temporal sulcus, and the arcuate fasciculus, considerable uncertainty remains about the precise computational function(s) of this circuitry. An explicit theoretical framework, combined with an empirical approach focusing on distinguishing between plausible alternative hypotheses, will be necessary for further progress.
Article
Full-text available
Generation of hierarchical structures, such as the embedding of subordinate elements into larger structures, is a core feature of human cognition. Processing of hierarchies is thought to rely on lateral prefrontal cortex (PFC). However, the neural underpinnings supporting active generation of new hierarchical levels remain poorly understood. Here, we created a new motor paradigm to isolate this active generative process by means of fMRI. Participants planned and executed identical movement sequences by using different rules: a Recursive hierarchical embedding rule, generating new hierarchical levels; an Iterative rule linearly adding items to existing hierarchical levels, without generating new levels; and a Repetition condition tapping into short term memory, without a transformation rule. We found that planning involving generation of new hierarchical levels (Recursive condition vs. both Iterative and Repetition) activated a bilateral motor imagery network, including cortical and subcortical structures. No evidence was found for lateral PFC involvement in the generation of new hierarchical levels. Activity in basal ganglia persisted through execution of the motor sequences in the contrast Recursive versus Iteration, but also Repetition versus Iteration, suggesting a role of these structures in motor short term memory. These results showed that the motor network is involved in the generation of new hierarchical levels during motor sequence planning, while lateral PFC activity was neither robust nor specific. We hypothesize that lateral PFC might be important to parse hierarchical sequences in a multi‐domain fashion but not to generate new hierarchical levels.
Article
Full-text available
The human ability to process hierarchical structures has been a longstanding research topic. However, the nature of the cognitive machinery underlying this faculty remains controversial. Recursion, the ability to embed structures within structures of the same kind, has been proposed as a key component of our ability to parse and generate complex hierarchies. Here, we investigated the cognitive representation of both recursive and iterative processes in the auditory domain. The experiment used a two-alternative forced-choice paradigm: participants were exposed to three-step processes in which pure-tone sequences were built either through recursive or iterative processes, and had to choose the correct completion. Foils were constructed according to generative processes that did not match the previous steps. Both musicians and non-musicians were able to represent recursion in the auditory domain, although musicians performed better. We also observed that general ‘musical’ aptitudes played a role in both recursion and iteration, although the influence of musical training was somehow independent from melodic memory. Moreover, unlike iteration, recursion in audition was well correlated with its non-auditory (recursive) analogues in the visual and action sequencing domains. These results suggest that the cognitive machinery involved in establishing recursive representations is domain-general, even though this machinery requires access to information resulting from domain-specific processes.
Article
Full-text available
An important aspect of animal perception and cognition is learning to recognize relationships between environmental events that predict others in time, a form of relational knowledge that can be assessed using sequence-learning paradigms. Humans are exquisitely sensitive to sequencing relationships, and their combinatorial capacities, most saliently in the domain of language, are unparalleled. Recent comparative research in human and nonhuman primates has obtained behavioral and neuroimaging evidence for evolutionarily conserved substrates involved in sequence processing. The findings carry implications for the origins of domain-general capacities underlying core language functions in humans. Here, we synthesize this research into a 'ventrodorsal gradient' model, where frontal cortex engagement along this axis depends on sequencing complexity, mapping onto the sequencing capacities of different species.
Article
1 Some basic neurophysiology 4 The neuron 1. 1 4 1. 1. 1 The axon 7 1. 1. 2 The synapse 9 12 1. 1. 3 The soma 1. 1. 4 The dendrites 13 13 1. 2 Types of neurons 2 Signals in the nervous system 14 2. 1 Action potentials as point events - point processes in the nervous system 15 18 2. 2 Spontaneous activi~ in neurons 3 Stochastic modelling of single neuron spike trains 19 3. 1 Characteristics of a neuron spike train 19 3. 2 The mathematical neuron 23 4 Superposition models 26 4. 1 superposition of renewal processes 26 4. 2 Superposition of stationary point processe- limiting behaviour 34 4. 2. 1 Palm functions 35 4. 2. 2 Asymptotic behaviour of n stationary point processes superposed 36 4. 3 Superposition models of neuron spike trains 37 4. 3. 1 Model 4. 1 39 4. 3. 2 Model 4. 2 - A superposition model with 40 two input channels 40 4. 3. 3 Model 4. 3 4. 4 Discussion 41 43 5 Deletion models 5. 1 Deletion models with 1nd~endent interaction of excitatory and inhibitory sequences 44 VI 5. 1. 1 Model 5. 1 The basic deletion model 45 5. 1. 2 Higher-order properties of the sequence of r-events 55 5. 1. 3 Extended version of Model 5. 1 - Model 60 5. 2 5. 2 Models with dependent interaction of excitatory and inhibitory sequences - MOdels 5. 3 and 5.
Article
It is clear that the left inferior frontal gyrus (LIFG) contributes in some fashion to sentence processing. While neuroimaging and neuropsychological evidence support a domain-general working memory function, recent neuroimaging data show that particular subregions of the LIFG, particularly the pars triangularis (pTri), show selective activation for sentences relative to verbal working memory and cognitive control tasks. These data suggest a language-specific function rather than a domain-general one. To resolve this apparent conflict, I propose separating claims of domain-generality and specificity independently for computations and representations—a given brain region may respond to a specific representation while performing a general computation over that representation, one shared with other systems. I hypothesize that the pTri underlies a language-specific working memory system, comprised of general memory retrieval/attention operations specialized for syntactic representations. There is a parallelism of top-down retrieval function among the phonological and semantic levels, localized to the pars opercularis and pars orbitalis, respectively. I further explore the idea of how such a system emerges in the human brain through the framework of neuronal retuning: the “borrowing” of domain-general mechanisms for language, either in evolution or development. The empirical data appear to tentatively support a developmental account of language-specificity in the pTri, possibly through connections to the posterior superior temporal sulcus (pSTS), a region that is both anatomically distinct for humans and functionally essential for language. Evidence of representational response specificity obtained from neuroimaging studies is useful in understanding how cognition is implemented in the brain. However, understanding the shared computations across domains and neural systems is necessary for a fuller understanding of this problem, providing potential answers to questions of how specialized systems, such as language, are implemented in the brain.
Book
A guide to using S environments to perform statistical analyses providing both an introduction to the use of S and a course in modern statistical methods. The emphasis is on presenting practical problems and full analyses of real data sets.