No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Online decoding of object-based attention using real-time fMRI
European Journal of Neuroscience
Abstract
Visual attention is used to selectively filter relevant information depending on current task demands and goals. Visual attention is called object-based attention when it is directed to coherent forms or objects in the visual field. This study used real-time functional magnetic resonance imaging for moment-to-moment decoding of attention to spatially overlapped objects belonging to two different object categories. First, a whole-brain classifier was trained on pictures of faces and places. Subjects then saw transparently overlapped pictures of a face and a place, and attended to only one of them while ignoring the other. The category of the attended object, face or place, was decoded on a scan-by-scan basis using the previously trained decoder. The decoder performed at 77.6% accuracy indicating that despite competing bottom-up sensory input, object-based visual attention biased neural patterns towards that of the attended object. Furthermore, a comparison between different classification approaches indicated that the representation of faces and places is distributed rather than focal. This implies that real-time decoding of object-based attention requires a multivariate decoding approach that can detect these distributed patterns of cortical activity.
... During the subsequent associative learning task (see Fig. 6), participants had to memorize pairs of images, consisting of the same objects as used in the neurofeedback blocks but now explicitly associated with a novel face or house. Each object was either associated with the same or the other category compared to the neurofeedback 28 . The data was exported in real-time into a FieldTrip buffer, and then immediately preprocessed. ...
... Thus, crucially, without explicitly notifying them when to think about what category. This is in line with findings from earlier studies in which it has been demonstrated that participants can learn to voluntarily control their across-voxel neural patterns by neurofeedback 7,[17][18][19]28 . Importantly, our results reveal the possibility of solely using neurofeedback to train participants to associate (more complex) stimulus categories in higher order regions to objects, which extends the neurofeedback study from Amano et al. 19 in which an association between a color and a grating orientation was created in the early visual cortex by training participants to modulate across-voxel neural patterns in their early visual cortex to represent a specific color while presenting a specific orientation on the screen. ...
... BrainStream software (see www. brain stream. nu) as was used before, see Ref. 28 , which is a Matlab-based software package developed at the Donders Centre for Cognition (Nijmegen, Netherlands). The toolbox builds on Psychtoolbox combined with an extension (StimBox) for adaptive stimulus presentation, FieldTrip toolboxes for raw and preprocessed data buffers, FSL and SPM8 for MR data analyses, a GUI streamer to access and export the raw MR volumes during acquisition, and the Donders Machine Learning Toolbox for online decoding. ...
Memory, one of the hallmarks of human cognition, can be modified when humans voluntarily modulate neural population activity using neurofeedback. However, it is currently unknown whether neurofeedback can influence the integration of memories, and whether memory is facilitated or impaired after such neural perturbation. In this study, participants memorized objects while we provided them with abstract neurofeedback based on their brain activity patterns in the ventral visual stream. This neurofeedback created an implicit face or house context in the brain while memorizing the objects. The results revealed that participants created associations between each memorized object and its implicit context solely due to the neurofeedback manipulation. Our findings shed light onto how memory formation can be influenced by synthetic memory tags with neurofeedback and advance our understanding of mnemonic processing.
... Previous psychophysical experiments have shown that attention to visual stimuli can alter the appearance of stimuli, in which the perceived contrast of attended stimuli is enhanced [6][7][8] . Neuroscience studies have revealed that attention to specific visual features induces modulations of brain activity associated with the attended feature representations [11][12][13][14][15][16][17][18][19][20][21] , which consequently enables decoding of attended information from brain activity patterns [14][15][16][17][18] . Although the link between the attentional modulation of perceptual contrast and that of neural activity was investigated using computational models 22 , the study focused on single-feature stimuli and simple tasks based on single-neuron responses. ...
... Previous psychophysical experiments have shown that attention to visual stimuli can alter the appearance of stimuli, in which the perceived contrast of attended stimuli is enhanced [6][7][8] . Neuroscience studies have revealed that attention to specific visual features induces modulations of brain activity associated with the attended feature representations [11][12][13][14][15][16][17][18][19][20][21] , which consequently enables decoding of attended information from brain activity patterns [14][15][16][17][18] . Although the link between the attentional modulation of perceptual contrast and that of neural activity was investigated using computational models 22 , the study focused on single-feature stimuli and simple tasks based on single-neuron responses. ...
Stimulus images can be reconstructed from visual cortical activity. However, our perception of stimuli is shaped by both stimulus-induced and top-down processes, and it is unclear whether and how reconstructions reflect top-down aspects of perception. Here, we investigate the effect of attention on reconstructions using fMRI activity measured while subjects attend to one of two superimposed images. A state-of-the-art method is used for image reconstruction, in which brain activity is translated (decoded) to deep neural network (DNN) features of hierarchical layers then to an image. Reconstructions resemble the attended rather than unattended images. They can be modeled by superimposed images with biased contrasts, comparable to the appearance during attention. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses, modulating neural representations to render reconstructions in accordance with subjective appearance.
... attentional shifts. Of note, previous rt-fMRI based decoding of categorybased attention only used whole brain classifiers (Niazi et al., 2014). We also added a temporal element to help further distinguish haemodynamic responses produced by deploying attention to quadrant-specific streams of stimuli. ...
... In contrast to the use of large areas of brain to extract signal for a BCI, using smaller cortical areas engaged in cognitive control processes (Hauschild et al., 2012) may enable a higher signal-to-noise ratio by reducing the incidence of unrelated brain activations. A alternative approach would be to use pattern recognition techniques to improve information extraction i.e. whole brain classifiers (Niazi et al., 2014). However in the ultimate translation to a surgically implanted BCI (see Fig. 8), using a smaller region of brain facilitates use of a smaller prosthesis, minimising surgical exposure, reducing operative time, surgical risk and inpatient stay. ...
Brain-computer-interfaces (BCI) provide a means of using human brain activations to control devices for communication. Until now this has only been demonstrated in primary motor and sensory brain regions, using surgical implants or non-invasive neuroimaging techniques. Here, we provide proof-of-principle for the use of higher-order brain regions involved in complex cognitive processes such as attention. Using realtime fMRI, we implemented an online 'winner-takes-all approach' with quadrant-specific parameter estimates, to achieve single-block classification of brain activations. These were linked to the covert allocation of attention to real-world images presented at 4-quadrant locations. Accuracies in three target regions were significantly above chance, with individual decoding accuracies reaching upto 70%. By utilising higher order mental processes, 'cognitive BCIs' access varied and therefore more versatile information, potentially providing a platform for communication in patients who are unable to speak or move due to brain injury.
... Attention enhances the features of the attended object, such as its motion, color, or shape, even when attended and unattended objects are superimposed [11], [12]. Early studies using functional magnetic resonance imaging (fMRI) have shown that it is possible to decode the category of the attended object in an image with superimposed objects from different categories [8], [13]. In more recent studies such as [14], the categories of unattended objects were also decoded and compared with the decoding accuracy of attended objects, showing that the latter were more accurately decodable. ...
Selective attention enables humans to efficiently process visual stimuli by enhancing important locations or objects and filtering out irrelevant information. Locating visual attention is a fundamental problem in neuroscience with potential applications in brain-computer interfaces. Conventional paradigms often use synthetic stimuli or static images, but visual stimuli in real life contain smooth and highly irregular dynamics. In this study, we show that these irregular dynamics in natural videos can be decoded from electroencephalography (EEG) signals to perform selective visual attention decoding. To this end, we propose an experimental paradigm in which participants attend to one of two superimposed videos, each showing a center-aligned person performing a stage act. We then train a stimulus-informed decoder to extract EEG signal components that are correlated with the motion patterns of the attended object, and show that this decoder can be used on unseen data to detect which of both objects is attended. Eye movements are also found to be correlated to the motion patterns in the attended video, despite the spatial overlap between the target and the distractor. We further show that these eye movements do not dominantly drive the EEG-based decoding and that complementary information exists in EEG and gaze data. Moreover, our results indicate that EEG also captures information about unattended objects. To our knowledge, this study is the first to explore EEG-based selective visual attention decoding on natural videos, opening new possibilities for experiment design in related fields.
... In summary, brain connectivity networks may be sensitive for predicting closedloop training effects, such as neural guided training, since closed-loop learning, in which online feedback of neural activation is provided to the participant for self-regulation, tends to affect behaviourally relevant functional network reorganization. [52][53][54][55][56] Besides, this study also indicated that predictive biomarkers for one intervention may not applicable for another type of intervention. ...
Predicting whether a chronic stroke patient is likely to benefit from a specific intervention can help patients establish reasonable expectations. It also provides the basis for candidates selecting for the intervention. Recent convergent evidence supports the value of network-based approach for understanding the relationship between dysfunctional neural activity and motor deficits after stroke. In this study, we applied resting-state brain connectivity networks to investigate intervention-specific predictive biomarkers of motor improvement in 22 chronic stroke participants who received either combined action observation with EEG-guided robot-hand training (Neural Guided-Action Observation Group, n = 12, age: 34–68 years) or robot-hand training without action observation and EEG guidance (non-Neural Guided-text group, n = 10, age: 42–57 years). The robot hand in Neural Guided-Action Observation training was activated only when significant mu suppression (8–12 Hz) was detected from participant's EEG signals in ipsilesional hemisphere while it was randomly activated in non-Neural Guided-text training. Only the Neural Guided-Action Observation group showed a significant long-term improvement in their upper-limb motor functions (p < 0.5). In contrast, no significant training effect on the paretic motor functions was found in the non-Neural Guided-text group (p > 0.5). The results of brain connectivity estimated via EEG coherence showed that the pre-training interhemispheric connectivity of delta, theta, alpha and contralesional connectivity of beta were motor improvement related in the Neural Guided-Action Observation group. They can not only differentiate participants with good and poor recovery (interhemispheric delta: p = 0.047, Hedges' g = 1.409; interhemispheric theta: p = 0.046, Hedges' g = 1.333; interhemispheric alpha: p = 0.038, Hedges' g = 1.536; contralesional beta: p = 0.027, Hedges' g = 1.613) but also significantly correlated with post-training intervention gains (interhemispheric delta: r = -0.901, p < 0.05; interhemispheric theta: r = -0.702, p < 0.05; interhemispheric alpha: r = -0.641, p < 0.05; contralesional beta: r = -0.729, p < 0.05). In contrast, no EEG coherence was significantly correlated with intervention gains in the non-Neural Guided-text group (all ps > 0.05). Partial least square regression showed that the combination of pre-training interhemispheric and contralesional local connectivity could precisely predict intervention gains in the Neural Guided-Action Observation group with a strong correlation between predicted and observed intervention gains (r = 0.82) and between predicted and observed intervention outcomes (r = 0.90). In summary, EEG-based resting-state brain connectivity networks may serve clinical decision-making by offering an approach to predicting Neural Guided-Action Observation training-induced motor improvement.
... While earlier BCI studies have mostly focused on modulating specific brain regions or signals, new developments in pattern classification of brain states are enabling real-time decoding and modulation of an entire functional network (Cox and Savoy, 2003;Peltier et al., 2009;Hollmann et al., 2011;Laconte, 2011;Sitaram et al., 2011;Rana et al., 2013;Sato et al., 2013;Niazi et al., 2014;Ruiz et al., 2014b). However, a major methodological concern arises from these approaches: the prior studies have focused on building pattern classifiers to decode subject-specific brain patterns, and it is not clear if a general approach could be developed such that a classifier trained on brain signals from a group of individuals could be used to distinguish between any two given specified brain states. ...
While earlier Brain-Computer Interface (BCI) studies have mostly focused on modulating specific brain regions or signals, new developments in pattern classification of brain states are enabling real-time decoding and modulation of an entire functional network. The present study proposes a new method for real-time pattern classification and neurofeedback of brain states from electroencephalographic (EEG) signals. It involves the creation of a fused classification model based on the method of Common Spatial Patterns (CSPs) from data of several healthy individuals. The subject-independent model is then used to classify EEG data in real-time and provide feedback to new individuals. In a series of offline experiments involving training and testing of the classifier with individual data from 27 healthy subjects, a mean classification accuracy of 75.30% was achieved, demonstrating that the classification system at hand can reliably decode two types of imagery used in our experiments, i.e., happy emotional imagery and motor imagery. In a subsequent experiment it is shown that the classifier can be used to provide neurofeedback to new subjects, and that these subjects learn to “match” their brain pattern to that of the fused classification model in a few days of neurofeedback training. This finding can have important implications for future studies on neurofeedback and its clinical applications on neuropsychiatric disorders.
... Decoding also affords the reconstruction of percepts that are not a direct consequence of external bottom-up input but rather of top-down internal drive. One may think here of representations induced by imagery (Horikawa & Kamitani, 2016;Naselaris, Olman, Stansbury, Ugurbil, & Gallant, 2015;Parkkonen, Andersson, Hämäläinen, & Hari, 2008;Stokes, Thompson, Cusack, & Duncan, 2010;Thirion et al., 2006;Vetter, Smith, & Muckli, 2014), selective attention (Kamitani & Tong, 2005;Niazi et al., 2014), perceptual filling-in (Chong, Familiar, & Shim, 2015), visual illusions (Corney & Lotto, 2007;Kok & de Lange, 2014), memory processing (Harrison & Tong, 2009;Polanía, Paulus, & Nitsche, 2012;Serences, Ester, Vogel, & Awh, 2009) and dreaming (Horikawa, Tamaki, Miyawaki, & Kamitani, 2013). Decoded representations may also be directly related to overt behavior (Haynes et al., 2007;Williams, Dang, & Kanwisher, 2007). ...
... For example, the amplitude of event-related potentials (ERPs) in electroencephalography data decreases when an auditory stream is unattended (Hillyard et al., 1973;Näätänen et al., 1992;Woldorff & Hillyard, 1991), and brain activity measured with functional magnetic resonance imaging (fMRI) increases in regions corresponding to the attended property (Degerman et al., 2006;Downar et al., 2001;Johnson & Zatorre, 2006;Paltoglou et al., 2009). In the visual domain, this biases neuronal population activity as measured by fMRI such that specifically the attended property can be decoded from an ambiguous stimulus (Jehee et al., 2011;Kamitani & Tong, 2005Niazi et al., 2014). Finally, neuronal firing rates of monkey single-unit recordings increase with selective spatial attention (Benson & Hienz, 1978). ...
Natural stimuli consist of multiple properties. However, not all of these properties are equally relevant in a given situation. In this study, we applied multivariate classification algorithms to intracranial electroencephalography data of human epilepsy patients performing an auditory Stroop task. This allowed us to identify neuronal representations of task-relevant and irrelevant pitch and semantic information of spoken words in a subset of patients. When properties were relevant, representations could be detected after about 350 ms after stimulus onset. When irrelevant, the association with gamma power differed for these properties. Patients with more reliable representations of irrelevant pitch showed increased gamma band activity (35–64 Hz), suggesting that attentional resources allow an increase in gamma power in some but not all patients. This effect was not observed for irrelevant semantics, possibly because the more automatic processing of this property allowed for less variation in free resources. Processing of different properties of the same stimulus seems therefore to be dependent on the characteristics of the property.
... While earlier BCI studies have mostly focused on modulating specific brain regions or signals, new developments in pattern classification of brain states are enabling real-time decoding and modulation of an entire functional network (Cox and Savoy, 2003;Peltier et al., 2009;Hollmann et al., 2011;Laconte, 2011;Sitaram et al., 2011;Rana et al., 2013;Sato et al., 2013;Niazi et al., 2014;Ruiz et al., 2014b). However, a major methodological concern arises from these approaches: the prior studies have focused on building pattern classifiers to decode subject-specific brain patterns, and it is not clear if a general approach could be developed such that a classifier trained on brain signals from a group of individuals could be used to distinguish between any two given specified brain states. ...
While earlier Brain-Computer Interface (BCI) studies have mostly focused on modulating specific brain regions or signals, new developments in pattern classification of brain states are enabling real-time decoding and modulation of an entire functional network. The present study proposes a new method for real-time pattern classification and neurofeedback of brain states from electroencephalographic (EEG) signals. It involves the creation of a fused classification model based on the method of Common Spatial Patterns (CSPs) from data of several healthy individuals. The subject-independent model is then used to classify EEG data in real-time and provide feedback to new individuals. In a series of offline experiments involving training and testing of the classifier with individual data from 27 healthy subjects, a mean classification accuracy of 75.30% was achieved, demonstrating that the classification system at hand can reliably decode two types of imagery used in our experiments, i.e., happy emotional imagery and motor imagery. In a subsequent experiment it is shown that the classifier can be used to provide neurofeedback to new subjects, and that these subjects learn to " match " their brain pattern to that of the fused classification model in a few days of neurofeedback training. This finding can have important implications for future studies on neurofeedback and its clinical applications on neuropsychiatric disorders.
... While earlier BCI studies have mostly focused on modulating specific brain regions or signals, new developments in pattern classification of brain states are enabling real-time decoding and modulation of an entire functional network (Cox and Savoy, 2003;Peltier et al., 2009;Hollmann et al., 2011;Laconte, 2011;Sitaram et al., 2011;Rana et al., 2013;Sato et al., 2013;Niazi et al., 2014;Ruiz et al., 2014b). However, a major methodological concern arises from these approaches: the prior studies have focused on building pattern classifiers to decode subject-specific brain patterns, and it is not clear if a general approach could be developed such that a classifier trained on brain signals from a group of individuals could be used to distinguish between any two given specified brain states. ...
... Thus, iSLR can rectify SLR's overpruning, as shown in the simulation (Fig. 3). We also confirmed that iSLR can provide prediction accuracy comparable to Elastic Net (84.1 ± 8.6%; Zou and Hastie, 2005;De Martino et al., 2008), which was also developed as a solution for over-pruning and recognized as a candidate for analyzing fMRI data (Niazi et al., 2014;Casanova et al., 2013; see Supplementary Material and Supplementary Fig. 2). Fig. 5A shows the rendered images of the selected voxels for a representative participant. ...
Background:
Recent functional magnetic resonance imaging (fMRI) decoding techniques allow us to predict the contents of sensory and motor events or participants' mental states from multi-voxel patterns of fMRI signals. Sparse logistic regression (SLR) is a useful pattern classification algorithm that has the advantage of being able to automatically select voxels to avoid over-fitting. However, SLR suffers from over-pruning, in which many voxels that are potentially useful for prediction are discarded.
New method:
We propose an ensemble solution for over-pruning, called "Iterative Recycling" (iRec), in which sparse classifiers are trained iteratively by recycling over-pruned voxels.
Results:
Our simulation demonstrates that iRec can effectively rectify over-pruning in SLR and improve its classification accuracy. We also conduct an fMRI experiment in which eight healthy volunteers perform a finger-tapping task with their index or middle fingers. The results indicate that SLR with iRec (iSLR) can predict the finger used more accurately than SLR. Further, iSLR is able to identify a voxel cluster representing the finger movements in the biologically plausible contralateral primary sensory-motor cortices in each participant. We also successfully dissociated the regularly arranged representation for each finger in the cluster.
Conclusion and comparison with other methods:
To the best of our knowledge, ours is the first study to propose a solution for over-pruning with ensemble-learning that is applicable to any sparse algorithm. In addition, from the viewpoint of machine learning, we provide the novel idea of using the sparse classification algorithm to generate accurate divergent base classifiers.
Objective. Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach. We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Main results. Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance. Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.
Functional magnetic resonance imaging (fMRI) has been extensively used to study brain activity. With advances in signal processing, fMRI data can be analyzed in real time and used in a brain–computer interface (BCI). The feasibility and applications of an fMRI-based BCI (fMRI-BCI) have been studied with increasing frequency in the last decade. This chapter describes aspects of the fMRI-BCI technology and provides a thorough overview of the topic. We review the foundations of the fMRI-BCI, including the fundamental physics and physiology behind fMRI, present a qualitative introduction to relevant statistical analysis techniques, and discuss current applications including control of an external device and therapeutic neurofeedback techniques. Innovations in study design and neuroscience research will increase the suitability for fMRI-BCI to be integrated into clinical methods.
Network science and engineering provide a flexible and generalizable tool set to describe and manipulate complex systems characterized by heterogeneous interaction patterns among component parts. While classically applied to social systems, these tools have recently proven to be particularly useful in the study of the brain. In this review, we describe the nascent use of these tools to understand human cognition, and we discuss their utility in informing the meaningful and predictable perturbation of cognition in combination with the emerging capabilities of neurofeedback. To blend these disparate strands of research, we build on emerging conceptualizations of how the brain functions (as a complex network) and how we can develop and target interventions or modulations (as a form of network control). We close with an outline of current frontiers that bridge neurofeedback, connectomics, and network control theory to better understand human cognition.
Neurofeedback is a psychophysiological procedure in which online feedback of neural activation is provided to the participant for the purpose of self-regulation. Learning control over specific neural substrates has been shown to change specific behaviours. As a progenitor of brain–machine interfaces, neurofeedback has provided a novel way to investigate brain function and neuroplasticity. In this Review, we examine the mechanisms underlying neurofeedback, which have started to be uncovered. We also discuss how neurofeedback is being used in novel experimental and clinical paradigms from a multidisciplinary perspective, encompassing neuroscientific, neuroengineering and learning-science viewpoints.
Neural activity was measured in 10 healthy volunteers by functional MRI while they viewed familiar and unfamiliar faces and listened to familiar and unfamiliar voices. The familiar faces and voices were those of people personally known to the subjects; they were not people who are more widely famous in the media. Changes in neural activity associated with stimulus modality irrespective of familiarity were observed in modules previously demonstrated to be activated by faces (fusiform gyrus bilaterally) and voices (superior temporal gyrus bilaterally). Irrespective of stimulus modality, familiarity of faces and voices (relative to unfamiliar faces and voices) was associated with increased neural activity in the posterior cingulate cortex, including the retrosplenial cortex. Our results suggest that recognizing a person involves information flow from modality-specific modules in the temporal cortex to the retrosplenial cortex. The latter area has recently been implicated in episodic memory and emotional salience, and now seems to be a key area involved in assessing the familiarity of a person. We propose that disturbances in the information flow described may underlie neurological and psychiatric disorders of the recognition of familiar faces, voices and persons (prosopagnosia, phonagnosia and Capgras delusion, respectively).
The perception of faces is sometimes regarded as a specialized task involving discrete brain regions. In an attempt to identi$ face-specific cortex, we used functional magnetic resonance imaging (fMRI) to measure activation evoked by faces presented in a continuously changing montage of common objects or in a similar montage of nonobjects. Bilateral regions of the posterior fusiform gyrus were activated by faces viewed among nonobjects, but when viewed among objects, faces activated only a focal right fusiform region. To determine whether this focal activation would occur for another category of familiar stimuli, subjects viewed flowers presented among nonobjects and objects. While flowers among nonobjects evoked bilateral fusiform activation, flowers among objects evoked no activation. These results demonstrate that both faces and flowers activate large and partially overlapping regions of inferior extrastriate cortex. A smaller region, located primarily in the right lateral fusiform gyrus, is activated specifically by faces.
Human inferior temporal cortex contains category-selective visual regions, including the fusiform face area (FFA) and the parahippocampal place area (PPA). These regions are defined by their greater category-average activation to the preferred category (faces and places, respectively) relative to nonpreferred categories. The approach of investigating category-average activation has left unclear to what extent category selectivity holds for individual object images. Here we investigate single-image activation profiles to address (1) whether each image from the preferred category elicits greater activation than any image outside the preferred category (categorical ranking), (2) whether there are activation differences within and outside the preferred category (gradedness), and (3) whether the activation profile falls off continuously across the category boundary or exhibits a discontinuity at the boundary (category step). We used functional magnetic resonance imaging to measure the activation elicited in the FFA and PPA by each of 96 object images from a wide range of categories, including faces and places, but also humans and animals, and natural and manmade objects. Results suggest that responses in FFA and PPA exhibit almost perfect categorical ranking, are graded within and outside the preferred category, and exhibit a category step. The gradedness within the preferred category was more pronounced in FFA; the category step was more pronounced in PPA. These findings support the idea that these regions have category-specific functions, but are also consistent with a distributed object representation emphasizing categories while still distinguishing individual images.
The Support Vector Machine (SVM) methodology is an effective, supervised, machine learning method that gives state-of-the-art performance for brain state classification from functional magnetic resonance brain images (fMRI). Due to the poor scalability of SVM (cubic in the number of training points) and the massive size of fMRI images, a SVM analysis is usually performed after data collection. Recent advances in real-time fMRI applications, such as Brain Computer Interfaces, require a fast and reliable classification method running in synchronization with the image collection. We design an online Kernel SVM (OKSVM) algorithm based on the Sequential Minimization Optimization (SMO) method, that is fast (training on each new image within 1 sec), has memory and time cost that scales linearly with the number of points used, and yields comparable prediction performance to an offline SVM. We analyze the method's performance by testing it on real fMRI data sets, and show that OKSVM performs well at greatly reduced computational cost. Our work provides a feasible online Kernel SVM for real-time fMRI experiments, and can be used to guide for the design of similar online classifiers in fMRI cognitive state classification.
Machine Learning (ML) methods applied to real-time functional MRI (rt-fMRI) data provide the ability to predict and detect online any changes in cognitive states. Applications based on rt-fMRI require appropriate selection of features, preprocessing routines, and models in order to both be practical to implement and deliver interpretable results. In the paper, we evaluate blind MVPA feature se- lection methods against a priori defined spatial IC maps, as well as the role of non-stationarity in real-time prediction. We also compare the performance of of- fline modeling and prediction with online, and the effectiveness of blind machine learning algorithms such as SVM with hybrid Bayesian-SVM models that utilize history of states to predict future occurrences. Collectively, we explore what is inside the “black-box of real-time fMRI, and examine both the advantages and shortcomings when ML methods are applied to predict and interpret cognitive states in the real-time context.
1. We have previously identified face-selective areas in the mid-fusiform and inferior temporal gyri in electrophysiological recordings made from chronically implanted subdural electrodes in epilepsy patients. In this study, functional magnetic resonance imaging (fMRI) was used to study the anatomic extent of face-sensitive brain regions and to assess hemispheric laterality. 2. A time series of 128 gradient echo echoplanar images was acquired while subjects continuously viewed an alternating series of 10 unfamiliar faces followed by 10 equiluminant scrambled faces. Each cycle of this alternating sequence lasted 12 s and each experimental run consisted of 14 cycles. The time series of each voxel was transformed into the frequency domain using Fourier analysis. Activated voxels were defined by significant peaks in their power spectra at the frequency of stimulus alternation and by a 180 degrees phase shift that followed changes in stimulus alternation order. 3. Activated voxels to faces were obtained in the fusiform and inferior temporal gyri in 9 of 12 subjects and were approximately coextensive with previously identified face-selective regions. Nine subjects also showed activation in the left or right middle occipital gyri, or in the superior temporal or lateral occipital sulci. Cortical volumes activated in the left and right hemispheres were not significantly different. Activated voxels to scrambled faces were observed in six subjects at locations mainly in the lingual gyri and collateral sulci, medial to the regions activated by faces. 4. Face stimuli activated portions of the midfusiform and inferior temporal gyri, including adjacent cortex within occipitotemporal sulci.(ABSTRACT TRUNCATED AT 250 WORDS)
Although attention usually enhances perceptual sensitivity, we found that it can also lead to relatively conservative detection biases and lower visibility ratings in discrimination tasks. These results are explained by a model in which attention reduces the trial-by-trial variability of the perceptual signal, and we determined how this model led to the observed behavior. These findings may partially reflect our impression of 'seeing' the whole visual scene despite our limited processing capacity outside of the focus of attention.
Negotiation and trade typically require a mutual interaction while simultaneously resting in uncertainty which decision the partner ultimately will make at the end of the process. Assessing already during the negotiation in which direction one's counterpart tends would provide a tremendous advantage. Recently, neuroimaging techniques combined with multivariate pattern classification of the acquired data have made it possible to discriminate subjective states of mind on the basis of their neuronal activation signature. However, to enable an online-assessment of the participant's mind state both approaches need to be extended to a real-time technique. By combining real-time functional magnetic resonance imaging (fMRI) and online pattern classification techniques, we show that it is possible to predict human behavior during social interaction before the interacting partner communicates a specific decision. Average accuracy reached approximately 70% when we predicted online the decisions of volunteers playing the ultimatum game, a well-known paradigm in economic game theory. Our results demonstrate the successful online analysis of complex emotional and cognitive states using real-time fMRI, which will enable a major breakthrough for social fMRI by providing information about mental states of partners already during the mutual interaction. Interestingly, an additional whole brain classification across subjects confirmed the online results: anterior insula, ventral striatum, and lateral orbitofrontal cortex, known to act in emotional self-regulation and reward processing for adjustment of behavior, appeared to be strong determinants of later overt behavior in the ultimatum game. Using whole brain classification we were also able to discriminate between brain processes related to subjective emotional and motivational states and brain processes related to the evaluation of objective financial incentives.
Considerable information about mental states can be decoded from noninvasive measures of human brain activity. Analyses of brain activity patterns can reveal what a person is seeing, perceiving, attending to, or remembering. Moreover, multidimensional models can be used to investigate how the brain encodes complex visual scenes or abstract semantic information. Such feats of "brain reading" or "mind reading," though impressive, raise important conceptual, methodological, and ethical issues. What does successful decoding reveal about the cognitive functions performed by a brain region? How should brain signals be spatially selected and mathematically combined to ensure that decoding reflects inherent computations of the brain rather than those performed by the decoder? We highlight recent advances and describe how multivoxel pattern analysis can provide a window into mind-brain relationships with unprecedented specificity, when carefully applied. However, as brain-reading technology advances, issues of neuroethics and mental privacy will be important to consider.
Forging new memories for facts and events, holding critical details in mind on a moment-to-moment basis, and retrieving knowledge in the service of current goals all depend on a complex interplay between neural ensembles throughout the brain. Over the past decade, researchers have increasingly utilized powerful analytical tools (e.g., multivoxel pattern analysis) to decode the information represented within distributed functional magnetic resonance imaging activity patterns. In this review, we discuss how these methods can sensitively index neural representations of perceptual and semantic content and how leverage on the engagement of distributed representations provides unique insights into distinct aspects of memory-guided behavior. We emphasize that, in addition to characterizing the contents of memories, analyses of distributed patterns shed light on the processes that influence how information is encoded, maintained, or retrieved, and thus inform memory theory. We conclude by highlighting open questions about memory that can be addressed through distributed pattern analyses.
When sensory input allows for multiple, competing perceptual interpretations, observers' perception can fluctuate over time, which is called bistable perception. Imaging studies in humans have revealed transient responses in a right-lateralized network in the frontal-parietal cortex (rFPC) around the time of perceptual transitions between interpretations, potentially reflecting the neural initiation of transitions. We investigated the role of this activity in male human observers, with specific interest in its relation to the temporal structure of transitions, which can be either instantaneous or prolonged by periods during which observers experience a mix of both perceptual interpretations. Using both bistable apparent motion and binocular rivalry, we show that transition-related rFPC activity is larger for transitions that last longer, suggesting that rFPC remains active as long as a transition lasts. We also replicate earlier findings that rFPC activity during binocular rivalry transitions exceeds activity during yoked transitions that are simulated using video replay. However, we show that this established finding holds only when perceptual transitions are replayed as instantaneous events. When replay, instead, depicts transitions with the actual durations reported during rivalry, yoked transitions and genuine rivalry transitions elicit equal activity. Together, our results are consistent with the view that at least a component of rFPC activation during bistable perception reflects a response to perceptual transitions, both real and yoked, rather than their cause. This component of activity could reflect the change in sensory experience and task demand that occurs during transitions, which fits well with the known role of these areas in attention and decision making.
Recent years have seen huge advancements in the methods available and used in neuroscience employing EEG or MEG. However, the standard approach is to average a large number of trials for experimentally defined conditions in order to reduce intertrial-variability, i.e., treating it as a source of “noise.” Yet it is now more and more accepted that trial-to-trial fluctuations bear functional significance, reflecting fluctuations of “brain states” that predispose perception and action. Such effects are often revealed in a pre-stimulus period, when comparing response variability to an invariant stimulus. However such offline analyses are disadvantageous as they are correlational by drawing conclusions in a post hoc-manner and stimulus presentation is random with respect to the feature of interest. A more direct test is to trigger stimulus presentation when the relevant feature is present. The current paper introduces Constance System for Online EEG (ConSole), a software package capable of analyzing ongoing EEG/MEG in real-time and presenting auditory and visual stimuli via internal routines. Stimulation via external devices (e.g., transcranial magnetic stimulation) or third-party software (e.g., PsyScope X) is possible by sending TTL-triggers. With ConSole it is thus possible to target the stimulation at specific brain states. In contrast to many available applications, ConSole is open-source. Its modular design enhances the power of the software as it can be easily adapted to new challenges and writing new experiments is an easy task. ConSole is already pre-equipped with modules performing standard signal processing steps. The software is also independent from the EEG/MEG system, as long as a driver can be written (currently two EEG systems are supported). Besides a general introduction, we present benchmark data regarding performance and validity of the calculations used, as well as three example applications of ConSole in different settings. ConSole can be downloaded at: http://console-kn.sf.net.
Previous work has suggested that object and place processing are neuroanatomically dissociated in ventral visual areas under conditions of passive viewing. It has also been shown that the hippocampus and parahippocampal gyrus mediate the integration of objects with background scenes in functional imaging studies, but only when encoding or retrieval processes have been directed toward the relevant stimuli. Using functional magnetic resonance adaptation, we demonstrated that object, background scene, and contextual integration of selectively repeated objects and background scenes could be dissociated during the passive viewing of naturalistic pictures involving object-scene pairings. Specifically, bilateral fusiform areas showed adaptation to object repetition, regardless of whether the associated scene was novel or repeated, suggesting sensitivity to object processing. Bilateral parahippocampal regions showed adaptation to background scene repetition, regardless of whether the focal object was novel or repeated, suggesting selectivity for background scene processing. Finally, bilateral parahippocampal regions distinct from those involved in scene processing and the right hippocampus showed adaptation only when the unique pairing of object with background scene was repeated, suggesting that these regions perform binding operations.
Large efforts are currently being made to develop and improve online analysis of brain activity which can be used, e.g., for brain–computer interfacing (BCI). A BCI allows a subject to control a device by willfully changing his/her own brain activity. BCI therefore holds the promise as a tool for aiding the disabled and for augmenting human performance. While technical developments obviously are important, we will here argue that new insight gained from cognitive neuroscience can be used to identify signatures of neural activation which reliably can be modulated by the subject at will. This review will focus mainly on oscillatory activity in the alpha band which is strongly modulated by changes in covert attention. Besides developing BCIs for their traditional purpose, they might also be used as a research tool for cognitive neuroscience. There is currently a strong interest in how brain-state fluctuations impact cognition. These state fluctuations are partly reflected by ongoing oscillatory activity. The functional role of the brain state can be investigated by introducing stimuli in real-time to subjects depending on the actual state of the brain. This principle of brain-state dependent stimulation may also be used as a practical tool for augmenting human behavior. In conclusion, new approaches based on online analysis of ongoing brain activity are currently in rapid development. These approaches are amongst others informed by new insight gained from electroencephalography/magnetoencephalography studies in cognitive neuroscience and hold the promise of providing new ways for investigating the brain at work.
Studies with real-time functional magnetic resonance imaging (fMRI) demonstrate that humans volitionally regulate hemodynamic signals from circumscribed regions of the brain, leading to area-specific behavioral consequences. Methods to better determine the nature of dynamic functional interactions between different brain regions and plasticity due to self-regulation training are still in development.
The authors investigated changes in brain states while training 6 healthy participants to self-regulate insular cortex by real-time fMRI feedback.
The authors used multivariate pattern analysis to observe spatial pattern changes and a multivariate Granger causality model to show changes in temporal interactions in multiple brain areas over the course of 5 repeated scans per subject during positive and negative emotional imagery with feedback about the level of insular activation.
Feedback training leads to more spatially focused recruitment of areas relevant for learning and emotion. Effective connectivity analysis reveals that initial training is associated with an increase in network density; further training "prunes" presumably redundant connections and "strengthens" relevant connections.
The authors demonstrate the application of multivariate methods for assessing cerebral reorganization during the learning of volitional control of local brain activity. The findings provide insight into mechanisms of training-induced learning techniques for rehabilitation. The authors anticipate that future studies, specifically designed with this hypothesis in mind, may be able to construct a universal index of cerebral reorganization during skill learning based on multiple similar criteria across various skilled tasks. These techniques may be able to discern recovery from compensation, dose-response curves related to training, and ways to determine whether rehabilitation training is actively engaging necessary networks.
This paper describes FieldTrip, an open source software package that we developed for the analysis of MEG, EEG, and other electrophysiological data. The software is implemented as a MATLAB toolbox and includes a complete set of consistent and user-friendly high-level functions that allow experimental neuroscientists to analyze experimental data. It includes algorithms for simple and advanced analysis, such as time-frequency analysis using multitapers, source reconstruction using dipoles, distributed sources and beamformers, connectivity analysis, and nonparametric statistical permutation tests at the channel and source level. The implementation as toolbox allows the user to perform elaborate and structured analyses of large data sets using the MATLAB command line and batch scripting. Furthermore, users and developers can easily extend the functionality and implement new algorithms. The modular design facilitates the reuse in other software packages.
Daily life continually confronts us with an exuberance of external, sensory stimuli competing with a rich stream of internal deliberations, plans and ruminations. The brain must select one or more of these for further processing. How this competition is resolved across multiple sensory and cognitive regions is not known; nor is it clear how internal thoughts and attention regulate this competition. Recording from single neurons in patients implanted with intracranial electrodes for clinical reasons, here we demonstrate that humans can regulate the activity of their neurons in the medial temporal lobe (MTL) to alter the outcome of the contest between external images and their internal representation. Subjects looked at a hybrid superposition of two images representing familiar individuals, landmarks, objects or animals and had to enhance one image at the expense of the other, competing one. Simultaneously, the spiking activity of their MTL neurons in different subregions and hemispheres was decoded in real time to control the content of the hybrid. Subjects reliably regulated, often on the first trial, the firing rate of their neurons, increasing the rate of some while simultaneously decreasing the rate of others. They did so by focusing onto one image, which gradually became clearer on the computer screen in front of their eyes, and thereby overriding sensory input. On the basis of the firing of these MTL neurons, the dynamics of the competition between visual images in the subject's mind was visualized on an external display.
We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic regression, and multi- nomial regression problems while the penalties include âÂÂ_1 (the lasso), âÂÂ_2 (ridge regression) and mixtures of the two (the elastic net). The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with sparse features. In comparative timings we find that the new algorithms are considerably faster than competing methods.
In this paper we present the R package deSolve to solve initial value problems (IVP) written as ordinary differential equations (ODE), differential algebraic equations (DAE) of index 0 or 1 and partial differential equations (PDE), the latter solved using the method of lines approach. The differential equations can be represented in R code or as compiled code. In the latter case, R is used as a tool to trigger the integration and post-process the results, which facilitates model development and application, whilst the compiled code significantly increases simulation speed. The methods implemented are efficient, robust, and well documented public-domain Fortran routines. They include four integrators from the ODEPACK package (LSODE, LSODES, LSODA, LSODAR), DVODE and DASPK2.0. In addition, a suite of Runge-Kutta integrators and special-purpose solvers to efficiently integrate 1-, 2- and 3-dimensional partial differential equations are available. The routines solve both stiff and non-stiff systems, and include many options, e.g., to deal in an efficient way with the sparsity of the Jacobian matrix, or finding the root of equations. In this article, our objectives are threefold: (1) to demonstrate the potential of using R for dynamic modeling, (2) to highlight typical uses of the different methods implemented and (3) to compare the performance of models specified in R code and in compiled code for a number of test cases. These comparisons demonstrate that, if the use of loops is avoided, R code can efficiently integrate problems comprising several thousands of state variables. Nevertheless, the same problem may be solved from 2 to more than 50 times faster by using compiled code compared to an implementation using only R code. Still, amongst the benefits of R are a more flexible and interactive implementation, better readability of the code, and access to R’s high-level procedures. deSolve is the successor of package odesolve which will be deprecated in the future; it is free software and distributed under the GNU General Public License, as part of the R software project.
How are objects represented in the human visual pathway? This question continues to elude the neuroimaging field due to at least two kinds of problems: first, the relatively low spatial resolution of fMRI and second, the bias inherent in prevailing statistical methods for analyzing the actual diagnosticity of cortical tissue. We collected high-resolution (1mm x 1mm) imaging data of the fusiform face area (FFA) from 4 subjects while they categorized images as 'animal', 'car', 'face', or 'sculpture.' We performed exploratory analysis to determine the nature of the distributions over classes and the similarity structure between classes. The FFA was visualized using nonmetric multidimensional scaling revealing "string-like" sequences of voxels, which appeared in small non-contiguous clusters of categories, intertwined with other categories. Since the feature space appeared highly nonlinear, we trained various statistical classifiers on the class conditional distributions (labelled) and separated the four categories with 100% reliability (over replications) and generalized to out of sample cases with high significance (45% to 51%; p<. 000001, chance=25%). The increased noise inherent in high-resolution neuroimaging data relative to standard resolution resisted any further gains in category performance above 60% (with "FACE" category often having the highest bias per category) even coupled with various feature extraction/selection methods. A sensitivity/diagnosticity analysis for each classifier per voxel showed: (1) reliable (with S.E.<3%) sensitivity present throughout the FFA for all 4 categories, and (2) showed multi-selectivity, that is, many voxels were selective for one category but responded to all 4 categories with some high diagnosticity but at lower intensity. This work further verifies the hypothesis that the FFA is a distributed, object-heterogeneous similarity structure and bolsters the view that the FFA response to "FACE" stimuli in standard resolution may be primarily due to a linear bias, which has resulted from an averaging artifact.
Previous studies of visual search in humans using event-related potentials (ERPs) have revealed an ERP component called ‘N2pc’ (180–280 ms) that reflects the focusing of attention onto potential target items in the search array. The present study was designed to localize the neuroanatomical sources of this component by means of magnetoencephalographic (MEG) recordings, which provide greater spatial precision than ERP recordings. MEG recordings were obtained with an array of 148 magnetometers from six normal adult subjects, one of whom was tested in multiple sessions so that both single-subject and group analyses could be performed. Source localization procedures revealed that the N2pc is composed of two distinct neural responses, an early parietal source (180–200 ms) and a later occipito-temporal source (220–240 ms). These findings are consistent with the proposal that parietal areas are used to initiate a shift of attention within a visual search array and that the focusing of attention is implemented by extrastriate areas of the occipital and inferior temporal cortex.
1. We have previously identified face-selective areas in the mid-fusiform and inferior temporal gyri in electrophysiological recordings made from chronically implanted subdural electrodes in epilepsy patients. In this study, functional magnetic resonance imaging (fMRI) was used to study the anatomic extent of face-sensitive brain regions and to assess hemispheric laterality. 2. A time series of 128 gradient echo echoplanar images was acquired while subjects continuously viewed an alternating series of 10 unfamiliar faces followed by 10 equiluminant scrambled faces. Each cycle of this alternating sequence lasted 12 s and each experimental run consisted of 14 cycles. The time series of each voxel was transformed into the frequency domain using Fourier analysis. Activated voxels were defined by significant peaks in their power spectra at the frequency of stimulus alternation and by a 180 degrees phase shift that followed changes in stimulus alternation order. 3. Activated voxels to faces were obtained in the fusiform and inferior temporal gyri in 9 of 12 subjects and were approximately coextensive with previously identified face-selective regions. Nine subjects also showed activation in the left or right middle occipital gyri, or in the superior temporal or lateral occipital sulci. Cortical volumes activated in the left and right hemispheres were not significantly different. Activated voxels to scrambled faces were observed in six subjects at locations mainly in the lingual gyri and collateral sulci, medial to the regions activated by faces. 4. Face stimuli activated portions of the midfusiform and inferior temporal gyri, including adjacent cortex within occipitotemporal sulci.(ABSTRACT TRUNCATED AT 250 WORDS)
The hippocampus has been proposed as the site of neural representation of large-scale environmental space, based upon the
identification of place cells (neurons with receptive fields for current position in the environment) within the rat hippocampus
and the demonstration that hippocampal lesions impair place learning in therat. The inability to identify place cells within
the monkey hippocampus and the observation that unilateral hippocampal lesions do not selectively impair topographic behavior
in humans suggest that alternate regions may subserve this function in man. To examine the contribution of the hippocampus
and adjacent medial-temporal lobe structures to topographic learning in the human, a ‘virtual’ maze was used as a task environment
during functional magnetic resonance imaging studies. During the learning and recall of topographic information, medial-temporal
activity was confined to the para- hippocampal gyri. This activity accords well with the lesion site known to produce topographical
disorientation in humans. Activity was also observed in cortical areas known to project to the parahippocampus and previously
proposed to contribute to a network subserving spatially guided behavior.
The cerebellum traditionally has been viewed as a neural device dedicated to motor control. Although recent evidence shows
that it is involved in nonmotor operations as well, an important question is whether this involvement is independent of motor
control and motor guidance. Functional magnetic resonance imaging was used to demonstrate that attention and motor performance
independently activate distinct cerebellar regions. These findings support a broader concept of cerebellar function, in which
the cerebellum is involved in diverse cognitive and noncognitive neurobehavioral systems, including the attention and motor
systems, in order to anticipate imminent information acquisition, analysis, or action.
Isolated, ventral brain lesions in humans occasionally produce specific impairments in the ability to use landmarks, particularly buildings, for way-finding. Using functional MRI, we tested the hypothesis that there exists a cortical region specialized for the perception of buildings. Across subjects, a region straddling the right lingual sulcus was identified that possessed the functional correlates predicted for a specialized building area. A series of experiments discounted several alternative explanations for the behavior of this site. These results are discussed in terms of their impact upon our understanding of the functional structure of visual processing, disorders of topographical disorientation, and the influence of environmental conditions upon neural organization.
Competing visual stimuli lead to slower responses to targets. This response competition must be resolved before correct responses are executed. Neuroimaging suggests that response competition monitoring may be subserved by an integrated neural network including the anterior cingulate cortex (ACC). In this study, 1 patient with a parietal lesion (Patient J.S.) and 1 with an ACC lesion (Patient G.M.) were presented with 2 flanker tasks; 1 required verbal identification of color targets, and the other required an opposite response to targets (e.g., see red and say "green"); a control group was also tested. For controls, perceptually incongruent flankers interfered with the ability to inhibit prepotent responses to targets. Patient J.S. performed in a similar manner, even when flankers appeared in the neglected field. Patient G.M. demonstrated reduced interference effects for contralesional flankers. Results are discussed in terms of goal-directed selective attention and response competition monitoring.
Selective attention is needed to prioritize the subset of sensory information that is most relevant to our goals. Unfortunately, selective attention is prone to lapses, even in situations where sustaining focused attention is crucial (e.g., when driving in traffic). We propose that such lapses occur partly because we lack a subjective sense of when we are or are not attending well, and that with an appropriate feedback signal, attention can be trained and enhanced. We report initial steps in the development of a closed-loop real-time fMRI system where we use multivariate pattern analysis to provide neurofeedback and train attention. During an fMRI session, observers were presented with a continuous stream of composite face/scene images; occasional shift cues were presented indicating which category should be attended and responded to. Data were pulled from the scanner and preprocessed in real-time (with motion correction, masking, smoothing, and temporal filtering). Whole-brain data obtained under each category cue were used to train a classifier to predict observers’ attentional state. Despite the fact that stimuli were identical in the attend-face and attend-scene conditions, we obtained highly reliable classification performance in real-time for individual trials. This successful real-time decoding allows us to provide immediate and time-varying feedback to observers regarding how well they are attending to the cued category (e.g., tinting the screen background more green or red to indicate increasingly correct or incorrect attentional focus, respectively). By staircasing behavioral performance to a fixed level of accuracy before providing feedback (adding phase-scrambled noise), we can also examine whether neurofeedback (vs. sham or no feedback) improves accuracy in the behavioral task. In sum, by applying multivariate pattern analysis in real-time to fMRI data, we can provide observers with a sophisticated and timely readout of attentional focus that may prove useful in training attention.
Meeting abstract presented at VSS 2012
Abstract When subjects are explicitly cued to focus attention on a particular location in visual space, targets presented at that location have been shown to elicit enhanced sensory-evoked activity in recordings of event-related brain potentials (ERPs). The present study sought to determine if this type of sensory facilitation also occurs during visual search tasks in which a feature conjunction target must be identified, presumably by means of focal attention, within an array of distractor items. In this experiment, subjects were required to discriminate the shape of a distinctively colored target item within an array containing 15 distractor items, and ERPs were elicited by task-irrelevant probe stimuli that were presented at the location of the target item or at the location of a distractor item on the opposite side of the array. When the delay between search-array onset and probe onset was 250 msec, the sensory-evoked responses in the latency range 75-200 msec were larger for probes presented at the location of the target than for probes presented at the location of the irrelevant distractor. These results indicate that sensory processing is modulated in a spatially restricted manner during visual search, and that focusing attention on a feature conjunction target engages neural systems that are shared with other forms of visual-spatial attention.
Theories of embodied cognition propose that facial expression recognition depends upon processing in modality-specific visual areas and also upon a simulation of the somatovisceral and motor responses associated with the perceived emotion. To test this proposal, we targeted transcranial magnetic stimulation (TMS) at the right occipital face area (rOFA) and right somatosensory cortex while participants discriminated facial expressions. TMS impaired discrimination of facial expressions at both sites but had no effect on a matched facial identity task. In a second experiment, double pulse TMS separated by 40ms was delivered at different times to rOFA and right somatosensory cortex during the expression discrimination task. Accuracy dropped when pulses were delivered at 60–100ms at rOFA and at 100–140ms and 130–170ms at right somatosensory cortex. These sequential impairments at rOFA and right somatosensory cortex provide strong support for embodied accounts of expression recognition and hierarchical models of face processing. The results also demonstrate that non-visual areas contribute to expression processing very soon after stimulus presentation.
Remembering object locations across different views is a fundamental competence for keeping oriented in large-scale space. Here we investigated such ability by comparing encoding and retrieval of locations across viewpoint changes relative to different spatial frames of reference. We acquired functional magnetic resonance images while subjects detected target displacements across consecutive views of a familiar virtual room, reporting changes in the target absolute position in the room (stable environmental frame), changes in its position relative to a set of movable objects (unstable object-based frame), and changes relative to their point of view (control viewer-centered frame). Behavioral costs were higher for the stable environmental frame, and a cortical network including the lingual/parahippocampal gyrus (LPHG) and the retrosplenial complex (RSC) selectively encoded spatial locations relative to this frame. Several regions, including the dorsal fronto-parietal cortex and the LPHG, were modulated by the amount of experienced viewpoint change, but only the RSC was selectively modulated by the amount of viewpoint change relative to the environmental frame, thus showing a special role in coding one's own position and heading in familiar environments.
In functional magnetic resonance imaging (fMRI) head motion can corrupt the signal changes induced by brain activation. This paper describes a novel technique called Prospective Acquisition CorrEction (PACE) for reducing motion-induced effects on magnetization history. Full three-dimensional rigid body estimation of head movement is obtained by image-based motion detection to a high level of accuracy. Adjustment of slice position and orientation, as well as regridding of residual volume to volume motion, is performed in real-time during data acquisition. Phantom experiments demonstrate a high level of consistency (translation < 40μm; rotation < 0.05°) for detected motion parameters. In vivo experiments were carried out and they showed a significant decrease of variance between successively acquired datasets compared to retrospective correction algorithms. Magn Reson Med 44:457–465, 2000. © 2000 Wiley-Liss, Inc.
The fusiform face area (FFA) is described as an easily identifiable module on the fusiform gyrus. However, the organization of face-selective regions in ventral temporal cortex (VTC) is more complex than this prevailing view. We highlight methodological factors contributing to these complexities and the extensive variability in how the FFA is identified. We suggest a series of constraints to aid researchers when defining any functionally specialized region with a pleasing realization: anatomy matters.
Reading systematically activates the left lateral occipitotemporal sulcus, at a site known as the visual word form area (VWFA). This site is reproducible across individuals/scripts, attuned to reading-specific processes, and partially selective for written strings relative to other categories such as line drawings. Lesions affecting the VWFA cause pure alexia, a selective deficit in word recognition. These findings must be reconciled with the fact that human genome evolution cannot have been influenced by such a recent and culturally variable activity as reading. Capitalizing on recent functional magnetic resonance imaging experiments, we provide strong corroborating evidence for the hypothesis that reading acquisition partially recycles a cortical territory evolved for object and face recognition, the prior properties of which influenced the form of writing systems.
Does the "fusiform face area" (FFA) code only for faces? This question continues to elude the neuroimaging field due to at least two kinds of problems: first, the relatively low spatial resolution of fMRI in which the FFA was defined and second, the potential bias inherent in prevailing statistical methods for analyzing the actual diagnosticity of cortical tissue. Using high-resolution (1 mm × 1 mm × 1 mm) imaging data of the fusiform face area (FFA) from 4 subjects who had categorized images as 'animal', 'car', 'face', or 'sculpture', we used multivariate linear and non-linear classifiers to decode the resultant voxel patterns. Prior to identifying the appropriate classifier we performed exploratory analysis to determine the nature of the distributions over classes and the voxel intensity pattern structure between classes. The FFA was visualized using non-metric multidimensional scaling revealing "string-like" sequences of voxels, which appeared in small non-contiguous clusters of categories, intertwined with other categories. Since this analysis suggested that feature space was highly non-linear, we trained various statistical classifiers on the class-conditional distributions (labelled) and separated the four categories with 100% reliability (over replications) and generalized to out-of-sample cases with high significance (up to 50%; p<.000001, chance=25%). The increased noise inherent in high-resolution neuroimaging data relative to standard resolution resisted any further gains in category performance above ~60% (with FACE category often having the highest bias per category) even coupled with various feature extraction/selection methods. A sensitivity/diagnosticity analysis for each classifier per voxel showed: (1) reliable (with S.E.<3%) sensitivity present throughout the FFA for all 4 categories, and (2) showed multi-selectivity; that is, many voxels were selective for more than one category with some high diagnosticity but at submaximal intensity. This work is clearly consistent with the characterization of the FFA as a distributed, object-heterogeneous similarity structure and bolsters the view that the FFA response to "FACE" stimuli in standard resolution may be primarily due to a linear bias, which has resulted from an averaging artefact.
We investigated the contribution of frontal and parietal cortices to bottom-up and top-down visual attention using electrophysiological measures in humans. Stimuli consisted of triangles, each with a different color and orientation. Subjects were presented with a sample triangle which served as the target for that trial. An array was subsequently presented with the target and three additional distractor stimuli, which were constructed to induce either automatic "pop-out" (50%) or effortful "search" (50%) behavior. For pop-out, both the color and orientation of the distractors differed from the target, which attracted attention automatically. For search, only the orientation of the distractors differed from the target, so effortful attention was required. Pop-out target detection generated a P300 event-related potential (ERP) with a peak amplitude over parietal sites whereas the search condition generated a fronto-centrally distributed P300. Reaction times and associated P300 latency in frontal areas were shorter for pop-out targets than for search targets. We used time-frequency analysis to compare pop-out and search conditions, within a 200-650 ms time-window and a 4-55 Hz frequency band. There was a double dissociation, with significantly increased power from 4 to 24 Hz in parietal areas for pop-out targets and increased power from 4 to 24 Hz in frontal regions for search targets. Taken together the ERP and time-frequency results provide evidence that the control of bottom-up and top-down attention depend on differential contributions from parietal and frontal cortices.
Category information for visually presented objects can be read out from multi-voxel patterns of fMRI activity in ventral-temporal cortex. What is the nature and reliability of these patterns in the absence of any bottom-up visual input, for example, during visual imagery? Here, we first ask how well category information can be decoded for imagined objects and then compare the representations evoked during imagery and actual viewing. In an fMRI study, four object categories (food, tools, faces, buildings) were either visually presented to subjects, or imagined by them. Using pattern classification techniques, we could reliably decode category information (including for non-special categories, i.e., food and tools) from ventral-temporal cortex in both conditions, but only during actual viewing from retinotopic areas. Interestingly, in temporal cortex when the classifier was trained on the viewed condition and tested on the imagery condition, or vice versa, classification performance was comparable to within the imagery condition. The above results held even when we did not use information in the specialized category-selective areas. Thus, the patterns of representation during imagery and actual viewing are in fact surprisingly similar to each other. Consistent with this observation, the maps of "diagnostic voxels" (i.e., the classifier weights) for the perception and imagery classifiers were more similar in ventral-temporal cortex than in retinotopic cortex. These results suggest that in the absence of any bottom-up input, cortical back projections can selectively re-activate specific patterns of neural activity.
Previous work using functional magnetic resonance imaging has shown that the identities of isolated objects viewed by human subjects can be extracted from distributed patterns of brain activity. Outside the laboratory, however, objects almost never appear in isolation; thus it is important to understand how multiple simultaneously occurring objects are encoded by the visual system. We used multivoxel pattern analysis to examine this issue, testing whether activity patterns in the lateral occipital complex (LOC) evoked by object pairs showed an ordered relationship to patterns evoked by their constituent objects. Applying a searchlight analysis to identify voxels with the highest signal-to-noise ratios, we found that responses to object pairs within these informative voxels were well predicted by the averages of responses to their constituent objects. Consistent with this result, we were able to classify object pairs by using synthetic patterns created by averaging single-object patterns. These results indicate that the representation of multiple objects in LOC is governed by a response normalization mechanism similar to that reported in visual areas of several nonhuman species. They also suggest a population coding scheme that preserves information about multiple objects under conditions of distributed attention, facilitating fast object and scene recognition during natural vision.
Color and motion serve as the prime examples of segregated processing in the visual brain, giving rise to the question how color-motion conjunctions are represented. This problem is also known as the "binding problem."
Human volunteers viewed visual displays containing colored dots rotating around the center. The dots could be red or green and rotate clockwise or counterclockwise, leading to four possible stimulus displays. Superimposed pairs of such stimuli provided two additional displays, each containing both colors and both directions of motion but differing in their feature conjunctions. We applied multivariate classifiers to voxel-activation patterns obtained while subjects viewed such displays. Our analyses confirm the presence of directional-motion information across visual cortex and provide evidence of hue coding in all early visual areas except V5/MT(+). Within each cortical area, information on color and motion appeared to be coded in distinct sets of voxels. Furthermore, our results demonstrate the explicit representation of feature conjunctions in the primary visual cortex and beyond.
The results show that conjunctions can be decoded from spatial activation patterns already in V1, indicating an explicit coding of conjunctions at early stages of visual processing. Our findings raise the possibility that the solution of what has been taken as the prime example of the binding problem engages neural mechanisms as early as V1.
Event-related potentials (ERPs) were recorded during visual discrimination tasks in which stimulus arrays were presented that contained one lateral target and 3 (experiment 1) or one (experiments 2 and 3) non-targets. In experiments 1 and 2, targets differed from non-targets with respect to their form or their color. In experiment 3, word pairs were presented, with targets differing from non-targets with respect to their content. Subjects were required to respond to the identity of the target. In all experiments, an enhanced negativity was elicited at posterior electrodes contralateral to the location of the target. In the form discrimination tasks, this effect was present in the N1, N2, and P3 time intervals. In the color discrimination tasks, it was confined to the N2 time range. In the word discrimination task (experiment 3), this effect could only be observed over the left posterior hemisphere. It is argued that these lateralized negativities reflect the N2pc component that is assumed to indicate attentional filtering processes during visual search tasks. The present results extend this assumption by showing that this component is also elicited when targets are presented together with just one non-target item. It is argued that the N2pc may reflect the attentional selection of task-relevant stimuli.
Using functional magnetic resonance imaging (fMRI), we found an area in the fusiform gyrus in 12 of the 15 subjects tested that was significantly more active when the subjects viewed faces than when they viewed assorted common objects. This face activation was used to define a specific region of interest individually for each subject, within which several new tests of face specificity were run. In each of five subjects tested, the predefined candidate "face area" also responded significantly more strongly to passive viewing of (1) intact than scrambled two-tone faces, (2) full front-view face photos than front-view photos of houses, and (in a different set of five subjects) (3) three-quarter-view face photos (with hair concealed) than photos of human hands; it also responded more strongly during (4) a consecutive matching task performed on three-quarter-view faces versus hands. Our technique of running multiple tests applied to the same region defined functionally within individual subjects provides a solution to two common problems in functional imaging: (1) the requirement to correct for multiple statistical comparisons and (2) the inevitable ambiguity in the interpretation of any study in which only two or three conditions are compared. Our data allow us to reject alternative accounts of the function of the fusiform face area (area "FF") that appeal to visual attention, subordinate-level classification, or general processing of any animate or human forms, demonstrating that this region is selectively involved in the perception of faces.
Medial temporal brain regions such as the hippocampal formation and parahippocampal cortex have been generally implicated in navigation and visual memory. However, the specific function of each of these regions is not yet clear. Here we present evidence that a particular area within human parahippocampal cortex is involved in a critical component of navigation: perceiving the local visual environment. This region, which we name the 'parahippocampal place area' (PPA), responds selectively and automatically in functional magnetic resonance imaging (fMRI) to passively viewed scenes, but only weakly to single objects and not at all to faces. The critical factor for this activation appears to be the presence in the stimulus of information about the layout of local space. The response in the PPA to scenes with spatial layout but no discrete objects (empty rooms) is as strong as the response to complex meaningful scenes containing multiple objects (the same rooms furnished) and over twice as strong as the response to arrays of multiple objects without three-dimensional spatial context (the furniture from these rooms on a blank background). This response is reduced if the surfaces in the scene are rearranged so that they no longer define a coherent space. We propose that the PPA represents places by encoding the geometry of the local environment.
The parahippocampal place area (PPA) has been demonstrated to respond more strongly in fMRI to scenes depicting places than to other kinds of visual stimuli. Here, we test several hypotheses about the function of the PPA. We find that PPA activity (1) is not affected by the subjects' familiarity with the place depicted, (2) does not increase when subjects experience a sense of motion through the scene, and (3) is greater when viewing novel versus repeated scenes but not novel versus repeated faces. Thus, we find no evidence that the PPA is involved in matching perceptual information to stored representations in memory, in planning routes, or in monitoring locomotion through the local or distal environment but some evidence that it is involved in encoding new perceptual information about the appearance and layout of scenes.
Contrasting theories of visual attention emphasize selection by spatial location, visual features (such as motion or colour) or whole objects. Here we used functional magnetic resonance imaging (fMRI) to test key predictions of the object-based theory, which proposes that pre-attentive mechanisms segment the visual array into discrete objects, groups, or surfaces, which serve as targets for visual attention. Subjects viewed stimuli consisting of a face transparently superimposed on a house, with one moving and the other stationary. In different conditions, subjects attended to the face, the house or the motion. The magnetic resonance signal from each subject's fusiform face area, parahippocampal place area and area MT/MST provided a measure of the processing of faces, houses and visual motion, respectively. Although all three attributes occupied the same location, attending to one attribute of an object (such as the motion of a moving face) enhanced the neural representation not only of that attribute but also of the other attribute of the same object (for example, the face), compared with attributes of the other object (for example, the house). These results cannot be explained by models in which attention selects locations or features, and provide physiological evidence that whole objects are selected even when only one visual attribute is relevant.
Recent imaging studies have implicated the anterior cingulate cortex (ACC) in various cognitive functions, including attention. However, until now, there was no evidence for changes in neuronal activity of individual ACC neurons during performance of tasks that require attention and effortful thought. We hypothesized these neurons must exist in the human ACC. In this study, we present electrophysiological data from microelectrode single neuron recordings in the human ACC of neuronal modulation during attention-demanding tasks in 19% of 36 neurons tested. These findings provide the first direct evidence of an influence of a cognitive state on the spontaneous neuronal activity of human ACC neurons.
Neural activity was measured in 10 healthy volunteers by functional MRI while they viewed familiar and unfamiliar faces and listened to familiar and unfamiliar voices. The familiar faces and voices were those of people personally known to the subjects; they were not people who are more widely famous in the media. Changes in neural activity associated with stimulus modality irrespective of familiarity were observed in modules previously demonstrated to be activated by faces (fusiform gyrus bilaterally) and voices (superior temporal gyrus bilaterally). Irrespective of stimulus modality, familiarity of faces and voices (relative to unfamiliar faces and voices) was associated with increased neural activity in the posterior cingulate cortex, including the retrosplenial cortex. Our results suggest that recognizing a person involves information flow from modality-specific modules in the temporal cortex to the retrosplenial cortex. The latter area has recently been implicated in episodic memory and emotional salience, and now seems to be a key area involved in assessing the familiarity of a person. We propose that disturbances in the information flow described may underlie neurological and psychiatric disorders of the recognition of familiar faces, voices and persons (prosopagnosia, phonagnosia and Capgras delusion, respectively).