Article
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Analyzing distributed patterns of brain activation using multivariate pattern analysis (MVPA) has become a popular approach for using functional magnetic resonance imaging (fMRI) data to predict mental states. While the majority of studies currently build separate classifiers for each participant in the sample, in principle a single classifier can be derived from and tested on data from all participants. These two approaches, within- and cross-participant classification, rely on potentially different sources of variability and thus may provide distinct information about brain function. Here, we used both approaches to identify brain regions that contain information about passively received monetary rewards (i.e., images of currency that influenced participant payment) and social rewards (i.e., images of human faces). Our within-participant analyses implicated regions in the ventral visual processing stream-including fusiform gyrus and primary visual cortex-and ventromedial prefrontal cortex (VMPFC). Two key results indicate these regions may contain statistically discriminable patterns that contain different informational representations. First, cross-participant analyses implicated additional brain regions, including striatum and anterior insula. The cross-participant analyses also revealed systematic changes in predictive power across brain regions, with the pattern of change consistent with the functional properties of regions. Second, individual differences in classifier performance in VMPFC were related to individual differences in preferences between our two reward modalities. We interpret these results as reflecting a distinction between patterns showing participant-specific functional organization and those indicating aspects of brain organization that generalize across individuals.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Making inferences about neural coding implicating fine-scale patterns of information by testing for generalization of brain patterns across individuals might seem impossible, given the well documented anatomical variability within healthy populations (e.g., Frost and Goebel, 2013;Zhen et al., 2017Zhen et al., , 2015. In fact, several studies have provided empirical evidence that spatially-structured information generalizes poorly across subjects if spatial normalization is used to bring activation patterns from different subjects into correspondence (Clithero et al., 2011;Cox and Savoy, 2003;Haxby et al., 2011). Haxby et al. however, showed that a novel approach to "hyperalign" brain activation patterns across subjects, could, in principle, enable researchers to harness fine-scale structure while testing for generalization across subjects. ...
... This dependence is especially unjustified in the case of fine-scale representations. In contrast, theoretical arguments (Chaimow et al., 2011;Kamitani and Tong, 2005;Kriegeskorte et al., 2010;Ramírez et al., 2014) and empirical evidence (Cox and Savoy, 2003;Clithero et al., 2011;Haxby et al., 2011;Frost and Goebel, 2013;Zhen et al., 2015Zhen et al., , 2017 suggest that this assumption is incorrect. We have advanced these arguments by demonstrating that residual information decoded with across-subject analyses relying on anatomical normalization can be fully explained in terms of total-signal imbalances across conditions. ...
... In sum, pattern analyses aiming to support claims regarding neural properties cannot escape the need to consider the process by which brain measurements were acquired, the nature of the system under study, as well as the impact of noise descriptors (be it multivariate or univariate) on relevant parameter estimates and ensuing statistical analyses. Clithero et al. (2011) studied the effect of positive and negative reward on brain activity. They conducted within-and across-subject classification analyses of fMRI data. ...
Article
A key challenge in human neuroscience is to gain information about patterns of neural activity using indirect measures. Multivariate pattern analysis methods testing for generalization of information across subjects have been used to support inferences regarding neural coding. One critical assumption of an important class of such methods is that anatomical normalization is suited to align spatially-structured neural patterns across individual brains. We asked whether anatomical normalization is suited for this purpose. If not, what sources of information are such across-subject cross-validated analyses likely to reveal? To investigate these questions, we implemented two-layered feedforward randomly-connected networks. A key feature of these simulations was a gain-field with a spatial structure shared across networks. To investigate whether total-signal imbalances across conditions—e.g. differences in overall activity— affect the observed pattern of results, we manipulated the energy-profile of images conforming to a pre-specified correlation structure. To investigate whether the level of granularity of the data also influences results, we manipulated the density of connections between network layers. Simulations showed that anatomical normalization is unsuited to align neural representations. Pattern similarity-relationships were explained by the observed total-signal imbalances across conditions. Further, we observed that deceptively complex representational structures emerge from arbitrary analysis choices, such as whether the data are mean-subtracted during preprocessing. These simulations also led to testable predictions regarding the distribution of low-level features in images used in recent fMRI studies that relied on leave-one-subject-out pattern analyses. Image analyses broadly confirmed these predictions. Finally, hyperalignment emerged as a principled alternative to test across-subject generalization of spatially-structured information. We illustrate cases in which hyperalignment proved successful, as well as cases in which it only partially recovered the latent correlation structure in the pattern of responses. Our results highlight the need for robust, high-resolution measurements from individual subjects. We also offer a way forward for across-subject analyses. We suggest ways to inform hyperalignment results with estimates of the strength of the signal associated with each condition. Such information can usefully constrain ensuing inferences regarding latent representational structures as well as population tuning dimensions.
... (uncorrected) for each normalization scheme. Normalization across features resulted in the largest number of significant searchlights (7,899), followed by normalization across samples (6,721) and no normalization (6,034). In section D, we estimate cluster-level significance controlling the familywise error rate. ...
... (uncorrected) for each normalization scheme. Normalization across features resulted in the largest number of significant searchlights (7,899), followed by normalization across samples (6,721) and no normalization (6,034). In section D, we estimate cluster-level significance controlling the familywise error rate. ...
... We implemented leave-one-participant-out cross-validation [6] due to having relatively few independent scanner runs, and because this provides straightforward second-level inference [7]. To perform cross-modal classification, we trained each classifier on samples from one sensory modality in N−1 participants, then tested the classifier on samples from the other sensory modality in the left-out participant (Fig. 1). ...
Conference Paper
Full-text available
Multivariate cross-classification is a powerful tool for decoding abstract or supramodal representations from distributed neural populations. However, this approach introduces several methodological challenges not encountered in typical multivariate pattern analysis and information-based brain mapping. In the current report, we review these challenges, recommend solutions, and evaluate alternative approaches where possible. We address these challenges with reference to an example fMRI data set where participants were presented with brief series of auditory and visual stimuli of varying predictability with the aim of decoding predictability across auditory and visual modalities. In analyzing this data set, we highlight four particular challenges: response normalization, cross-validation, direction of cross-validation, and permutation testing.
... Many tasks have been considered for the classification of multiple subject fMRI data. From coarse grained visual, motor, auditory tasks ( [43], [42], [40], [25]), reward distinction ( [4]), to fine grained categories of objects ( [12], [35], [39]) and also very fine-grained category like concrete noun distinction ( [26], [15], [36]) have been used to demonstrate the feasibility of classifying a subject's thought based on training from other subjects. ...
... Functional images from all subjects must be transformed into co-ordinates of a standard brain. Transforming into Talairach Tournoux co-ordinates ( [25], [46]) anatomically defined regions using AAL atlas ( [26]) or creation of a whole brain mask based on all participants ( [4]) are some approaches to standardising. ...
... Feature selection/extraction plays a crucial role in obtaining the commonalities that are comparable across subjects. Most active voxels ( [25]), most discriminating voxels ( [40], [39]), most stable voxels ( [15], parcellation ( [42]), searchlight ( [4]) are the common techniques. Conversions into an intermediate form like canonical correlates( [36]), similarity relations ( [35]) or factors ( [15]) try to extract the latent variables that underlie the activations. ...
Article
Mind reading or thought prediction is a promising application of functional neuroimaging studies. The emergence of functional magnetic resonance imaging (fMRI) has, in the last two decades given a boost to these studies. In order to improve the accuracy, predictability and repeatability of thought prediction, it is important to have a representation that can capture the nuances of fMRI activations with respect to a particular cognitive state. In this paper, the process of creating a geometrical representation of the activations using non-linear manifolds is described. Manifold learning brings out the geometry of the activated voxels in the fMRI image. It is shown that this kind of representation is able to give high accuracy in classification studies as compared to using activation profiles.
... Trials were separated by a variable intertrial interval (ITI) of 4-8 s. The scanner session also contained another, passive viewing task (results reported elsewhere: Clithero et al., 2011b). Stimuli were projected onto a screen at the back of the scanner bore and participants viewed the stimuli through mirrored goggles. ...
... Following the scanner session, participants performed an economic exchange task ( Figure 1B) Clithero et al., 2011b). On each trial, participants chose whether to spend more money to view a novel high attractiveness face or less money to view a novel low attractiveness face. ...
... Tasks were programed using the Psychophysics Toolbox version 2.54 (Brainard, 1997) for MATLAB. Cash payment was determined by adding the cumulative total of a randomly selected run from the passive viewing task (Clithero et al., 2011b) to a base payment of $50 and then subtracting the total amount spent during the economic exchange task. Participants received an average of $16 for their bonus reward and spent an average of $2.12 to view new faces in the economic exchange task, resulting in a total mean payment of $$66 (range $53-92). ...
Article
Full-text available
According to many studies, the ventromedial prefrontal cortex (VMPFC) encodes the subjective value of disparate rewards on a common scale. Yet, a host of other reward factors-likely represented outside of VMPFC-must be integrated to construct such signals for valuation. Using functional magnetic resonance imaging (fMRI), we tested whether the interactions between posterior VMPFC and functionally-connected brain regions predict subjective value. During fMRI scanning, participants rated the attractiveness of unfamiliar faces. We found that activation in dorsal anterior cingulate cortex, anterior VMPFC, and caudate increased with higher attractiveness ratings. Using data from a post-scan task in which participants spent money to view attractive faces, we quantified each individual's subjective value for attractiveness. We found that connectivity between posterior VMPFC and regions frequently modulated by social information-including the temporal-parietal junction (TPJ) and middle temporal gyrus-was correlated with individual differences in subjective value. Crucially, these additional regions explained unique variation in subjective value, beyond that extracted from value regions alone. These findings indicate not only that posterior VMPFC interacts with additional brain regions during valuation, but also that these additional regions carry information employed to construct the subjective value for social reward.
... The analysis was made on two models: (i) combined global model (CGM), reflecting participant-independent factors where data of all the participants were pooled together, and (ii) personalized average model (PAM) where data from each participants were treated individually reflecting participantdependent factors (see Ref. [30] for a similar attempt on evaluating within-participant or participant-dependent and cross-participant or participant-independent classifiers). In CGM analysis, all trials across participants were analyzed together in a single classifier. ...
... This was also reflected by the fact that the number of features that optimally predict individual's preference decisions widely varied across participants (See Table S1). These distinctions between CGM and PAM models are not too surprising as sharp differences were also reported between within-and cross-participant classifiers of reward related fMRI responses [30]. ...
... It is important here to note that since response mapping to preference was counterbalanced over participants this issue is mostly relevant for the individualized PAM results but not for the group-based CGM results. Finally, instead of adopting regions of interest-based prediction techniques, which are often used in fMRI-based brain decoding [30,[37][38][39], we adopted a mechanistic machine learning approach where one searches for the best set of features which yield the best classification in a validating dataset. We treated sites of brain activity and the nature of brain oscillations agnostically -that is, without any reference to prior hypotheses. ...
Article
Full-text available
Preference formation is a complex problem as it is subjective, involves emotion, is led by implicit processes, and changes depending on the context even within the same individual. Thus, scientific attempts to predict preference are challenging, yet quite important for basic understanding of human decision making mechanisms, but prediction in a group-average sense has only a limited significance. In this study, we predicted preferential decisions on a trial by trial basis based on brain responses occurring before the individuals made their decisions explicit. Participants made a binary preference decision of approachability based on faces while their electrophysiological responses were recorded. An artificial neural network based pattern-classifier was used with time-frequency resolved patterns of a functional connectivity measure as features for the classifier. We were able to predict preference decisions with a mean accuracy of 74.3 ± 2.79% at participant-independent level and of 91.4 ± 3.8% at participant-dependent level. Further, we revealed a causal role of the first impression on final decision and demonstrated the temporal trajectory of preference decision formation.
... With respect to the current study, an additional aspect of the multivariate pattern classification method is the finding that the multivariate neural activation patterns in fMRI and also in EEG ⁄ MEG contain sufficient information to not only decode mental states within subjects, but also to accurately decode mental states of individuals based on the activation patterns of different individuals (Mourao-Miranda et al., 2005;Poldrack et al., 2009;Chan et al., 2011;Clithero et al., 2011;Schaefer et al., 2011). The results of existing studies, however, are mixed, with some reporting differences between the decoding of brain states within and across subjects (Shinkareva et al., 2008;Clithero et al., 2011;Schaefer et al., 2011). ...
... With respect to the current study, an additional aspect of the multivariate pattern classification method is the finding that the multivariate neural activation patterns in fMRI and also in EEG ⁄ MEG contain sufficient information to not only decode mental states within subjects, but also to accurately decode mental states of individuals based on the activation patterns of different individuals (Mourao-Miranda et al., 2005;Poldrack et al., 2009;Chan et al., 2011;Clithero et al., 2011;Schaefer et al., 2011). The results of existing studies, however, are mixed, with some reporting differences between the decoding of brain states within and across subjects (Shinkareva et al., 2008;Clithero et al., 2011;Schaefer et al., 2011). Withinsubject decoding could, in principle, be driven completely by subject-specific brain activation patterns, but in similar brain regions across subjects. ...
... Withinsubject decoding could, in principle, be driven completely by subject-specific brain activation patterns, but in similar brain regions across subjects. Across-subject decoding, on the other hand, reveals brain regions in which the neural activation patterns largely generalize across individuals (Clithero et al., 2011). Thus, highly similar withinsubject and across-subject decoding performance and spatial distributions would indicate a functional organization that is generalizable between subjects, whereas its divergence would separate subjectspecific from generalized activation patterns. ...
Article
The present magnetoencephalography study investigated whether the brain states of early syntactic and auditory-perceptual processes can be decoded from single-trial recordings with a multivariate pattern classification approach. In particular, it was investigated whether the early neural activation patterns in response to rule violations in basic auditory perception and in high cognitive processes (syntax) reflect a functional organization that largely generalizes across individuals or is subject-specific. On this account, subjects were auditorily presented with correct sentences, syntactically incorrect sentences, correct sentences including an interaural time difference change, and sentences containing both violations. For the analysis, brain state decoding was carried out within and across subjects with three pairwise classifications. Neural patterns elicited by each of the violation sentences were separately classified with the patterns elicited by the correct sentences. The results revealed the highest decoding accuracies over temporal cortex areas for all three classification types. Importantly, both the magnitude and the spatial distribution of decoding accuracies for the early neural patterns were very similar for within-subject and across-subject decoding. At the same time, across-subject decoding suggested a hemispheric bias, with the most consistent patterns in the left hemisphere. Thus, the present data show that not only auditory-perceptual processing brain states but also cognitive brain states of syntactic rule processing can be decoded from single-trial brain activations. Moreover, the findings indicate that the neural patterns in response to syntactic cognition and auditory perception reflect a functional organization that is highly consistent across individuals.
... A few more recent studies have now accomplished this. One such study used MVPA to predict whether participants had received monetary or social rewards on individual trials of a reward task (Clithero et al., 2010). In this study, within-and between-individual classification was directly compared. ...
... Our findings contrast with those of Clithero et al. (2010), who have argued that classification within and between individuals implicate different brain regions. In their study a classifier predicted which of several kinds of rewards participants had received, and results showed differing patterns of information distribution for the between-individual and withinindividual analyses. ...
... As mentioned in the Introduction section, while some studies have shown good classifier performance across individuals at coarser level of detail (Davatzikos et al., 2005; MouraoMiranda et al., 2005; Poldrack et al., 2009), there are now a growing number that show common patterns across individuals on a smaller scale as well (Clithero et al., 2010; Quadflieg et al., 2011; Shinkareva et al., 2008). The present data confirm these previous studies in showing that MVPA can uncover invariant patterns of neural activity across subjects even within restricted sets of voxels, and, additionally, they show an interesting trend. ...
Article
In a recent study we found that multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data could predict which of several touch-implying video clips a subject saw, only using voxels from primary somatosensory cortex. Here, we re-analyzed the same dataset using cross-individual MVPA to locate patterns of information that were common across participants' brains. In this procedure a classifier learned to distinguish the neural patterns evoked by each stimulus based on the data from a sub-group of the subjects and was then tested on data from an individual that was not part of that sub-group. We found prediction performance to be significantly above chance both when using voxels from the whole brain and when only using voxels from the postcentral gyrus. SVM voxel weight maps established based on the whole-brain analysis as well as a separate searchlight analysis suggested foci of especially high information content in medial and lateral occipital cortex and around the intraparietal sulcus. Classification across individuals appeared to rely on similar brain areas as classification within individuals. These data show that observing touch leads to stimulus-specific patterns of activity in sensorimotor networks and that these patterns are similar across individuals. More generally, the results suggest that cross-individual MVPA can succeed even when applied to restricted regions of interest.
... Instead of constructing new features from the original ones, a few studies opted to frame a searchlight strategy for fMRI data. Local patterns -defined in the spherical region where the searchlight beam focuses -can be obtained to make inference in the single-subject or inter-subject setting [49,25]. Recursive feature elimination (RFE) is another commonly used method for feature selection. ...
... Such multi-source standardization (or other types of feature normalization performed on a subject-by-subject basis) has often been used in the past (see e.g [96]; [121]; [25]; [67]; [17], we also believe it is being used in other studies without being explicitly reported) without having been methodologically motivated. In fact, multi-source transductive transfer offers the first explicit justification for this common operation, and this first study demonstrates its importance in practice. ...
Thesis
In functional neuroimaging experiments, participants perform a set of tasks while their brain activity is recorded, e.g. with electroencephalography (EEG), magnetoencephalography (MEG) or functional magnetic resonance imaging (fMRI). Analysing data from a group of participants, which is often denoted as group-level analysis, aims at identifying traits in the data that relate with the tasks performed by the participant and that are invariant within the population. This allows understanding the functional organization of the brain in healthy subjects and its dysfunctions in pathological populations. While group-level analyses for classical univariate statistical inference schemes, such as the general linear model, have been heavily studied, there are still many open questions for group-level strategies based on multivariate machine learning methods. This thesis therefore focuses on multivariate group-level analysis of functional neuroimaging and brings four contributions. The first contribution is a comparison of the results provided by two classifier-based multivariate group-level strategies: i) the standard one in which one aggregates the performances of within-subject models in a hierarchical analysis, and ii) the scheme we denote as inter-subject pattern analysis, where a population-level predictive model is directly estimated from data recorded on multiple subjects. An extensive set of experiments are conducted on both a large number of artificial datasets - where we parametrically control the size of the multivariate effect and the amount of inter-individual variability - as well as on two real fMRI datasets. Our results show that the two strategies can provide different results and that inter-subject analysis both offers a greater ability to small multivariate effects and facilitates the interpretation of the obtained results at a comparable computational cost.We then provide a survey of the methods that have been proposed to improve inter-subject pattern analysis, which is actually a hard task due to the largely heterogeneous vocabulary employed in the literature dedicated to this topic. Our second contribution consists in first introducing an unifying formalization of this framework, that we cast as a multi-source transductive transfer learning problem, and then in reviewing more than 500 related papers to offer a first comprehensive view of the existing literature where inter-subject pattern analysis was used in task-based functional neuroimaging experiments.Our third contribution is an experimental study that examines the well-foundedness of our multi-source transductive transfer formalization of inter-subject pattern analysis. With fMRI and MEG data recorded from numerous subjects, we demonstrate that between-subject variability impairs the generalization ability of classical machine learning algorithms and that a standard multi-source transductive learning strategy improves the generalization performances of such algorithms. Based on these promising results we further investigate the use of two more advanced machine learning methods to deal with the multi-source problem.The fourth contribution of this thesis is a new multivariate group-level analysis method for functional neuroimaging datasets. Our method is based on optimal transport, which leverages the geometrical properties of multivariate brain patterns to overcome inter-individual differences impacting the traditional group-level analyses. We extend the concept of Wasserstein barycenter, which was initially meant to average probability measures, to make it applicable to arbitrary data that do not necessarily fulfill the properties of a true probability measure. For this, we introduce a new algorithm that estimates a barycenter and provide an experimental study on artificial and real functional MRI.
... Kriegeskorte et al. used RSA to analyze the similarity patterns in the inferior temporal areas of man and monkey [39]. More recently, Clithero et al. performed cross-subject analysis by using RSA combined with feature-selection methods [9]. They used data obtained from visual stimuli when subjects were viewing faces. ...
... We list here a few of the researches who use PyMVPA in their study. As we mentioned earlier, Clithero et al. used PyMVPA for performing similarity analysis across subjects [9]. Sun et al. used SMLR classifier in PyMPVA to analyze the fMRI data obtained from patients having recent onset of psychosis [66]. ...
... Ideally, in order to identify consistent and discriminable activations associated with each emotion category, multiple emotions should be elicited and contrasted within a single neuroimaging study [40,41]. However, very few neuroimaging studies have examined multiple emotions in the same experiment in a manner that allows testing directly the predictions of basic emotion theories [42,43], leading to a search for other methods to test these predictions . ...
... 16, No. 9 studies targeted at the issues discussed in this review will thus be essential. In addition, neuroimaging methods assess correlations between emotions and brain regions, not whether these regions are functionally essential [41,86,87]. Thus, converging evidence from studies using other methods , such as neuropsychological studies of patients with focal brain lesions (Box 2) and studies with nonhuman animals (Box 3), will ultimately need to be integrated with findings from neuroimaging to arrive at comprehensive emotion models [88,89]. ...
Article
A longstanding controversy in the field of emotion research has concerned whether emotions are better conceptualized in terms of discrete categories, such as fear and anger, or underlying dimensions, such as arousal and valence. In the domain of neuroimaging studies of emotion, the debate has centered on whether neuroimaging findings support characteristic and discriminable neural signatures for basic emotions or whether they favor competing dimensional and psychological construction accounts. This review highlights recent neuroimaging findings in this controversy, assesses what they have contributed to this debate, and offers some preliminary conclusions. Namely, although neuroimaging studies have identified consistent neural correlates associated with basic emotions and other emotion models, they have ruled out simple one-to-one mappings between emotions and brain regions, pointing to the need for more complex, network-based representations of emotion.
... They typically contain multiple folds to support cross-validation; in different folds, different samples are used for training and testing. CoSMoMVPA provides functions to generate a variety of partitioning schemes including those for crossmodal (Oosterhof, Wiggett, Diedrichsen, tipper, & Downing, 2010, Oosterhof, tipper, & Downing, 2012 and crossparticipant (Clithero, Smith, Carter, & Huettel, 2011;Haxby et al., 2011) decoding (Figure 11). Using a crossvalidation measure (explained below) with classifier provides functionality for cross-validation using a variety of cross-validation schemes and classifiers. ...
Preprint
Full-text available
Recent years have seen an increase in the popularity of multivariate pattern (MVP) analysis of functional magnetic resonance (fMRI) data, and, to a much lesser extent, magneto-and electro-encephalography (M/EEG) data. We present CoSMoMVPA, a lightweight MVPA (MVP analysis) toolbox implemented in the intersection of the Matlab and GNU Octave languages, that treats both fMRI and M/EEG data as first-class citizens. CoSMoMVPA supports all state-of-the-art MVP analysis techniques, including searchlight analyses, classification, correlations, representational similarity analysis, and the time generalization method. These can be used to address both data-driven and hypothesis-driven questions about neural organization and representations, both within and across: space, time, frequency bands, neuroimaging modalities, individuals, and species. It uses a uniform data representation of fMRI data in the volume or on the surface, and of M/EEG data at the sensor and source level. Through various external toolboxes, it directly supports reading and writing a variety of fMRI and M/EEG neuroimaging formats, and, where applicable, can convert between them. As a result, it can be integrated readily in existing pipelines and used with existing preprocessed datasets. CoSMoMVPA overloads the traditional volumetric searchlight concept to support neighborhoods for M/EEG and surface-based fMRI data, which supports localization of multivariate effects of interest across space, time, and frequency dimensions. CoSMoMVPA also provides a generalized approach to multiple comparison correction across these dimensions using Threshold-Free Cluster Enhancement with state-of-the-art clustering and permutation techniques. CoSMoMVPA is highly modular and uses abstractions to provide a uniform interface for a variety of MVP measures. Typical analyses require a few lines of code, making it accessible to beginner users. At the same time, expert programmers can easily extend its functionality. CoSMoMVPA comes with extensive documentation, including a variety of runnable demonstration scripts and analysis exercises (with example data and solutions). It uses best software engineering practices including version control, distributed development, an automated test suite, and continuous integration testing. It can be used with the proprietary Matlab and the free GNU Octave software, and it complies with open source distribution platforms such as NeuroDebian. CoSMoMVPA is Free/Open Source Software under the permissive MIT license. Website: https://cosmomvpa.org Source code: https://github.com/CoSMoMVPA/CoSMoMVPA
... Indeed, it has been reported in several articles (see e.g. [29]; [36]; [24]; [11]; [8]; [46]), but we believe it is also vastly used without being reported because researchers might consider it to be trivial practice. Our formalization highlights that it is in fact not trivial, and we therefore recommend researchers to explicitly report in their publications whether they perform subject-bysubject feature standardization using this multi-source transductive setting. ...
Preprint
Background and objective In medical imaging, population studies have to overcome the differences that exist between individuals to identify invariant image features that can be used for diagnosis purposes. In functional neuroimaging, an appealing solution to identify neural coding principles that hold at the population level is inter-subject pattern analysis, i.e. to learn a predictive model on data from multiple subjects and evaluate its generalization performance on new subjects. Although it has gained popularity in recent years, its widespread adoption is still hampered by the blatant lack of a formal definition in the literature. In this paper, we precisely introduce the first principled formalization of inter-subject pattern analysis targeted at multivariate group analysis of functional neuroimaging. Methods We propose to frame inter-subject pattern analysis as a multi-source transductive transfer question, thus grounding it within several well defined machine learning settings and broadening the spectrum of usable algorithms. We describe two sets of inter-subject brain decoding experiments that use several open datasets: a magnetoencephalography study with 16 subjects and a functional magnetic resonance imaging paradigm with 100 subjects. We assess the relevance of our framework by performing model comparisons, where one brain decoding model exploits our formalization while others do not. Results The first set of experiments demonstrates the superiority of a brain decoder that uses subject-by-subject standardization compared to state of the art models that use other standardization schemes, making the case for the interest of the transductive and the multi-source components of our formalization The second set of experiments quantitatively shows that, even after such transformation, it is more difficult for a brain decoder to generalize to new participants rather than to new data from participants available in the training phase, thus highlighting the transfer gap that needs to be overcome. Conclusion This paper describes the first formalization of inter-subject pattern analysis as a multi-source transductive transfer learning problem. We demonstrate the added value of this formalization using proof-of-concept experiments on several complementary functional neuroimaging datasets. This work should contribute to popularize inter-subject pattern analysis for functional neuroimaging population studies and pave the road for future methodological innovations.
... Few studies have explored the effect of different input modalities in brain activation decoding tasks, especially when coupled with different word embedding models. However, both embodied and symbolic theories of conceptual organization hold that there is partial sharing and partial differentiation between the neural activation patterns observed for concepts activated via different stimulus modalities [2,10,13] . Therefore, we assume a difference between different input modalities in both the decoding performance and the neural basis. ...
Article
fMRI word decoding refers to decode what the human brain is thinking by interpreting functional Magnetic Resonance Imaging (fMRI) scans from people watching or listening to words, representing a sort of mind-reading technology. Existing works decoding words from imaging data have been largely limited to concrete nouns from a relatively small number of semantic categories. Moreover, such studies use different word-stimulus presentation paradigms and different computational models, lacking a comprehensive understanding of the influence of different factors on fMRI word decoding. In this paper, we present a large-scale evaluation of eight word embedding models and their combinations for decoding fine-grained fMRI data associated with three classes of words recorded from three stimulus-presentation paradigms. Specifically, we investigate the following research questions: (1) How does the brain-image decoder perform on different classes of words? (2) How does the brain-image decoder perform in different stimulus-presentation paradigms? (3) How well does each word embedding model allow us to decode neural activation patterns in the human brain? Furthermore, we analyze the most informative voxels associated with different classes of words, stimulus-presentation paradigms and word embedding models to explore their neural basis. The results have shown the following: (1) Different word classes can be decoded most effectively with different word embedding models. Concrete nouns and verbs are more easily distinguished than abstract nouns and verbs. (2) Among the three stimulus-presentation paradigms (picture, sentence and word clouds), the picture paradigm achieves the highest decoding accuracy, followed by the sentence paradigm. (3) Among the eight word embedding models, the model that encodes visual information obtains the best performance, followed by models that encode textual and contextual information. (4) Compared to concrete nouns, which activate mostly vision-related brain regions, abstract nouns activate broader brain regions such as the visual, language and default-mode networks. Moreover, both the picture paradigm and the model that encodes visual information have stronger associations with vision-related brain regions than other paradigms and word embedding models, respectively.
... One potential disadvantage is that population-level models are not always as predictive as individualized models (e.g., see direct comparisons in Clithero et al., 2011;Haxby et al., 2011;Lindquist et al., 2017;Shinkareva et al., 2008). One important limitation is inter-subject variability in structural and functional anatomy that reduces generalizability across subjects (Cox and Savoy, 2003). ...
Article
Human neuroimaging research has transitioned from mapping local effects to developing predictive models of mental events that integrate information distributed across multiple brain systems. Here we review work demonstrating how multivariate predictive models have been utilized to provide quantitative, falsifiable predictions; establish mappings between brain and mind with larger effects than traditional approaches; and help explain how the brain represents mental constructs and processes. Although there is increasing progress toward the first two of these goals, models are only beginning to address the latter objective. By explicitly identifying gaps in knowledge, research programs can move deliberately and programmatically toward the goal of identifying brain representations underlying mental states and processes.
... Classification was performed using linear support vector machines (SVMs; Boser et al., 1992) with the soft-margin parameter C automatically scaled to the norm of the data. All classification analyses were performed using leave-one-participant-out cross-validation (e.g., Clithero et al., 2011;Mourao-Miranda et al., 2005). That is, for each cross-validation fold, the decision boundary was constructed based on samples from 24 of the 25 participants, and tested on the left-out participant. ...
... Although the forward encoding approach (e.g., Huth et al., 2012Huth et al., , 2016Mitchell et al., 2008) assumes a common semantic model across individuals (e.g., based on word cooccurrence statistics in text), this model is fit differentially to each voxel in each individual's brain. The critical benchmark of common neural representation is to train an encoding or decoding model on a group of participants and test this model on a novel, left-out participant (Clithero et al., 2011;Mourão-Miranda et al., 2005). Although this may sound overly stringent, it is the de facto standard required for predictive models to provide diagnostic information in individual people (Dubois and Adolphs, 2016;Poldrack, 2011). ...
Chapter
Full-text available
Semantic memory comprises our conceptual knowledge of the world and provides a critical interface between perception, action, and language. Acquiring and later deploying semantic knowledge in service of behavior relies on the coordinated function of distributed cortical and hippocampal circuitry. Semantic memory is typically delineated from episodic memory in that its content is divorced from the autobiographical or experiential context at acquisition. Neuropsychological work has related this distinction to gross anatomical substrates: the hippocampus is required during context-rich acquisition and subsequent consolidation, but is over time superseded by more stable, context-free cortical encoding. Neuroimaging research has since proceeded to comprehensively map the cortical organization of semantic memory, demonstrating that the macroanatomical substrates parallel the sensorimotor systems most relevant to particular domains of knowledge. Finally, methodological advances in neuroimaging have enabled us to leverage sophisticated computational models of semantic representation and decode fine-grained neural representations of semantic content from distributed patterns of brain activity.
... They typically contain multiple folds to support cross-validation; in different folds, different samples are used for training and testing. CoSMoMVPA provides functions to generate a variety of partitioning schemes including those for crossmodal (Oosterhof et al., 2010 and cross-participant (Clithero et al., 2011;Haxby et al., 2011) decoding (Figure 11). Using a cross-validation measure (explained below) with classifier provides functionality for cross-validation using a variety of cross-validation schemes and classifiers. ...
Article
Full-text available
Recent years have seen an increase in the popularity of multivariate pattern (MVP) analysis of functional magnetic resonance (fMRI) data, and, to a much lesser extent, magneto- and electro-encephalography (M/EEG) data. We present CoSMoMVPA, a lightweight MVPA (MVP analysis) toolbox implemented in the intersection of the Matlab and GNU Octave languages, that treats both fMRI and M/EEG data as first-class citizens. CoSMoMVPA supports all state-of-the-art MVP analysis techniques, including searchlight analyses, classification, correlations, representational similarity analysis, and the time generalization method. These can be used to address both data-driven and hypothesis-driven questions about neural organization and representations, both within and across: space, time, frequency bands, neuroimaging modalities, individuals, and species. It uses a uniform data representation of fMRI data in the volume or on the surface, and of M/EEG data at the sensor and source level. Through various external toolboxes, it directly supports reading and writing a variety of fMRI and M/EEG neuroimaging formats, and, where applicable, can convert between them. As a result, it can be integrated readily in existing pipelines and used with existing preprocessed datasets. CoSMoMVPA overloads the traditional volumetric searchlight concept to support neighborhoods for M/EEG and surface-based fMRI data, which supports localization of multivariate effects of interest across space, time, and frequency dimensions. CoSMoMVPA also provides a generalized approach to multiple comparison correction across these dimensions using Threshold-Free Cluster Enhancement with state-of-the-art clustering and permutation techniques. CoSMoMVPA is highly modular and uses abstractions to provide a uniform interface for a variety of MVP measures. Typical analyses require a few lines of code, making it accessible to beginner users. At the same time, expert programmers can easily extend its functionality. CoSMoMVPA comes with extensive documentation, including a variety of runnable demonstration scripts and analysis exercises (with example data and solutions). It uses best software engineering practices including version control, distributed development, an automated test suite, and continuous integration testing. It can be used with the proprietary Matlab and the free GNU Octave software, and it complies with open source distribution platforms such as NeuroDebian. CoSMoMVPA is Free/Open Source Software under the permissive MIT license. Website: http://cosmomvpa.org Source code: https://github.com/CoSMoMVPA/CoSMoMVPA
... Analyses pertaining to the discrimination of visual stimuli included discrimination of stimulus orientation, position, color, and form. Additional analyses pertaining to the semantic category of the visual stimulus (e.g., animals vs. tools; Simanova, Hagoort, Oostenveld, & van Gerven, 2014) and stimuli that were consistently associated with different rewards (e.g., face vs. currency, where a picture of currency indicated a monetary reward; Clithero, Smith, Carter, & Huettel, 2011) were included in our lenient categorization but excluded from the strict categorization. In our strict categorization, we also excluded two further studies in which there was a possibility that the visual stimulus could evoke representation of motor actions. ...
Article
How is the processing of task information organized in the brain? Many views of brain function emphasize modularity, with different regions specialized for processing different types of information. However, recent accounts also highlight flexibility, pointing especially to the highly consistent pattern of frontoparietal activation across many tasks. Although early insights from functional imaging were based on overall activation levels during different cognitive operations, in the last decade many researchers have used multivoxel pattern analyses to interrogate the representational content of activations, mapping out the brain regions that make particular stimulus, rule, or response distinctions. Here, we drew on 100 searchlight decoding analyses from 57 published papers to characterize the information coded in different brain networks. The outcome was highly structured. Visual, auditory, and motor networks predominantly (but not exclusively) coded visual, auditory, and motor information, respectively. By contrast, the frontoparietal multiple-demand network was characterized by domain generality, coding visual, auditory, motor, and rule information. The contribution of the default mode network and voxels elsewhere was minor. The data suggest a balanced picture of brain organization in which sensory and motor networks are relatively specialized for information in their own domain, whereas a specific frontoparietal network acts as a domain-general "core" with the capacity to code many different aspects of a task.
... When examining patterns at a finer spatial scale, inter-individual variability is yet larger and performing such inter-subject predictions becomes even more challenging. At such scale, the alignement between cortical folding and the underlying functional organization vary between subjects [Essen et al. 2007;Sabuncu, Singer, et al. 2010], in a way that the potentially poor voxel-to-voxel correspondance provided by spatial normalization procedures limits the generalization power of classifiers that use voxel values as features [Clithero et al. 2011]. To our knowledge, only two studies describe methods specifically designed for inter-subject classification without the need for spatial normalization. ...
Thesis
Full-text available
L’apprentissage inter-sujet intervient dans l’analyse des données enregistrées chez des sujets humains, lorsque le sujet chez lequel on doit faire une prédiction ne faisait pas partie de la base d’apprentissage. Le plus typique de ces problèmes est l’aide au diagnostic, lorsque on demande à un outil informatique si un sujet, inconnu jusque là, est sain ou malade. Dans cette thèse, nous défendons le point de vue que le problème d’apprentissage inter-sujet doit être formalisé comme un problème multi-source dans lequel chaque sujet de la base d’apprentissage fournit une source de données enregistrées dans un espace d’entrée potentiellement différent et qui sont des réalisations de distributions différentes. Le cadre multi-source est ainsi une généralisation du problème d’adaptation de domaine, dans lequel une seule source de données est disponible. Nous présentons ensuite trois contributions motivées par des problèmes d’apprentissage inter-sujet en neuroimagerie.Le résultat de notre première contribution est une méthode qui permet de produire des prédictions inter-sujet sur des données d’IRM fonctionnelle en utilisant les patrons d’activation disponibles à des échelles spatiales relativement fines disponibles dans une région d’intérêt du cortex. Du à la forte variabilité fonctionnelle inter-sujet, les espaces d’entrée dans lesquels vivent ces patrons sont différents au travers des sujets. Notre contribution consiste à construire un espace commun pour tous les sujets en utilisant une représentation graphique des patrons d’activation ainsi qu’un noyau de graphe qui projette implicitement ces représentations dans un espace de hilbert à noyau reproduisant. Nous avons démontré l’efficacité de cette approche grâce à l’amélioration de la performance de classification dans un tâche de prédiction inter-sujet construite pour étudier l’organisation fonctionnelle du cortex auditif.La deuxième contribution présentée dans cette thèse est une nouvelle méthode qui permet l’identification de différences de formes locales du cortex entre plusieurs groupes d’observations. Les objets utilisés sont, une fois de plus, des représentations graphiques, cette fois construites à partir des points correspondant à des extrema de profondeur des sillons corticaux. L’utilisation d’un noyau de graphe adapté à ces objets permet, dans l’ espace de hilbert à noyau reproduisant correspondant, de quantifier les différences entre groupes d’observations par la performance d’un classifieur entraîné à reconnaître ces groupes. Une méthode d’inférence spatial non paramétrique permet ensuite la détection, c’est à dire l’identification des zones du cortex qui présentent des différences significatives. Nous validons cette méthode en démontrant qu’elle permet d’identifier, sur une large population de sujets sains, des asymétries corticales ainsi que des différences inter-sexe.La troisième contribution est une méthode d’adaptation de domaine pour le cas multi-source. Notre méthode se base sur le kernel mean matching, une procédure d’appariement de distributions qui adapte la distribution de l’ensemble d’entrainement à celle de l’ensemble de test par une pondération des exemples d’apprentissage. Nous décrivons une extension du kernel mean matching au cas où l’ensemble d’apprentissage se compose de plusieurs sources de données. Nous présentons des résultats préliminaires sur une tâche de classification inter-sujet dans une expérience de magnéto-encéphalographie.
... We observed an average classification rate of 55.8% for six category classification using two types of visual inductions. These levels are in the same range as prior classification studies on individual participants or crossparticipants (e.g., Kassam et al., 2013;Chikazoe et al., 2014), although interpretation of classification accuracies across studies should be taken with caution since accuracy rates can be sensitive to the particularities of different experimental designs and analytical techniques (Clithero et al., 2011). Taken together, our findings provide both univariate and multivariate support for notion that neural activity separates affective episodes apart along the lines of their sensory qualities. ...
Article
Full-text available
A growing body of work suggests that sensory processes may also contribute to affective experience. In this study, we performed a meta-analysis of affective experiences driven through visual, auditory, olfactory, gustatory, and somatosensory stimulus modalities including study contrasts that compared affective stimuli to matched neutral control stimuli. We found, first, that limbic and paralimbic regions, including the amygdala, anterior insula, pre-supplementary motor area, and portions of orbitofrontal cortex were consistently engaged across two or more modalities. Second, early sensory input regions in occipital, temporal, piriform, mid-insular, and primary sensory cortex were frequently engaged during affective experiences driven by visual, auditory, olfactory, gustatory, and somatosensory inputs. A classification analysis demonstrated that the pattern of neural activity across a contrast map diagnosed the stimulus modality driving the affective experience. These findings suggest that affective experiences are constructed from activity that is distributed across limbic and paralimbic brain regions and also activity in sensory cortical regions.
... Recently, it has become more common to apply multivariate techniques such as pattern classification (Haynes and Rees 2006; Norman et al. 2006) to predict decisions (Clithero, Carter, and Huettel 2009; Clithero et al. 2011; Tusche, Bode, and Haynes 2010). For example, Smith et al. (2012) use such an approach to predict out-of-sample choices from " nonchoice " neural responses to different products, and Chua et al. (2011) do so to predict quitting behavior of smokers four months later. ...
Article
The first decade of consumer neuroscience research has produced groundbreaking work in identifying the basic neural processes underlying human judgment and decision making, with the majority of such studies published in neuroscience journals and influencing models of brain function. Yet for the field of consumer neuroscience to thrive in the next decade, the current emphasis on basic science research must be extended into marketing theory and practice. The authors suggest five concrete ways that neuroscientific methods can be fruitfully applied to marketing. They then outline three fundamental challenges facing consumer neuroscientists and offer potential solutions for addressing them. The authors conclude by describing how consumer neuroscience can become an important complement to research and practice in marketing.
... MVPA has been used to compare topographic patterns of neural activity across experimental conditions in a number of cognitive domains, including task performance (Esterman, Chiu, Tamber-Rosenau, & Yantis, 2009) and subjective remembering (Rissman, Greely, & Wagner, 2010). Recent demonstrations of between-subject decoding show that MVPA can be used to determine whether patterns of neural activity (and the cognitive processes presumed to be reflected therein) are common across individuals (Clithero, Smith, Carter, & Huettel, 2011;Haxby et al., 2011). ...
Article
Neuroimaging studies of recognition memory have identified distinct patterns of cortical activity associated with two sets of cognitive processes: Recollective processes supporting retrieval of information specifying a probe item's original source are associated with the posterior hippocampus, ventral posterior parietal cortex, and medial pFC. Familiarity processes supporting the correct identification of previously studied probes (in the absence of a recollective response) are associated with activity in anterior medial-temporal lobe (MTL) structures including the perirhinal cortex and anterior hippocampus, in addition to lateral prefrontal and dorsal posterior parietal cortex. Here, we address an open question in the cognitive neuroscientific literature: To what extent are these same neurocognitive processes engaged during an internally directed memory search task like free recall? We recorded fMRI activity while participants performed a series of free recall and source recognition trials, and we used a combination of univariate and multivariate analysis techniques to compare neural activation profiles across the two tasks. Univariate analyses showed that posterior MTL regions were commonly associated with recollective processes during source recognition and with free recall responses. Prefrontal and posterior parietal regions were commonly associated with familiarity processes and free recall responses, whereas anterior MTL regions were only associated with familiarity processes during recognition. In contrast with the univariate results, free recall activity patterns characterized using multivariate pattern analysis did not reliably match the neural patterns associated with recollective processes. However, these free recall patterns did reliably match patterns associated with familiarity processes, supporting theories of memory in which common cognitive mechanisms support both item recognition and free recall.
... Such connections are thought to be involved in a number of cognitive processes that may be vital to metacognition [5], including associating events across time, orientation, implicit and explicit learning, temporal binding of experience, and assessment of the motivational state of an individual [43]. Imaging studies suggest that VS is also differentially active during novelty or salience tasks [44], as well as those involving social rewards [45] and encoding facial emotion [46,47]. These processes could be expected to support the complex integration of personal and social information as needed for metacognition. ...
Article
Individuals in the early phases of psychotic illness have disturbed metacognitive capacity, which has been linked to a number of poor outcomes. Little is known, however, about the neural systems associated with metacognition in this population. The purpose of this study was to elucidate the neuroanatomical correlates of metacognition. We anticipated that higher levels of metacognition may be dependent upon gray matter density (GMD) of regions within the prefrontal cortex. Examining whole-brain structure in 25 individuals with early phase psychosis, we found positive correlations between increased medial prefrontal cortex and ventral striatum GMD and higher metacognition. These findings represent an important step in understanding the path through which the biological correlates of psychotic illness may culminate into poor metacognition and, ultimately, disrupted functioning. Such a path will serve to validate and promote metacognition as a viable treatment target in early phase psychosis.
... Among these, most rely on global (i.e full-brain) analysis, using large-scale features ( [12], [15]). The few studying fine-scale local patterns achieve very low inter-subject generalization performances [3], or use implicit abstract models [9]. None attempt to characterize the inter-subject functional variability (i.e the fact that the correlation between cortical folding and the underlying functional organization vary between subjects [4]) and use such characterization in the classification process. ...
Conference Paper
Full-text available
Classification of medical images in multi-subjects settings is a difficult challenge due to the variability that exists between individuals. Here we introduce a new graph-based framework designed to deal with inter-subject functional variability present in fMRI data. A graphical model is constructed to encode the functional, geometric and structural properties of local activation patterns. We then design a specific graph kernel, allowing to conduct SVM classification in graph space. Experiments conducted in an inter-subject classification task of patterns recorded in the auditory cortex show that it is the only approach to perform above chance level, among a wide range of tested methods.
... Though there are subject variations in anatomy and methodological difficulties in inter-subject coregistration, neural similarities arose in terms of the locations and activation amplitudes of voxels utilized by the classifier to identify the category of a stimulus [36]. There are now a growing number of studies in showing that MVPA can uncover invariant patterns of neural activity across subjects [20,37,38]. ...
Article
Full-text available
Perception of sound categories is an important aspect of auditory perception. The extent to which the brain's representation of sound categories is encoded in specialized subregions or distributed across the auditory cortex remains unclear. Recent studies using multivariate pattern analysis (MVPA) of brain activations have provided important insights into how the brain decodes perceptual information. In the large existing literature on brain decoding using MVPA methods, relatively few studies have been conducted on multi-class categorization in the auditory domain. Here, we investigated the representation and processing of auditory categories within the human temporal cortex using high resolution fMRI and MVPA methods. More importantly, we considered decoding multiple sound categories simultaneously through multi-class support vector machine-recursive feature elimination (MSVM-RFE) as our MVPA tool. Results show that for all classifications the model MSVM-RFE was able to learn the functional relation between the multiple sound categories and the corresponding evoked spatial patterns and classify the unlabeled sound-evoked patterns significantly above chance. This indicates the feasibility of decoding multiple sound categories not only within but across subjects. However, the across-subject variation affects classification performance more than the within-subject variation, as the across-subject analysis has significantly lower classification accuracies. Sound category-selective brain maps were identified based on multi-class classification and revealed distributed patterns of brain activity in the superior temporal gyrus and the middle temporal gyrus. This is in accordance with previous studies, indicating that information in the spatially distributed patterns may reflect a more abstract perceptual level of representation of sound categories. Further, we show that the across-subject classification performance can be significantly improved by averaging the fMRI images over items, because the irrelevant variations between different items of the same sound category are reduced and in turn the proportion of signals relevant to sound categorization increases.
... When examining patterns at a finer spatial scale, inter-individual variability is yet larger and performing such inter-subject predictions becomes even more challenging. At such scale, the alignement between cortical folding and the underlying functional organization vary between subjects [22,23], in a way that the potentially poor voxel-to-voxel correspondance provided by spatial normalization procedures limits the generalization power of classifiers that use voxel values as features [24]. To our knowledge, only two studies describe methods specifically designed for inter-subject classification without the need for spatial normalization. ...
Article
Full-text available
In brain imaging, solving learning problems in multi-subjects settings is difficult because of the differences that exist across individuals. Here we introduce a novel classification framework based on group-invariant graphical representations, allowing to overcome the inter-subject variability present in functional magnetic resonance imaging (fMRI) data and to perform multivariate pattern analysis across subjects. Our contribution is twofold: first, we propose an unsupervised representation learning scheme that encodes all relevant characteristics of distributed fMRI patterns into attributed graphs; second, we introduce a custom-designed graph kernel that exploits all these characteristics and makes it possible to perform supervised learning (here, classification) directly in graph space. The well-foundedness of our technique and the robustness of the performance to the parameter setting are demonstrated through inter-subject classification experiments conducted on both artificial data and a real fMRI experiment aimed at characterizing local cortical representations. Our results show that our framework produces accurate inter-subject predictions and that it outperforms a wide range of state-of-the-art vector- and parcel-based classification methods. Moreover, the genericity of our method makes it is easily adaptable to a wide range of potential applications. The dataset used in this study and an implementation of our framework are available at http://dx.doi.org/10.6084/m9.figshare.1086317.
... On the one side, evidence does suggest a certain degree of uniformity. For instance it is possible to train pattern recognition algorithms to distinguish two brain-states in one subject and then successfully apply the learned model on another subject (Shinkareva et al., 2008;Clithero et al., 2011;Kaplan and Meyer, 2012). Such a crosssubject classification can only be possible if the spatio-temporal patterns of brain activation are quite similar across different subjects. ...
Article
Full-text available
Functional magnetic resonance imaging (fMRI) is the workhorse of imaging-based human cognitive neuroscience. The use of fMRI is ever-increasing; within the last 4 years more fMRI studies have been published than in the previous 17 years. This large body of research has mainly focused on the functional localization of condition- or stimulus-dependent changes in the blood-oxygenation-level dependent signal. In recent years, however, many aspects of the commonly practiced analysis frameworks and methodologies have been critically reassessed. Here we summarize these critiques, providing an overview of the major conceptual and practical deficiencies in widely used brain-mapping approaches, and exemplify some of these issues by the use of imaging data and simulations. In particular, we discuss the inherent pitfalls and shortcomings of methodologies for statistical parametric mapping. Our critique emphasizes recent reports of excessively high numbers of both false positive and false negative findings in fMRI brain mapping. We outline our view regarding the broader scientific implications of these methodological considerations and briefly discuss possible solutions.
... Both studies focus exclusively on within-subject classification or prediction, and they do not attempt to predict average behavior for groups, or choices across subjects. Tucshe, Bode, and Haynes (2010) study the neural correlates of hypothetical choices rather than real 7 Our study is also related to Hampton and O'Doherty (2007); Grosenick, Greer, and Knutson (2008); Krajbich et al. (2009);Clithero, Carter, and Huettel (2009);and Clithero et al. (2011). These papers employ the same class of methods from the statistical learning literature used here. ...
Article
Full-text available
We investigate the feasibility of inferring the choices people would make (if given the opportunity) based on their neural responses to the pertinent prospects when they are not engaged in actual decision making. The ability to make such inferences is of potential value when choice data are unavailable, or limited in ways that render standard methods of estimating choice mappings problematic. We formulate prediction models relating choices to “nonchoice” neural responses, and use them to predict out-of-sample choices for new items and for new groups of individuals. The predictions are sufficiently accurate to establish the feasibility of our approach.
... However, some authors have reported that the 2 approaches can sometimes produce markedly different results (Jimura and Poldrack 2012;also see, e.g., Tamber-Rosenau et al. 2011, 2013. Only a subset of fMRI studies using multivariate methods focus on individual differences (e.g., Carp et al. 2011;Clithero et al. 2011;Tong et al. 2012) and still fewer have explicitly compared these methods (Coutanche et al. 2011;Hoeft et al. 2011;Coutanche 2013). ...
Article
Full-text available
Expertise effects for nonface objects in face-selective brain areas may reflect stable aspects of neuronal selectivity that determine how observers perceive objects. However, bottom-up (e.g., clutter from irrelevant objects) and top-down manipulations (e.g., attentional selection) can influence activity, affecting the link between category selectivity and individual performance. We test the prediction that individual differences expressed as neural expertise effects for cars in face-selective areas are sufficiently stable to survive clutter and manipulations of attention. Additionally, behavioral work and work using event related potentials suggest that expertise effects may not survive competition; we investigate this using functional magnetic resonance imaging. Subjects varying in expertise with cars made 1-back decisions about cars, faces, and objects in displays containing one or 2 objects, with only one category attended. Univariate analyses suggest car expertise effects are robust to clutter, dampened by reducing attention to cars, but nonetheless more robust to manipulations of attention than competition. While univariate expertise effects are severely abolished by competition between cars and faces, multivariate analyses reveal new information related to car expertise. These results demonstrate that signals in face-selective areas predict expertise effects for nonface objects in a variety of conditions, although individual differences may be expressed in different dependent measures depending on task and instructions.
... These hypotheses are based not only on previous work in dietary choice, but also on findings from the previous literature on goal-directed choice. First, areas of vmPFC have consistently been shown to correlate with stimulus values at the time of choice across a wide variety of decision contexts (Tom et al., 2007;Boorman et al., 2009;Lebreton et al., 2009;Basten et al., 2010;Hare et al., 2010;Plassmann et al., 2010;Shenhav and Greene, 2010;Clithero et al., 2011;Kahnt et al., 2011;Park et al., 2011), including decisions involving intertemporal tradeoffs (Kable and Glimcher, 2007;Ballard and Knutson, 2009;Hare et al., 2009Hare et al., , 2011aCarter et al., 2010;Peters and Buchel, 2010). Second, previous studies have associated responses in left dlPFC with choosing to wait for delayed monetary rewards using transcranial magnetic stimulation (TMS) and fMRI (McClure et al., 2004;Figner et al., 2010;Luo et al., 2012). ...
Article
Full-text available
There is widespread interest in identifying computational and neurobiological mechanisms that influence the ability to choose long-term benefits over more proximal and readily available rewards in domains such as dietary and economic choice. We present the results of a human fMRI study that examines how neural activity relates to observed individual differences in the discounting of future rewards during an intertemporal monetary choice task. We found that a region of left dorsolateral prefrontal cortex (dlPFC) BA-46 was more active in trials where subjects chose delayed rewards, after controlling for the subjective value of those rewards. We also found that the connectivity from dlPFC BA-46 to a region of ventromedial prefrontal cortex (vmPFC) widely associated with the computation of stimulus values, increased at the time of choice, and especially during trials in which subjects chose delayed rewards. Finally, we found that estimates of effective connectivity between these two regions played a critical role in predicting out-of-sample, between-subject differences in discount rates. Together with previous findings in dietary choice, these results suggest that a common set of computational and neurobiological mechanisms facilitate choices in favor of long-term reward in both settings.
... The remaining participant was used as an independent test set. This procedure (leave-oneparticipant-out cross-validation with N = 22 folds; Clithero, Smith, Carter, & Huettel, 2011) was repeated until every participant was used in the test set once. The classification analysis was performed separately for seven ROIs identified by the functional localizer and two taken from the literature, which correspond to the putative location of mirror neurons in the lateral pFC (Oosterhof, Tipper, & Downing, 2012;Kilner, Neal, Wiskopf, Friston, & Frith, 2009). ...
Article
Full-text available
Language content and action/perception have been shown to activate common brain areas in previous neuroimaging studies. However, it is unclear whether overlapping cortical activation reflects a common neural source or adjacent, but distinct, sources. We address this issue by using multivoxel pattern analysis on fMRI data. Specifically, participants were instructed to engage in five tasks: (1) execute hand actions (AE), (2) observe hand actions (AO), (3) observe nonbiological motion (MO), (4) read action verbs, and (5) read nonaction verbs. A classifier was trained to distinguish between data collected from neural motor areas during (1) AE versus MO and (2) AO versus MO. These two algorithms were then used to test for a distinction between data collected during the reading of action versus nonaction verbs. The results show that the algorithm trained to distinguish between AE and MO distinguishes between word categories using signal recorded from the left parietal cortex and pre-SMA, but not from ventrolateral premotor cortex. In contrast, the algorithm trained to distinguish between AO and MO discriminates between word categories using the activity pattern in the left premotor and left parietal cortex. This shows that the sensitivity of premotor areas to language content is more similar to the process of observing others acting than to acting oneself. Furthermore, those parts of the brain that show comparable neural pattern for action execution and action word comprehension are high-level integrative motor areas rather than low-level motor areas.
... The statistical significance of the pooled sample accuracy can be assessed using a simple classical binomial test (assuming the standard case of binary classification) that is based on the likelihood of obtaining the observed number of correct trials (or more) by chance (Langford, 2005). A less frequent variant of this analysis uses the average sample accuracy instead of the pooled sample accuracy (Clithero et al., 2011). A second approach, more commonly used, is to consider subjectspecific sample accuracies and estimate their distribution in the population . ...
Article
Full-text available
Multivariate classification algorithms are powerful tools for predicting cognitive or pathophysiological states from neuroimaging data. Assessing the utility of a classifier in application domains such as cognitive neuroscience, brain-computer interfaces, or clinical diagnostics necessitates inference on classification performance at more than one level, i.e., both in individual subjects and in the population from which these subjects were sampled. Such inference requires models that explicitly account for both fixed-effects (within-subjects) and random-effects (between-subjects) variance components. While models of this sort are standard in mass-univariate analyses of fMRI data, they have not yet received much attention in multivariate classification studies of neuroimaging data, presumably because of the high computational costs they entail. This paper extends a recently developed hierarchical model for mixed-effects inference in multivariate classification studies and introduces an efficient variational Bayes approach to inference. Using both synthetic and empirical fMRI data, we show that this approach is equally simple to use as, yet more powerful than, a conventional t-test on subject-specific sample accuracies, and computationally much more efficient than previous sampling algorithms and permutation tests. Our approach is independent of the type of underlying classifier and thus widely applicable. The present framework may help establish mixed-effects inference as a future standard for classification group analyses.
... To address these aims, we modified a previously used paradigm 11 , in which we optimized the design for multivoxel pattern analysis (MVPA) techniques. MPVA has been applied in many decision-making paradigms; economic value 17 , associative value 18 , reward modality 19 , value-based decisions 20 and consumer choices 21,22 have all been decoded from fMRI data. In this study, participants were scanned with fMRI while they reported their 'willingness to pay' (a proxy measure of their stimulus valuation obtained via a Becker-DeGroot-Marschack auction process 23 ) for three different classes of goods: food, money and noncomestible consumer items or 'trinkets' (Fig. 1a). ...
Article
Full-text available
To choose between manifestly distinct options, it is suggested that the brain assigns values to goals using a common currency. Although previous studies have reported activity in ventromedial prefrontal cortex (vmPFC) correlating with the value of different goal stimuli, it remains unclear whether such goal-value representations are independent of the associated stimulus categorization, as required by a common currency. Using multivoxel pattern analyses on functional magnetic resonance imaging (fMRI) data, we found a region of medial prefrontal cortex to contain a distributed goal-value code that is independent of stimulus category. More ventrally in the vmPFC, we found spatially distinct areas of the medial orbitofrontal cortex to contain unique category-dependent distributed value codes for food and consumer items. These results implicate the medial prefrontal cortex in the implementation of a common currency and suggest a ventral versus dorsal topographical organization of value signals in the vmPFC.
... Certainly such cross-learning is more challenging (Wang et al., 2004; Aron et al., 2006; Lee et al., 2009) and typically yields lower classification accuracies, perhaps due to differences in experimental paradigm, but also more prosaic discrepancies in the shape and timing of the BOLD responses across participants (Aguirre et al., 1998; Duann et al., 2002; Handwerker et al., 2004) and sessions (McGonigle et al., 2000; Smith et al., 2005). But assuming a shared semantic basis the similarity structure should show some consistency (Wang et al., 2004; Kriegeskorte and Bandettini, 2007a,b; Clithero et al., 2011; Haxby et al., 2011). ...
Article
Full-text available
Both embodied and symbolic accounts of conceptual organization would predict partial sharing and partial differentiation between the neural activations seen for concepts activated via different stimulus modalities. But cross-participant and cross-session variability in BOLD activity patterns makes analyses of such patterns with MVPA methods challenging. Here, we examine the effect of cross-modal and individual variation on the machine learning analysis of fMRI data recorded during a word property generation task. We present the same set of living and non-living concepts (land-mammals, or work tools) to a cohort of Japanese participants in two sessions: the first using auditory presentation of spoken words; the second using visual presentation of words written in Japanese characters. Classification accuracies confirmed that these semantic categories could be detected in single trials, with within-session predictive accuracies of 80-90%. However cross-session prediction (learning from auditory-task data to classify data from the written-word-task, or vice versa) suffered from a performance penalty, achieving 65-75% (still individually significant at p « 0.05). We carried out several follow-on analyses to investigate the reason for this shortfall, concluding that distributional differences in neither time nor space alone could account for it. Rather, combined spatio-temporal patterns of activity need to be identified for successful cross-session learning, and this suggests that feature selection strategies could be modified to take advantage of this.
Preprint
Full-text available
High-profile studies claim to assess mental states across individuals using multi-voxel decoders of brain activity. The fixed, fine-grained, multi-voxel patterns in these "optimized" decoders are purportedly necessary for discriminating between, and accurately identifying, mental states. Here, we present compelling evidence that the efficacy of these decoders is overstated. Across a variety of tasks, decoder patterns were not necessary. Not only were "optimized decoders" spatially imprecise and 90% redundant, but they also performed similarly to simpler decoders, built from average brain activity. We distinguish decoder performance when used for discriminating between, in contrast to identifying, mental states, and show even when discrimination performance is strong, identification can be poor. Using similarity rules, we derived novel and intuitive discriminability metrics that capture 95% and 68% of discrimination performance within- and across-subjects, respectively. These findings demonstrate that current across-subject decoders remain inadequate for real-life decision making.
Article
Background and objective: In medical imaging, population studies have to overcome the differences that exist between individuals to identify invariant image features that can be used for diagnosis purposes. In functional neuroimaging, an appealing solution to identify neural coding principles that hold at the population level is inter-subject pattern analysis, i.e. to learn a predictive model on data from multiple subjects and evaluate its generalization performance on new subjects. Although it has gained popularity in recent years, its widespread adoption is still hampered by the blatant lack of a formal definition in the literature. In this paper, we precisely introduce the first principled formalization of inter-subject pattern analysis targeted at multivariate group analysis of functional neuroimaging. Methods: We propose to frame inter-subject pattern analysis as a multi-source transductive transfer question, thus grounding it within several well defined machine learning settings and broadening the spectrum of usable algorithms. We describe two sets of inter-subject brain decoding experiments that use several open datasets: a magneto-encephalography study with 16 subjects and a functional magnetic resonance imaging paradigm with 100 subjects. We assess the relevance of our framework by performing model comparisons, where one brain decoding model exploits our formalization while others do not. Results: The first set of experiments demonstrates the superiority of a brain decoder that uses subject-by-subject standardization compared to state of the art models that use other standardization schemes, making the case for the interest of the transductive and the multi-source components of our formalization The second set of experiments quantitatively shows that, even after such transformation, it is more difficult for a brain decoder to generalize to new participants rather than to new data from participants available in the training phase, thus highlighting the transfer gap that needs to be overcome. Conclusion: This paper describes the first formalization of inter-subject pattern analysis as a multi-source transductive transfer learning problem. We demonstrate the added value of this formalization using proof-of-concept experiments on several complementary functional neuroimaging datasets. This work should contribute to popularize inter-subject pattern analysis for functional neuroimaging population studies and pave the road for future methodological innovations.
Article
Full-text available
Reward processing is a fundamental human activity. The basal ganglia are recognized for their role in reward processes; however, specific roles of the different nuclei (e.g., nucleus accumbens, caudate, putamen and globus pallidus) remain unclear. Using quantitative meta-analyses we assessed whole-brain and basal ganglia specific contributions to money, erotic, and food reward processing. We analyzed data from 190 fMRI studies which reported stereotaxic coordinates of whole-brain, within-group results from healthy adult participants. Results showed concordance in overlapping and distinct cortical and sub-cortical brain regions as a function of reward type. Common to all reward types was concordance in basal ganglia nuclei, with distinct differences in hemispheric dominance and spatial extent in response to the different reward types. Food reward processing favored the right hemisphere; erotic rewards favored the right lateral globus pallidus and left caudate body. Money rewards engaged the basal ganglia bilaterally including its most anterior part, nucleus accumbens. We conclude by proposing a model of common reward processing in the basal ganglia and separate models for money, erotic, and food rewards.
Article
Humans are social creatures, engaging almost constantly in social behaviors that serve ultimate social goals, such as forming strong bonds with one another. However, most social behaviors provide only incremental progress toward an ultimate goal. Instead, the drive to engage in any individual social act may derive from its proximal value rather than its ultimate goal. Thus, this proximal value forms the foundation on which the complexities of human sociality are built. We describe two complementary approaches for using proximal social rewards to understand social behaviors and their ultimate goals: (a) decontextualizing social rewards-paring down complex social interactions can help identify which basic building blocks remain valuable even in minimalistic contexts-and (b) recontextualizing social rewards-reintroducing motivational and contextual factors into the study of social experience can help identify how proximal rewards serve their ultimate function. We discuss how this dual-approach framework can inform future research by bridging basic social building blocks and real-world social goals.
Article
Full-text available
The prefrontal cortex (pFC) is central to flexible, goal-directed cognition, and understanding its representational code is an important problem in cognitive neuroscience. In humans, multivariate pattern analysis (MVPA) of fMRI blood oxygenation level-dependent (BOLD) measurements has emerged as an important approach for studying neural representations. Many previous studies have implicitly assumed that MVPA of fMRI BOLD is just as effective in decoding information encoded in pFC neural activity as it is in visual cortex. However, MVPA studies of pFC have had mixed success. Here we estimate the base rate of decoding information from pFC BOLD activity patterns from a meta-analysis of published MVPA studies. We show that pFC has a significantly lower base rate (55.4%) than visual areas in occipital (66.6%) and temporal (71.0%) cortices and one that is close to chance levels. Our results have implications for the design and interpretation of MVPA studies of pFC and raise important questions about its functional organization.
Preprint
Full-text available
Current neurobiological models assign a central role to predictive processes calibrated to environmental statistics. Neuroimaging studies examining the encoding of stimulus uncertainty have relied almost exclusively on manipulations in which stimuli were presented in a single sensory modality, and further assumed that neural responses vary monotonically with uncertainty. This has left a gap in theoretical development with respect to two core issues: i) are there cross-modal brain systems that encode input uncertainty in way that generalizes across sensory modalities, and ii) are there brain systems that track input uncertainty in a non-monotonic fashion? We used multivariate pattern analysis to address these two issues using auditory, visual and audiovisual inputs. We found signatures of cross-modal encoding in frontoparietal, orbitofrontal, and association cortices using a searchlight cross-classification analysis where classifiers trained to discriminate levels of uncertainty in one modality were tested in another modality. Additionally, we found widespread systems encoding uncertainty non-monotonically using classifiers trained to discriminate intermediate levels of uncertainty from both the highest and lowest uncertainty levels. These findings comprise the first comprehensive report of cross-modal and non-monotonic neural sensitivity to statistical regularities in the environment, and suggest that conventional paradigms testing for monotonic responses to uncertainty in a single sensory modality may have limited generalizability.
Article
Full-text available
In current volatile environment, advanced marketing research methods are needed to identify preferences or constraints for fulfilling the needs of customers. Neuromarketing research tools have capacity to study the brain of consumers and may provide answer of many unanswered questions related with consumers. This research demonstrates the complementary role of neuromarketing techniques played in understanding the aspects of consumer behavior. It focuses on the prospective usage of Eye Tracking as techniques of neuromarketing for conducting market research. This research is conceptual in nature and based on literature review. The outcomes of this paper suggest prospective uses for Eye Tracking in marketing strategies, such as segmentation, targeting and positioning. It can be alleged that in the current and upcoming scenarios neuromarketing techniques especially Eye Tracking will be a part of marketing research. Keywords: neuromarketing, eye tracking, visual attention, consumer behavior, marketing research
Article
Information about potential rewards in the environment is essential for guiding adaptive behavior, and understanding neural reward processes may provide insights into neuropsychiatric dysfunctions. Over the past 10 years, multivoxel pattern analysis (MVPA) techniques have been used to study brain areas encoding information about expected and experienced outcomes. These studies have identified reward signals throughout the brain, including the striatum, medial prefrontal cortex, orbitofrontal cortex, dorsolateral prefrontal cortex, and parietal cortex. This review article discusses some of the assumptions and models that are used to interpret results from these MVPA studies, and how they relate to findings from animal electrophysiology. The article reviews and summarizes some of the key findings from MVPA studies on reward. In particular, it first focuses on studies that, in addition to mapping out the brain areas that process rewards, have provided novel insights into the coding mechanisms of value and reward. Finally, it discusses examples of how multivariate imaging approaches are being used more recently to decode features of expected rewards that go beyond value, such as the identity of the expected outcome or the action required to obtain it. The study of such complex and multifaceted reward representations highlights the key advantage of using representational methods, which are uniquely able to reveal these signals and may narrow the gap between animal and human research. Applied in a clinical context, MVPA may advance our understanding of neuropsychiatric disorders and the development of novel treatment strategies.
Article
Significance statement: We address a key question in predictive visual cognition: how does the brain combine multiple concurrent expectations for different features of a single object, like its color and motion trajectory? By combining a behavioral protocol that independently varies expectation of (and attention to) multiple object features with computational modeling and function magnetic resonance imaging (fMRI), we demonstrate that behavior and fMRI activity patterns in visual cortex are best accounted for by a model where prediction error in one object feature spreads to other object features. These results demonstrate how predictive vision forms object-level expectations out of multiple independent features.
Article
Full-text available
While there is accumulating evidence for the existence of distinct neural systems supporting goal-directed and habitual action selection in the mammalian brain, much less is known about the nature of the information being processed in these different brain regions. Associative learning theory predicts that brain systems involved in habitual control, such as the dorsolateral striatum, should contain stimulus and response information only, but not outcome information, while regions involved in goal-directed action, such as ventromedial and dorsolateral prefrontal cortex and dorsomedial striatum, should be involved in processing information about outcomes as well as stimuli and responses. To test this prediction, human participants underwent fMRI while engaging in a binary choice task designed to enable the separate identification of these different representations with a multivariate classification analysis approach. Consistent with our predictions, the dorsolateral striatum contained information about responses but not outcomes at the time of an initial stimulus, while the regions implicated in goal-directed action selection contained information about both responses and outcomes. These findings suggest that differential contributions of these regions to habitual and goal-directed behavioral control may depend in part on basic differences in the type of information that these regions have access to at the time of decision making. Copyright © 2015 the authors 0270-6474/15/353764-08$15.00/0.
Article
Full-text available
The first decade of consumer neuroscience research has produced groundbreaking work in identifying the basic neural processes underlying human judgment and decision making, with the majority of such studies published in neuroscience journals and influencing models of brain function. Yet for the field of consumer neuroscience to thrive in the next decade, the current emphasis on basic science research must be extended into marketing theory and practice. The authors suggest five concrete ways that neuroscientific methods can be fruitfully applied to marketing. They then outline three fundamental challenges facing consumer neuroscientists and offer potential solutions for addressing them. The authors conclude by describing how consumer neuroscience can become an important complement to research and practice in marketing.
Article
Full-text available
Attention, the prioritization of goal-relevant stimuli, and expectation, the modulation of stimulus processing by probabilistic context, represent the two main endogenous determinants of visual cognition. Neural selectivity in visual cortex is enhanced for both attended and expected stimuli, but the functional relationship between these mechanisms is poorly understood. Here, we adjudicated between two current hypotheses of how attention relates to predictive processing, namely, that attention either enhances or filters out perceptual prediction errors (PEs), the PE-promotion model versus the PE-suppression model. We acquired fMRI data from category-selective visual regions while human subjects viewed expected and unexpected stimuli that were either attended or unattended. Then, we trained multivariate neural pattern classifiers to discriminate expected from unexpected stimuli, depending on whether these stimuli had been attended or unattended. If attention promotes PEs, then this should increase the disparity of neural patterns associated with expected and unexpected stimuli, thus enhancing the classifier's ability to distinguish between the two. In contrast, if attention suppresses PEs, then this should reduce the disparity between neural signals for expected and unexpected percepts, thus impairing classifier performance. We demonstrate that attention greatly enhances a neural pattern classifier's ability to discriminate between expected and unexpected stimuli in a region- and stimulus category-specific fashion. These findings are incompatible with the PE-suppression model, but they strongly support the PE-promotion model, whereby attention increases the precision of prediction errors. Our results clarify the relationship between attention and expectation, casting attention as a mechanism for accelerating online error correction in predicting task-relevant visual inputs.
Article
A variety of exciting scientific achievements have been made in the last few decades in brain encoding and decoding via functional magnetic resonance imaging (fMRI). This trend continues to rise in recent years, as evidenced by the increasing number of published papers in this topic and several published survey papers addressing different aspects of research issues. Essentially, these survey articles were mainly from cognitive neuroscience and neuroimaging perspectives, although computational challenges were briefly discussed. To complement existing survey articles, this paper focuses on the survey of the variety of image analysis methodologies, such as neuroimage registration, fMRI signal analysis, ROI (regions of interest) selection, machine learning algorithms, reproducibility analysis, structural and functional connectivity, and natural image analysis, which were employed in previous brain encoding/decoding research works. This paper also provides discussions of potential limitations of those image analysis methodologies and possible future improvements. It is hoped that extensive discussions of image analysis issues could contribute to the advancements of the increasingly important brain encoding/decoding field.
Article
Full-text available
Research in neuroscience faces the challenge of integrating information across different spatial scales of brain function. A promising technique for harnessing information at a range of spatial scales is multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data. While the prevalence of MVPA has increased dramatically in recent years, its typical implementations for classification of mental states utilize only a subset of the information encoded in local fMRI signals. We review published studies employing multivariate pattern classification since the technique's introduction, which reveal an extensive focus on the improved detection power that linear classifiers provide over traditional analysis techniques. We demonstrate using simulations and a searchlight approach, however, that non-linear classifiers are capable of extracting distinct information about interactions within a local region. We conclude that for spatially localized analyses, such as searchlight and region of interest, multiple classification approaches should be compared in order to match fMRI analyses to the properties of local circuits.
Over the past decade, pattern classification methods have become widespread in functional magnetic resonance imaging (fMRI). These methods, typically referred to as multivoxel pattern analysis (MVPA) or multivariate pattern decoding, are now applied to a wide range of neuroscientific questions. There has been particular interest in applying these approaches, e.g., in detecting deception or for diagnostic purposes. In this review, we will focus on what can be achieved by pattern classification analyses of fMRI data; the strengths and weaknesses of this approach; and the biological processes giving rise to the signals measured by this method. Finally, we will discuss how these multivariate approaches are starting to be applied to the analysis of anatomical magnetic resonance imaging (MRI) and magnetoencephalographic (MEG) data. WIREs Cogni Sci 2011 2 568-579 DOI: 10.1002/wcs.141 For further resources related to this article, please visit the WIREs website. Copyright © 2011 John Wiley & Sons, Ltd.
Article
Full-text available
Over the past decade, functional Magnetic Resonance Imaging (fMRI) has emerged as a powerful new instrument to collect vast quantities of data about activity in the human brain. A typical fMRI experiment can produce a three-dimensional image related to the human subject's brain activity every half second, at a spatial resolution of a few millimeters. As in other modern empirical sciences, this new instrumentation has led to a flood of new data, and a corresponding need for new data analysis methods. We describe recent research applying machine learning methods to the problem of classifying the cognitive state of a human subject based on fRMI data observed over a single time interval. In particular, we present case studies in which we have successfully trained classifiers to distinguish cognitive states such as (1) whether the human subject is looking at a picture or a sentence, (2) whether the subject is reading an ambiguous or non-ambiguous sentence, and (3) whether the word the subject is viewing is a word describing food, people, buildings, etc. This learning problem provides an interesting case study of classifier learning from extremely high dimensional (105 features), extremely sparse (tens of training examples), noisy data. This paper summarizes the results obtained in these three case studies, as well as lessons learned about how to successfully apply machine learning methods to train classifiers in such settings.
Article
Full-text available
The perception of faces is sometimes regarded as a specialized task involving discrete brain regions. In an attempt to identi$ face-specific cortex, we used functional magnetic resonance imaging (fMRI) to measure activation evoked by faces presented in a continuously changing montage of common objects or in a similar montage of nonobjects. Bilateral regions of the posterior fusiform gyrus were activated by faces viewed among nonobjects, but when viewed among objects, faces activated only a focal right fusiform region. To determine whether this focal activation would occur for another category of familiar stimuli, subjects viewed flowers presented among nonobjects and objects. While flowers among nonobjects evoked bilateral fusiform activation, flowers among objects evoked no activation. These results demonstrate that both faces and flowers activate large and partially overlapping regions of inferior extrastriate cortex. A smaller region, located primarily in the right lateral fusiform gyrus, is activated specifically by faces.
Article
Full-text available
Hemisphericity refers to the idea that people may rely on a preferred mode of cognitive processing, which is linked to activity on the part of the left or right cerebral hemispheres. Four methods have been used in hemisphericity research: lateral eye-movements, electrophysiological measures, questionnaires and cognitive tests. It is demonstrated that none has been properly validated with respect to the purposes for which they are employed in studies of hemisphericity. It is argued that the idea of hemisphericity lacks adequate foundation and that, because of the assumptions implicit in the idea of hemisphericity, it will never be possible to provide such a foundation. The idea is a misleading one which should be abandoned.
Article
Full-text available
Efficient recognition of odorous objects universally shapes animal behavior and is crucial for survival. To distinguish kin from nonkin, mate from nonmate and food from nonfood, organisms must be able to create meaningful perceptual representations of odor qualities and categories. It is currently unknown where and in what form the brain encodes information about odor quality. By combining functional magnetic resonance imaging (fMRI) with multivariate (pattern-based) techniques, we found that spatially distributed ensemble activity in human posterior piriform cortex (PPC) coincides with perceptual ratings of odor quality, such that odorants with more (or less) similar fMRI patterns were perceived as more (or less) alike. We did not observe these effects in anterior piriform cortex, amygdala or orbitofrontal cortex, indicating that ensemble coding of odor categorical perception is regionally specific for PPC. These findings substantiate theoretical models emphasizing the importance of distributed piriform templates for the perceptual reconstruction of odor object quality.
Article
Full-text available
Patterns of brain activity during deception have recently been characterized with fMRI on the multi-subject average group level. The clinical value of fMRI in lie detection will be determined by the ability to detect deception in individual subjects, rather than group averages. High-dimensional non-linear pattern classification methods applied to functional magnetic resonance (fMRI) images were used to discriminate between the spatial patterns of brain activity associated with lie and truth. In 22 participants performing a forced-choice deception task, 99% of the true and false responses were discriminated correctly. Predictive accuracy, assessed by cross-validation in participants not included in training, was 88%. The results demonstrate the potential of non-linear machine learning techniques in lie detection and other possible clinical applications of fMRI in individual subjects, and indicate that accurate clinical tests could be based on measurements of brain function with fMRI.
Article
Full-text available
The core feature of an economic exchange is a decision to trade one good for another, based on a comparison of relative value. Economists have long recognized, however, that the value an individual ascribes to a good during decision making (i.e., their relative willingness to trade for that good) does not always map onto the reward they actually experience. Here, we show that experienced value and decision value are represented in distinct regions of ventromedial prefrontal cortex (VMPFC) during the passive consumption of rewards. Participants viewed two categories of rewards-images of faces that varied in their attractiveness and monetary gains and losses-while being scanned using functional magnetic resonance imaging. An independent market task, in which participants exchanged some of the money that they had earned for brief views of attractive faces, determined the relative decision value associated with each category. We found that activation of anterior VMPFC increased with increasing experienced value, but not decision value, for both reward categories. In contrast, activation of posterior VMPFC predicted each individual's relative decision value for face and monetary stimuli. These results indicate not only that experienced value and decision value are represented in distinct regions of VMPFC, but also that decision value signals are evident even in the absence of an overt choice task. We conclude that decisions are made by comparing neural representations of the value of different goods encoded in posterior VMPFC in a common, relative currency.
Article
Full-text available
Although orientation columns are less than a millimeter in width, recent neuroimaging studies indicate that viewed orientations can be decoded from cortical activity patterns sampled at relatively coarse resolutions of several millimeters. One proposal is that these differential signals arise from random spatial irregularities in the columnar map. However, direct support for this hypothesis has yet to be obtained. Here, we used high-field, high-resolution functional magnetic resonance imaging (fMRI) and multivariate pattern analysis to determine the spatial scales at which orientation-selective information can be found in the primary visual cortex (V1) of cats and humans. We applied a multiscale pattern analysis approach in which fine- and coarse-scale signals were first removed by ideal spatial lowpass and highpass filters, and the residual activity patterns then analyzed by linear classifiers. Cat visual cortex, imaged at 0.3125 mm resolution, showed a strong orientation signal at the scale of individual columns. Nonetheless, reliable orientation bias could still be found at spatial scales of several millimeters. In the human visual cortex, imaged at 1 mm resolution, a majority of orientation information was found on scales of millimeters, with small contributions from global spatial biases exceeding approximately 1 cm. Our high-resolution imaging results demonstrate a reliable millimeters-scale orientation signal, likely emerging from irregular spatial arrangements of orientation columns and their supporting vasculature. fMRI pattern analysis methods are thus likely to be sensitive to signals originating from other irregular columnar structures elsewhere in the brain.
Article
Full-text available
In this, the 21st century, human-brain mapping celebrates 21 years of cognitive activation studies. This review looks at imaging neuroscience and key ideas it has pursued; some ideas portend exciting developments, and others have failed gloriously. In terms of achievements, there is much to celebrate, in the sense that it is difficult to imagine modern neuroscience without brain imaging. I will look at recent advances from the perspectives of functional segregation and integration in the brain, paying special attention to approaches that deal with the distributed and integrated nature of neuronal processing and the questions they address.
Article
Full-text available
Efficient recognition of odorous objects universally shapes animal behavior and is crucial for survival. To distinguish kin from nonkin, mate from nonmate and food from nonfood, organisms must be able to create meaningful perceptual representations of odor qualities and categories. It is currently unknown where and in what form the brain encodes information about odor quality. By combining functional magnetic resonance imaging (fMRI) with multivariate (pattern-based) techniques, we found that spatially distributed ensemble activity in human posterior piriform cortex (PPC) coincides with perceptual ratings of odor quality, such that odorants with more (or less) similar fMRI patterns were perceived as more (or less) alike. We did not observe these effects in anterior piriform cortex, amygdala or orbitofrontal cortex, indicating that ensemble coding of odor categorical perception is regionally specific for PPC. These findings substantiate theoretical models emphasizing the importance of distributed piriform templates for the perceptual reconstruction of odor object quality.
Article
Full-text available
In order for stimuli to be perceptually discriminable, their representations in the brain must be distinct. Investigating the task of discriminating the syllables /ra/ and /la/, we hypothesized that the more distinct a person's neural representations of those sounds were, the better their behavioral ability to discriminate them would be. Standard neuroimaging approaches are ill-suited to testing this hypothesis as they have problems differentiating between neural representations spatially intermingled within the same brain area. We therefore performed multi-voxel pattern-based analysis of the functional magnetic resonance imaging (fMRI) activity elicited by these syllables, in native speakers of English and Japanese. In right primary auditory cortex, the statistical separability of these fMRI patterns predicted subjects' behavioral ability to tell the sounds apart, not only across groups but also across individuals. This opens up a new approach for identifying neural representations and for quantifying their task suitability.
Article
Full-text available
Visual working memory provides an essential link between perception and higher cognitive functions, allowing for the active maintenance of information about stimuli no longer in view. Research suggests that sustained activity in higher-order prefrontal, parietal, inferotemporal and lateral occipital areas supports visual maintenance, and may account for the limited capacity of working memory to hold up to 3-4 items. Because higher-order areas lack the visual selectivity of early sensory areas, it has remained unclear how observers can remember specific visual features, such as the precise orientation of a grating, with minimal decay in performance over delays of many seconds. One proposal is that sensory areas serve to maintain fine-tuned feature information, but early visual areas show little to no sustained activity over prolonged delays. Here we show that orientations held in working memory can be decoded from activity patterns in the human visual cortex, even when overall levels of activity are low. Using functional magnetic resonance imaging and pattern classification methods, we found that activity patterns in visual areas V1-V4 could predict which of two oriented gratings was held in memory with mean accuracy levels upwards of 80%, even in participants whose activity fell to baseline levels after a prolonged delay. These orientation-selective activity patterns were sustained throughout the delay period, evident in individual visual areas, and similar to the responses evoked by unattended, task-irrelevant gratings. Our results demonstrate that early visual areas can retain specific information about visual features held in working memory, over periods of many seconds when no physical stimulus is present.
Article
Full-text available
The Python programming language is steadily increasing in popularity as the language of choice for scientific computing. The ability of this scripting environment to access a huge code base in various languages, combined with its syntactical simplicity, make it the ideal tool for implementing and sharing ideas among scientists from numerous fields and with heterogeneous methodological backgrounds. The recent rise of reciprocal interest between the machine learning (ML) and neuroscience communities is an example of the desire for an inter-disciplinary transfer of computational methods that can benefit from a Python-based framework. For many years, a large fraction of both research communities have addressed, almost independently, very high-dimensional problems with almost completely non-overlapping methods. However, a number of recently published studies that applied ML methods to neuroscience research questions attracted a lot of attention from researchers from both fields, as well as the general public, and showed that this approach can provide novel and fruitful insights into the functioning of the brain. In this article we show how PyMVPA, a specialized Python framework for machine learning based data analysis, can help to facilitate this inter-disciplinary technology transfer by providing a single interface to a wide array of machine learning libraries and neural data-processing methods. We demonstrate the general applicability and power of PyMVPA via analyses of a number of neural data modalities, including fMRI, EEG, MEG, and extracellular recordings.
Article
Full-text available
Decoding patterns of neural activity onto cognitive states is one of the central goals of functional brain imaging. Standard univariate fMRI analysis methods, which correlate cognitive and perceptual function with the blood oxygenation-level dependent (BOLD) signal, have proven successful in identifying anatomical regions based on signal increases during cognitive and perceptual tasks. Recently, researchers have begun to explore new multivariate techniques that have proven to be more flexible, more reliable, and more sensitive than standard univariate analysis. Drawing on the field of statistical learning theory, these new classifier-based analysis techniques possess explanatory power that could provide new insights into the functional properties of the brain. However, unlike the wealth of software packages for univariate analyses, there are few packages that facilitate multivariate pattern classification analyses of fMRI data. Here we introduce a Python-based, cross-platform, and open-source software toolbox, called PyMVPA, for the application of classifier-based analysis techniques to fMRI datasets. PyMVPA makes use of Python's ability to access libraries written in a large variety of programming languages and computing environments to interface with the wealth of existing machine learning packages. We present the framework in this paper and provide illustrative examples on its usage, features, and programmability.
Article
Full-text available
Conventional statistical analysis methods for functional magnetic resonance imaging (fMRI) data are very successful at detecting brain regions that are activated as a whole during specific mental activities. The overall activation of a region is usually taken to indicate involvement of the region in the task. However, such activation analysis does not consider the multivoxel patterns of activity within a brain region. These patterns of activity, which are thought to reflect neuronal population codes, can be investigated by pattern-information analysis. In this framework, a region's multivariate pattern information is taken to indicate representational content. This tutorial introduction motivates pattern-information analysis, explains its underlying assumptions, introduces the most widespread methods in an intuitive way, and outlines the basic sequence of analysis steps.
Article
Full-text available
Despite growing interest in applying machine learning to neuroimaging analyses, few studies have gone beyond classifying sensory input to directly predicting behavioral output. With spatial resolution on the order of millimeters and temporal resolution on the order of seconds, functional magnetic resonance imaging (fMRI) is a promising technology for such applications. However, fMRI data's low signal-to-noise ratio, high dimensionality, and extensive spatiotemporal correlations present formidable analytic challenges. Here, we apply different machine-learning algorithms to previously acquired data to examine the ability of fMRI activation in three regions-the nucleus accumbens (NAcc), medial prefrontal cortex (MPFC), and insula-to predict purchasing. Our goal was to improve spatiotemporal interpretability as well as classification accuracy. To this end, sparse penalized discriminant analysis (SPDA) enabled automatic selection of correlated variables, yielding interpretable models that generalized well to new data. Relative to logistic regression, linear discriminant analysis, and linear support vector machines, SPDA not only increased interpretability but also improved classification accuracy. SPDA promises to allow more precise inferences about when specific brain regions contribute to purchasing decisions. More broadly, this approach provides a general framework for using neuroimaging data to build interpretable models, including those that predict choice.
Article
Full-text available
A FUNDAMENTAL CHALLENGE FOR SYSTEMS NEUROSCIENCE IS TO QUANTITATIVELY RELATE ITS THREE MAJOR BRANCHES OF RESEARCH: brain-activity measurement, behavioral measurement, and computational modeling. Using measured brain-activity patterns to evaluate computational network models is complicated by the need to define the correspondency between the units of the model and the channels of the brain-activity data, e.g., single-cell recordings or voxels from functional magnetic resonance imaging (fMRI). Similar correspondency problems complicate relating activity patterns between different modalities of brain-activity measurement (e.g., fMRI and invasive or scalp electrophysiology), and between subjects and species. In order to bridge these divides, we suggest abstracting from the activity patterns themselves and computing representational dissimilarity matrices (RDMs), which characterize the information carried by a given representation in a brain or model. Building on a rich psychological and mathematical literature on similarity analysis, we propose a new experimental and data-analytical framework called representational similarity analysis (RSA), in which multi-channel measures of neural activity are quantitatively related to each other and to computational theory and behavior by comparing RDMs. We demonstrate RSA by relating representations of visual objects as measured with fMRI in early visual cortex and the fusiform face area to computational models spanning a wide range of complexities. The RDMs are simultaneously related via second-level application of multidimensional scaling and tested using randomization and bootstrap techniques. We discuss the broad potential of RSA, including novel approaches to experimental design, and argue that these ideas, which have deep roots in psychology and neuroscience, will allow the integrated quantitative analysis of data from all three branches, thus contributing to a more unified systems neuroscience.
Article
Full-text available
Interpreting brain image experiments requires analysis of complex, multivariate data. In recent years, one analysis approach that has grown in popularity is the use of machine learning algorithms to train classifiers to decode stimuli, mental states, behaviours and other variables of interest from fMRI data and thereby show the data contain information about them. In this tutorial overview we review some of the key choices faced in using this approach as well as how to derive statistically significant results, illustrating each point from a case study. Furthermore, we show how, in addition to answering the question of 'is there information about a variable of interest' (pattern discrimination), classifiers can be used to tackle other classes of question, namely 'where is the information' (pattern localization) and 'how is that information encoded' (pattern characterization).
Article
Full-text available
1. We have previously identified face-selective areas in the mid-fusiform and inferior temporal gyri in electrophysiological recordings made from chronically implanted subdural electrodes in epilepsy patients. In this study, functional magnetic resonance imaging (fMRI) was used to study the anatomic extent of face-sensitive brain regions and to assess hemispheric laterality. 2. A time series of 128 gradient echo echoplanar images was acquired while subjects continuously viewed an alternating series of 10 unfamiliar faces followed by 10 equiluminant scrambled faces. Each cycle of this alternating sequence lasted 12 s and each experimental run consisted of 14 cycles. The time series of each voxel was transformed into the frequency domain using Fourier analysis. Activated voxels were defined by significant peaks in their power spectra at the frequency of stimulus alternation and by a 180 degrees phase shift that followed changes in stimulus alternation order. 3. Activated voxels to faces were obtained in the fusiform and inferior temporal gyri in 9 of 12 subjects and were approximately coextensive with previously identified face-selective regions. Nine subjects also showed activation in the left or right middle occipital gyri, or in the superior temporal or lateral occipital sulci. Cortical volumes activated in the left and right hemispheres were not significantly different. Activated voxels to scrambled faces were observed in six subjects at locations mainly in the lingual gyri and collateral sulci, medial to the regions activated by faces. 4. Face stimuli activated portions of the midfusiform and inferior temporal gyri, including adjacent cortex within occipitotemporal sulci.(ABSTRACT TRUNCATED AT 250 WORDS)
Article
Full-text available
How long does it take for the human visual system to process a complex natural image? Subjectively, recognition of familiar objects and scenes appears to be virtually instantaneous, but measuring this processing time experimentally has proved difficult. Behavioural measures such as reaction times can be used, but these include not only visual processing but also the time required for response execution. However, event-related potentials (ERPs) can sometimes reveal signs of neural processing well before the motor output. Here we use a go/no-go categorization task in which subjects have to decide whether a previously unseen photograph, flashed on for just 20 ms, contains an animal. ERP analysis revealed a frontal negativity specific to no-go trials that develops roughly 150 ms after stimulus onset. We conclude that the visual processing needed to perform this highly demanding task can be achieved in under 150 ms.
Article
Despite an increasing focus on the neural basis of human decision making in neuroscience, relatively little attention has been paid to decision making in social settings. Moreover, although human social decision making has been explored in a social psychology context, few neural explanations for the observed findings have been considered. To bridge this gap and improve models of human social decision making, we investigated whether acquiring a good reputation, which is an important incentive in human social behaviors, activates the same reward circuitry as monetary rewards. In total, 19 subjects participated in functional magnetic resonance imaging (fMRI) experiments involving monetary and social rewards. The acquisition of one's good reputation robustly activated reward-related brain areas, notably the striatum, and these overlapped with the areas activated by monetary rewards. Our findings support the idea of a "common neural currency" for rewards and represent an important first step toward a neural explanation for complex human social behaviors.
Article
Although previous studies have implicated a diverse set of brain regions in reward-related decision making, it is not yet known which of these regions contain information that directly reflects a decision. Here, we measured brain activity using functional MRI in a group of subjects while they performed a simple reward-based decision-making task: probabilistic reversal-learning. We recorded brain activity from nine distinct regions of interest previously implicated in decision making and separated out local spatially distributed signals in each region from global differences in signal. Using a multivariate analysis approach, we determined the extent to which global and local signals could be used to decode subjects' subsequent behavioral choice, based on their brain activity on the preceding trial. We found that subjects' decisions could be decoded to a high level of accuracy on the basis of both local and global signals even before they were required to make a choice, and even before they knew which physical action would be required. Furthermore, the combined signals from three specific brain areas (anterior cingulate cortex, medial prefrontal cortex, and ventral striatum) were found to provide all of the information sufficient to decode subjects' decisions out of all of the regions we studied. These findings implicate a specific network of regions in encoding information relevant to subsequent behavioral choice.
Article
To make sound economic decisions, the brain needs to compute several different value-related signals. These include goal values that measure the predicted reward that results from the outcome generated by each of the actions under consideration, decision values that measure the net value of taking the different actions, and prediction errors that measure deviations from individuals' previous reward expectations. We used functional magnetic resonance imaging and a novel decision-making paradigm to dissociate the neural basis of these three computations. Our results show that they are supported by different neural substrates: goal values are correlated with activity in the medial orbitofrontal cortex, decision values are correlated with activity in the central orbitofrontal cortex, and prediction errors are correlated with activity in the ventral striatum.
Article
Neuroeconomics is the study of the neurobiological and computational basis of value-based decision making. Its goal is to provide a biologically based account of human behaviour that can be applied in both the natural and the social sciences. This Review proposes a framework to investigate different aspects of the neurobiology of decision making. The framework allows us to bring together recent findings in the field, highlight some of the most important outstanding problems, define a common lexicon that bridges the different disciplines that inform neuroeconomics, and point the way to future applications.
Article
It has been proposed that activity in human primary visual cortex (V1) is necessary but not sufficient for conscious awareness, but direct physiological evidence for the activation of human V1 in the absence of awareness has been elusive. Here, we used functional MRI to measure activity in V1 while participants viewed masked and invisible orientated gratings. Remarkably, when participants were unable to report the orientation of a masked grating they were viewing, we could nevertheless predict its orientation significantly better than chance, based only on a single brief measurement of activity from their primary visual cortex. This was achieved by using multivariate pattern-recognition to substantially improve the ability to detect orientation signals in V1 (see also Kamitani & Tong, VSS 2004). Thus, activity in primary visual cortex conveys orientation-specific information that is inaccessible to conscious report. Orientation can therefore be represented unconsciously in V1, suggesting that information must be relayed to another region of the brain to be represented in conscious experience.
Article
The human orbitofrontal cortex is an important brain region for the processing of rewards and punishments, which is a prerequisite for the complex and flexible emotional and social behaviour which contributes to the evolutionary success of humans. Yet much remains to be discovered about the functions of this key brain region, and new evidence from functional neuroimaging and clinical neuropsychology is affording new insights into the different functions of the human orbitofrontal cortex. We review the neuroanatomical and neuropsychological literature on the human orbitofrontal cortex, and propose two distinct trends of neural activity based on a meta-analysis of neuroimaging studies. One is a mediolateral distinction, whereby medial orbitofrontal cortex activity is related to monitoring the reward value of many different reinforcers, whereas lateral orbitofrontal cortex activity is related to the evaluation of punishers which may lead to a change in ongoing behaviour. The second is a posterior–anterior distinction with more complex or abstract reinforcers (such as monetary gain and loss) represented more anteriorly in the orbitofrontal cortex than simpler reinforcers such as taste or pain. Finally, we propose new neuroimaging methods for obtaining further evidence on the localisation of function in the human orbitofrontal cortex.
Article
The techniques available for the interrogation and analysis of neuroimaging data have a large influence in determining the flexibil- ity, sensitivity and scope of neuroimaging experiments. The development of such methodologies has allowed investigators to address scientific questions which could not previously be answered and, as such, has become an important research area in its own right. In this paper, we present a review of the research carried out by the Analysis Group at the Oxford Centre for Functional MRI of the Brain (FMRIB). This research has focussed on the development of new methodologies for the analysis of both structural and functional magnetic resonance imaging data . The majority of the research laid out in this paper has been implemented as freely available software tools within FMRIB's Software Library (FSL).