Linear Reconstruction of Perceived Images from Human Brain Activity.

Radboud University Nijmegen, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands. Electronic address: .
NeuroImage (Impact Factor: 6.36). 07/2013; 83. DOI: 10.1016/j.neuroimage.2013.07.043
Source: PubMed


With the advent of sophisticated acquisition and analysis techniques, decoding the contents of someone's experience has become a reality. We propose a straightforward linear Gaussian approach, where decoding relies on the inversion of properly regularized encoding models, which can still be solved analytically. In order to test our approach we acquired functional magnetic resonance imaging data under a rapid event-related design in which subjects were presented with handwritten characters. Our approach is shown to yield state-of-the-art reconstructions of perceived characters as estimated from BOLD responses. This even holds for previously unseen characters. We propose that this framework serves as a baseline with which to compare more sophisticatedmodels for which analytical inversion is infeasible.

Download full-text


Available from: Marcel van Gerven,
  • Source
    • "We tested the model on a previously acquired and preprocessed fMRI dataset [8] "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent work has shown that it is possible to reconstruct perceived stimuli from human brain activity. At the same time, studies have indicated that perception and imagery share the same neural substrate. This could bring cognitive brain computer interfaces (BCIs) that are driven by direct readout of mental images within reach. A desirable feature of such BCIs is that subjects gain the ability to construct arbitrary messages. In this study, we explore whether words can be generated from neural activity patterns that reflect the perception of individual characters. To this end, we developed a graphical model where low-level properties of individual characters are represented via Gaussian mixture models and high-level properties reflecting character co-occurrences are represented via a hidden Markov model. With this work we provide the initial outline of a model that could allow the development of cognitive BCIs driven by direct decoding of internally generated messages.
    Pattern Recognition in NeuroImaging, Stanford; 06/2015
  • Source
    • "penalizing squared differences between adjacent voxels) is a special case of a more general definition given in Grosenick et al. (2013). This definition offers versatile applications, for instance in connectivity analysis (Watanabe et al., 2014) and encoding models (Schoenmakers et al., 2013). Furthermore, Michel et al. (2011) introduced Total Variation (TV) as an alternative type of spatial regularization for fMRI. "
    [Show abstract] [Hide abstract]
    ABSTRACT: By exploiting information that is contained in the spatial arrangement of neural activations, multivariate pattern analysis (MVPA) can detect distributed brain activations which are not accessible by standard univariate analysis. Recent methodological advances in MVPA regularization techniques have made it feasible to produce sparse discriminative whole-brain maps with highly specific patterns. Furthermore, the most recent refinement, the Graph Net, explicitly takes the 3D-structure of fMRI data into account. Here, these advanced classification methods were applied to a large fMRI sample (N = 70) in order to gain novel insights into the functional localization of outcome integration processes. While the beneficial effect of differential outcomes is well-studied in trial-and-error learning, outcome integration in the context of instruction-based learning has remained largely unexplored. In order to examine neural processes associated with outcome integration in the context of instruction-based learning, two groups of subjects underwent functional imaging while being presented with either differential or ambiguous outcomes following the execution of varying stimulus–response instructions. While no significant univariate group differences were found in the resulting fMRI dataset, L1-regularized (sparse) classifiers performed significantly above chance and also clearly outperformed the standard L2-regularized (dense) Support Vector Machine on this whole-brain between-subject classification task. Moreover, additional L2-regularization via the Elastic Net and spatial regularization by the Graph Net improved interpretability of discriminative weight maps but were accompanied by reduced classification accuracies. Most importantly, classification based on sparse regularization facilitated the identification of highly specific regions differentially engaged under ambiguous and differential outcome conditions, comprising several prefrontal regions previously associated with probabilistic learning, rule integration and reward processing. Additionally, a detailed post-hoc analysis of these regions revealed that distinct activation dynamics underlay the processing of ambiguous relative to differential outcomes. Together, these results show that L1-regularization can improve classification performance while simultaneously providing highly specific and interpretable discriminative activation patterns.
    NeuroImage 10/2014; 104. DOI:10.1016/j.neuroimage.2014.10.025 · 6.36 Impact Factor
  • Source
    • "Most previous studies have used fMRI to decode visual information (Brouwer and Heeger, 2009; Haynes and Rees, 2005; Kamitani and Tong, 2005; Kay et al., 2008; Miyawaki et al., 2008; Naselaris et al., 2009; Nishimoto et al., 2011; Schoenmakers et al., 2013; Thirion et al., 2006). Rather than the metabolic correlates of fMRI data, this study utilized neural activity measurements of MEG to enable an accurate reconstruction of visual patterns in a trial-by-trial manner. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Visual decoding and encoding are crucial aspects in investigating the representation of visual information in the human brain. This paper proposes a bidirectional model for decoding and encoding of visual stimulus based on manifold representation of the temporal and spatial information extracted from magnetoencephalographic data. In the proposed decoding process, principal component analysis is applied to extract temporal principal components (TPCs) from the visual cortical activity estimated by a beamforming method. The spatial distribution of each TPC is in a high-dimensional space and can be mapped to the corresponding spatiotemporal component (STC) on a low-dimensional manifold. Once the linear mapping between the STC and the wavelet coefficients of the stimulus image is determined, the decoding process can synthesize an image resembling the stimulus image. The encoding process is performed by reversing the mapping or transformation in the decoding model and can predict the spatiotemporal brain activity from a stimulus image. In our experiments using visual stimuli containing eleven combinations of checkerboard patches, the information of spatial layout in the stimulus image was revealed in the embedded manifold. The correlation between the reconstructed and original images was 0.71 and the correlation map between the predicted and original brain activity was highly correlated to the map between the original brain activity for different stimuli (r=0.89). These results suggest that the temporal component is important in visual processing and manifolds can well represent the information related to visual perception.
    NeuroImage 07/2014; 102. DOI:10.1016/j.neuroimage.2014.07.046 · 6.36 Impact Factor
Show more