Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies

Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA 94720, USA.
Current biology: CB (Impact Factor: 9.57). 09/2011; 21(19):1641-6. DOI: 10.1016/j.cub.2011.08.031
Source: PubMed


Quantitative modeling of human brain activity can provide crucial insights about cortical representations [1, 2] and can form the basis for brain decoding devices [3-5]. Recent functional magnetic resonance imaging (fMRI) studies have modeled brain activity elicited by static visual patterns and have reconstructed these patterns from brain activity [6-8]. However, blood oxygen level-dependent (BOLD) signals measured via fMRI are very slow [9], so it has been difficult to model brain activity elicited by dynamic stimuli such as natural movies. Here we present a new motion-energy [10, 11] encoding model that largely overcomes this limitation. The model describes fast visual information and slow hemodynamics by separate components. We recorded BOLD signals in occipitotemporal visual cortex of human subjects who watched natural movies and fit the model separately to individual voxels. Visualization of the fit models reveals how early visual areas represent the information in movies. To demonstrate the power of our approach, we also constructed a Bayesian decoder [8] by combining estimated encoding models with a sampled natural movie prior. The decoder provides remarkable reconstructions of the viewed movies. These results demonstrate that dynamic brain activity measured under naturalistic conditions can be decoded using current fMRI technology.

Download full-text


Available from: An Thanh Vu, Oct 10, 2015
71 Reads
  • Source
    • "Understanding brain function during continuous, complex ongoing sensory stimulation like that experienced in the natural world is a challenge for neuroscientific research. In order to address this challenge, there has been increasing activity in the development of tools to identify the brain mechanisms activated by complex audiovisual stimuli, such as movies (Bartels and Zeki, 2004; Hasson et al., 2004; Kauppi et al., 2010; Nishimoto et al., 2011; Zacks et al., 2001). These movies, particularly those involving feature films, typically depict complex scenes that change over time and incorporate extensive use of video editing. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Intersubject correlation (ISC) analysis of functional magnetic resonance imaging (fMRI) data provides insight into how continuous streams of sensory stimulation are processed by groups of observers. Although edited movies are frequently used as stimuli in ISC studies, there has been little direct examination of the effect of edits on the resulting ISC maps. In this study we showed 16 observers two audiovisual movie versions of the same dance. In one experimental condition there was a continuous view from a single camera (Unedited condition) and in the other condition there were views from different cameras (Edited condition) that provided close up views of the feet or face and upper body. We computed ISC maps for each condition, as well as created a map that showed the difference between the conditions. The results from the Unedited and Edited maps largely overlapped in the occipital and temporal cortices, although more voxels were found for the Edited map. The difference map revealed greater ISC for the Edited condition in the Postcentral Gyrus, Lingual Gyrus, Precentral Gyrus and Medial Frontal Gyrus, while the Unedited condition showed greater ISC in only the Superior Temporal Gyrus. These findings suggest that the visual changes associated with editing provide a source of correlation in maps obtained from edited film, and highlight the utility of using maps to evaluate the difference in ISC between conditions. Copyright © 2015. Published by Elsevier Ltd.
    Cortex 06/2015; 71. DOI:10.1016/j.cortex.2015.06.026 · 5.13 Impact Factor
  • Source
    • "Previous work on decoding of brain signals has shown that it is important to incorporate prior knowledge in order to improve reconstruction quality [5], [6], [8]. Furthermore, purely discriminative models that do not make use of prior knowledge have been shown to yield less accurate reconstructions [4]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent work has shown that it is possible to reconstruct perceived stimuli from human brain activity. At the same time, studies have indicated that perception and imagery share the same neural substrate. This could bring cognitive brain computer interfaces (BCIs) that are driven by direct readout of mental images within reach. A desirable feature of such BCIs is that subjects gain the ability to construct arbitrary messages. In this study, we explore whether words can be generated from neural activity patterns that reflect the perception of individual characters. To this end, we developed a graphical model where low-level properties of individual characters are represented via Gaussian mixture models and high-level properties reflecting character co-occurrences are represented via a hidden Markov model. With this work we provide the initial outline of a model that could allow the development of cognitive BCIs driven by direct decoding of internally generated messages.
    Pattern Recognition in NeuroImaging, Stanford; 06/2015
  • Source
    • "The Gallant lab at UC Berkeley has conducted a program of research that utilizes decoding extensively and to great effect (Huth, Nishimoto, Vu, & Gallant, 2012; Naselaris, Prenger, Kay, Oliver, & Gallant, 2009; Naeselaris et al., 2011), including predicting visual features of movies from brain activity with surprising fidelity (Nishimoto et al., 2011). For instance, using a collection of natural images collected from the Internet, they produced a Bayesian reconstruction algorithm that selects the known image which is most structurally and semantically similar to a randomly-chosen image presented to a participant in an fMRI scanner. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Core communication research questions are increasingly being investigated using brain imaging techniques. A majority of these studies apply a functional magnetic resonance imaging (fMRI) approach. This trend raises two important questions that we address in this article. First, under what conditions can fMRI methodology increase knowledge and refine communication theory? Second, how can editors, reviewers, and readers of communication journals discriminate sound and relevant fMRI research from unsound or irrelevant fMRI research? To address these questions, we first discuss what can and cannot be accomplished with fMRI. Subsequently, we provide a pragmatic introduction to fMRI data collection and analysis for social-science-oriented communication scholars. We include practical guidelines and a checklist for reporting and evaluating fMRI studies.
    Communication Methods and Measures 03/2015; 9(1-2):5-29. DOI:10.1080/19312458.2014.999754
Show more