Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies

Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA 94720, USA.
Current biology: CB (Impact Factor: 9.57). 09/2011; 21(19):1641-6. DOI: 10.1016/j.cub.2011.08.031
Source: PubMed


Quantitative modeling of human brain activity can provide crucial insights about cortical representations [1, 2] and can form the basis for brain decoding devices [3-5]. Recent functional magnetic resonance imaging (fMRI) studies have modeled brain activity elicited by static visual patterns and have reconstructed these patterns from brain activity [6-8]. However, blood oxygen level-dependent (BOLD) signals measured via fMRI are very slow [9], so it has been difficult to model brain activity elicited by dynamic stimuli such as natural movies. Here we present a new motion-energy [10, 11] encoding model that largely overcomes this limitation. The model describes fast visual information and slow hemodynamics by separate components. We recorded BOLD signals in occipitotemporal visual cortex of human subjects who watched natural movies and fit the model separately to individual voxels. Visualization of the fit models reveals how early visual areas represent the information in movies. To demonstrate the power of our approach, we also constructed a Bayesian decoder [8] by combining estimated encoding models with a sampled natural movie prior. The decoder provides remarkable reconstructions of the viewed movies. These results demonstrate that dynamic brain activity measured under naturalistic conditions can be decoded using current fMRI technology.

Download full-text


Available from: An Thanh Vu
  • Source
    • "Neural decoding approaches have demonstrated the efficacy of using statistical prediction models trained with brain activity associated with a task to decode subjective contents of task parameters (Kamitani and Tong, 2005), move robotic limbs (Wessberg et al., 2000;Hochberg et al., 2006Hochberg et al., , 2012Schwartz et al., 2006), reconstruct visually presented stimuli (Miyawaki et al., 2008), and elicit the contents of dreams (Horikawa et al., 2013). Neural encoding approaches, on the other hand, have shown that brain activity can be matched to databases of images (Kay et al., 2008) and videos (Nishimoto et al., 2011). While data-driven approaches have proven useful for revealing to some extent the structure of information representation in the brain, performing experiments is costly and time consuming, and often has an associated moral cost, such as when experiments result in the death or reduced lifespan of animals. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Data-driven neuroscience aims to find statistical relationships between brain activity and task behavior from large-scale datasets. To facilitate high-throughput data processing and modeling, we created BrainLiner as a web platform for sharing time-aligned, brain-behavior data. Using an HDF5-based data format, BrainLiner treats brain activity and data related to behavior with the same salience, aligning both behavioral and brain activity data on a common time axis. This facilitates learning the relationship between behavior and brain activity. Using a common data file format also simplifies data processing and analyses. Properties describing data are unambiguously defined using a schema, allowing machine-readable definition of data. The BrainLiner platform allows users to upload and download data, as well as to explore and search for data from the web platform. A WebGL-based data explorer can visualize highly detailed neurophysiological data from within the web browser, and a data-driven search feature allows users to search for similar time windows of data. This increases transparency, and allows for visual inspection of neural coding. BrainLiner thus provides an essential set of tools for data sharing and data-driven modeling.
    Preview · Article · Jan 2016 · Frontiers in Neuroinformatics
  • Source
    • "Understanding brain function during continuous, complex ongoing sensory stimulation like that experienced in the natural world is a challenge for neuroscientific research. In order to address this challenge, there has been increasing activity in the development of tools to identify the brain mechanisms activated by complex audiovisual stimuli, such as movies (Bartels and Zeki, 2004; Hasson et al., 2004; Kauppi et al., 2010; Nishimoto et al., 2011; Zacks et al., 2001). These movies, particularly those involving feature films, typically depict complex scenes that change over time and incorporate extensive use of video editing. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Intersubject correlation (ISC) analysis of functional magnetic resonance imaging (fMRI) data provides insight into how continuous streams of sensory stimulation are processed by groups of observers. Although edited movies are frequently used as stimuli in ISC studies, there has been little direct examination of the effect of edits on the resulting ISC maps. In this study we showed 16 observers two audiovisual movie versions of the same dance. In one experimental condition there was a continuous view from a single camera (Unedited condition) and in the other condition there were views from different cameras (Edited condition) that provided close up views of the feet or face and upper body. We computed ISC maps for each condition, as well as created a map that showed the difference between the conditions. The results from the Unedited and Edited maps largely overlapped in the occipital and temporal cortices, although more voxels were found for the Edited map. The difference map revealed greater ISC for the Edited condition in the Postcentral Gyrus, Lingual Gyrus, Precentral Gyrus and Medial Frontal Gyrus, while the Unedited condition showed greater ISC in only the Superior Temporal Gyrus. These findings suggest that the visual changes associated with editing provide a source of correlation in maps obtained from edited film, and highlight the utility of using maps to evaluate the difference in ISC between conditions. Copyright © 2015. Published by Elsevier Ltd.
    Full-text · Article · Jun 2015 · Cortex
  • Source
    • "Previous work on decoding of brain signals has shown that it is important to incorporate prior knowledge in order to improve reconstruction quality [5], [6], [8]. Furthermore, purely discriminative models that do not make use of prior knowledge have been shown to yield less accurate reconstructions [4]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent work has shown that it is possible to reconstruct perceived stimuli from human brain activity. At the same time, studies have indicated that perception and imagery share the same neural substrate. This could bring cognitive brain computer interfaces (BCIs) that are driven by direct readout of mental images within reach. A desirable feature of such BCIs is that subjects gain the ability to construct arbitrary messages. In this study, we explore whether words can be generated from neural activity patterns that reflect the perception of individual characters. To this end, we developed a graphical model where low-level properties of individual characters are represented via Gaussian mixture models and high-level properties reflecting character co-occurrences are represented via a hidden Markov model. With this work we provide the initial outline of a model that could allow the development of cognitive BCIs driven by direct decoding of internally generated messages.
    Full-text · Conference Paper · Jun 2015
Show more