, 639 (2013);
et al.T. Horikawa
Neural Decoding of Visual Imagery During Sleep
This copy is for your personal, non-commercial use only.
clicking here.colleagues, clients, or customers by
, you can order high-quality copies for your
If you wish to distribute this article to others
The following resources related to this article are available online at
here.following the guidelines
can be obtained by
Permission to republish or repurpose articles or portions of articles
Updated information and services,
): December 11, 2013 www.sciencemag.org (this information is current as of
version of this article at:
including high-resolution figures, can be found in the online
can be found at:
Supporting Online Material
related to this article
A list of selected additional articles on the Science Web sites
, 10 of which can be accessed free:
cites 34 articles
2 articles hosted by HighWire Press; see:
This article has been
This article appears in the following
registered trademark of AAAS.
is aScience2013 by the American Association for the Advancement of Science; all rights reserved. The title
CopyrightAmerican Association for the Advancement of Science, 1200 New York Avenue NW, Washington, DC 20005.
(print ISSN 0036-8075; online ISSN 1095-9203) is published weekly, except the last week in December, by the Science
on December 11, 2013
on December 11, 2013
on December 11, 2013
on December 11, 2013
on December 11, 2013
antigen-presenting cells or cytokines. In either
case, the current observations exclude models in
which each naïve T cell yields progeny with the
same distribution of cells of either short- or long-
term potential. Thus, although our data do not
evaluate the role of asymmetric division as a
mechanism to generate daughter cells with dif-
ferent fates, they do show that asymmetric divi-
sion by itself cannot explain the disparity between
individual T cell families that is experimentally
observed. Rather, a strong variation between fam-
ilies in the expansion of proximal and distal
heterogeneity observed here. Lastly, the observa-
tion of strong heterogeneity at the single-cell lev-
el indicates that T cell responses are made up of
ses of stem-cell renewal (19). Thus, although the
differentiation and expansion of the combined
T cell population follow a uniform course, the
fate of individual naïve Tcells is unpredictable.
References and Notes
1. J. J. Obar, K. M. Khanna, L. Lefrançois, Immunity 28, 859
2. J. N. Blattman et al., J. Exp. Med. 195, 657 (2002).
3. K. Murali-Krishna et al., Immunity 8, 177 (1998).
4. E. A. Butz, M. J. Bevan, Immunity 8, 167 (1998).
5. M. A. Williams, M. J. Bevan, Annu. Rev. Immunol. 25,
6. S. Sarkar et al., J. Exp. Med. 205, 625 (2008).
7. J. J. Obar, L. Lefrançois, Int. Immunol. 22, 619 (2010).
8. C. Stemberger et al., Immunity 27, 985 (2007).
9. C. Gerlach et al., J. Exp. Med. 207, 1235 (2010).
10. J. T. Chang et al., Science 315, 1687 (2007).
11. J. W. J. van Heijst et al., Science 325, 1265
12. T. N. M. Schumacher, C. Gerlach, J. W. J. van Heijst,
Nat. Rev. Immunol. 10, 621 (2010).
13. K. Schepers et al., J. Exp. Med. 205, 2309 (2008).
14. Materials and methods are available as supplementary
materials on Science Online.
15. W. N. D’Souza, S. M. Hedrick, J. Immunol. 177, 777
16. T. E. Schlub et al., Eur. J. Immunol. 39, 67 (2009).
17. V. R. Buchholz et al., Science 340, 630 (2013).
18. K. R. Duffy et al., Science 335, 338 (2012).
19. B. D. Simons, H. Clevers, Cell 145, 851 (2011).
Acknowledgments: The authors thank A. Pfauth and
F. van Diepen for cell sorting; E. Borg for technical assistance;
D. Zehn for provision of LM-OVA (T4); S. Harkema for helpful
discussions; and G. Bendle, M. C. Wolkers, and E. A. Moseman
for critically reading the manuscript. The data presented in
this manuscript are tabulated in the main paper and in the
supplementary materials. The authors declare no conflict of
interest. This research was funded by European Research Council
AdG Life-his-T to T.N.M.S., a Walter-Hitzig fellowship from the
CCI (BMBF 01EO0803), a Deutsche Forschungsgemeinschaft (DFG
RO 4120/1-1) research fellowship to J.C.R., a Marie Curie Intra
European Fellowship to L.P., a Nederlandse Organisatie voor
Wetenschappelijk Onderzoek Veni grant (916.86.080) to J.B.B.,
and a Leukemia and Lymphoma Society Career Development
Fellowship to S.H.N.
Materials and Methods
22 January 2013; accepted 7 March 2013
Published online 14 March 2013;
Neural Decoding of Visual
Imagery During Sleep
T. Horikawa,1,2M. Tamaki,1* Y. Miyawaki,3,1† Y. Kamitani1,2‡
Visual imagery during sleep has long been a topic of persistent speculation, but its private
nature has hampered objective analysis. Here we present a neural decoding approach in which
machine-learning models predict the contents of visual imagery during the sleep-onset period,
given measured brain activity, by discovering links between human functional magnetic
resonance imaging patterns and verbal reports with the assistance of lexical and image databases.
Decoding models trained on stimulus-induced brain activity in visual cortical areas showed
accurate classification, detection, and identification of contents. Our findings demonstrate that
specific visual experience during sleep is represented by brain activity patterns shared by
stimulus perception, providing a means to uncover subjective contents of dreaming using
objective neural measurement.
ing (1–3), but none has demonstrated how spe-
cific visual contents are represented in brain
activity. The advent of machine-learning–based
analysis allows for the decoding of stimulus- and
task-induced brain activity patterns to reveal vi-
sleep (Fig. 1A). Although dreaming has often
been associated with the rapid-eye movement
reaming is a subjective experience dur-
ing sleep often accompanied by vivid
visual contents. Previous research has
(REM) sleep stage, recent studies have demon-
strated that dreaming is dissociable from REM
sleep and can be experienced during non-REM
periods (10). We focused on visual imagery (hal-
lucination) experienced during the sleep-onset
(hypnagogic) period (sleep stage 1 or 2) (11, 12)
because it allowed us to collect many observa-
tions by repeating awakenings and recording par-
ticipants’ verbal reports of visual experience.
Reports at awakenings in sleep-onset and REM
periods share general features such as frequency,
length, and contents, while differing in several as-
pects, including the affective component (13–15).
We analyzed verbal reports using a lexical data-
base to create systematic labels for visual con-
tents. We hypothesized that contents of visual
imagery during sleep are represented at least
partly by visual cortical activity patterns shared
by stimulus representation. Thus, we trained de-
coders on brain activity induced by natural im-
ages from Web image databases.
Three people participated in the functional
magnetic resonance imaging (fMRI)sleep exper-
an electroencephalogram signature was detected
(16) (fig. S1), and they were asked to give a
verbal report freely describing their visual ex-
perience before awakening [table S1; duration,
34 T 19 s (mean T SD)]. We repeated this pro-
cedure to attain at least 200 awakenings with a
awakened participants every 342.0 s, and visual
ings (Fig. 1B). Offline sleep stage scoring (fig.
S2) further selected awakenings to exclude con-
tamination from the wake stage in the period
immediately before awakening (235, 198, and
186 awakenings for participants 1 to 3, respec-
tively, used for decoding analyses) (16).
From the collected reports, words describing
visual objects or scenes were manually extracted
and mapped to WordNet, a lexical database in
which semantically similar words are grouped
as “synsets” in a hierarchical structure (17, 18)
extracted visual words into base synsets that ap-
pearedinatleast 10reportsfromeach participant
(26, 18, and 16 synsets for participants 1 to 3, re-
spectively; tables S2 to S4) (16). The fMRI data
a visual content vector, each element of which
also collected images depicting each base synset
from ImageNet (19), an image database in which
Web images are grouped according to WordNet,
or from Google Images, for decoder training.
We constructed decoders by training linear
support vector machines (SVMs) (20) on fMRI
data measured while each person viewed Web
images for each base synset. Multivoxel patterns
in the higher visual cortex [HVC; the ventral re-
fusiform face area (FFA), and parahippocampal
place area (PPA); 1000 voxels], the lower visual
1ATR Computational Neuroscience Laboratories, Kyoto 619-
0288, Japan.2Nara Institute of Science and Technology, Nara
630-0192, Japan.3National Institute of Information and Com-
munications Technology, Kyoto 619-0288, Japan.
*Present address: Brown University, 190 Thayer Street,
Providence, RI 02912, USA.
†Present address: The University of Electro-Communications,
Tokyo 182-8585, Japan.
‡Corresponding author. E-mail: email@example.com
VOL 3403 MAY 2013
cortex (LVC; V1 to V3 combined; 1000 voxels),
or the subareas (400 voxels for each area) were
used as the input for the decoders (16).
First, a binary classifier was trained on the
fMRI responses to stimulus images of two base
synsets (three-volume averaged data correspond-
ing to the 9-s stimulus block) and tested on the
sleep samples [three-volume (9-s) averaged data
immediately before awakening] that contained
other concurrent synsets (16) (Fig. 3A). We only
used synset pairs in which one of the synsets ap-
with the other (201, 118, and 86 pairs for par-
ticipants 1 to 3, respectively). The distribution of
the pairwise decoding accuracies for the HVC is
on the same stimulus-induced fMRI data with
randomly shuffled synset labels (Fig. 3B; fig. S4,
individual participants). The mean decoding ac-
curacy was 60.0%, 95% confidence interval (CI)
[(59.0, 61.0%); three participants pooled], which
was significantly higher than that of the label-
shuffled decoders with both Wilcoxon rank-sum
and permutation tests (P < 0.001).
between perception and sleep-onset imagery, we
focused on the synset pairs that produced content-
specificpatternsineach ofthe stimulusandsleep
sification accuracy within each of the stimulus
and sleep data sets; figs. S5 and S6) (16). With
the selected pairs, even higher accuracies were
obtained [mean = 70.3%, CI (68.5, 72.1); Fig.
3B, dark blue; fig. S4, individual participants;
tables S5 to S7, lists of the selected pairs], indi-
cating that content-specific patterns are highly
consistent between perception and sleep-onset
imagery. The selection of synset pairs, which
used knowledge of the test (sleep) data, does not
bias the null distribution by the label-shuffled
decoders (Fig. 3B, black), because the content
specificity in the sleep data set alone does not
imply commonality between the two data sets.
Additional analyses revealed that the multi-
voxel pattern, rather than the average activity
level, was critical for decoding (figs. S7 and S8).
We also found that the variability of decoding
for at least partly by the semantic differences
between paired synsets. The decoding accuracy
for synsets paired acrossmeta-categories(human,
object, scene, and others; tables S2 to S4) was
significantly higher than that for synsets within
meta-categories (Wilcoxon rank-sum test, P <
0.001; Fig. 3C and fig. S9). However, even with-
in a meta-category, the mean decoding accuracy
significantly exceeded chance level, indicating
specificity to fine object categories.
The mean decoding accuracies for different
visual areas are shown in Fig. 3D (fig. S10,
individual participants). The LVC scored 54.3%,
CI (53.4, 55.2) for all pairs, and 57.2%, CI (54.2,
60.2) for selected pairs (three participants pooled).
The performance was significantly above chance
level but worse than that for the HVC. Individual
Fig. 1. Experimental overview. (A) fMRI data were acquired from sleeping partic-
stage 1 or 2 (red dashed line) and verbally reported their visual experience during
sleep. fMRI data immediately before awakening [an average of three volumes (= 9 s)] were used as the input for main decoding analyses (sliding time windows
were used for time course analyses). Words describing visual objects or scenes (red letters) were extracted. The visual contents were predicted using machine-
learning decoders trained on fMRI responses to natural images. (B) The numbers of awakenings with and without visual contents are shown for each participant
(with numbers of experiments in parentheses).
Fig. 2. Visualcontentlabeling.(A)Wordsdescribingvisualobjectsor
grouped into base synsets (blue frames) located higher in the tree. (B) Visual reports (participant 2) are represented by visual content vectors, in which the
are shown for some of the base synsets.
3 MAY 2013VOL 340
areas (V1 to V3, LOC, FFA, and PPA) showed a
gradual increase in accuracy along the visual pro-
plex response properties from low-level image
features to object-level features (21). When the
time window was shifted, the decoding accuracy
peaked around 0 to 10 s before awakening (Fig.
3E and fig. S11; no correction for hemodynamic
delay). The high accuracies after awakening may
be due to hemodynamic delay and the large time
window. Thus, verbal reports are likely to reflect
brain activity immediately before awakening.
To read out richer contents given arbitrary
sleep data, we next performed a multilabel de-
coding analysis in which the presence or absence
Fig. 3. Pairwise decoding. (A) Schematic overview. (B) Distributions of
decoding accuracies with original and label-shuffled data for all pairs
(light blue and gray) and selected pairs (dark blue and black) (three
participants pooled). (C) Mean accuracies for the pairs within and across
meta-categories (synsets in others were excluded; numbers of pairs are in
parentheses). (D) Accuracies across visual areas (numbers of selected pairs
for V1, V2, V3, LOC, FFA, PPA, LVC, and HVC: 45, 50, 55, 70, 48, 78, 55,
and 97). (E) Time course (HVC and LVC; averaged across pairs and par-
window centered at each point (gray window and arrow for main analyses).
For all results, error bars or shadings indicate 95% CI, and dashed lines
denote chance level.
Fig. 4. Multilabel decoding. (A) Schematic overview. (B) ROC curves
(left) and AUCs (right) are shown for each synset (participant 2; asterisks,
Wilcoxon rank-sum test, P < 0.05). (C) AUC averaged within meta-
categories for different visual areas (three participants pooled; numbers
of synsets in parentheses). (D) Example time course of synset scores for a
single sleep sample (participant 2, 118th; color legend as in (B); reported
reported synsets (red) and unreported synsets with high or low (blue or
gray) co-occurrence with reported synsets (averaged across awakenings
and participants). Scores are normalized by the mean magnitude in each
participant. (F) Identification analysis. Accuracies are plotted against
candidate set size for original and extended visual content vectors (av-
eraged across awakenings and participants). Because Pearson’s correla-
such samples were excluded. For all results, error bars or shadings in-
dicate 95% CI, and dashed lines denote chance level.
VOL 340 3 MAY 2013
of each base synset was predicted by a synset Download full-text
detector constructed from a combination of pair-
provided a continuous score indicating how like-
ly the synset is to be present in each report. We
curves for each base synset by shifting the detec-
tion threshold for the output score (Fig. 4B, the
HVC in participant 2, time window immediately
before awakening; fig. S12, all participants), and
the detection performance was quantified by the
area under the curve (AUC). Although the per-
60 synsets were detected with above-chance lev-
els (Wilcoxon rank-sum test, uncorrected P <
0.05), greatly exceeding the number of synsets
expected by chance (0.05 × 60 = 3).
Using the AUC, we compared the decoding
performance for individual synsets grouped into
meta-categories in different visual areas. Overall,
the performance was better in the HVC than in
the LVC, consistent with the pairwise decoding
performance [fig. S13; three participants pooled;
analysis of variance (ANOVA), P = 0.003]. Al-
though V1 to V3 did not showdifferent perform-
ances across meta-categories, the higher visual
areas showed a marked dependence on meta-
categories (Fig. 4C and fig. S13). In particular,
the FFA showed better performance with human
ance with scene synsets [ANOVA (interaction),
P = 0.001], consistent with the known response
characteristics of these areas (22, 23). The LOC
and FFA showed similar results, presumably be-
cause our functional localizers selected partially
diverse and dynamic profiles in each sleep sam-
ple (Fig. 4D, fig. S14, and movies S1 and S2)
(16). These profiles may reflect a dynamic varia-
tion of visual contents, including those expe-
rienced even before the period near awakening.
On average, there was a general tendency for the
scores for reported synsets to increase toward the
that did not appear in reports showed greater
scores if they had a high co-occurrence relation-
ship with reported synsets (Fig. 4E; synsets with
the top 15% conditional probabilities given a re-
ported synset, calculated from the whole-content
vectors in each participant). The effect of co-
occurrence is rather independent of that of se-
mantic similarity (Fig. 3C), because both factors
(high/low co-occurrence and within/across meta-
categories) had highly significant effects on the
diately before awakening; two-way ANOVA, P <
0.001, three participants pooled) with moderate
interaction (P = 0.016). The scores for reported
synsets were significantly higher than those for
unreported synsets even within the same meta-
category (Wilcoxon rank-sum test, P < 0.001).
Verbal reports are unlikely to describe full details
of visual experience during sleep, and it is pos-
(such as street and car) tend to be experienced
together even when all are not reported. There-
fore, high scores for the unreported synsets may
indicate unreported but actual visual contents
Finally, to explore the potential of multilabel
decoding to distinguish numerous contents, we
performed identification analysis (7, 8). The
output scores (score vector) were used to identify
the true visual content vector among a variable
number of candidates (true vector + random vec-
tors with matched probabilities for each synset)
by selecting the candidate most correlated with
the score vector (repeated 100 times for each
sleep sample to obtain the correct identification
rate) (16). The performance exceeded chance lev-
el across all set sizes (Fig. 4F, HVC, three par-
ticipants pooled; fig. S16, individual participants),
although the accuracies were not as high as those
achieved using stimulus-induced brain activity in
previous studies (7, 8). The same analysis was
performed with extended visual content vectors
in which unreported synsets having a high co-
occurrence with reported synsets (top 15% con-
ditional probability) were assumed to be present.
The results showed that extended visual content
vectors were better identified (Fig. 4F and fig.
S16), suggesting that multilabel decoding out-
Together, our findings provide evidence that
are represented by, and can be read out from,
visual cortical activity patterns shared with stim-
ulus representation. Our approach extends previ-
ous research on the (re)activation of the brain
during sleep (24–27) and the relationship be-
tween dreaming and brain activity (2, 3, 28) by
ity patterns and unstructured verbal reports using
results suggest that the principle of perceptual
equivalence (29), which postulates a common
neural substrate for perception and imagery, gen-
eralizes to spontaneously generated visual expe-
semantic decoding with the HVC, this does not
rule out the possibility of decoding low-level
features with the LVC. The decoding presented
here is retrospective in nature: Decoders were
constructed after sleep experiments based on the
sets largely overlap between the first and the last
halves of the experiments (59 out of 60 base
synsets appeared in both), the same decoders
may apply to future sleep data. The similarity
between REM and sleep-onset reports (13–15)
sleep (24, 25, 28) suggest that the same decoders
could also be used to decode REM imagery. Our
method may further work beyond the bounds of
the dynamics of spontaneous brain activity in
association with stimulus representation. We ex-
pect that it will lead to a better understanding of
the functions of dreaming and spontaneous neu-
ral events (10, 30).
References and Notes
1. W. Dement, N. Kleitman, J. Exp. Psychol. 53, 339
2. M. Dresler et al., Curr. Biol. 21, 1833 (2011).
3. C. Marzano et al., J. Neurosci. 31, 6674 (2011).
4. J. V. Haxby et al., Science 293, 2425 (2001).
5. Y. Kamitani, F. Tong, Nat. Neurosci. 8, 679 (2005).
6. S. A. Harrison, F. Tong, Nature 458, 632 (2009).
7. K. N. Kay, T. Naselaris, R. J. Prenger, J. L. Gallant, Nature
452, 352 (2008).
8. Y. Miyawaki et al., Neuron 60, 915 (2008).
9. M. Stokes, R. Thompson, R. Cusack, J. Duncan,
J. Neurosci. 29, 1565 (2009).
10. Y. Nir, G. Tononi, Trends Cogn. Sci. 14, 88 (2010).
11. T. Hori, M. Hayashi, T. Morikawa, in Sleep Onset: Normal
and Abnormal Processes, R. D. Ogilvie, J. R. Harsh, Eds.
(American Psychological Association, Washington, DC,
1994), pp. 237–253.
12. R. Stickgold, A. Malia, D. Maguire, D. Roddenberry,
M. O’Connor, Science 290, 350 (2000).
13. D. Foulkes, G. Vogel, J. Abnorm. Psychol. 70, 231
14. G. W. Vogel, B. Barrowclough, D. D. Giesler, Arch. Gen.
Psychiatry 26, 449 (1972).
15. D. Oudiette et al., Conscious. Cogn. 21, 1129
16. Materials and methods are available as supplementary
materials on Science Online.
17. C. Fellbaum, Ed., WordNet: An Electronic Lexical
Database (MIT Press, Cambridge, MA, 1998).
18. A. G. Huth, S. Nishimoto, A. T. Vu, J. L. Gallant, Neuron
76, 1210 (2012).
19. J. Deng et al., IEEE Conference on Computer Vision and
Pattern Recognition, 248 (2009).
20. V. N. Vapnik, Statistical Learning Theory (Wiley, New
21. E. Kobatake, K. Tanaka, J. Neurophysiol. 71, 856
22. R. Epstein, N. Kanwisher, Nature 392, 598 (1998).
23. N. Kanwisher, J. McDermott, M. M. Chun, J. Neurosci. 17,
24. A. R. Braun et al., Science 279, 91 (1998).
25. P. Maquet, J. Sleep Res. 9, 207 (2000).
26. Y. Yotsumoto et al., Curr. Biol. 19, 1278 (2009).
27. M. A. Wilson, B. L. McNaughton, Science 265, 676
28. S. Miyauchi, M. Misaki, S. Kan, T. Fukunaga, T. Koike,
Exp. Brain Res. 192, 657 (2009).
29. R. A. Finke, Principles of Mental Imagery (MIT Press,
Cambridge, MA, 1989).
30. J. A. Hobson, Nat. Rev. Neurosci. 10, 803 (2009).
Acknowledgments: We thank Y. Onuki, T. Beck, Y. Fujiwara,
G. Pandey, and T. Kubo for assistance with early experiments
and M. Takemiya and P. Sukhanov for comments on the
manuscript. This work was supported by grants from the
Strategic Research Program for Brain Science (MEXT), the
Strategic Information and Communications R&D Promotion
Programme (SOUMU), the National Institute of Information
and Communications Technology, the Nissan Science Foundation,
and the Ministry of Internal Affairs and Communications (Novel
and innovative R&D making use of brain structures).
Materials and Methods
Figs. S1 to S16
Tables S1 to S7
Movies S1 and S2
20 December 2012; accepted 5 March 2013
Published online 4 April 2013;
3 MAY 2013VOL 340