Scene Buildup From Latent Memory Representations Across Eye Movements

Abstract

An unresolved problem in eye movement research is how a representation is constructed on-line from several consecutive fixations of a scene. Such a scene representation is generally understood to be sparse; yet, for meeting behavioral goals a certain level of detail is needed. We propose that this is achieved through the buildup of latent representations acquired at fixation. Latent representations are retained in an activity-silent manner, require minimal energy expenditure for their maintenance, and thus allow a larger storage capacity than traditional, activation based, visual working memory. The latent representations accumulate and interact in working memory to form to the scene representation. The result is rich in detail while sparse in the sense that it is restricted to the task-relevant aspects of the scene sampled through fixations. Relevant information can quickly and flexibly be retrieved by dynamical attentional prioritization. Latent representations are observable as transient functional connectivity patterns, which emerge due to short-term changes in synaptic weights. We discuss how observing latent representations could benefit from recent methodological developments in EEG-eye movement co-registration.

Full text

fpsyg-09-02701 January 7, 2019 Time: 16:56 # 1
PERSPECTIVE
published: 11 January 2019
doi: 10.3389/fpsyg.2018.02701
Edited by:
Paul Sajda,
Columbia University, United States
Reviewed by:
Joseph Charles Schmidt,
University of Central Florida,
United States
Thomas Sanocki,
University of South Florida,
United States
*Correspondence:
Andrey R. Nikolaev
Andrey.Nikolaev@kuleuven.be
Specialty section:
This article was submitted to
Perception Science,
a section of the journal
Frontiers in Psychology
Received: 01 August 2018
Accepted: 17 December 2018
Published: 11 January 2019
Citation:
Nikolaev AR and van Leeuwen C
(2019) Scene Buildup From Latent
Memory Representations Across Eye
Movements. Front. Psychol. 9:2701.
doi: 10.3389/fpsyg.2018.02701
Scene Buildup From Latent Memory
Representations Across Eye
Movements
Andrey R. Nikolaev*and Cees van Leeuwen
Laboratory for Perceptual Dynamics, Brain & Cognition Research Unit, KU Leuven, Leuven, Belgium
An unresolved problem in eye movement research is how a representation is
constructed on-line from several consecutive fixations of a scene. Such a scene
representation is generally understood to be sparse; yet, for meeting behavioral goals a
certain level of detail is needed. We propose that this is achieved through the buildup
of latent representations acquired at fixation. Latent representations are retained in an
activity-silent manner, require minimal energy expenditure for their maintenance, and
thus allow a larger storage capacity than traditional, activation based, visual working
memory. The latent representations accumulate and interact in working memory to
form to the scene representation. The result is rich in detail while sparse in the
sense that it is restricted to the task-relevant aspects of the scene sampled through
fixations. Relevant information can quickly and flexibly be retrieved by dynamical
attentional prioritization. Latent representations are observable as transient functional
connectivity patterns, which emerge due to short-term changes in synaptic weights. We
discuss how observing latent representations could benefit from recent methodological
developments in EEG-eye movement co-registration.
Keywords: eye movement, visual scene, brain activity, latent representations, working memory
SCENE BUILDUP ACROSS EYE MOVEMENTS
From finding our keys in a room to appreciating a work of art, various aims for visually exploring a
scene seem to require a rich, integral representation. Task-relevant details acquired during fixation
would accumulate therein, across multiple eye movements, on a time scale of, roughly, one to a
few minutes (Melcher and Kowler, 2001;Hollingworth and Henderson, 2002;Tatler et al., 2003;
Melcher, 2006).
However, it remains a matter of debate how much of the information obtained at fixation is, in
fact, memorized across saccades. Some authors have argued that such memorization is not needed
at all, as the world itself serves as an “outside memory” (O’Regan and Noë, 2001). Others claim
that memory is used only to a limited extent; object information is likely to be preserved between
saccades, but only as long as it remains in the focus of attention (Rensink, 2000). The majority of
researchers assign memory a somewhat larger role, and propose that representations of several
fixated objects are accumulated in visual short term memory (VSTM) (Irwin, 1996;Irwin and
Zelinsky, 2002;Prime et al., 2011;Tatler and Land, 2011;Higgins and Rayner, 2015).
Visual short term memory may function as “trans-saccadic memory” to provide visual stability
across saccades. This function could be achieved through remapping between pre- and post-
saccadic visual representations (Mathôt and Theeuwes, 2011;Marino and Mazer, 2016). However,
Frontiers in Psychology | www.frontiersin.org 1January 2019 | Volume 9 | Article 2701

fpsyg-09-02701 January 7, 2019 Time: 16:56 # 2
Nikolaev and van Leeuwen Latent Representations of Scenes
VSTM capacity limitations restrict the information accumulating
to 3–5 items, and when this number is exceeded, newly incoming
information will overwrite the old. These limitations would
typically keep the resulting scene representation sparse (Irwin,
1996;Irwin and Zelinsky, 2002). The presumed sparseness
clashes with average observers’ ability to successfully recognize
thousands of scene images (Standing, 1973;Konkle et al., 2010).
Large amounts of information, moreover, can be retained about
both a scene’s spatial layout and the objects therein (Friedman,
1979;Sanocki et al., 2010). These findings suggest a special
aptness of memory for scenes may exist.
Scene representation across eye movement, according to
the ongoing, task-related character of information processing
could be a function of “working memory” (Baddeley, 1992;
Melcher, 2006). Besides information in the focus of attention, for
which capacity is limited, working memory also encompasses an
activated portion of visual long-term memory (VLTM) (Cowan,
1988;Oberauer, 2002). The latter may have a role in maintaining
information in a heightened state of availability, in particular
information about recently attended items or information
associated with items in the attentional focus. The involvement
of VLTM would allow detailed information from many locations
to be used for constructing a rich scene representation
(Hollingworth and Henderson, 2002;Hollingworth, 2007).
It seems implausible, however, that long-term memory is
extensively involved in constructing goal-driven, momentary
representations of scenes. Encoding into VLTM requires
consolidation, the fastest type of which is “synaptic
consolidation.” As it encompasses protein synthesis, synaptic
consolidation takes several minutes (Dudai, 2004) – incompatible
with the construction rate required for scene representation
across eye movements. Moreover, VLTM, by definition, preserves
information for a long time. Even though VLTM capacity may
be huge, the myriad scene fragments it would have to store will
unduly clutter it up. A more parsimonious solution is called for;
one that combines high encoding speeds and ease of availability
with a sufficiently large capacity.
To satisfy these seemingly contradictory requirements, a
“proto-LTM,” or “medium-term” memory was postulated by
Melcher (2006). Proto-LTM allows large amounts of information
to be kept available over a period of minutes, but this information
is not consolidated. At the time, no neural mechanism for such
type of memory could be offered. Recent discoveries, however,
suggest the existence of a retention mechanism that could satisfy
these requirements. Information acquired at fixation may be
retained afterward in an “activity-silent” neural state, which
results in latent mental representations (Stokes, 2015;Postle,
2016).
We propose that building up a rich representation of scenes
involves integration of local representations that exist in latent
memory states. Latent representations are residual traces of
former active representations. Unlike VSTM representations,
latent representations do not require persistent activity for
their maintenance and consequently can be much more
numerous than typical VSTM capacity allows. They can
quickly be retrieved by dynamical attentional prioritization,
flexibly depend on task demands, and thus offer the resources
necessary for obtaining a detailed representation of entire
scene.
ACTIVITY-BASED AND ACTIVITY-SILENT
WORKING MEMORY
Maintenance in working memory has traditionally been
associated with increased and unremitting levels of neural
activity, such as sustained neuronal spiking or persistent neural
population oscillations (Brunel, 2003). This conception stems
from experiments, in which animals or humans typically have
to remember several items and, after a couple of seconds’ delay,
memory for these items is tested. While information is retained
in memory during the delay period, brain activity is found to be
enhanced, in proportion to the amount of retained information
(“memory load”). Results like this are taken as evidence that
the memory system has to work harder to maintain more
information (Postle, 2016).
However in the last 3–5 years, evidence increasingly
suggests that working memory can also be maintained without
elevated brain activity. An “activity-silent” neural state allows
maintenance of memory representations in a hidden or latent
form (Stokes, 2015). Although silent, the neural mechanism
underlying such neural states is anything but inert. According
to the recent understanding of synaptic connections, short-term
changes in synaptic efficacy may carry memory information
in absence of persistent neuronal spiking or oscillatory activity
(Mongillo et al., 2008). These synaptic changes give rise to
evanescent circuits that are constantly being replicated in
different network locations (Routtenberg, 2013). This mechanism
allows for configuration and integration of representational
networks (Schacter and Addis, 2007). Accordingly, the latent
representations may be maintained in the patterns of synaptic
connectivity, which are based on short-term modulation of
synaptic weights during encoding (Stokes, 2015;Postle, 2016).
Activity-silent and activity-based maintenance are not
mutually exclusive. In fact these mechanisms may work
in tandem (Stokes, 2015). In active memory maintenance,
oscillatory dynamics and spiking fluctuate, disappearing
and reappearing intermittently (van Leeuwen and Raffone,
2001). Between the active states, information is maintained
by temporary changes in synaptic weights of the recurrent
connections (Lundqvist et al., 2016;Trübutschek et al., 2017).
In the time course of a typical memory task, the activity-based
states are more prominent in the initial period after presentation
of to-be-memorized items, which mostly involves encoding. This
period is followed by activity-silent maintenance, during which,
however, activity may intermittently resurface (Trübutschek
et al., 2017).
The alternation of activity-based and activity-silent neural
states may be controlled by attention. Attention dynamically
prioritizes representations in working memory whenever they
become relevant to behavior (Rose et al., 2016;Myers et al., 2017;
van Ede et al., 2017). As a result, representations are moved from
an activity-silent state into the focus of attention, i.e., into an
active neural state and vice versa. Dynamical prioritization of
Frontiers in Psychology | www.frontiersin.org 2January 2019 | Volume 9 | Article 2701

fpsyg-09-02701 January 7, 2019 Time: 16:56 # 3
Nikolaev and van Leeuwen Latent Representations of Scenes
one item does not impair the maintenance of unprioritized items,
allowing the operation to be reversed (van Ede et al., 2017).
Dynamical prioritization may involve two component
processes: first, selecting a memory and, second, reconfiguring its
state according to the current task demands (Myers et al., 2017).
Thus, maintenance of currently attended items is accompanied
by enhanced activity, whereas unattended items are maintained
in an activity-silent state. As soon as attention is shifted to them,
they switch to an active state. This allows latent representations to
be temporally precise, i.e., items appear in the focus of attention
at the most relevant times (van Ede et al., 2017).
LATENT MEMORY REPRESENTATIONS
ARE BUILDING BLOCKS OF SCENE
REPRESENTATIONS
Rapid shifts of attention are a key feature of eye movement
control. Just before the onset of a saccade, attention shifts
focus to the new fixation target (Deubel and Schneider, 1996).
Visual information about the new fixation target is acquired after
the saccade, as long as the target remains in focus. The new
information is encoded in working memory by active neuronal
firing, which results in temporary change of synaptic weights
(Nee and Jonides, 2013;Trübutschek et al., 2017). Toward the end
of the fixation, transfer of attention to the next fixation location
deprioritizes the current item. Subsequently, its representation
turns from an active to an activity-silent, latent state, while firing
activity encodes the information at the new fixation location.
Items that are preserved in a latent state have elsewhere
become known as “accessory memory items” (Olivers et al.,
2011). Unlike in scene perception, however, these were not
supposed to be kept for immediate use. This, however, is the
essential role of our latent representations. Our key hypothesis
is that building up a scene representation involves integration
of multiple latent memory representations retained across eye
movements, in order to support fine-grained operations with
task-relevant items of the scene. Thus, latent representations
carry information about recently visited items that are directly
related to completion of the visual task at hand.
The latent representation does not require sustained activity
for maintenance and is, therefore, energy-efficient. This enables
significantly larger number of items to be retained than in a
classical VSTM. But this number is still limited to task-relevant
ones, as attention operates as a gatekeeper; representations can
only enter the latent state from a prior activated state. This keeps
scene representation relatively sparse and selective, opening the
door for effects of attentional and change blindness (Rensink,
2000).
The selectivity of a latent representation is a function of the
distribution of attention within the entire visual field. The spatial
extent of the attention field scales with eccentricity (Puckett and
DeYoe, 2015) and its size and position is flexible depending on
the current visual task (Herrmann et al., 2010). It will allow,
for instance, more than one location to be selected at once. In
visually demanding tasks, there is a tradeoff between the number
of items in memory and the precision of their selection: the more
precision is required, the fewer locations can be selected at once
(Hogeboom and van Leeuwen, 1997;Franconeri et al., 2007). If
a task requires high precision, the latent representation includes
more details about the items represented, at the expense of their
quantity. Vice versa, when precision could be sacrificed, more
than a few locations could be selected at once.
The selectivity of latent representation renders it unlikely
to contain many low-level sensory characteristics. However, it
likely preserves the heterogeneity of the visual field, which may
be shaped by the typical eccentricity-dependent degradation of
acuity and color sensitivity from fovea to periphery. Furthermore,
memory capacity is much larger for scene layout information
than for single objects in a scene (Sanocki et al., 2010). Therefore,
latent representations may come associated with a substantial
amount of scene layout information.
A series of latent representations should remain in memory
long enough to enable task completion, e.g., for several tens of
eye movements. Because multiple latent representations coexist
in time, this allows them to interact. Such interactions may have
automatic and implicit effects on the performance of the current
task. Moreover, the interaction may result in a characteristic form
of location priming, which may be called latent priming. Since
the latent representations are accumulated across sequential eye
movements, the locations encoded at the previous representation
may prime that for the following one. This is in analogy
to priming as known in scene perception. A scene presented
as a prime facilitates subsequent spatial processing of target
objects within a following test scene. This finding was explained
by activation of a prime-induced representation of the scene’s
layout, a representation which integrates information about task-
relevant objects and their layout (Sanocki, 2003).
Since latent memory representation is based on changes of
synaptic weights, these may be the loci of integration of sequential
views. During a sequence of fixations, the weights of synaptic
connections established at prior fixations will be modulated by
subsequent ones. The strength of modulation may depend on
the behavioral relevance of the fixated item, as expressed by the
amount of attention it received. The integration may involve
changes in supple synapses, i.e., synapses that are highly malleable
and give rise to evanescent circuits (Routtenberg, 2013). These
transient memory networks do not depend on consolidation and
so there is no need to retrieve them – this provides instant access
and high flexibility to meet the momentary visual goals.
Our hypothesis is consistent with an understanding of
working memory without separation between processing and
information storage units. Previous information continually
modulates the presently processed one. This results in the active
and ongoing accumulation and integration of information that
occurs in natural stimulation conditions (van Leeuwen and
Raffone, 2001;Hasson et al., 2015;Voss et al., 2017).
Whereas latent representations support completion of the task
at hand, after many repetitions of a task they may enter long-term
memory storage. There, latent representations may be retained
as experience instrumental for handling a category of similar
tasks. This experience may constitute the “selection history”
that biases attentional prioritization of items previously attended
in similar contexts (Awh et al., 2012). The “selection history”
Frontiers in Psychology | www.frontiersin.org 3January 2019 | Volume 9 | Article 2701

fpsyg-09-02701 January 7, 2019 Time: 16:56 # 4
Nikolaev and van Leeuwen Latent Representations of Scenes
may contradict current selection goals. As a result, complex
interactions may occur between “selection history” and latent
representations related to the current task.
DETECTING SCENE BUILDUP ACROSS
EYE MOVEMENTS
Studying the integration of scene information across saccades
has been made possible by recent advances in combining video-
based eye-tracking with EEG measurement. Eye movement-EEG
co-registration allows addressing research questions inaccessible
with either technique alone. Visual processing can now be studied
in naturalistic visual conditions – a major step from traditional
stimulus-response paradigms. Consequently, co-registration is
increasingly being adopted for studying, for example, memory
encoding (Nikolaev et al., 2011), reading (Dimigen et al., 2011),
attention (Fischer et al., 2013), visual search (Körner et al., 2014),
and emotional responses (Simola et al., 2015).
Co-registration of free viewing behavior is methodologically
demanding. Sequential eye movements systematically affect EEG:
EEG responses to the current saccade overlap with those to the
preceding and following ones, giving rise to spurious effects.
Since fixation durations have a non-uniform distribution, this
problem cannot be solved by averaging EEG across fixation-
related epochs (Dimigen et al., 2011;Dias et al., 2013;Nikolaev
et al., 2016). The established solutions to this problem involves
matching of eye movement characteristics between experimental
conditions (Nikolaev et al., 2016) or statistically considering eye
movement effects using Generalized Additive Mixed Modelling
(GAMM) (Van Humbeeck et al., 2018).
To reveal latent memory representations across eye
movements, EEG-eye movement co-registration can be
combined with methods allowing the detection of activity-silent
neural states. We suggest two major approaches. The first is
based on assumption that latent representations are maintained
in patterns of functional connectivity (Stokes, 2015). Functional
connectivity is manifested in frequency-specific patterns of
phase synchrony, which support neural communication and
plasticity (Fell and Axmacher, 2011). Accordingly, the analysis
of functional connectivity during fixation intervals may reveal
scene buildup from latent representations.
Synchrony measures of scalp EEG are sensitive to various
stages of the activity-based memory process. Synchrony is
higher during encoding for information remembered than during
encoding for information subsequently forgotten (Summerfield
and Mangels, 2005). During memory maintenance, widespread
increase of synchrony is proportional to memory load (Payne
and Kounios, 2009). Recently we explored the dynamic
reconfiguration of functional connectivity in free viewing during
encoding and retrieval (Seidkhani et al., 2017). We evaluated
the functional connectivity after fixation onset through graph-
theoretical measures. Encoding involved a more segregated mode
of operation than retrieval, as it was evident from such measures
as mean path length, radius, closeness, and eccentricity.
However, since between-area synchronization is a prerequisite
of memory formation (Fell and Axmacher, 2011) EEG synchrony
may also reflect activity-silent retention. To reveal latent
representations across eye movements, functional connectivity
analysis could be applied to fixation intervals in free viewing
exploration of a scene, for instance preceding a memory test.
The other approach proposed for identifying latent
representations involves multivariate pattern analysis (MVPA).
Initially developed for and intensively used in fMRI research,
MVPA has been increasing in popularity for application to
EEG/MEG (King and Dehaene, 2014;Trübutschek et al., 2017;
Wolff et al., 2017). Multiple data points are jointly analyzed in
order to isolate the topographical patterns that differentiate best
the experimental conditions. MVPA is implemented via machine
learning, where a classifier is trained to decode specific mental
states from the patterns of brain activity. To extract time-course
information from EEG, a series of classifiers can be trained,
each applied on successive time slices of the data allowing the
researcher to trace how mental representations unfold over time.
The activity-silent representation during retention interval
could be divulged by presenting a probe impulse, which pings a
hidden neural state. MVPA of the EEG response to this impulse
could revive item-specific activity, like observed during an item’s
encoding (Stokes, 2015;Postle, 2016). For example, Wolff et al.
(2017) asked participants to remember two oriented grating
stimuli. Then, a cue indicated which of the stimuli will be tested
after a 1 s retention interval. During the retention interval, a
probe impulse was flashed. The EEG response to this impulse,
decoded with MVPA, reflected not only the attended (cued)
but also the unattended (uncued) stimulus, which resided in a
hidden state of memory. An MVPA experiment for detection of
latent representations across eye movements may involve gaze-
contingent presentation of probe (flash) impulses during scene
inspection.
CONCLUSION
Taking up the visual information from a scene involves more than
a snapshot. Visual system, memory and attention work together
to achieve a goal-oriented representation. How this is achieved
has been studied for decades, but has still largely remained
an unresolved problem. Recent advances in the understanding
of memory mechanisms, along with developments in the
methodology of simultaneous recording eye movement and
brain activity and novel, computationally intensive approaches to
decoding hidden patterns of brain activity, offer a perspective for
solving this intriguing puzzle.
AUTHOR CONTRIBUTIONS
AN contributed the central ideas to this paper. AN and CvL wrote
the manuscript.
FUNDING
AN and CvL were supported by an Odysseus grant from the
Flemish Organization for Science (FWO) to CvL.
Frontiers in Psychology | www.frontiersin.org 4January 2019 | Volume 9 | Article 2701

fpsyg-09-02701 January 7, 2019 Time: 16:56 # 5
Nikolaev and van Leeuwen Latent Representations of Scenes
REFERENCES
Awh, E., Belopolsky, A. V., and Theeuwes, J. (2012). Top-down versus bottom-
up attentional control: a failed theoretical dichotomy. Trends Cogn. Sci. 16,
437–443. doi: 10.1016/j.tics.2012.06.010
Baddeley, A. (1992). Working memory. Science 255, 556–559. doi: 10.1126/science.
1736359
Brunel, N. (2003). Dynamics and plasticity of stimulus-selective persistent activity
in cortical network models. Cereb. Cortex 13, 1151–1161. doi: 10.1093/cercor/
bhg096
Cowan, N. (1988). Evolving conceptions of memory storage, selective attention,
and their mutual constraints within the human information-processing system.
Psychol. Bull. 104, 163–191. doi: 10.1037/0033-2909.104.2.163
Deubel, H., and Schneider, W. X. (1996). Saccade target selection and object
recognition: evidence for a common attentional mechanism. Vis. Res. 36,
1827–1837. doi: 10.1016/0042-6989(95)00294- 4
Dias, J. C., Sajda, P., Dmochowski, J. P., and Parra, L. C. (2013). EEG precursors
of detected and missed targets during free-viewing search. J. Vis. 13:13.
doi: 10.1167/13.13.13
Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, A. M., and Kliegl, R.
(2011). Coregistration of eye movements and EEG in natural reading:
analyses and review. J. Exp. Psychol. Gen. 140, 552–572. doi: 10.1037/a002
3885
Dudai, Y. (2004). The neurobiology of consolidations, or, how stable is the engram?
Annu. Rev. Psychol. 55, 51–86. doi: 10.1146/annurev.psych.55.090902.142050
Fell, J., and Axmacher, N. (2011). The role of phase synchronization in memory
processes. Nat. Rev. Neurosci. 12, 105–118. doi: 10.1038/nrn2979
Fischer, T., Graupner, S. T., Velichkovsky, B. M., and Pannasch, S. (2013).
Attentional dynamics during free picture viewing: evidence from oculomotor
behavior and electrocortical activity. Front. Syst. Neurosci. 7:17. doi: 10.3389/
fnsys.2013.00017
Franconeri, S. L., Alvarez, G. A., and Enns, J. T. (2007). How many locations can
be selected at once? J. Exp. Psychol. Hum. Percept. Perform. 33, 1003–1012.
doi: 10.1037/0096-1523.33.5.1003
Friedman, A. (1979). Framing pictures: the role of knowledge in automatized
encoding and memory for gist. J. Exp. Psychol. Gen. 108, 316–355. doi: 10.1037/
0096-3445.108.3.316
Hasson, U., Chen, J., and Honey, C. J. (2015). Hierarchical process memory:
memory as an integral component of information processing. Trends Cogn. Sci.
19, 304–313. doi: 10.1016/j.tics.2015.04.006
Herrmann, K., Montaser-Kouhsari, L., Carrasco, M., and Heeger, D. J. (2010).
When size matters: attention affects performance by contrast or response gain.
Nat. Neurosci. 13, 1554–1559. doi: 10.1038/nn.2669
Higgins, E., and Rayner, K. (2015). Transsaccadic processing: stability, integration,
and the potential role of remapping. Attent. Percept. Psychophys. 77, 3–27.
doi: 10.3758/s13414-014- 0751-y
Hogeboom, M., and van Leeuwen, C. (1997). Visual search strategy and perceptual
organization covary with individual preference and structural complexity. Acta
Psychol. 95, 141–164. doi: 10.1016/S0001-6918(96)00049-2
Hollingworth, A. (2007). Object-position binding in visual memory for natural
scenes and object arrays. J. Exp. Psychol. Hum. Percept. Perform. 33, 31–47.
doi: 10.1037/0096-1523.33.1.31
Hollingworth, A., and Henderson, J. M. (2002). Accurate visual memory for
previously attended objects in natural scenes. J. Exp. Psycho. Hum. Percept.
Perform. 28, 113–136. doi: 10.1037/0096-1523.28.1.113
Irwin, D. E. (1996). Integrating information across saccadic eye movements. Curr.
Dir. Psychol. Sci. 5, 94–100. doi: 10.1111/1467-8721.ep10772833
Irwin, D. E., and Zelinsky, G. J. (2002). Eye movements and scene perception:
memory for things observed. Percept. Psychophys. 64, 882–895. doi: 10.3758/
BF03196793
King, J. R., and Dehaene, S. (2014). Characterizing the dynamics of mental
representations: the temporal generalization method. Trends Cogn. Sci. 18,
203–210. doi: 10.1016/j.tics.2014.01.002
Konkle, T., Brady, T. F., Alvarez, G. A., and Oliva, A. (2010). Scene memory is
more detailed than you think: the role of categories in visual long-term memory.
Psychol. Sci. 21, 1551–1556. doi: 10.1177/0956797610385359
Körner, C., Braunstein, V., Stangl, M., Schlogl, A., Neuper, C., and Ischebeck, A.
(2014). Sequential effects in continued visual search: using fixation-related
potentials to compare distractor processing before and after target detection.
Psychophysiology 51, 385–395. doi: 10.1111/psyp.12062
Lundqvist, M., Rose, J., Herman, P., Brincat, S. L., Buschman, T. J., and Miller, E. K.
(2016). Gamma and beta bursts underlie working memory. Neuron 90, 152–164.
doi: 10.1016/j.neuron.2016.02.028
Marino, A. C., and Mazer, J. A. (2016). Perisaccadic updating of visual
representations and attentional states: linking behavior and neurophysiology.
Front. Syst. Neurosci. 10:3. doi: 10.3389/fnsys.2016.00003
Mathôt, S., and Theeuwes, J. (2011). Visual attention and stability. Philos. Trans. R.
Soc. Lond. Ser. B Biol. Sci. 366, 516–527. doi: 10.1098/rstb.2010.0187
Melcher, D. (2006). Accumulation and persistence of memory for natural scenes.
J. Vis. 6, 8–17. doi: 10.1167/6.1.2
Melcher, D., and Kowler, E. (2001). Visual scene memory and the guidance of
saccadic eye movements. Vis. Res. 41, 3597–3611. doi: 10.1016/S0042-6989(01)
00203-6
Mongillo, G., Barak, O., and Tsodyks, M. (2008). Synaptic theory of working
memory. Science 319, 1543–1546. doi: 10.1126/science.1150769
Myers, N. E., Stokes, M. G., and Nobre, A. C. (2017). Prioritizing information
during working memory: beyond sustained internal attention. Trends Cogn. Sci.
21, 449–461. doi: 10.1016/j.tics.2017.03.010
Nee, D. E., and Jonides, J. (2013). Trisecting representational states in short-term
memory. Front. Hum. Neurosci. 7:796. doi: 10.3389/fnhum.2013.00796
Nikolaev, A. R., Meghanathan, R. N., and van Leeuwen, C. (2016). Combining EEG
and eye movement recording in free viewing: pitfalls and possibilities. Brain
Cogn. 107, 55–83. doi: 10.1016/j.bandc.2016.06.004
Nikolaev, A. R., Nakatani, C., Plomp, G., Jurica, P., and van Leeuwen, C. (2011).
Eye fixation-related potentials in free viewing identify encoding failures in
change detection. Neuroimage 56, 1598–1607. doi: 10.1016/j.neuroimage.2011.
03.021
Oberauer, K. (2002). Access to information in working memory: exploring the
focus of attention. J. Exp. Psychol. Learn. Mem. Cogn. 28, 411–421. doi: 10.1037/
0278-7393.28.3.411
Olivers, C. N., Peters, J., Houtkamp, R., and Roelfsema, P. R. (2011).
Different states in visual working memory: when it guides attention and
when it does not. Trends Cogn. Sci. 15, 327–334. doi: 10.1016/j.tics.2011.
05.004
O’Regan, J. K., and Noë, A. (2001). A sensorimotor account of vision and visual
consciousness. Behav. Brain Sci. 24, 939–973. doi: 10.1017/S0140525X01000115
Payne, L., and Kounios, J. (2009). Coherent oscillatory networks supporting short-
term memory retention. Brain Res. 1247, 126–132. doi: 10.1016/j.brainres.2008.
09.095
Postle, B. R. (2016). How does the brain keep information “in Mind”? Curr. Dir.
Psychol. Sci. 25, 151–156. doi: 10.1177/0963721416643063
Prime, S. L., Vesia, M., and Crawford, J. D. (2011). Cortical mechanisms for trans-
saccadic memory and integration of multiple object features. Philos. Trans. R.
Soc. Lond. Ser. B Biol. Sci. 366, 540–553. doi: 10.1098/rstb.2010.0184
Puckett, A. M., and DeYoe, E. A. (2015). The attentional field revealed by single-
voxel modeling of fmri time courses. J. Neurosci. 35, 5030–5042. doi: 10.1523/
JNEUROSCI.3754-14.2015
Rensink, R. A. (2000). The dynamic representation of scenes. Vis. Cogn. 7, 17–42.
doi: 10.1080/135062800394667
Rose, N. S., LaRocque, J. J., Riggall, A. C., Gosseries, O., Starrett, M. J., Meyering,
E. E., et al. (2016). Reactivation of latent working memories with transcranial
magnetic stimulation. Science 354, 1136–1139. doi: 10.1126/science.aah
7011
Routtenberg, A. (2013). Lifetime memories from persistently supple synapses.
Hippocampus 23, 202–206. doi: 10.1002/hipo.22088
Sanocki, T. (2003). Representation and perception of scenic layout. Cogn. Psychol.
47, 43–86. doi: 10.1016/S0010-0285(03)00002- 1
Sanocki, T., Sellers, E., Mittelstadt, J., and Sulman, N. (2010). How high is visual
short-term memory capacity for object layout? Attent. Percept. Psychophys. 72,
1097–1109. doi: 10.3758/APP.72.4.1097
Schacter, D. L., and Addis, D. R. (2007). Constructive memory: the ghosts of past
and future. Nature 445:27. doi: 10.1038/445027a
Seidkhani, H., Nikolaev, A. R., Meghanathan, R. N., Pezeshk, H., Masoudi-
Nejad, A., and van Leeuwen, C. (2017). Task modulates functional connectivity
networks in free viewing behavior. Neuroimage 159, 289–301. doi: 10.1016/j.
neuroimage.2017.07.066
Frontiers in Psychology | www.frontiersin.org 5January 2019 | Volume 9 | Article 2701

fpsyg-09-02701 January 7, 2019 Time: 16:56 # 6
Nikolaev and van Leeuwen Latent Representations of Scenes
Simola, J., Le Fevre, K., Torniainen, J., and Baccino, T. (2015). Affective processing
in natural scene viewing: valence and arousal interactions in eye-fixation-
related potentials. Neuroimage 106, 21–33. doi: 10.1016/j.neuroimage.2014.
11.030
Standing, L. (1973). Learning 10,000 pictures. Q. J. Exp. Psychol. 25, 207–222.
doi: 10.1080/14640747308400340
Stokes, M. G. (2015). ’Activity-silent’ working memory in prefrontal cortex: a
dynamic coding framework. Trends Cogn. Sci. 19, 394–405. doi: 10.1016/j.tics.
2015.05.004
Summerfield, C., and Mangels, J. A. (2005). Coherent theta-band EEG activity
predicts item-context binding during encoding. Neuroimage 24, 692–703.
doi: 10.1016/j.neuroimage.2004.09.012
Tatler, B. W., Gilchrist, I. D., and Rusted, J. (2003). The time course of abstract
visual representation. Perception 32, 579–592. doi: 10.1068/p3396
Tatler, B. W., and Land, M. F. (2011). Vision and the representation of the
surroundings in spatial memory. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci.
366, 596–610. doi: 10.1098/rstb.2010.0188
Trübutschek, D., Marti, S., Ojeda, A., King, J. R., Mi, Y., Tsodyks, M., et al. (2017).
A theory of working memory without consciousness or sustained activity. eLife
6:e23871. doi: 10.7554/eLife.23871
van Ede, F., Niklaus, M., and Nobre, A. C. (2017). Temporal expectations
guide dynamic prioritization in visual working memory through attenuated
alpha oscillations. J. Neurosci. 37, 437–445. doi: 10.1523/JNEUROSCI.2272-16.
2016
Van Humbeeck, N., Meghanathan, R. N., Wagemans, J., van Leeuwen, C., and
Nikolaev, A. R. (2018). Presaccadic EEG activity predicts visual saliency in free-
viewing contour integration. Psychophysiology 55:e13267. doi: 10.1111/psyp.
13267
van Leeuwen, C., and Raffone, A. (2001). Coupled nonlinear maps as models of
perceptual pattern and memory trace dynamics. Cogn. Proces. 2, 67–116.
Voss, J. L., Bridge, D. J., Cohen, N. J., and Walker, J. A. (2017). A closer look at the
hippocampus and memory. Trends Cogn. Sci. 21, 577–588. doi: 10.1016/j.tics.
2017.05.008
Wolff, M. J., Jochim, J., Akyurek, E. G., and Stokes, M. G. (2017). Dynamic
hidden states underlying working-memory-guided behavior. Nat. Neurosci. 20,
864–871. doi: 10.1038/nn.4546
Conflict of Interest Statement: The authors declare that the research was
conducted in the absence of any commercial or financial relationships that could
be construed as a potential conflict of interest.
Copyright © 2019 Nikolaev and van Leeuwen. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (CC BY).
The use, distribution or reproduction in other forums is permitted, provided the
original author(s) and the copyright owner(s) are credited and that the original
publication in this journal is cited, in accordance with accepted academic practice.
No use, distribution or reproduction is permitted which does not comply with these
terms.
Frontiers in Psychology | www.frontiersin.org 6January 2019 | Volume 9 | Article 2701