ArticlePDF Available

Abstract and Figures

Visual mentation is the experience of visual images in the mind and includes visual aspects of perception, mental imagery, mind wandering and dream-ing. We propose an Integrative Theory of visual mentation (VM) that unifies biopsychological theories perception, dreaming and mental imagery. Un-der the theory, we make three major hypotheses where VM (1) involves the activation of perceptual representations in the temporal lobe, (2) is expe-rienced phenomenologically due to the activation of these representations, and (3) depends on shared mechanisms of simulation — dependent on a subset of the Default Network — that exploit these perceptual representa-tions. The resulting Integrative Theory informs the development of a com-putational model — and generative site-specific artwork — that generates visual images from perceptual, mind wandering and dreaming processes. These images are composed of shared perceptual representations learned during waking. Perception, mind wandering and dreaming are contiguous simulations of sensory reality modulated by varying degrees of exogenous and endogenous activation impacting a predictive model.
Content may be subject to copyright.
An Integrative Theory and Computational Model of Visual
Benjamin David Robert Bogart
School of Interactive Art and Technology, Simon Fraser University
Philippe Pasquier
School of Interactive Art and Technology, Simon Fraser University
Steven J. Barnes
Department of Psychology, University of British Columbia
Visual mentation is the experience of visual images in the mind and includes
visual aspects of perception, mental imagery, mind wandering and dream-
ing. We propose an Integrative Theory of visual mentation (VM) that unifies
biopsychological theories perception, dreaming and mental imagery. Un-
der the theory, we make three major hypotheses where VM (1) involves the
activation of perceptual representations in the temporal lobe, (2) is expe-
rienced phenomenologically due to the activation of these representations,
and (3) depends on shared mechanisms of simulation — dependent on a
subset of the Default Network — that exploit these perceptual representa-
tions. The resulting Integrative Theory informs the development of a com-
putational model — and generative site-specific artwork — that generates
visual images from perceptual, mind wandering and dreaming processes.
These images are composed of shared perceptual representations learned
during waking. Perception, mind wandering and dreaming are contiguous
simulations of sensory reality modulated by varying degrees of exogenous
and endogenous activation impacting a predictive model.
Perception does not produce a perfect replica of the world: Percepts are the results of sam-
pled sensory information that is integrated and extended according to expectations occurring at
multiple levels of abstraction and in relation to task demands (Triesch et al., 2003). Indeed, the
visual perceptual system has often been described as a creative process — e.g. Kandel & Wurtz
(2000). Under this conception, visual perception is the result of the construction of highly detailed
impressions of our external reality that are not reducible to either the world, or the organism’s
expectations. While perception is constrained by external sensory information, mental imagery in-
volves the construction of images experienced by the mind occurring in the absence of, or despite,
available sensory information. Through a review of a number of theories of dreaming and mental
imagery, the authors conclude that the mechanisms that allow constructive perception are the same
as those that allow mental imagery (Borst & Kosslyn, 2012), and are exploited in dreaming and
Dreams are meaningful simulations, however abstract, of a somewhat familiar world. These
simulations are sequences of events whose order is determined by feedback in a predictive model
of the world. Dreams can be as simple as banal thoughts or images, or as complex as long recurring
melodramas. They are often thought of as bizarre, and may include chimeric elements — fusions of
multiple places or people. In this review, we focus on two popular and conflicting biopsychological
conceptions of dreaming: Dreams result from our perceptual system’s attempt to make sense of
activation in sensory regions of the brain, or dreams are akin to mental imagery and are largely
independent of the early sensory systems. Visual mentation is the phenomenological experience
of images in the mind, be them self-generated (in the case of dreaming, mind-wandering and
mental imagery) or the result of external stimuli (perception). Mind wandering (day-dreaming)
and dreaming overlap in their content and involve similar modulations of cognition (Domhoff,
2011; Fox et al., 2013) — e.g. lack of self-reflective awareness. Perception is highly constrained and
anchored by external stimuli while dreams occur largely in the absence thereof. Mind-wandering
is a shift of attention away from external stimuli toward internally generated imagery.
The contributions of the present work are threefold: First, we review biopsychological mech-
anisms of visual mentation, in particular perception, mental imagery, mind-wandering and dream-
ing. Second, we propose an Integrative Theory of visual mentation where constituent phenomena
exploit overlapping mechanisms of simulation.These mechanisms are enabled by a common set
of perceptually-oriented cortical representations which are contextualized by a predictive model.
Third, our theoretical proposal serves as the foundation of a computational model (situated agent)
and artwork that constructs images from sensory components through generative processes mani-
fest during each of three cognitive processes: (1) perception, (2) mind wandering and (3) dream-
ing. It is assumed that the dreams and mind wanderings of the system will appear similar to those
of children, and perhaps non-human animals, and not human adults.
Our computational model serves three major purposes in itself. First, it formalizes the Inte-
grative Theory and provides a site for the evaluation and refinement of that theory. Second, it is also
an artwork, titled Dreaming Machine #3, contextualized in a series of generative site-specific com-
putational installations (Bogart & Pasquier, 2013). Thirdly, the manifestation of the theory in an
artwork broadens discourse regarding visual mentation beyond traditional scientific contexts. The
interdisciplinary nature of this work is expected to be mutually beneficial: (a) an artistic perspec-
tive and critique of biopsychological theories of dreaming is expected to broaden and generalize
existing theory and (b) a rigorous analysis of biopsychological theories enriches and deepens the
This section of the paper reviews a number of biopsychological theories of visual menta-
tion. It will be shown that these theories all share particular characteristics that serve as the basis
of the Integrative Theory, to be presented in a later section. We describe two major theories of
dreaming: (1) According to Hobson’s Activation-Synthesis (2000) and AIM (2009) theories, dream
experiences are the result of high-level perceptual processes making sense of activations of early
sensory regions of the brain, and (2) Nir and Tononi’s (2010) proposal that dreams are more closely
related to mental imagery than perception. Kosslyn and Thompson’s (2003) theory of mental im-
agery compliments Nir and Tononi’s proposal. Additionally, we present Domhoff’s (2011) proposal
that dreaming and mind-wandering recruit overlapping components of the default mode network
Visual Mental Representation
In the introduction, it was stated that visual mentation is dependent on a common set of
perceptually-oriented cortical representations, learned from perceptual experience. Additionally,
they are characterized by being high-level and abstract. By high-level we mean that these represen-
tations are highly invariant to shifts in scale, orientation and lighting. This invariance results from
increasing cortical processing in the ventral stream. These representations are abstract because
increasing cortical processing causes them to decreasingly resemble the patterns of light projected
upon the retina. These representations are diffusely encoded in the medial temporal lobe (Graham
et al., 2010), in particular the perirhinal cortex, and active during visual mentation. At a cognitive
level of description, these representations can be considered percepts that correspond to concrete
concepts, such as table, ball, sky, et cetera. Through the remainder of this paper, we will refer to
these representations as perceptual representations.
Sleep, Dreaming and Mental Imagery
A description of dreaming is incomplete without an overview of sleep, a behavioral state dis-
played by many mammals and birds (Siegel, 2008) that is modulated by a circadian clock entrained
by zeitgebers — e.g. visual brightness and social interaction (Morin & Allen, 2006). Sleep is usually
divided into four stages: Rapid eye movement sleep, REM Sleep (Aserinsky & Kleitman, 1953), and
non-rapid eye movement sleep, NREM stages 1 through 3. Sleep begins with NREM stage 1, which
is quite similar to waking, in terms of the electroencephalogram (EEG), and is characterized by
high frequency and low amplitude EEG. Sleep then descends through increasing NREM stages (2 to
3). Each subsequent stage is characterized by an EEG with greater amplitude and lower frequency,
with stage 3 being associated with the lowest frequency and the highest amplitude waves, and
also known as slow-wave sleep (SWS). Once a sleeping individual has progressed through NREM
stages 1 to 3, the progression reverses back through the stages where REM sleep may occur after
NREM stage 1. In addition to rapid eye movements, REM sleep also tends to be accompanied by a
variety of other physiological changes (e.g. the loss of muscle tone). An entire cycle typically lasts
approximately 90 minutes. The sleeper spends the remainder of the night repeating these cycles
and oscillating between sleep stages. However, an important change occurs as a typical night of
sleep unfolds: The first half of a night’s sleep contains much more NREM stage 3 sleep, whereas
the second half contains much more NREM stage 1 and 2, and REM sleep.
Dreaming can occur during any stage of sleep (Nielsen, 2000). Nevertheless, there are some
general conclusions that can be made about the distribution of dream content over a typical night
of sleep. Early on in the night, narrative dreams are more likely to be reported when awaking
subjects during REM sleep. By contrast, when waking a subject from NREM sleep early in the
night, their dreams are more likely to be reported as “. . . short, thought-like, less vivid, less visual
and more conceptual, less motorically animated, under greater volitional control, more plausible,
more concerned with current issues, less emotional and less pleasant” (Nir & Tononi, 2010). Later
in the night, dreams are reported as being “considerably longer and more hallucinatory”, (Nir et
al., 2013) irrespective of the stage from which a subject is woken.
In this section, we describe two popular biopsychological theories of dreaming, as well
as their relation to perception. First, Hobson’s Activation, Input/Output Gating, and Modulation
Early Visual Cortex
Prefrontal Cortex (PFC)
Temporal (TL)
Early Visual Cortex
Prefrontal Cortex (PFC)
Temporal (TL)
Figure 1.: Information flow during: (1) dreaming as perception, initiated by PGO waves originating
in the brainstem, according to Hobson (left) and (2) dreaming as imagination, according to Nir and
Tononi (right).
(AIM) (J. A. Hobson, 2009) theory proposes that dreams are the result of high-level cognitive
processes that shape activations of early sensory regions into narrative sequences. The resulting
perceptual experience is a functional simulation of reality. Second, Nir and Tononi (2010; 2013)
propose that dreams are more similar to mental imagery than to perception, and are not dependent
on the activation of early sensory regions.
Dreams as Perception. Hobson’s Activation, Input/Output Gating and Modulation (AIM)
model (2009) is the successor to the activation-synthesis theory (J. A. Hobson & McCarley, 1977).
AIM proposes that, during early development, dreams are important for the emergence of the pro-
toself (a precursor to the sense of self); dreams provide a “. . . virtual reality model of the world
that is of functional use to the development and maintenance of waking consciousness” (J. A. Hob-
son, 2009). This virtual reality is a free-running simulator of possible sensory and motor scenarios
which is initiated by the activation of early sensory regions. The protoself develops to account,
and take responsibility, for unconscious cognitive operations that respond to both external (during
waking) and internal (during dreaming) stimuli. It is presumed that the protoself develops through
the incremental growth of executive mechanisms (e.g. working memory, attention and planning)
in the prefrontal cortex (PFC) that structure and contextualize automatic and predictive activity
in the temporal, occipital and parietal cortices. AIM constitutes a three dimensional state-space of
neurological properties:
Activation: During waking and REM sleep, the whole brain is highly activated and during
NREM sleep it is minimally activated. Activation during REM sleep is due to PGO waves (Callaway
et al., 1987), so named because they have been measured in the pontine brain-stem (P), the lateral
geniculate nucleus (G), and in the occipital cortex (O), and originate in the pontine brainstem. PGO
waves have also been shown to “. . . occur in sensori-motor systems in the forebrain” (J. A. Hobson,
2009) and cause activation of the various sensory systems, in particular vision, “. . .from the motor
side up.” (J. A. Hobson, 2009) This activation is not driven by external (exogenous) stimuli. The
PGO waves “. . . encode saccadic eye movement directions that are commanded by the oculomotor
brainstem network” and cause the early sensory regions to generate imagery in the absence of
sensory information from the eyes. Thus, PGO waves drive eye movements during REM sleep that
also result in activation of early sensory regions. This activity is then interpreted by the same
mechanisms as external perception, pictured by the top black arrow in Figure 1 (left) which shows
the ventral stream of visual processing.
Input-output gating: During REM sleep, PGO waves begin and the reticular activating system
(in the brainstem) disconnects the body from the brain, resulting in temporary paralysis (motor
output) and a loss of most sensory afferents (sensory input).
Modulation refers to the change of neurotransmitter levels: During REM sleep, aminergic
neurons are inhibited and cholinergic neurons are activated. This results in an attenuated influence
of the PFC, which accounts for the poor recall of dream material and a lack of self-awareness.
By plotting the degree of activation, gating and neurotransmitter modulation in the AIM state
space, it is possible to contextualize exceptional conscious states: In lucid dreaming the balance
between aminergic and cholinergic neurons is even, gating is partial, and activation is as high as
during waking. States of conscious deficit, such as coma, include a lack of aminergic activation, a
wide range of degrees of gating, and a low degree of activation.
According to Hobson’s proposal, perceptual functions transform activity in early sensory re-
gions into a cohesive, and even narrative, subjective experience that is an exploitation of the organ-
isms model of the world. According to this conception, a dream is the output of our sensory and
perceptual functions, which are necessarily intertwined with the learning and exploitation of the
predictive model. Hobson (2009) provides little discussion of the structure of PGO waves beyond
their relation to saccadic eye movements. It is unclear how these saccadic control signals could
cause early sensory activations that would lead to the complexity and narrative qualities of many
dreams. Hobson’s theory depends on a strong correlation between REM sleep and dreaming, yet
this correlation is weaker than was once believed (Siegel, 2011). Hobson himself acknowledges this
inconsistency: “[a]n important caveat is that although the distinctive features of dream conscious-
ness. . . are maximally correlated with REM sleep, they are also found — to a limited degree — in
NREM sleep. . . ” (J. A. Hobson, 2009). So if PGO waves cause the narrative qualities of dream ex-
perience, then what causes the dream-like qualities of phenomenological experience during NREM
AIM and Free Energy Minimization J. Hobson & Friston (2012) have developed an alternate
conception that links the existing AIM theory and free energy minimization. The free-energy for-
mulation posits that biological systems resist disorder by minimizing surprise (Ashby, 1947). Fris-
ton (2010) defines surprise as the improbability of sensations given a model of the world, i.e.
surprising sensations are those that have not been predicted. Since an organism can’t control the
environment, surprise cannot be directly minimized; The minimization of free energy improves the
organisms ability to predict: “. . . free energy is always greater than surprise, which means that min-
imizing free energy implicitly minimizes surprise. . . ” (J. Hobson & Friston, 2012). Free energy is
a way of looking at an organisms ability to stabilize itself in opposition to changing environmental
factors (homoeostasis); For example, a warm-blooded mammal’s ability to keep a constant internal
temperature despite the changing temperature of the environment.
According to the free energy formulation of AIM, predictions are learned during waking
consciousness and contribute to the organisms “virtual reality model” of the world. The model is
optimized during REM sleep through the reduction of its complexity. The function of sleep (and
in particular REM sleep dreaming) is to refine and compress the predictive model of the world on
which the organism depends. The most significant contribution of the free energy formulation is
that the aspect of prediction is central, and dreaming is described functionally as the optimization of
the predictive model. The AIM proposal is that the early sensory regions, in particular vision, cause
activation of higher level perceptual systems, which entails the simulation of a “virtual reality”
learned from external stimulus, but experienced phenomenologically in the absence of external
Dreams as Mental Imagery. Nir and Tononi (2010) provide an alternative account, rooted in
a criticism of Hobson’s theory, that proposes that dreams are more similar to mental imagery than
perception. Hobson’s theory depends on a correlation between REM sleep and dreams, and does
not explain, and in fact is weakened by, the notion of dreams in NREM sleep. The qualities of the
dreams that occur in REM and NREM sleep have been reported as having different characteristics,
see Nir & Tononi (2010, page 94), but Hobson does not explain how NREM dream-like mentation
could occur in the absence of PGO waves and saccadic eye movements.
Additionally, Hobson’s model, according to Nir and Tononi’s interpretation of J. A. Hobson
et al. (2000), involves a change in the directionality of signal propagation, suppressing feedback
and enhancing feed-forward connections: “High levels of acetylcholine in the absence of aminergic
neuromodulation might enhance feed-forward transmission and suppress back-propagation.” (Nir
& Tononi, 2010) This change in directionality could also explain the weakened activity in parts of
the PFC during REM sleep, and high inhibition during NREM sleep (Muzur et al., 2002). Nir and
Tononi cite lesion studies that “. . . suggest that dreaming is more closely related to imagination
than it is to perception.” These studies indicate that dreaming depends more on the forebrain than
the “brain-stem REM generator” (PGO waves). In many cases, damage to the temporo-parieto-
occipital junction also leads to total cessation of dreaming (Solms, 2000), and “. . . supports various
cognitive processes that are essential for mental imagery.” Damage to the prefrontal cortex, for
example due to leucotomy, leads to total cessation of dreams in 70–90% of subjects, who also
exhibited a “. . . lack of initiative, curiosity and fantasy in waking life.” (Nir & Tononi, 2010) In
general, damage to perceptual areas leads to deficits in both perception and dreaming: “. . . lesions
leading to impairments in waking have parallel deficits in dreaming.” (Nir & Tononi, 2010)
The development of dreams in infants appears to correlate with the development of mental
imagery, and not linguistic nor memory ability. Analysis of the content of children’s dreams shows
that the younger the child, the more simplistic the dream experiences. This is not due to a lack
of linguistic ability: “. . . although children of age 2–5 years can see and speak of everyday people,
objects and events, they apparently cannot dream of them.” (Nir & Tononi, 2010) The dreams
of children of this age are further characterized by having “. . . no characters that move, no social
interactions, little feeling, and they do not include the dreamer as an active character. There are also
no autobiographic, [or] episodic memories. .. ” (Nir & Tononi, 2010). Children under 7 years report
dreams when awakened from sleep only 20% of the time, when compared to 80–90% in adults.
Nir and Tononi provide a compelling argument that dreaming may in fact be more closely related
to mental imagery (top-down) than perception (bottom-up), and therefore that neural mechanisms
used in mental imagery are in play in the case of dreaming.
Nir and Tononi do not provide citations that make the direct connection between abilities on
mental imagery tests — e.g. the Block Design Test (Wechsler, 1967) — and the content of children’s
dreams. Indeed, a significant difficulty with their argument is the childrens’ reporting ability: How
could a child understand the concept of a dream without experiencing one? If there is a correlation
between dreams and mental imagery but not linguistic ability, it implies that the system that allows
both mental imagery and dreams is different (in terms of connections, access or network dynamics)
than the system that leads to linguistic ability. Nir & Tononi (2010) note that there is at least one
significant difference between dreaming and mental imagery: “. . . while imagining, one is aware
that the images are internally generated (preserved reflective thought).” The lack of comparable
reflective thought in dreams could be explained by the relative lack of activation in the PFC during
non-lucid dream sleep.
What Causes Activation During Dreaming? A significant issue with Nir and Tononi’s account
is the absence of a specified cause of the activation during dreaming. Hobson locates the cause
of dreaming in PGO waves. According to Nir & Tononi (2010), activation could be due to inten-
tional prefrontal control during dreaming, but where self-awareness is inhibited — dreams are the
same as mental images except they are not recognized as being intentional. This is interesting to
consider in relation to the lack of self-reflective thought in (non-lucid) dreaming, and the role of
the temporo-parieto-occipital junction in both mental imagery and dreaming. Nir & Tononi (2010)
also make reference to a possible role of the DMN, due to the partial overlap of associated brain
regions active during dreaming and the DMN structures, in particular the PFC and temporo-parieto-
occipital junction. Possible links between dreaming and the default network will be discussed later.
Visual imagination and mental imagery are used interchangeably by Nir & Tononi (2010), and are
not explicitly defined. It is clear that they consider perception as being “bottom-up”, where sensory
information is processed by increasingly high-level perceptual processes, and mental imagery as
“top-down”, where processing happening in higher level abstract brain areas (presumably along
the ventral visual stream) causes the experience of visual images in the mind.
Whether dreams are similar to perception or mental imagery, there appears to be a consensus
that dreams are the result of the activation of high-level perceptual representations in the medial
temporal cortex, including the hippocampus, parahippocampus, entorhinal, and perirhinal areas.
Hobson, Nir and Tononi’s theories appear to be overly concerned with brain directionality (bottom-
up vs. top-down). Studies in attention make the case that high-level brain function effects low-
level perception (Triesch et al., 2003), and the opposite is obviously the case. Perhaps there is little
difference between these two options: (a) PGO waves activate early sensory regions and lead to
perceptual activation (J. A. Hobson, 2009), and (b) the activation of perceptual representations
leads to the activation of early sensory regions (Nir & Tononi, 2010). In both cases, perceptual and
sensory systems are activated, and causality is difficult to disentangle. While there is a lack of an
explicit definition of mental imagery, Nir & Tononi (2010) cite S. M. Kosslyn (1994), who proposes
a theoretical account for the presence of mental images in the mind functionally dependent on the
early visual cortex and explains their resemblance to perception. However, it is disputed whether
there is a functional role for the early sensory regions in dreaming and metal imagery.
Perceptual Anticipation Theory. Kosslyn (2003; 2006) proposes the Perceptual Anticipation
Theory of mental imagery which proposes a functional role of the early visual system: “. . . mental
images arise when one anticipates perceiving an object or scene so strongly that a depictive repre-
sentation of the stimulus is created in [the] early visual cortex” (S. M. Kosslyn & Thompson, 2003).
The act of imagining an object involves the construction of a sensory impression of that object in
sensory cortex.
The patterns that define these mental images (visual long-term memory) are not stored in
the visual cortex (VC), but are encoded, using “population coding” (Stokes et al., 2009; Young &
Yamane, 1992), in the temporal lobe (TL). In population coding, a pattern is not reflected in the
firing of a particular neuron, but in the firing of a group of neurons. Unlike the arrangement of
the VC, these representations are non-topographical and therefore implicit. They can only be made
explicit through the constructive activation of the topographically oriented early visual system:
“. . .image generation is not simply ‘playing backward’ stored information, but rather is necessarily
a constructive activity” (S. M. Kosslyn & Thompson, 2003). These representations do not include
spatial information, which is encoded elsewhere.1The constructive activation of early VC from TL
representations is analogous to that pictured as a black arrow in Figure 1 (right).
Once the TL representations are decoded into the VC, they are perceived using the same
mechanisms as external visual perception, not pictured in Figure 1 (right). Kosslyn’s proposal
that the early VC is functionally required in mental imagery is due to the overlapping con-
straints between mental imagery and perception, in particular the results of visual scanning ex-
periments (S. Kosslyn, 1973; S. M. Kosslyn et al., 1978; Pinker & Kosslyn, 1978; Pinker, 1980), to
be discussed in later. In many of these studies it is noted that those subjects most adept at imag-
ining images, as measured by the Vividness of Visual Imagery Questionnaire (Marks, 1973), showed
the smallest differences between perceptual and mental imagery constraints.
Reconstructed mental images can be used to further conceptualize images propositionally
or linguistically: “. . . reconstructing the shape in topographically organized early cortex affords an
opportunity to reinterpret the pattern." (S. M. Kosslyn & Thompson, 2003) Activation in the early
VC is expected to occur when the task requires: (1) a higher resolution representation than is
afforded by the linguistic system, (2) a specific example of such an object — not a prototype of a
class and (3) the inspection of object-centric properties (e.g. color and size) — not spatial relations
(e.g. position).
Are Mental Images Visual or Propositional? Since the 80’s Kosslyn (1979; 1980; 1981; 2003)
and Pylyshyn (1981; 2002; 2003a; 2003b) have been engaged in a long standing debate on the
nature of mental images. Pylyshyn’s “propositional” theory contends that mental images are not
images, but symbolic / propositional descriptions of visual properties that do not depend on the
visual system. Rather than being rooted in visual experience, these representations are composed
of the same kinds of abstract symbolic codes used in language. According to Pylyshyn, any acti-
vation in the early visual system during mental imagery is spurious and nonfunctional. Dominic
Gregory (2010) attempts to resolve these two views through a philosophical framework in which
the intrinsically visual content of mental images are at the forefront, and that the underlying format,
or encoding, contributes to an impasse in resolving imagistic and propositional conceptions.
Shared Representations in Perception and Imagery Kosslyn’s account is specifically focused on
the early visual system and assumes that long term visual memory is located in the inferior TL. The
role of the inferior and medial TL in perception and memory is questioned by the Emergent Memory
Account (EMA) as proposed by Graham et al. (2010). According to the EMA, the medial temporal
lobe is not simply a storehouse for memory, but is specialized for perceptual representations that
are used in both memory and perception. If the EMA is correct, then visual mental imagery would
involve much of the medial temporal lobe (MTL) and may not even depend on the early visual
system. Under the EMA hypothesis, perceptual information leads to the activation of perceptual
representations in the MTL. This account is incompatible with Kosslyn’s theory that predicts a
functional role of the early VC following the overlapping constraints in visual perception and mental
imagery, to be discussed in detail later.
1For more information regarding the processing of object and spatial information see Mishkin et al. (1983); Goodale
& Westwood (2004); Murray et al. (2007).
Summary of Sleep, Dreaming and Mental Imagery
Hobson theorizes that dreams involve mechanisms similar to external perception, while Nir
and Tononi consider that dreams involve mechanisms that more closely overlap with those of men-
tal imagery. Both theories posit a key role for perceptual representations in long term memory.
Kosslyn proposes that mental imagery exploits the same functions as external perception, and men-
tal images are decoded into early visual cortex from encoded TL representations. Thus we can
consider dreaming, mental imagery and external perception as highly related and that they may
share neural mechanisms, including a role for perceptual representations situated in the TL. The
mechanisms that cause the activation of these representations is, as yet, unclear. The default net-
work, which is active in both dreaming and mind wandering, may explain the source of activation
of perceptual representations in dreaming.
The Default Mode Network and Dreaming
As noted by Domhoff (2011), current research on dreaming certainly emphasizes a continuity
between dreaming and waking cognition. Although notions of dreams as bizarre narratives have
captured our collective imagination for a significant period, even early dream content studies from
the 1970s indicated that bizarre dreams are exceptional; most dream reports are “. . . clear, coherent,
and detailed account[s] of a realistic situation involving the dreamer and other people caught
up in very ordinary activities. . . (Domhoff (2011) citing Snyder (1970)). A consideration of
dream content as ordinary and the inclusion of terms describing meta-awareness (e.g. contemplate,
decide,realize,ponder, etc.) in dream reports supports a continuity between dreaming and waking
consciousness. Studies of relaxed waking (such as mind-wandering) in laboratory settings have
shown that these states can be “. . . as fragmented or unusual as dreams” (Domhoff, 2011). Physical
impossibilities and disconnected thoughts can be present in both dreaming and in mind-wandering.
The DMN is a set of neural structures that are highly active when the subject is resting (dur-
ing mind-wandering) and are inhibited during goal-oriented activity. Domhoff (2011) proposes
that dreams result from an activation of a subset of the DMN that “. . . is active when the mind is
wandering, daydreaming, or simulating past or future events.” Regions that are associated with the
DMN include the “. . .medial prefrontal cortex, the anterior cingulate cortex, and the temporopari-
etel junction. . . ”, which are also implicated in dreaming. Not all neural structures implicated in
the DMN are highly active during REM and NREM sleep. Domhoff proposes this lack of activation
during dreaming is because sensation, locomotion and executive functions are not necessary during
Domhoff proposes that DMN activity serves as a bridge between waking and dreaming con-
sciousness: Just before the onset of sleep, the DMN is likely to be active due to a relaxed state.
The shift from waking relaxed thought to sleep is rapid and DMN activity continues into NREM.
The central component of Domhoff’s proposal (2011) is the correlation between dreams and the
DMN, which provides a link to mind-wandering. This link is compelling as the characteristics of
mind-wandering (in particular the lack of volition and self-reflection) resemble those of dreaming.
The proposal that dreams and mind-wandering have common mechanisms is also supported
by a recent meta-study by Fox et al. (2013), which concludes that there are “. . . large overlaps in
activation patterns of cortical regions. . . in REM sleep, mind-wandering and DMN activity. Areas
associated with the DMN that are activated during mind-wandering and dreaming include medial
PFC, posterior cingulate, hippocampus, parahippocampus and entorhinal cortex (the latter three
being constituents of the medial TL). Note that the temporo-parieto-occipital junction (inferior
parietal lobule) was not found to be active during REM sleep in this meta-study. This is difficult
to resolve with evidence that damage to this area often leads to a cessation of dreaming according
to Solms (2000). The association between mind-wandering and dreaming is not limited to neu-
roimaging studies, but also supported by first-person reports where the phenomenology of both
include an emphasis on visual and auditory information, the inclusion ’bizarre’ and implausible
events, an emphasis on “ongoing waking concerns”, the inclusion of social interactions, and a lack
of meta-awareness (the awareness of being aware).
The implication of the DMN in dreaming proposed by Domhoff (2011) and extended by
Fox et al. (2013), and therefore that mind-wandering and dreaming may be enabled by shared
mechanisms, is compelling. It resolves the colloquial relation between dreaming and day-dreaming
(mind-wandering) with empirical evidence. If it is accepted, then “. . . dreams can be seen as a
unique and more fully developed form of mind-wandering, and therefore as the quintessential cog-
nitive simulation. [emphasis added]” (Domhoff, 2011). It is important to note that the overlap
between dreaming and the default network reported by Fox et al. (2013) is tied specifically to REM
sleep: “. . .the observed overlap with the DMN is not common to all sleep stages, but specific to REM
sleep—the only sleep stage truly reliably associated with dream mentation.[emphasis added]” (Fox et
al., 2013). Thus the link between the DMN and dreaming becomes muddy in NREM sleep. Still,
a role of the DMN network, in particular the associated PFC regions, at least partially answers the
open question as to what causes the activation of TL representations if we consider dreaming as
mental imagery. Additionally, Domhoff (2011) notes that the activity of the DMN through relaxed
waking into sleep indicates a continuity of mind-wandering, and dreaming where the difference is
due to network dynamics and not independent mechanisms.
Dreaming, Mind-wandering and Simulation Simulation and prediction have being considered
as frameworks for the organization of general cognition (Hesslow, 2002). Both J. A. Hobson (2009)
and Schooler et al. (2011) describe dreams and mind-wandering, respectively, as having predictive
functions informed by learned models of sensory reality, which is supported by overlapping activity
during dreaming and mind wandering. It has been proposed that subsets of the DMN function as
simulators: “the default network thus comprises at least two distinct interacting subsystems — one
subsystem functions to provide information from memory; the second participates to derive self-
relevant mental simulations.[emphasis added]” (Buckner et al., 2008) The former system being cor-
related with the TL and the latter the PFC. Together, these sub-systems of the DMN use components
of long-term memory in the construction of simulations. Such simulations are not only valuable for
the rehearsal of social interactions (Schooler et al., 2011) and threat response (Revonsuo, 2000),2
but could also contribute to waking perception. The combination of sensory information and a sim-
ulation, informed by a predictive model of the world, facilitates the ability to anticipate changes
in the environment and thus have obvious adaptive value. The predictions learned through wak-
ing perception that form the basis of these simulations also allow the unconscious modulation of
attention for particular scenarios, as evidenced by priming experiments, e.g. (Davenport & Potter,
In both dreaming and mind-wandering, our attention shifts to internal images where external
stimuli is largely ignored and we show impairments in executive functions. For example, during
mind-wandering the DMN is most active when subjects are not aware of their mind-wandering (e.g.
a lack of meta-awareness) (Christoff et al., 2009). The DMN is also active when people engage in
2Revonsuo (2000) proposes the threat simulation hypothesis where dreaming provides an adaptive function by allow-
ing the practice of threat perception and avoidance in the absence of real danger.
“personal planning concerning the future” Domhoff (2011). When we dream and let our minds
wander, we enter the simulated worlds of our imaginations.
Summary of Theories
This section covered significant territory, so we will summarize the key aspects of the re-
viewed theories:
1. Hobson (2009; 2012) proposes that dreams are the result of our perceptual mechanisms
attempting to make sense of the PGO activation of sensory regions during REM sleep. REM sleep
is characterized by: a similar degree of activation to waking, a disconnection of the brain from the
rest of the body, and a suppression of feed-back mechanisms. Dreams are considered functional
2. Nir and Tononi (2010; 2013) propose that dreaming is less like perception and more like
mental imagery.
3. Kosslyn (2003; 2006) proposes that mental images and external perception are subject
to similar constraints. During mental imagery, encoded representations in the TL are rendered in
early VC and perceived by the same mechanisms as external perception.
4. Domhoff (2011) proposes that dreams are enabled by the same mechanisms that support
mind-wandering, specifically a subset of the DMN, a position which is also supported by Fox et al.
(2013). Dreams result from the activation of perceptual information and provide the quintessential
mechanism of simulation, thus extending the function and phenomenology of mind-wandering.
The following two sections detail the contributions of the research presented in this paper. First, the
Integrative Theory is described, after which its companion computational and artistic realization is
Integrative Theory
The previous sections include discussions and a selection of key points of theory regarding
possible relations between visual aspects of external perception, mental imagery, mind-wandering
and dreaming (visual mentation). This section unifies theoretical points made in the previous
sections into the Integrative Theory where three central hypotheses are made:
1. Visual mentation involves the activation of perceptual representations located in the TL.
2. The phenomenological experience of visual mentation is due to activity of perceptual rep-
resentations within the TL.
3. Visual mentation is the simulation of perceptual information exploiting perceptual repre-
sentations and modulated by varying degrees of exogenous activation.
All theories discussed above predict some functional role of perceptual representations in the TL —
be they encoded or explicit, conceptual or perceptual, and thus support Hypothesis 1:
1. J. A. Hobson (2009) notes that TL epileptics experience seizures commonly characterized
as dreamy states. He also contends that dreams are perceived using the same mechanisms as
external perception, which implies a degree of TL processing.
2. Nir & Tononi (2010) link dreaming with mental imagery as the activation of representa-
tions implicated in mental imagery.
3. For S. M. Kosslyn & Thompson (2003), the TL is the storehouse for visual representations,
and overlaps with Hobson in proposing a functional role of early VC.
Early Visual Cortex
Prefrontal Cortex
Figure 2.: The Integrative Theory unifies external perception, mental imagery, mind-wandering
and dreaming.
4. Graham et al. (2010) link perception and memory and propose that TL damage leads to
deficits in memory recall and perception. The representations implied in mental imagery are shared
with external perception, and therefore presumably also in dreams.
5. Domhoff (2011) and Nir & Tononi (2010) cite lesion studies that show that damage to the
TL subsystem of the default network, in particular at the temporal-occipito-parieto junction, leads
to deficit in dreams, initiative, curiosity and fantasy, and even total cessation of dreaming.
Together, these points support the argument for a central functional role of perceptual represen-
tations located in the TL in visual mentation. While Hypothesis 1 is fairly well established, the
cause of the activation of perceptual representations is disputed. In order to arrive at a cohesive
conception, some aspects of the discussed theories must be rejected. We identify two explanations:
perceptual representation activation results from (1) early VC activation, as in external perception
(Hobson, Kosslyn), (2) prefrontal control (Domhoff, Nir and Tononi).
The functional role of early VC in mental imagery and dreaming is key to the theories pro-
posed by Hobson and Kosslyn. The relation between mental imagery and external perception has
been a topic of study since the 1970s and often conflate image recall (the recall of a particular
visual memory), imagery of a memorized image (mental imagery of a learned visual image) and
novel imagery (the construction of a new mental image not in memory nor perception). Stud-
ies have shown that mental imagery and perception share similar constraints, including field of
view (Finke, 1980; Marzi et al., 2006) and scan time (S. Kosslyn, 1973; Lea, 1975; S. M. Kosslyn
et al., 1978). Scanning time experiments, such as documented by S. Kosslyn (1973), involve the
subject memorizing a map containing a number of landmarks. The subject is then asked to imagine
the map with a virtual cursor superimposed on a particular landmark. The subject is then asked to
smoothly shift their mental image, while keeping the cursor static, to a target landmark, and report
when they arrive. The actual distance between landmarks is correlated with the mental scanning
time. This was also reproduced in three dimensions by Pinker (1980).
It has also been shown that the scale of objects in perceptual and mental images appear to
take up similar amounts of a virtual field of view of limited resolution: objects imagined at relatively
small scales take longer to interpret than objects at large scales (S. Kosslyn, 1976). Studies also have
shown that the ability to attend to details of objects imagined at a small scale is more difficult than
when imagined at a large scales (Moyer, 1973; S. M. Kosslyn, 1975; Paivio, 1975; S. Kosslyn, 1976;
S. M. Kosslyn et al., 1977). There is also evidence for interactions between perception and imagery,
including visual illusions (Mohr et al., 2011), perceptual deficits during imagery (Wais et al., 2010)
and the effect of the mental imagery on the perception of subsequent perceptual images (Diekhof
et al., 2011). The general argument for a functional role of the early VC in mental imagery is
that similar constraints imply similar mechanisms. These experiments support the notion of mental
imagery as depictive rather than symbolic, although it has been demonstrated in a computational
model by Sima (2011) that a symbolic system could still explain the mental image scanning results.
In Marzi et al.’s study (2006), a subject with damage to the early VC had no perceptual ability
in one visual quadrant, and yet was able to construct whole mental images. Most interestingly,
perceptual constraints in the blind quadrant — reaction time effects dependent on location of
stimulus in the field of view (Chelazzi et al., 1988) — did not apply to mental images as in normal
subjects. This indicates that the early VC modulates mental images but is not functionally required.
The lack of a functional requirement for the early VC is also supported recent studies which found
that multiple patients that acquired total cortical blindness have preserved mental imagery (Zago
et al., 2010; Bridge et al., 2012).
Recent fMRI decoding studies have allowed a more detailed examination of the role of the
early VC in mental imagery. Decoding studies have attempted to correlate patterns of brain activity
with particular visual stimuli. An analysis of brain activity can predict which visual stimulus a
subject is currently viewing. S. Lee et al. (2012) demonstrated that activity in the VC and TL could
predict an image either seen or imagined (after memorization) by a subject. During imagery, they
found a high degree of correlated activation relevant to the memorized stimulus in the TL, and low
stimulus-correlated activity in the VC. During perception, they found the opposite pattern, greater
stimulus-correlated activation in the VC and less stimulus-correlated activation in the TL.
As dreams occur in the absence of PGO waves, we can conclude that the experience of images
in the mind (mental imagery and dreams) is likely due to activity of perceptual representations in
the TL (Hypothesis 2) that is independent of the early VC. We are then left with two possibilities in
the case of dreaming: the activation of perceptual representations is due to intentional control from
the PFC, or it is due to endogenous activation of the TL. The DMN spans structures in both the PFC
and the TL, and due to the established link between mental imagery, dreaming, mind-wandering
and the default network, we can conclude that activation in the TL is due to activation in the DMN
system including portions of the PFC. The difference between the various states of visual mentation
are due to differing dynamics of the DMN: mental imagery results from intentional functions of
the PFC, external perception is highly dependent on external stimuli impacting the early VC, and
mind-wandering and dreaming are the result of non-uniform endogenous activation within the
Figure 2 depicts the causal patterns of three modes of visual mentation: External perception
is the result of exogenous activation of early VC which in turn causes perceptual representation
activation in the TL. Mental imagery is the result of PFC control mechanisms causing the activation
of perceptual representations, which result in the experience of mental images. Visual aspects of
dreaming and mind-wandering are the result of endogenous activation within the DMN, modulated
by varying degrees of control initiated by the PFC. This endogenous activation could be structured
as a feedback loop between the TL and PFC aspects of the default network. The PFC initiates activa-
tion in the TL, which results in further activation of the PFC, and results in subsequent activation of
the TL. Hesslow (2002) describes a similar feedback loop in terms of simulating chains of behavior
operating in perception where, “. . .during normal behavior, we will always, ‘in our thoughts’, be a
few steps ahead of the actual events.”
The exploitation of shared perceptual representations in visual mentation supports the var-
ious constituents of visual mentation are contiguous and can causally effect one another. For ex-
ample, it has been shown that waking perception in the hours before sleep has a significant effect
on dream content (Stickgold et al., 2000). Antti Revonsuo describes the continuity of perception
and dreaming: “We are dreaming all the time, it’s just that our dreams are shaped by our per-
ceptions when awake, and therefore constrained” (Tucker et al., 2009). We can then consider the
differences between the various constituents of visual mentation as due to the same or similar
neural mechanisms engaged in differing dynamics. These dynamics can even extend beyond the
waking / sleeping barrier, where some parts of our brain engage in sleep, while others remain vigi-
lant, simultaneously: “[S]leep or wakefulness may regularly occur independently in different brain
regions.” (Nir et al., 2013)
Hypothesis 3 states that a key functional attribute shared between processes of visual men-
tation is simulation. Considering the constructive aspects of external perception, we can conceive
of our experience of the world as a simulation that is anchored in sensory information. By con-
trast, mental imagery is a simulation relatively decoupled from external stimuli, but intentionally
controlled and constrained by task demands. Dreaming and mind-wandering result from these
same mechanisms of simulation, but operating independently of both task demands and sensory
Dreaming and mind-wandering are then free-running simulations that exploit our predictive
models of sensory reality. These simulations provide banal experiences related to waking concerns
due to their shared mechanisms with external perception. They may also provide the bizarre as-
pects of dreams such as chimeric elements and discontinuity because they lack the stability provided
by external sensory information. The predictions themselves have functional value in perception
(perceptual priming) and social interaction: “Perhaps a primary function of mind-wandering is
to generate the autobiographical predictions necessary to successfully navigate the complex so-
cial world.” (Schooler et al., 2011) Despite the possibility of their lack of cohesion (compared to
waking perception), the free running simulations could also provide adaptive functions including
the development of the “protoself” (J. A. Hobson, 2009) and the rehearsal of threatening situa-
tions Revonsuo (2000).
The particular brain systems that enable all of these simulations are a subset of the DMN;
which has been implicated in dreaming and mind-wandering. Domhoff characterizes dreams as
the quintessential cognitive simulation (Domhoff, 2011) because they are a more fully developed
form of mind-wandering. In summary, the Integrative Theory proposes that visual mentation is set of
closely related phenomena that all exploit the same mechanisms of representation and simulation.
The next section discusses the computational model and artwork that manifests key components of
the Integrative Theory.
Computational Model: A Machine that Dreams
The Integrative Theory is the foundation for the computational model. While the theory
includes a role for executive function in task-oriented mental imagery, the computational model
is centrally focused on aspects of visual mentation (perception, mind-wandering and dreaming)
independent of executive mechanisms such as working memory, attention and planning. Addition-
ally, the focus of the system design is the dynamics of percept activation that leads to dreaming
and mind wandering images, not the construction of percepts themselves. The process leading to
percepts is simply a method to provide the system a variety of perceptual information, and not a
model perceptual processes. The key features of the proposed theory manifest in the computational
model include: shared representations and processes utilized in perception, mind-wandering and
dreaming, and the explicit contiguity of those three processes of simulation.
Computational modeling provides a compelling framework for the theorization and critique
of biopsychological conceptions. The implementation of these ideas in formal language requires
sufficient detail as to force the specification of tacit aspects. The computational model is realized
as a situated agent that captures visual images with a video camera, and displays a window into its
’mental’ images. This display is a visualization of the perceptual, mind-wandering and dreaming
processes of the system.
During the day, the system memorizes and learns a predictive model of the visual world’s
spatial and temporal properties from visual information collected by the camera. ’Dreaming’ (at
night) and ’mind wandering’ (during low arousal) is the result of the exploitation of the predictive
model. The system’s processes simulate reality as constructed from perceptual representations
(learned during waking) and differ only in the degree to which these processes are anchored in
sensory information. For simplicity, sleep is not broken into stages and all sleep entails dreaming.
The arousal of the system (the degree of change in external stimuli) and a circadian clock cause the
transition between these contiguous processes. The limitations of the system’s experience, memory
and recognition ability lead to impoverished visual mentation. The design of the computational
model manifests the following key attributes:
Dreaming and mind-wandering are enabled by endogenous activation of predictive mech-
anisms causing activation of perceptual representations, following the Integrative Theory.
External perception, mind-wandering and dreaming are contiguous processes modulated
by varying degrees of influence from exogenous and endogenous activation.
Visual mentation is a simulation of the external visual world resulting from the activation
of perceptual representations as a result of feedback in the predictive model.
The system is a partial artificial agent that lacks executive mechanisms.
System Architecture
Figure 3 shows the architecture of the system. STIMU LUS provides images (St) captured
by the camera at time t, which are passed onto SEGMENTATION. For each frame, SEGMENTATION
breaks Stinto color regions using mean shift segmentation (Comaniciu & Meer, 2002), and passes
the set of segmented regions (Rst) onto CLUS TERING, which clusters segmented regions according
to features and constructs new PERCEPUNITS that represent similar regions in subsequent frames.
A stimulus is recognized when it is associated with an existing cluster. The set of PERCEPUNITs
(P st) is passed onto SUPP RE SSOR, which determines the balance between exogenous (caused by
external stimuli) and endogenous (caused by feedback from PREDICTION) activation. This bal-
ance determines the weighting and combination of inputs from CLUS TERING and PREDICTION
External Visual Context
Figure 3.: System Architecture
as controlled by AROU SA L and the circadian CLOC K in the absence of executive mechanisms. The
CLO CK indicates day or night as determined by the brightness of STI MU LUS and ARO US AL indicates
the change in STIM ULUS over time. Activated PERCEPUNITS are made visible on the display by
the RE NDERER. PREDICTION learns to predict the next state of PERCEPUNIT activation considering
the current state of PERCEPUNIT activation. These predictions are feed back into PREDICTION, as
mediated by SUPP RE SSION. Each of these modules is described in detail in the following sections.
STI MULUS. This module provides visual stimuli to the system, an example of which is pic-
tured in Figure 4. The stimulus is a single video frame (St,where tis the frame number). The
current stimulus is passed to SEGM EN TATI ON and AROUSAL, while its mean luminosity (Sl
t) feeds
the circadian CLOC K.
CLO CK. The CLOCK is a discrete square wave that indicates day when the mean luminosity of
STI MULUS (mean(Sl
t)) is above a threshold (Tl), and night otherwise: if Sl
t> Tlthen clock = 1,
otherwise clock = 0. The CLO CK provides the onset of day and night and modulates SU PP RESSION,
to be discussed later.
ARO USAL. The ARO USAL produces a discrete square wave that indicates change in STIMULUS
between t1and t. It is calculated from the absolute difference in mean luminosity of all pixels in
t1and Sl
t. If the difference is above a threshold, kmean(Sl
t1), mean(Sl
t)k> Ta, then arousal =
1, otherwise arousal = 0.
SEG MENTATIO N. SE GM ENTAT IO N breaks STIMULU S into color regions according to the mean
shift segmentation method described by Comaniciu & Meer (2002). The resulting regions, an ex-
ample of which is pictured in Figure 6, are extracted through thresholding and contour finding as
Figure 4.: Example STIMULUS provided to system.
follows and detailed in Algorithm 1: The mean shift algorithm (Line 2) is run on a hue, saturation
and value (HSV) color-space version of STIM ULUS (Line 1). A histogram (H) is computed from
the value channel of the resulting image (Line 4), and used to determine the pixel values of seg-
mented regions. For each bin (b) of the histogram (Line 5), if there are pixels in this bin (Line 6) a
binary image (BI ) is calculated where white pixels indicate the value of the bin (bv) matches the
segmented region (Line 7); As each pixel value may contain multiple discontinuous color regions,
contour finding is used to determine the bounding box around each contiguous region (Line 8),
and its center position (Line 9); A region instance (R) is constructed from STIM ULUS (Line 10),
using the bounding box (bb) and binary image (BI ) to calculate a mask, and is associated with an
x, y position in the frame, as pictured in Figure 6.
1: Shsv
2: Sms
tmeanShif tSegmentation(Shsv
3: Sl, ms
4: Hhist(Sl, ms
5: for bHdo
6: if bc>0then
7: BI Sl, ms
8: bb boundingBox(findContours(BI))
9: x, y position(bb)
10: RregionU nit(St, BI, x, y)
The resulting region instances, an example of which is pictured in Figure 6b, are stored in a
(a) Results of mean shift segmentation of STIMU LUS.
(b) Grey scale version of mean shift output (Sl, ms
Figure 5.: ST IMULU S is broken into regions according to the mean shift segmentation method.
(a) Boolean image (BI ) resulting from a comparison
of histogram values and gray-scale mean shift output.
(b) The image in Figure 6a is used to calculate a num-
ber of contours. For each contour is used to extract St,
where the mask is applied as an alpha channel.
Figure 6.: An example region extracted from mean shift output (Sms
set (Rst← {R0, . . . , Rn}) and passed onto CLUS TE RING. Each region is associated with an image,
a mask and the following features (f):
Rx,y Position of the center of the segmented bounding box around the contour calculated
during SE GMENTATION, normalized to range from 0 to 1, where x= 1, y = 1 corresponds to the far
bottom right corner of STIM ULUS.
Rh,s,v Mean color values in hue, saturation and value color-space of St, normalized to
range from 0 to 1.
CLUSTERI NG. Each Rextracted from Strepresents a contiguous region of pixels at a particular
moment in time. Clustering allows the recognition of regions that persist over time, through the
construction of PERCEPUNITS from regions captured at differing times. PERCEPUNITS, an example
of which is pictured in Figure 7, are abstractions of the raw stimuli provided by SEGM EN TATI ON, and
are, like regions, associated with images, masks, and the region features. CLUSTERI NG compares
regions segmented from the current frame (Rst) to PERCEPUNIT clusters (P s) composed of regions
segmented from previous frames. These PERCEPUNITS are destructive in that all RE GIONS belonging
to a particular cluster are represented by a single PERCEPUNIT whose properties are a weighted
average of constituent regions, including images, masks and features. Hereafter, this process of
destructive clustering will be referred to as merging. When a newly segmented pixel region is
merged with a PERCEPUNIT, it is analogous to the system’s recognition of that sensory information,
where the PERCEPUNIT is updated to take into account the new sensory information. For each
merge, the resulting PERCEPUNIT is set with an initial activation (PA= 1, where 0PA1).
The clustering method is a modified version of the Basic Sequential Algorithmic
Scheme (Theodoridis & Koutroumbas, 2009). Clustering begins with an empty set of PERCEPUNITS
(P st) and is detailed in Algorithm 2, where the maximum number of PERCEPUNIT S (q) is specified a
priori and clustering proceeds for each tand nis the current number of PERCEPUNITS: loop through
each newly segmented region (Line 1) and determine which PERCEPUNIT is closest to this region
(Rt) over all features (Line 2); If we have not reached the maximum number of PERCEPUNIT S, and
the distance between the region (Rt) and the nearest PERCEPUNIT is above a threshold (Line 3),
then make a new cluster from the region (Line 4), activate it (Line 5) and increment n; otherwise
update the closest cluster (C) by merging it with the region (Lines 8 and 9) and activate it.
The result is a set of PERCEPUNIT S (P st) that eventually converges to a fixed size (q) where
Algorithm 2 Clustering
1: for RtRstdo
2: CargminikRf
t, P f
iP stk
3: if kRf
t, Cfk> Tsand n<qthen
4: P st+1 P stRt
5: RA
6: nn+ 1
7: else
8: Pt+1 merge(Pt, C )
9: CPt+1
10: CA1
11: tt+ 1
each approximates a point in feature space and is represented by the weighted average of the
constituent regions. Figure 8 shows a collection of PERCEPUNIT S constructed over 150 frames,
where q= 900. The output of CLUS TERING is the state of activation of all PERCEPUNIT S (At
{P st|P sA
t= 1}) and is passed to both SUPP RE SSION and PREDICTION.
Figure 7.: An example of the image corresponding to a clustered PERCEPUNIT which is constructed
from pixel regions segmented at different times.
PREDICTION. The PREDICTION module is a Multilayer Perceptron (Rumelhart & Williams,
1986) which serves as a generic prediction mechanism. The network consists of the same number
of input units as there are clusters (q), q/2hidden units in one layer and qnumber of output
units. The hidden and output units use symmetric sigmoid activation functions with slopes of 1.
This module buffers the state of activation that corresponds to the previous frame (At1), which
is fed to the network as a training sample. The current state of activation (At) provides the target
vector for supervised back-propagation learning. The output units are discretized as a vector of
Boolean values where each element corresponds to each PERCEPUNIT (P sP
t1), where 1 indicates
the PERCEPUNIT is predicted to be present at tand 0 otherwise. The learning rate is fixed at 1 as
the network is learning from continuous live data and learning samples are not repeated.3
The training delta (error) is the binary distance between P sP
t1and At. In order for the net-
work to descend towards convergence, Atis only fed to the network when the difference between
3The system does not buffer activation patterns over time and thus learning is continuous, rather than epoch, oriented.
Figure 8.: A collection of approximately 900 PERCEPUNIT S drawn in the location in which they
were segmented. The lack of temporal stability in the mean-shift edges cause fragmented percepts
which is compounded by arbitrary stacking of percepts in this rendering, e.g. the bicycle without a
rider above.
P sp
t1and Atis above a threshold. At time t, the network provides a prediction of what PER-
CEPUNITS are expected to be activated at t+ 1 such that when the prediction is perfect (delta = 0)
then P sP
t1=At. PREDICTION learns from current perceptual activation from CLUS TERING, and
exploits that learning by calculating a prediction of the next state (P sP
t), based on the current state
(At), which is passed onto SUPP RE SSION along with the expected prediction for this t(P sP
SUP PRESSIO N. SUPPRE SS ION determines the balance between activation caused by feed-
back from PREDICTION (endogenous) and activation caused by external STIMULU S (exogenous),
as shown in Figure 3. It’s input from CLUS TERING is the set of all activated PERCEPUNIT S (At) while
it’s input from PREDICTION is the set of all percepts predicted to be currently activated (P sP
When the system is waking (clock = 1 and arousal = 1), the output of SU PP RESSOR is both pre-
dicted (P sP
t) and activated percepts (At), which are passed to RENDERER, to be explained later.
When the system is mind wandering or dreaming (clock = 1 0and arousal = 0), activation from
STI MULUS (At) is suppressed and predicted PERCEPUNITS are fed back to PREDICTION as inputs for
the next frame (AtP sP
REN DERER. This module visualizes the system’s current state of activation (At) and the degree
to which those activations are expected (P sP
t1) for this t. During waking, all activated percepts
are rendered. If the percepts are not predicted, then they are rendered 100% opaque, indicating
they are a surprise. If the percepts are predicted, then they are rendered with 50% opacity. As
a subset of PERCEPUNIT S is active at any one moment during perception, mind wandering and
dreaming, there are portions of the rendered image that do not include perceptual information.
arousal = 1 (High)
arousal = 0 (Low)
clock = 1 (Day)
clock = 0 (Night)
Mind Wandering
Figure 9.: Cognitive processes in relation to CLOCK and AR OU SAL states.
In order to deemphasize these areas of the image, activated percepts are drawn on top of a back-
ground image, except during dreaming: During waking, the background is the result of a running
average filter of current stimulus over time (Pt
t10 St/10), as pictured in Figure 10. During mind-
wandering, the background is the result of a running average filter of mean shift segmentation
results (Pt
t10 Sms
t/10), as pictured in Figure 12. During dreaming, the background is solid black,
as pictured in Figure 11.
Cognitive Processes
The modules described above are the framework for the three cognitive processes of the
system: Perception, dreaming and mind-wandering. These processes are contiguous, as they are
enabled by the same mechanisms with differing dynamics. The differing dynamics of the system
are controlled by the weighting between internal feedback from prediction (endogenous) and acti-
vation caused by STIM ULUS (exogenous) as mediated by SUPPRESSIO N, which is controlled by the
states of CLOC K and AR OUSAL, as pictured in Figure 9.
Perception. This is the initial process of the system, as without collecting any sensory informa-
tion there would be no mind wandering or dreaming content. The sub-processes of SEGM EN TATI ON
and CLUSTER IN G occur only during day-time perception when high fidelity sensory information is
available. Waking perception occurs when the CLOC K indicates daylight (clock = 1) and AROUSAL
Figure 10.: An example perceptual image. Moving foreground objects appear blurry as they aug-
ment perceptual clusters such that STIM ULUS is less weighted than the existing cluster.
indicates change in stimuli over time (arousal = 1). Activation of PERCEPUNI TS results from the
clustering process. During perception, SUPPRESSI ON inhibits feedback to PREDICTION, such that
endogenous activation is inhibited. The result is that the rendering of activated PERCEPUNITS
causes the display to resemble the image captured by the camera. Perception is as constructive as
dreaming and mind-wandering, but is constrained and anchored in sensory information.
Dreaming. Dreaming occurs during night time (when clock = 0) and the activation of percep-
tual representations are caused by feedback in the prediction mechanism while activation caused
by ST IMULU S is suppressed. Latent activation from STIM ULUS is the baseline ground from which
both dreaming and mind-wandering arise. AR OU SAL allows PREDICTION to generate feedback as
mediated by SUPP RE SSION. The resulting dream is a chain of predictions, seeded by recent percep-
tual activation. The inhibition of activation from STIMULUS results in sequences of activation that
become increasingly divorced from the latent perceptual activation that seeded them. Perceptual
information that is tenuously related to the current context is included in dreams, whose result-
ing images may appear divorced from plausible external reality. As during perception, activated
PERCEPUNITS are rendered on the display, as shown in Figure 11.
Mind-Wandering. The mind-wandering process is analogous to dreaming, except the CLOCK
indicates day-time (clock = 1). A lack of AROUSAL (arousal = 0), due to static stimuli, causes
SUP PRESSIO N to increase feedback activation from PREDICTION and inhibits perceptual activation
caused by STIM ULUS. This leads to the activation of PERCEPUN ITS not related to current stimuli, as
pictured in Figure 12. When changes in ST IM ULUS are manifest in an increase of arousal (arousal =
1), mind-wandering ends and the perceptual process resumes. This allows the system to switch
its cognitive process away from internally oriented imagery (mind wandering) towards external
Figure 11.: An example dream image where endogenous activations in the PREDICTION module
construct sequences of activation that are increasingly unrelated to perceptual activations that ini-
tiated them.
STI MULUS (perception).
The computational model is written in C++ using the openFrameworks (Zach Lieberman &
Castro, 2014) environment to manage OpenGL (Silicon Graphics Incorporated, 2014) rendering
for visualization and OpenCV (Bradski, 2014) for segmentation and pixel operations. The MLP in
the prediction module is written using FANN (Nissen, 2003). The system is fed visual stimulus by
an Avigilon IP camera which provides 1440 ×1080 pixel video images as a JPEG sequence. Through
development stimulus was provided by a set of video frames (1 per second recorded over three
consecutive days) captured during an artistic exhibition, described below.
Various revisions of the computational system have been exhibited internationally in artistic
and academic contexts. The system is meant for long-term installation in a public setting in order
to be exposed to as much variety of visual material as possible. An early associative version of the
system was shown at the New Forms Festival (2012) in Vancouver, Canada, an example of which
is pictured in Figure 13. Work-in-progress on the perceptual subsystem was presented at ACM
Creativity and Cognition (Bogart et al., 2013) in Sydney, Australia.
Figure 12.: An example mind-wandering image where endogenous activations in the PREDICTION
module cause a shift away from external stimulus toward internally generated imagery.
Behavior of the Computational Model
The model behaves as expected, with the exception of the observation that mind wandering
appears to generate more variation and complex imagery than dreaming. Perceptual processes
result in cohesive simulations that resemble stimulus images. The feedback mechanism in the
predictor leads to complex system dynamics that range from static, through periodic to complex
patterns of percept activity. The dynamics are highly sensitive to the state of the predictor’s learning
and the initial conditions — the previous stimuli and latent perception that initiates dreaming and
mind wandering. In the following paragraphs we describe how these dynamics of the simulator are
visually manifest.
In the perceptual mode, the simulation (e.g. Figure 10) resembles the visual stimulus pro-
vided to the system. The clustering process leads to blurry moving objects, where clusters and
newly segmented regions are weighed equally and clusters are not spatially invariant. Over time,
the perceptual process generates simulations similar to a sequence of long photographic exposures,
where clustering causes static objects to slowly fade in and moving objects to slowly fade away.
This aesthetic is reinforced by the rendering of percepts on top of a running average of the unpro-
cessed stimulus. The video footage used to train the system contains bursts of activity (e.g. rush
hour on weekdays) that appear intermittently in what is predominantly a static image with little
change between frames. During these periods of static stimuli, the system’s prediction error is low
and many percepts are rendered transparently. As arousal stays low while there is little change in
stimuli, the majority of the system’s daylight time is spent mind wandering, and not perceiving.
Figure 13.: Sample image from an exhibition of work-in-progress on the perceptual subsystem of
the computational model, exhibited under the title: An Artist in Process: A Computational Sketch of
Dreaming Machine #3
During mind wandering, the simulation begins by continuing the stimulus from latent per-
ception. This makes the transition between perceiving and mind wandering appear smooth and
contiguous. As feedback activation from the predictor takes over from activation caused by exter-
nal stimuli, the simulations tends to drift away from the initial state of activation. The percepts that
are activated by external stimuli are evenly distributed over the field of view while the simulation is
composed of percepts that are less evenly distributed. As the mind wandering percepts are drawn
on top of a running average of the mean shift segmentation results, as pictured in Figure 12, the
simulated portions of the image — those that correspond to percept activation — appear detailed
and nearly photographic. The unoccupied areas show simplified color patches as generated by the
segmentation system. As the simulation drifts, images appear both recognizable (clearly show a
place in space and time) and abstracted, where percepts not perceived at the same time may be
simultaneously present. The activation of percepts at different locations over time changes the
distribution of percepts over the field of view, leading to dynamism in the simulations. Subsequent
images in the simulation may include large changes in composition due to changes in percept ac-
tivation. Mind wandering is often interrupted by increases in arousal — e.g. due to a person or
car entering the frame — which ends the mind wandering process. This transition is sudden as the
state of activation in the simulation differs significantly from the perceptual activation, due to drift
in the feedback process, and changes in stimuli.
As night falls, the external context is more likely to be static. Thus there is an increase of
mind wandering during the periods before the night. This is consistent with Domhoff’s (2011)
proposal and shows mind wandering through to dreaming is a contiguous process involving the
constant activation of a subset of the default network. Intermittent increases in arousal occur, and
the system shows very short periods of perception during this time. Due to the complexities of
the environment, the total luminosity of the image tends to shift dramatically. In the test footage,
the sun sets in front of the camera passing behind a pair of trees. When the sun is behind the
trees, the system enters dreaming, from which it is awakened when the sun is visible again. This
leads to a complex oscillations where perception, mind wandering and dreaming all occur for short
periods in close succession. Eventually, the sun sets and once stimuli is dark, the system enters
a stable dreaming period — e.g. as pictured in Figure 11. Due to the dynamics of the system
while the sun sets — where drastic changes in light and reflection lead to large changes in percept
content — there tends to be a lack of diversity in percepts at the onset of dreaming. Dreams
show the same internal dynamics of mind wandering — the activation of percepts not seen in the
same stimulus and the uneven distribution of percepts over the visual field. The lack of diversity
of percepts, and the lack of intermittent perceptual states, causes dreams to appear more static
and periodic than mind wandering. The drift seen in mind wandering appears to stabilize over
a number of iterations, and thus the long periods of dreaming tend to involve the simulation of
repeated sequences of similar images.
Limitations of The Computational Model
The computational model described above is a formalization of the Integrative Theory —-
which is oriented toward dreaming in mammalian brains. While the perceptual representations
described in the Integrative Theory describe classes of concrete objects (e.g. concepts referring
to perceptual information such as ’dog’ or ’chair’ and not abstract concepts such as ’justice’ or
’freedom’), those implemented in the computational model are simplistic. The percepts generated
by the system are restricted to sensory information and lack the context dependence and dynamism
of concepts. While mammalian visual processing involves a degree of separation between spatial
properties (such as the position in space) and object-centric properties (such as color) (Murray et
al., 2007), perceptual representations in the model are fixed in space and not invariant to scale
and orientation. For example, if two identical objects are presented in two different parts of the
visual field, they would be represented by two different percepts. As the computational model is
an on-line system that learns continuously, computer vision methods for generating scale invariant
features such as SIFT (Lowe, 2004) and SURF (Bay et al., 2006) are too computationally taxing.
The mean shift method of segmentation is highly sensitive to subtle changes in lighting over time.
This results in variable percept boundaries, which adds additional noise to the clustering process
which assumes changes in region features are due to significant changes in the environment, not
changes in the borders around regions. The more noise in the percepts, the more likely they are to
represent redundant information.
These limitations in the model’s perceptual representations lead to a significant deficit in
terms of its perceptual ability compared to humans, children and likely even non-human mam-
mals. As the dreams reflect the perceptual abilities of the dreamer, the perceptual deficits of the
computational model result in parallel deficits in dreams and mental imagery. In humans with
face-blindness (prosopagnosia), dreams contain indistinct and unrecognizable faces. The dreams
generated by the computational model can only ’imagine’ percepts in the same positions in which
they were perceived. The world as perceived and dreamt by the system is simplified and highly
The label “Default Network” implies that the default state of the brain is related to the
self-oriented process of simulation we call mind-wandering, rather than external perception. In
the computational model, the central integrative process is simulation which runs continuously
whether the system is perceiving, mind-wandering or dreaming. Thus the default state of the sys-
tem is simulation, with the caveat that the simulation requires periods of perception from which
to train the predictive model. In terms of function, we do not seek a special role for dreaming
and mind-wandering, but rather propose that the simulation processes that enable dreaming and
mind-wandering are always occurring. The metabolic cost associated with dreaming and mind-
wandering is the base metabolic cost of the brain functioning normally. The low activation (and
therefore low metabolic cost) during slow wave sleep is then the special case, while REM sleep
reflects normal waking function.
Constructive Imagery
Nir & Tononi (2010) and Fosse et al. (2003) note that dreaming does not involve the replay
of episodic memories. As dreams obviously involve the experience of familiar places and people, we
can conclude that dreams are composed of concepts, just concepts not contextualized in episodes.
The sequences of events supplied by the predictive model are implicit and not accessible to con-
sciousness in the way that episodic memory is accessible. If dreams are simply the result of our
implicit predictions of what to expect next, then why can dreams be experienced as bizarre and
discontinuous? As stated above, dreams are predominantly more banal and related to daily lived
experience than we tend to think. That being said, dreams may still unify strange, implausible
events and chimeric elements (fusions of multiple places or people) into a cohesive experience that
would be unlikely to occur in reality. This paradox is resolved by the requirement for a predic-
tive model to be a reduction of reality, always containing imperfections. The predictive model is
exploited in the computational model through feedback, which amplifies these imperfections and
causes predictions in the absence of sensory information to potentially diverge significantly from
plausible experience. The discontinuity of dreams can be explained by the absence of anchoring
in sensory information provided by reality which mitigates the effects of the imperfections in the
predictive model.
Future Work
This section is divided into three subsections, each concerned with future plans for a par-
ticular aspect of this research. The Integrative Theory itself could be extended in a number of
ways. The computational model deserves significant refinement. While the computational model
is framed as an artwork, the research process has inspired plans for a number of additional artistic
Integrative Theory
The Integrative Theory holds that the experience of perception, mind-wandering and dream-
ing result from the activation of visual representations of concrete objects. It is expected that
dreaming in brains involves not just the activation of representations of concrete perceptual objects
but also higher level conceptual representations. Barsalou (1999) proposes that the difficulty in
pinning down the definition and origin of concepts is that they are not fixed representations but
simulations in themselves. These simulations shift their content to match the particular needs of
the current task. On the surface, the focus on simulation in Barsalou’s conception is highly com-
plimentary to the Integrative Theory. Still, there remains a major issue in the task-dependence
of concepts in Barsalou’s conception. Mind-wandering and dreaming, with their correlation with
the default network, are particularly non-task oriented and stimulus independent. It is unclear
how those simulations would be constrained independently of task demands. Perhaps conceptual
simulations are not task dependent so much as they are context dependent.
The Integrative Theory, as currently conceived, has associated working memory, executive
and attentional mechanisms with the PFC in general. This particular conception glosses over dif-
ferences in activity across the various regions of the PFC, including dorsolateral and ventromedial
areas. The Integrative Theory would benefit from a finer grained conception sensitive to constituent
regions of the PFC and an integration of executive functions associated with the PFC, which is out
of scope of the research described in this paper.
Computational Model
The computational model is framed as a situated agent, although its agency is extremely im-
poverished in the absence of intention and other executive functions, which are out of scope of the
computational model. One avenue of development would be the inclusion of executive mechanisms
and integrating the system into an existing cognitive agent architecture (such as CLARION (Hélie
& Sun, 2010) or SOAR (Laird, 2012)). This would also entail a richer and hierarchical system of
representations, although cognitive architectures tend to lack a basis in brain anatomy which is
central to the Integrative Theory.
The MLP was chosen to facilitate implementation and for its proven flexibility and perfor-
mance. As prediction is inherently about patterns occurring over time, a machine learning method
specifically developed for temporal prediction could improve the ability of the system to predict
sensory information, and therefore enrich dream sequences. The authors will examine Recurrent
Neural Networks (Dorffner, 1996) and Deep Belief Networks (H. Lee et al., 2009) as possible
methods to improve the model’s predictor. Additionally, the authors will examine replacing the
hard threshold used in the circadian clock with a proper oscillator entrained by visual brightness.
The combination of the noise sensitivity of mean shift segmentation over time, and the rela-
tively small number of perceptual representations, has lead to quite unstable region edges, even in
cases where the edges appear quite distinct to a human observer. This is due to a lack of temporal
information used in the segmentation, which simply treats each frame as a new and unique image.
Bailer et al. (2005) have proposed a method to improve the temporal stability in mean shift image
segmentation, which could be included in the system.
While the discussion section above describes initial results of the dynamics of the system, no
formal analysis has yet been undertaken. Such an analysis would provide valuable data in refining
the system and increasing the diversity of dream, in particular, and mind wandering content. The
model, as described in this paper, involves no random variation in the activation of percepts. This
differs significantly from brains where neurons associated with perceptual representations are likely
to be constantly, but weakly, active. These weak but constant activations are likely to significantly
impact the feedback process by constantly shifting the current state of the system. This could
mitigate the tendency of long periods of dreaming seen on our model to converge to stability.
The inclusion of a small degree of random variation in the computational model is expected to
significantly increase the diversity and complexity of dreams.
The research process that lead to the Integrative Theory began with an artistic intention to
create the artwork Dreaming Machine #3. The development of the theory, and implementation
of the computational model, resulted in concepts for further artworks. While Dreaming Machine
#3 is meant for situated installation in the real world, artifacts of visual culture could also be
used as source material. For example, time-based narrative media such as film and television.
One artwork, tentatively titled Watching and Dreaming would be fed frames from film or television
shows. The segmentation and arousal modules would be tuned to the movement and scene changes
present in film and television. The system’s circadian clock would be on a fixed schedule where the
system would alternate between being fed frames from film or T.V. (during perception and mind-
wandering) and black frames (during dreaming). The repetition of the film or T.V. show would
allow epoch learning, which would change the quality of dreams with increasing exposure of the
visual material.
Another proposed artwork, titled the Dreaming Mirror, would be tuned to the segmentation
and reconstruction of facial images in particular. The system would break faces into multiple
regions stored in a corpus where the regions would be used to reconstruct the face of the person
currently in front of the work. The system would learn correlations between regions of the same
face (prediction over space rather than time). In the absence of a face, the system would then
’dream’ by reconstructing faces from the visual corpus, using the most recent face as a starting
Dreaming Machine #3 (Landscape), has been accepted to be exhibited at the International
Symposium on Electronic Art (2014) in Dubai. This version of the system is tuned to perceiving
the landscape and predict its changes over longer time-scales. The system will ignore foreground
objects and the patterns of their short-term movement to emphasize shifts in light and inanimate
objects in the urban landscape of Dubai.
Perception, mind-wandering and dreaming are all simulations of reality. While we are awake
and interacting with the world, we learn concepts and build predictive models of what we should
expect to occur. These predictions tell us what is routine, what surprises we should attend to, and
assist us in making sense of ambiguous sensory information. Predictive mechanisms are continu-
ously exploited, with the possible exception of slow-wave sleep. In the absence of sensory infor-
mation, predictions feed back on themselves leading to simulations of reality that result in dreams.
When sensory information is static, for example due to habituation, mind-wandering occurs as a
shift of attention away from external stimulus toward endogenously simulations.
Perhaps the search for the adaptive functions of dreaming is misguided. If dreaming is con-
structed from predictive mechanisms exploited and learned in waking consciousness, then perhaps
simulation really is the “default” mode of the brain. The value of these simulations in both waking
and sleeping offsets their metabolic cost. Simulation is the default mode of the brain, and occurs
whether or not external stimulus is present.
In this paper, we have reviewed a number of biological mechanisms of visual mentation
that are unified in the proposed Integrative Theory. According to this proposal, dreams, mental
imagery and mind-wandering all entail the activation of perceptual representations that result in
the phenomenology of visual images in the mind. As these representations are shared between
various modes of visual mentation, activity in one mode effects activity in another. This explains
how dreams reflect normal waking concerns, in particular the effect of waking stimulus in the hours
before sleep. Dreams are indeed mental imagery, and perceptual representations are activated via
endogenous activation from the PFC in the default network.
Dreams can be narrative-like because they exploit predictive mechanisms of the default net-
work (rather than PGO waves) we use to make sense of the complexity of the world. Dreams can
be bizarre and discontinuous because constituent predictions lack the anchor of ongoing sensory
reality to constrain them. Predictions are centrally important for their role in priming perceptual
processes. In dreams and mind-wandering, predictions show their true potentially discontinuous
character. Our perceptual simulation of reality is cohesive because of stability and structure in the
sensory information we receive from the world. Our dream simulations are approximate carica-
tures of normal waking experience. They are thus inherently meaningful as they encapsulate tacit
knowledge over a life-time of experiences and have a central role in how we make sense of the
The presentation of the computational theory as an artwork is meant to expand public en-
gagement in brain-science in general, and the science of perception and dreaming in particular. A
visually enticing and on-line model of dreaming provides a powerful entry-point for the viewer to
consider the illusion of their perceptual simulation of reality. When looking at Dreaming Machine
#3, the viewer is looking through cultural, technical and scientific systems toward the very pro-
cesses that allow us all to make sense of the world. As we look into the machine to seek meaning
and understanding, so too does the machine look into us.
Aserinsky, E., & Kleitman, N. (1953). Regularly occurring periods of eye motility, and concomitant phenom-
ena during sleep. Science,118, 273–274.
Ashby, W. R. (1947). Principles of the self-organizing dynamic system. The Journal of General Psychology,
37(2), 125–128.
Bailer, W., Schallauer, P., Haraldsson, H. B., & Rehatschek, H. (2005). Optimized mean shift algorithm for
color segmentation in image sequences. In Image and video communications and processing 2005 (Vol.
5685, pp. 522–529).
Barsalou, L. (1999). Perceptual symbol systems. Behavioral and brain sciences,22(04), 577–660.
Bay, H., Tuytelaars, T., & Gool, L. (2006). Surf: Speeded up robust features. In A. Leonardis, H. Bischof, &
A. Pinz (Eds.), Computer Vision — ECCV 2006 (Vol. 3951, pp. 404–417). Springer Berlin Heidelberg.
Bogart, B. D. R., & Pasquier, P. (2013). Context machines: A series of situated and self-organizing artworks.
Leonardo,46(2), 114–122.
Bogart, B. D. R., Pasquier, P., & Barnes, S. J. (2013). An integrative theory of visual mentation and sponta-
neous creativity. In Proceedings of the 9th ACM Conference on Creativity & Cognition (pp. 264–273). New
York, NY, USA: ACM.
Borst, G., & Kosslyn, S. (2012). From mental imagery to spatial cognition and language: Essays in honour
of Michel Denis. In V. Gyselinck & F. Pazzaglia (Eds.), (pp. 19–42). Psychology Press.
Bradski, G. (2014). OpenCV. Available from
Bridge, H., Harrold, S., Holmes, E., Stokes, M., & Kennard, C. (2012). Vivid visual mental imagery in the
absence of the primary visual cortex. Journal of Neurology,259(6), 1062–1070.
Buckner, R. L., Andrews-Hanna, J. R., & Schacter, D. L. (2008). The brain’s default network. Annals of the
New York Academy of Sciences,1124(1), 1–38.
Callaway, C. W., Lydic, R., Baghdoyan, H. A., & Hobson, J. A. (1987, June). Pontogeniculooccipital waves:
spontaneous visual system activity during rapid eye movement sleep. Cellular and Molecular Neurobiology,
7(2), 105–149.
Chelazzi, L., Marzi, C., Panozzo, G., Pasqualini, N., Tassinari, G., & Tomazzoli, L. (1988). Hemiretinal
differences in speed of light detection in esotropic amblyopes. Vision Research,28(1), 95–104.
Christoff, K., Gordon, A. M., Smallwood, J., Smith, R., & Schooler, J. W. (2009). Experience sampling during
fMRI reveals default network and executive system contributions to mind wandering. Proceedings of the
National Academy of Sciences,106(21), 8719–8724.
Comaniciu, D., & Meer, P. (2002). Mean shift: a robust approach toward feature space analysis. Pattern
Analysis and Machine Intelligence, IEEE Transactions on,24(5), 603–619.
Davenport, J. L., & Potter, M. C. (2004). Scene consistency in object and background perception. Psycholog-
ical Science,15(8), 559–564.
Diekhof, E. K., Kipshagen, H. E., Falkai, P., Dechent, P., Baudewig, J., & Gruber, O. (2011, January). The
power of imagination — how anticipatory mental imagery alters perceptual processing of fearful facial
expressions. NeuroImage,54(2), 1703–1714.
Domhoff, G. W. (2011). The neural substrate for dreaming: Is it a subsystem of the default network?
Consciousness and Cognition,20(4), 1163–1174.
Dorffner, G. (1996). Neural networks for time series processing. Neural Network World,6, 447–468.
Finke, R. (1980). Levels of equivalence in imagery and perception. Psychological Review,87(2), 113–132.
Fosse, M. J., Fosse, R., Hobson, J. A., & Stickgold, R. J. (2003, January). Dreaming and episodic memory: A
functional dissociation? Journal of Cognitive Neuroscience,15(1), 1–9.
Fox, K. C. R., Nijeboer, S., Solomonova, E., Domhoff, G. W., & Christoff, K. (2013). Dreaming as mind
wandering: evidence from functional neuroimaging and first-person content reports. Frontiers in Human
Neuroscience,7(412), 1–18.
Friston, K. (2010, February). The free-energy principle: a unified brain theory? Nature Reviews Neuroscience,
11(2), 127–138.
Goodale, M. A., & Westwood, D. A. (2004). An evolving view of duplex vision: separate but interacting
cortical pathways for perception and action. Current Opinion in Neurobiology,14(2), 203–211.
Graham, K. S., Barense, M. D., & Lee, A. C. H. (2010). Going beyond LTM in the MTL: A synthesis of
neuropsychological and neuroimaging findings on the role of the medial temporal lobe in memory and
perception. Neuropsychologia,48(4), 831–853.
Gregory, D. (2010, September). Visual imagery: Visual format or visual content? Mind & Language,25(4),
Hélie, S., & Sun, R. (2010). Incubation, insight, and creative problem solving: A unified theory and a
connectionist model. Psychological Review,117(3), 994–1024.
Hesslow, G. (2002). Conscious thought as simulation of behaviour and perception. Trends in Cognitive
Sciences,6(6), 242–247.
Hobson, J., & Friston, K. (2012). Waking and dreaming consciousness: Neurobiological and functional
considerations. Progress in Neurobiology,98(1), 82–98.
Hobson, J. A. (2009). REM sleep and dreaming: towards a theory of protoconsciousness. Nature Reviews
Neuroscience,10(11), 803–813.
Hobson, J. A., & McCarley, R. W. (1977). The brain as a dream state generator: an activation-synthesis
hypothesis of the dream process. American Journal of Psychiatry,134(12), 1335–1348.
Hobson, J. A., Pace-Schott, E. F., & Stickgold, R. (2000). Dreaming and the brain: Toward a cognitive
neuroscience of conscious states. Behavioral and Brain Sciences,23, 793–842.
International Symposium on Electronic Art. (2014). Available from
Kandel, E., & Wurtz, R. (2000). Constructing the visual image. Principles of neural science,4, 492–506.
Kosslyn, S. (1973, February). Scanning visual images: Some structural implications. Attention, Perception,
& Psychophysics,14(1), 90–94.
Kosslyn, S. (1976, May). Can imagery be distinguished from other forms of internal representation? Evi-
dence from studies of information retrieval times. Memory & Cognition,4(3), 291–297.
Kosslyn, S. M. (1975, July). Information representation in visual images. Cognitive Psychology,7(3), 341–
Kosslyn, S. M. (1980). Image and mind. Harvard University Press.
Kosslyn, S. M. (1981). The medium and the message in mental imagery: A theory. Psychological Review,
88(1), 46–66.
Kosslyn, S. M. (1994). Image and brain: the resolution of the imagery debate. MIT Press.
Kosslyn, S. M., Ball, T. M., & Reiser, B. J. (1978). Visual images preserve metric spatial information: Evidence
from studies of image scanning. Journal of Experimental Psychology: Human Perception and Performance,
4(1), 47–60.
Kosslyn, S. M., Murphy, G. L., Bemesderfer, M. E., & Feinstein, K. J. (1977). Category and continuum in
mental comparisons. Journal of Experimental Psychology: General,106(4), 341–375.
Kosslyn, S. M., Pinker, S., Smith, G. E., & Shwartz, S. P. (1979). On the demystification of mental imagery.
Behavioral and Brain Sciences,2(4), 535–581.
Kosslyn, S. M., & Thompson, W. L. (2003). When is early visual cortex activated during visual mental
imagery? Psychological Bulletin,129(5), 723–746.
Kosslyn, S. M., Thompson, W. L., & Ganis, G. (2006). The case for mental imagery. Oxford University Press.
Laird, J. (2012). The Soar Cognitive Architecture. MIT Press.
Lea, G. (1975). Chronometric analysis of the method of loci. Journal of Experimental Psychology: Human
Perception and Performance,1(2), 95–104.
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009). Convolutional deep belief networks for scalable
unsupervised learning of hierarchical representations. In Proceedings of the 26th annual international
conference on machine learning (pp. 609–616). New York, NY, USA: ACM.
Lee, S., Kravitz, D. J., & Baker, C. I. (2012, February). Disentangling visual imagery and perception of
real-world objects. NeuroImage,59(4), 4064–4073.
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of
Computer Vision,60(2), 91–110.
Marks, D. (1973, February). Visual imagery differences in the recall of pictures. British Journal of Psychology,
64(1), 17–24.
Marzi, C., Mancini, F., Metitieri, T., & Savazzi, S. (2006). Retinal eccentricity effects on reaction time to
imagined stimuli. Neuropsychologia,44(8), 1489–1495.
Mishkin, M., Ungerleider, L. G., & Macko, K. A. (1983). Object vision and spatial vision: Two cortical
pathways. Trends in neurosciences,6, 414–417.
Mohr, H. M., Linder, N. S., Dennis, H., & Sireteanu, R. (2011). Orientation-specific aftereffects to mentally
generated lines. Perception,40(3), 272–290.
Morin, L., & Allen, C. (2006). The circadian visual system, 2005. Brain Research Reviews,51(1), 1–60.
Moyer, R. (1973). Comparing objects in memory: Evidence suggesting an internal psychophysics. Attention,
Perception, & Psychophysics,13(2), 180–184.
Murray, E. A., Bussey, T. J., & Saksida, L. M. (2007). Visual Perception and Memory: A New View of Medial
Temporal Lobe Function in Primates and Rodents. Annual review of neuroscience,30, 99–122.
Muzur, A., Pace-Schott, E. F., & Hobson, J. (2002). The prefrontal cortex in sleep. Trends in Cognitive
Sciences,6(11), 475–481.
New Forms Festival. (2012). Available from
Nielsen, T. A. (2000, 12). A review of mentation in REM and NREM sleep: REM sleep as a possible
reconciliation of two opposing models. Behavioral and Brain Sciences,23, 851–866.
Nir, Y., Massimini, M., Boly, M., & Tononi, G. (2013). Sleep and consciousness. In A. E. Cavanna, A. Nani,
H. Blumenfeld, & S. Laureys (Eds.), Neuroimaging of consciousness (pp. 133–182). Springer Berlin Heidel-
Nir, Y., & Tononi, G. (2010). Dreaming and the brain: from phenomenology to neurophysiology. Trends in
cognitive sciences,14(2), 88–100.
Nissen, S. (2003). Implementation of a fast artificial neural network library (FANN) (Tech. Rep.). Department
of Computer Science University of Copenhagen (DIKU). Available from
Paivio, A. (1975, November). Perceptual comparisons through the mind’s eye. Memory & Cognition,3(6),
Pinker, S. (1980). Mental imagery and the third dimension. Journal of Experimental Psychology: General,
109(3), 354–371.
Pinker, S., & Kosslyn, S. (1978). The representation and manipulation of three-dimensional space in mental
images. Journal of Mental Imagery,2(1), 69–84.
Pylyshyn, Z. W. (1981). The imagery debate: Analogue media versus tacit knowledge. Psychological Review,
88(1), 16–45.
Pylyshyn, Z. W. (2002). Mental imagery: In search of a theory. Behavioral and brain sciences,25(02),
Pylyshyn, Z. W. (2003a). Return of the mental image: are there really pictures in the brain? Trends in
cognitive sciences,7(3), 113–118.
Pylyshyn, Z. W. (2003b). Seeing and visualizing: It’s not what you think. The MIT Press.
Revonsuo, A. (2000, 12). The reinterpretation of dreams: An evolutionary hypothesis of the function of
dreaming. Behavioral and Brain Sciences,23, 877–901.
Rumelhart, G., D.E. Hinton, & Williams, R. (1986). Parallel distributed processing: Explorations in the
microstructure of cognition. In (pp. 318–363). MIT Press, Cambridge, MA.
Schooler, J. W., Smallwood, J., Christoff, K., Handy, T. C., Reichle, E. D., & Sayette, M. A. (2011). Meta-
awareness, perceptual decoupling and the wandering mind. Trends in Cognitive Sciences,15(7), 319–326.
Siegel, J. M. (2008). Do all animals sleep? Trends in Neurosciences,31(4), 208–213.
Siegel, J. M. (2011). REM sleep: A biological and psychological paradox. Sleep Medicine Reviews,15,
Silicon Graphics Incorporated. (2014). OpenGL. Available from
Sima, J. (2011). The nature of mental images — an integrative computational theory. In Proceedings of the
33rd annual meeting of the cognitive science society.
Snyder, F. (1970). The psychodynamic implications of the physiological studies on dreams. In L. Madow &
L. Snow (Eds.), (pp. 124–151). Thomas.
Solms, M. (2000). Dreaming and REM sleep are controlled by different brain mechanisms. Behavioral and
Brain Sciences,23, 843–850.
Stickgold, R., Malia, A., Maguire, D., Roddenberry, D., & O’Connor, M. (2000). Replaying the game: Hypna-
gogic images in normals and amnesics. Science,290(5490), 350–353.
Stokes, M., Thompson, R., Cusack, R., & Duncan, J. (2009, February). Top-Down activation of Shape-Specific
population codes in visual cortex during mental imagery. The Journal of Neuroscience,29(5), 1565–1572.
Theodoridis, S., & Koutroumbas, K. (2009). Pattern recognition. Academic Press.
Triesch, J., Ballard, D. H., Hayhoe, M. M., & Sullivan, B. T. (2003). What you see is what you need. Journal
of Vision,3(1), 94–86.
Tucker, L., Thompson, A., Smith, J., Cohen, A., Clark, M., & Barrett, M. (2009, Feburary). Why do we dream?
Documentary Television Program. (BBC Horizon Documentary)
Wais, P. E., Rubens, M. T., Boccanfuso, J., & Gazzaley, A. (2010, June). Neural mechanisms underlying
the impact of visual distraction on retrieval of Long-Term memory. The Journal of Neuroscience,30(25),
Wechsler, D. (1967). Manual for the wechsler preschool and primary scale of intelligence. The Psychological
Young, M., & Yamane, S. (1992, May). Sparse population coding of faces in the inferotemporal cortex.
Science,256(5061), 1327–1331.
Zach Lieberman, T. W., & Castro, A. (2014). openFrameworks. Available from
Zago, S., Corti, S., Bersano, A., Baron, P., Conti, G., Ballabio, E., et al. (2010). A cortically blind patient with
preserved visual imagery. Cognitive and behavioral neurology,23(1), 44–48.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Sleep offers a unique opportunity to relate changes in brain activity to changes in consciousness. Indeed, if it were not for sleep, when consciousness fades in and out on a regular basis, it might be hard to imagine that consciousness is not a given but depends somehow on the way our brain is functioning. At the same time as changes in consciousness occur, brain activity undergoes major changes through an orderly progression of sleep stages, which can be identified by recording the electroencephalogram (EEG), eye movements (EOG), and muscle tone (EMG). Within each sleep stage, there are frequent, short-lasting electrophysiological phenomena, such as slow oscillations and spindles representing moments at which brain activity undergoes important fluctuations. There are also orderly spatial changes in the activation of many brain regions, as indicated by imaging studies. Importantly, similar brain activities occur in animals, and this has spearheaded detailed studies of the underlying neural mechanisms.
Full-text available
Conference Paper
It has been suggested that creativity can be functionally segregated into two processes: spontaneous and deliberate. In this paper, we propose that the spontaneous aspect of creativity is enabled by the same neural simulation mechanisms that have been implicated in visual mentation (e.g. visual perception, mental imagery, mind-wandering and dreaming). This proposal is developed into an Integrative Theory that serves as the foundation for a computational model of dreaming and site-specific artwork: A Machine that Dreams.
This book is about how we see and how we visualize. But it is equally about how we are easily misled by our everyday experiences of these faculties. Galileo is said to have proclaimed (Galilei, 1610/1983; quoted in Slezak, submitted), "dots if men had been born blind, philosophy would be more perfect, because it would lack many false assumptions that have been taken from the sense of sight." Many deep puzzles arise when we try to understand the nature of visual perception, visual imagery or visual thinking. As we try to formulate scientific questions about these human capacities we immediately find ourselves being entranced by the view from within. This view, which the linguist Kenneth Pike (Pike, 1967) has referred to as the emic perspective (as opposed to the external or etic perspective), is both essential and perilous. As scientists we cannot ignore the contents of our conscious experience because this is one of the principle ways of knowing what we see and what our thoughts are about. On the other hand, the contents of our conscious experience are also insidious because they lead us to believe that we can see directly into our own minds and observe the causes of our cognitive processes. Such traps are nothing new; psychology is used to being torn by the duality of mental life --- its subjective and its objective (causal) side. Since people first began to think about the nature of mental states, such as thinking, seeing and imagining, they have had to contend with the fact that knowing how these achievements appear to us on the inside often does us little good, and indeed often leads us in entirely the wrong direction, when we seek a scientific explanation. Of course we have the option of putting aside the quest for a scientific explanation and set our goal towards finding a satisfying description in terms that are consonant with how seeing and imagining appear to us. This might be called a phenomenological approach to understanding the workings of the mind or the everyday folk understanding of vision. There is nothing wrong with such a pursuit. Much popular psychology revels in it, as do a number of different schools of philosophical inquiry (e.g., ordinary language philosophy, phenomenological philosophy). Yet in the long term few of us would be satisfied with an analysis or a natural history of phenomenological regularities. One reason is that characterizing the systematic properties of how things seem to us does not allow us to connect with the natural sciences, to approach the goal of unifying psychology with biology, chemistry and physics. It does not help us to answer the how and why questions; How does vision work? Or, Why do things look the way they do? Or, What happens when we think visually? The problem with trying to understand vision and visual imagery is that on the one hand these phenomena are intimately familiar to us from the inside so it is difficult to objectify them, even though the processes involved are also too fast and too ephemeral to be observed introspectively. On the other hand, what we do observe is misleading because it is always the world as it appears to us that we see, not the real work that is being done by the mind in going from the proximal stimuli, generally optical patterns on the retina, to the familiar experience of seeing (or imagining) the world. The question, How do we see appears very nearly nonsensical: Why, we see by just looking, and the reason that things look as they do to us is that this is the way that they actually are. It is only by objectifying the phenomena, by "making them strange" that we can turn the question into puzzle that can be studied scientifically. One good way to turn the mysteries of vision and imagery into a puzzle is to ask what it would take for a computer to see or imagine. But this is not the only way and indeed this way is often itself laden with our preconceptions, as I will try to show throughout this book. The title of this book is meant to be ambiguous. It means both that seeing and visualizing are different from thinking (and from each other), and that our intuitive views about seeing and visualizing rest largely on a grand illusion. The message of this book is that seeing is different from thinking and to see is not, as it often seems to us, to create an inner replica of the world we are observing or thinking about or visualizing. But this is a long and not always an intuitively compelling story. In fact, its counterintuitive nature is one reason it may be worth telling. When things seem clearly a certain way it is often because we are subject to a general shared illusion. To stand outside this illusion requires a certain act of will and an open-minded and determined look at the evidence. Few people are equipped to do this, and I am not deluded enough to believe that I am the only one who can. But some things about vision and mental imagery are by now clear enough that only deeply ingrained prejudices keep them from being the received view. It is these facts, which seem to me (if not to others) to be totally persuasive, that I concern myself with in this book. If any of the claims appear radical it is not because they represent a leap into the dark caverns of speculative idealism, but only that some ways of looking at the world are just too comfortable and too hard to dismiss. Consequently what might be a straightforward story about how we see, becomes a long journey into the data and theory developed over the past 30 years, as well into the conceptual issues that surround them.
Describes a computational theory of imagery that posits that visual mental images are transitory data structures that occur in an analog spatial medium. These "surface" representations are generated from more abstract "deep" representations in long-term memory and, once formed, can be operated upon in various ways. The theory is described in terms of detailed claims about the mental structures and processes invoked during imagery. In addition, the philosophical and empirical roots of the present theory are briefly reviewed. Further, arguments and data that have been offered against the theory are critically examined, and none are found damaging. An alternative account of the data that purportedly support the theory is also examined and found deficient in several respects. Finally, the current status of the "analog-propositional" debate is reviewed. (40 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)