Mixing realities: reflections on presence and embodiment in intermedial performance design of Blue Hour VR

Abstract

Created for the 2019 Prague Quadrennial’s 36Q°, Blue Hour VR was a site-responsive mixed reality performative installation that placed the spectator, as experiencer, within a hybrid landscape of real-time three-dimensional computer graphics and 360-degree video. This article describes the design process, staging and experience of Blue Hour VR from the vantage point of its creators. Using a phenomenological perspective, the article discusses how Blue Hour VR staged presence and embodiment within an intermedial haptic experience. Blue Hour VR demonstrates how virtual reality technology can be harnessed by a mixed reality performance design, which includes both the material and virtual environment, creating a complex stratigraphy of intermedial textures and visual dramaturgies that co-exist inside, outside and in between perceptual realities. In doing so, the article aims to contribute to the limited body of work on mixed and virtual reality in the context of theatre and performance design.

Full text

Page 1 of 20
Mixing realities: reflections on presence and embodiment
in intermedial performance design of Blue Hour VR
Paul Cegys
a,b
and Joris Weijdom
c
aDepartment of Film, Television and Scenography, Aalto University, Aalto, Finland;
b
Department of Communication
Arts, University of Waterloo Faculty of Arts, Waterloo, Canada;
c
Department of Theatre,
c
HKU University of the Arts
Utrecht, Utrecht, Netherlands
ABSTRACT
Created for the 2019 Prague Quadrennial’s 36Q°, Blue Hour VR was a site-responsive
mixed reality performative installation that placed the spectator, as experiencer, within
a hybrid landscape of real- time three-dimensional computer graphics and 360-degree
video. This article describes the design process, staging and experience of Blue Hour VR
from the vantage point of its creators. Using a phenomenological perspective, the
article discusses how Blue Hour VR staged presence and embodiment within an
intermedial haptic experience. Blue Hour VR demonstrates how virtual reality
technology can be harnessed by a mixed reality performance design, which includes
both the material and virtual environment, creating a complex stratigraphy of
intermedial textures and visual dramaturgies that co-exist inside, outside and in
between perceptual realities. In doing so, the article aims to contribute to the limited
body of work on mixed and virtual reality in the context of theatre and performance
design.
Introduction
Intermedial, mixed media, site-responsive works are forming new trajectories of performance
design and experience and expanding scenographic potential. Because analysis frameworks
and mapping approaches of these complex and unwieldy events are in their infancy, they are
difficult to present, analyse and archive. Their interactive nature challenges us with relaying
the haptic and phenomenological dimensions of subjective experience. This article will present
Blue Hour VR, as a site-responsive mixed reality (MR) performative installation, which blurred
the boundaries between performance, spectator- ship, and real and virtual environments, and
piloted interactive virtual reality (VR) technology and hybrid environments that merge 360
Video with real-time three-dimensional computer graphics. It will show how Blue Hour VR
created a performative meditative context, an experiment of embodiment and presence in
between realities, which explored the social relevance of VR technology as a tool of
performance.
After briefly introducing Blue Hour VR and defining key terms relevant to the experiential
qualities of MR installations and their technological characteristics, we will turn to a
thorough
presentation of the Blue Hour VR installation. This article will focus on the
This is an Accepted Manuscript of an article published by Taylor & Francis in
Theatre and Performance Design on 25th September 2020, available at:
https://www.tandfonline.com/doi/full/10.1080/23322551.2020.1785710.

Page 2 of 20
phenomenological aspects of the embodied experience, and on the visual perceptions
constructed and experienced in the hybrid virtual environments. We will describe how we
incorporated the consequences of VR’s potentially inherent performative quality in the
design process of the installation through the concepts of embodied improvisations,
puppeteering and technodramaturgy. Finally, we will discuss the conceptualizations of
meaning, which arose from the intermedial interweave, the phenomenological experience
and the visual perceptions of Blue Hour VR. We will conclude with how our findings and
questions concerning the design and presentation of Blue Hour VR could potentially offer
helpful pointers for the development of other site-responsive MR installations.
Blue Hour VR within the 36Q°
Blue Hour VR was a site-responsive, mixed reality, performative installation embedded within
a larger spatial installation, the 36Q°, curated for the 2019 Prague Quadrennial of Performance
Design and Space by Markéta Fantová and Jan Rolnik. The 36Q° brought together several
prominent performance designers under the artistic guidance of Romain Tardy to create a
large-scale interactive environment in the Malá Sports Hall next to the Výstaviště Praha Palace.
Blue Hour VR, a 10-minute experience staged in tactile sand pools, immersed individual
experiencers, wearing wireless head-mounted displays (HMDs), in virtual environments. The
dramaturgical development of Blue Hour VR explored perceptions of space, time, and the
phenomenon of simultaneous immersion in virtual and material worlds. The dramaturgy of
the virtual environments explored our anthropocentric relationship to the natural world and
contemplated the temporality of human existence. Placing the body at the centre of the
experience, Blue Hour VR relied on low-fidelity computer graphics showcasing how these
accessible and lower cost techniques can lead to compelling immersive hybrid mixed reality
performances. The perspectives of the experiencers were shifted dramatically, from those of
perceiving spectators, to those of single autonomous embodied insiders, who navigated their
points of view and physically performed their experiences of virtual immersion.
The 36Q° 2019 intermedial environment. Photo © David Kumermann.

Page 3 of 20
This repositioning of the spectator/experiencer at the locus of performance enabled a
heightened sense of bodily engagement with the material space, as experiencers physically
responded to the perceptual incongruencies that arose from the alignment between real and
virtual world environments.
These perceptual incongruencies led to relational dissonances and new experiences of
embodiment and presence that opened up new modes of being and knowing within VR
and
between realities. Blue Hour VR opened the incredible potential of VR to ‘refresh our
perception’ (Kattenbelt 2008) and inject new momentum into theatre and performance, which
in turn can open up new dramaturgical potentials and scenographic opportunities. In Blue Hour
VR the spectator was the performer, and was called upon to perceive in
ways that went
beyond the visual (spectator) and/or auditory (audience) apparatuses.
With this distinction at
the core of our design process, we called our spectator-performer
an experiencer, in a similar
way to how Robin Nelson conceives of the experiencer in intermedial multisensory
performances (in Bay-Cheng et al. 2010). His use of the term experiencer draws upon Merleau-
Ponty’s assertion that the body is a medium for perception (1962), and Deleuze and Guattari’s
idea of haptic space (1987), which denies opposition
between the senses (Nelson, in Bay-
Cheng et al. 2010). The dichotomy of the spectator-
performer conflated into a single vantage
of experience offers a new form of performative presence, and raises questions about the
ontology of performance. Blue Hour VR was an embodied spatio-temporal event with multiple
simultaneous layers of experience and performance. We will attend to three dimensions of
experience – the phenomenological
experience, visual perceptions, and conceptualizations of
meaning – which all fold into an irreducible intermedial stratigraphy, a Gestalt of direct
experience.
We proceed with brief definitions of key terms and concepts, which ground our artistic
process, its presentation and analysis. Readers who wish to first attend to a detailed
description of the Blue Hour VR are encouraged to skip ahead to these sections and refer back
to the terms as needed.
Terms of reference
We categorize Blue Hour VR as a site-responsive mixed reality installation. In order to explain
our choice of terminology, we first need to define mixed reality (MR) and virtual reality (VR).
In the context of a technologically mediated performance experience, we use the concept of
the virtuality continuum by Milgram and Kishino (1994), which is a scale with two extremes –
real environment and virtual environment – at either end, whereby any kind of mix in between
these two extremes can be considered MR. Milgram and Kishino consider VR to be an
environment ‘in which the participant-observer is totally immersed in, and able to interact
with, a completely synthetic world’ (1994, 2). Here, VR is considered the virtual environment
extreme and for this reason not part of MR (Speicher, Hall, and Nebeling 2019). However, in
this article we use MR as a qualification for the experience, while using technology that is
normally associated with VR.
Terms like immersion, embodiment and presence are often used when discussing VR
experiences to describe the degree to which a person feels they are within a virtual
environment generated by a computer and shown on an HMD. In the context of VR, a physical
state of immersion can be experienced through the idea of perceptual immersion, whereby

Page 4 of 20
the ‘perceptual system of the user’ is ‘submerged’, or rather mediated by technology in a
virtual environment (Biocca and Levy 1995, 57).
We use the term embodiment both in the context of sensing one’s own physical body in a real
environment and the perception of one’s body in a virtual environment. This sensation
emerges from bottom-up influences, sensory information like visual, haptic and proprioceptive
stimuli, and top-down influences, cognitive processes that interpret sensory information
(Kilteni, Groten, and Slater 2012, 377). According to Kilteni et al., a sense of embodiment is
dependent on three concepts: the sense of self-location, sense of agency and the sense of body
ownership.
Presence is used in many contexts and understood in many ways. Lee defines presence as ‘a
psychological state in which virtual (para-authentic or artificial) objects are experienced as
actual objects in either sensory or non-sensory ways’ (2004, 37). Here, para-authentic means
that a technologically mediated object ‘holds a valid connection with the object that it
represents’ (34), while artificial refers to objects that ‘do not actually exist in the real world’
but their technologically generated representations are perceived as real (35). Mel Slater
explains that the sense of presence is ‘a perceptual but not a cognitive illusion’; cognitively
knowing what is perceived is not real, it is still perceptually experienced as real (Slater 2018,
432). Both definitions above consider presence only in the context of technological mediation,
and therefore it is useful to turn to Cormac Power for an understanding of how presence
might be described in theatre and performance.
Power qualifies presence as ‘a key term underlying the articulation of theatrical experience
within Consciousness Studies’, whereby presence can be defined as ‘being the simultaneity
between consciousness and an object of attention’ (2008, 3). He suggests three modes of
presence in theatre: fictional, auratic and literal. Especially in the theatre, it is this third
literal mode of presence that always exists next to the fictional and auratic modes of
presence, staging a double experience of both the actual present and the represented.
When using the terms immersion, embodiment or presence in this article, we refer to ‘a certain
degree’ of these states, rather than a ‘full’ or ‘singular’ state, acknowledging the multiple levels
of perception of being in several places simultaneously.
Intermediality arises at the ‘meeting point in-between the performers, the observers, and the
confluence of media involved in a performance at a particular moment in time’ (Chapple and
Kattenbelt 2006, 12). Boenisch describes intermediality as ‘an effect created in the perception
of observers that is triggered by performance – and not simply by the media, machines,
projections or computers that are used in performance’ (in Chapple and Kattenbelt 2006, 113).
We see that intermediality is conceptualized within the experience of the spectator, or rather
experiencer, and thus it becomes conceptualized through what the spectator visually
perceives and experiences through all of the senses.
The physical experience
We turn now to the phenomenological experience of Blue Hour VR, by prioritizing those
aspects of the experience and design that focus on the bodily (embodied) experience.
Physical site-responsivity was a central condition of our design. The Malá Sports Hall was
arranged into a grid-like pattern of 36 circular pools. Filled with sand, from the Vltava river,
cedar, bamboo, local flora, salt and water, these tactile ecosystems offered material ways of

Page 5 of 20
exploring embodied memories through touch and scent. Two enormous scaffolding structures
towered over the space, textured with a 24-minute media loop, a weave of animations, lighting
effects and audio landscapes, inspired by the human circadian rhythm. Climbing these
scaffolding structures afforded a precipitous vantage of the pools below. The ceiling of the
Malá Sports Hall had large curved ribs, like the bones of a whale. The physical architecture of
the hall, as well as the atmospheric architecture of light which alluded to the nautical states of
blue hour, were the starting points for the design of the real-time 3D computer-generated
(3DCG) virtual environment.
Of the 36, we occupied four sand-filled pools as the sites of the Blue Hour VR installation.
Passers-by could see the experiencer, singularly, standing barefoot in the sand, lit by a small
spotlight. The experiencer wore an HMD, which blocked their eyes and ears, and held a
lantern-like object (orb-lantern). The first-person perspective of the experiencer was visible in
real-time on a screen on the periphery of the pool.
Performativity of the experiencer. Photo © Tomáš Brabec.
The onboarding1 was an important part of the experience, as an assistant helped the
experiencer to find their grounding, assume the HMD and the orb-lantern. The orb- lantern
rooted the experiencer to a real object for the duration of the experience. It was both a real-
world object in the hands of the experiencer and replicated as a para-authentic interactive
object tracked in real-time in the virtual world. The experience of the HMD itself has prominent
physical sensations, with its weighty plastic and glass sci-fi mask construction, wireless antler-
like antenna, and its immersive optical screen system. Because of the one-to-one mapping of
the sports hall, putting the HMD on had the illusion of putting on a thick but transparent pair
of cumbersome glasses.
For the first three minutes of the Blue Hour VR experience, the virtual environment was a 3DCG

Page 6 of 20
recreation of the sports hall. These first few minutes gave time for the experiencers to navigate
and explore the incongruities of sensations from the real and virtual environments. This
introduction propelled most experiencers to engage with their bodies, to explore the place
illusion physically with their bodies as well as their perceptions. We think that this initial
interaction and physical exploration grounded the experiencer’s sense of embodiment and
immersion; it established the experiencer’s sense of self- location and body ownership
(Kilteni, Groten, and Slater 2012).
There are psychological and emotional dimensions to the phenomenological experience
of wearing the HMD. We can speak of the novelty of the apparatus itself, the sense of
disembodiment that comes with the virtual ‘blindfold’, the deprivation of the real line of
sight, the sense of enclosure and sensory deprivation. There is an important trust aspect
to all this, as the experiencers are deprived of their sight, cut off from real- time awareness
of what is taking place around them. Each experiencer felt a different degree of
vulnerability during this process and were aware of being watched by others. Their
experience became performative.
The Blue Hour VR installation within the 36Q° 2019 environment. Photo © Joris Weijdom
In our visual design there was an absence of a representation of the experiencer’s body.
Looking down at the virtual sand pool and surrounding environment, the experiencer’s view
was unobstructed (neither the image of their body nor that of a 3D avatar was rep- resented).
We had expected this to contribute perhaps to a feeling of disembodiment, and we were
surprised that this was not the case. Many of the experiencers communicated a strong
feeling of embodiment within the virtual environment, even with the absence of a
representation of their body. It seems that the lack of body representation led to a heightened
sense of body awareness (Bakk 2019).
Furthermore, we suggest that the unstable nature of sand, which required the constant
rebalancing of their feet, invoked a heightened kinaesthetic sense that helped to ground the
experiencers in their body, despite the absence of a body representation. Interestingly, only a

Page 7 of 20
few of the experiencers of Blue Hour VR felt nausea or vertigo. We believe this is due to the
steps we took to ensure grounding of the feet, grounding to a tactile object, the orb-lantern,
and our design choice to encourage experiencers to engage with their bodies in the initial
sequence of the HMD virtual environment, and the hybrid blend between 3DCG and 360
Video.
Sound was also a crucial auditory guide in the experience. We can speak of three categories of
sound: the ambient sounds of ongoing conversations in the sports hall, the omnipresent 24-
minute looped audio track played outside of the HMD on the loudspeakers, and the internal
soundtrack played within the HMD, scoring the linear visual dramaturgy of the experience.
Visual perceptions created by the hybrid use of 360 Video and
3DCG
In general, there are two categories of VR: 360 Video and real-time 3DCG. Both types of VR
position experiencers at the centre of a virtual environment, affording them the possibility of
physically looking around by rotating their heads and bodies. This ability to control one’s
viewing direction in an immersive electronically mediated environment is what Kurt Vanhoutte
and Nele Wynants posit as being embodied in the narrative (Bay-Cheng et al. 2010, 47).
However, both types of VR have fundamental technical differences that impact on an
experiencer’s perception of interactivity. For example, 360 Video is com- posed of pre-
recorded footage shot in-camera and then projected onto a singular all- encompassing fixed-
depth sphere in a virtual environment, while 3DCG simulates a three-dimensional
environment in real-time that enables a greater sense of verisimilitude because of its ability to
replicate depth, mass and the parallactic response to an experiencer’s movements.
However, achieving photo-realism with 3DCG is extremely labour intensive in the
reconstruction of virtual environments through computer modelling, texturing, shading and
lighting. In contrast, 360 Video records whatever real environment surrounds its optical lenses,
making it less onerous to create. Both types of VR position the experiencer at the centre of a
virtual environment as the locus of a potentially performative act.
In Blue Hour VR, all experiencers entered visually into a one-to-one para-authentic
representation of the sports hall. This challenged the experiencer to make sense of the
verisimilitude of the virtual representation, grappling simultaneously with the task of walking
on unstable ground and realizing that they have no feet. As experiencers reached the centre
of the sand pool, they triggered the commencement of the timed dramaturgical sequence
through the lifting of the lantern-orb. This act of finding, grasping and lifting the real lantern-
orb once again questioned the veracity of the virtual representation as the experiencer’s visual
apparatus recalibrated to accommodate the incongruencies between real and virtual objects.
Picking up the lantern-orb triggered the directional proliferation of small particles into the
empty arena, casting light into darkened areas. The experiencer was then lifted upwards
towards the ceiling of the sports hall.
Light particles became energized with an upwards motion, congregating en masse to initiate
the peeling open of the roof, which allowed shafts of voluminous light to pour into the hall. In
the most dynamic 3DCG animated sequence, as the experiencer reached the roof line, the
roof’s beam structure peeled away like the bones of a whale opening up and back and
eventually falling endlessly away into the abyss below.

Page 8 of 20
The para-authentic pool, lantern-orb and ‘ascension scene’. 3D renders © Joris Weijdom.
This sequence, which we termed ‘the ascension’, introduced 360 Video and 3DCG into a hybrid
blend, enabling highly realistic video environments in the background, while a series of
complex object animations of the deconstruction of the hall were super- imposed on the
foreground. This layering of imagery between 3DCG and 360 Video, as one experiencer
commented, created a sense of expansiveness and visual depth that is unexpected from
screen-based media, evoking a similarly false spatial depth achieved by the layering of two-
dimensional scenic drops to force a perspective within a proscenium theatre.
Next, the experiencer became surrounded by four spherical portals that grew out of the orb-
lantern and became pinned to each of the cardinal directions. Once activated, the portals
continually rescaled based on the experiencer’s proximity. Experiencers could interact with
each of the portals by approaching them, peering into them, looking around, and eventually
entering them. Once inside, the experiencer engaged with one of four two-minute-long 360
Video poems, inhabiting the narrative environment as they continued to have complete
control of their individual viewpoint.
The 360 Video poems, tethered to the four seasons, were compiled from footage shot in 12
locations over two years. Each poem contemplated the beauty of nature, as well as the
dissonant influence of human dominance, and of temporality and destruction in the
Anthropocene. Three of the poems immersed experiencers in the ecology of a natural place:
a forest or a beach. These were ‘universal’ places in the sense that they engaged memories
and associations which most of us possess as part of the human experience. Winter
contemplated the fragmentation and dematerialization of nature with the increasing presence
of technology in our lives. Spring was a plea to consider the seventh generation. Autumn
questioned the resilience of a forest in a changing climate. The fourth video poem stood apart,
as it immersed the experiencer in the non-iconic but recognizable sinking city of Venice. The
urban human experience was evoked, as were the themes of the demise of the city, and the
questionable evolution of our civilization. Summer lamented the impermanence of human

Page 9 of 20
systems and the temporality of their organizations.
The four 360 Video poems, autumn, winter, spring, summer. Video stills © Paul Cegys.
The poems began as realistic representations of the places. Then, digital manipulations
intruded and began to dematerialize them, deconstructing and transforming them digitally.
The musical score augmented these transformations. The digital manipulations began to
intrude into the perceptual and cognitive experience. If the experiencer achieved an illusion of
place, and through it a sense of embodiment and presence in the ecology of an environment,
that illusion was then shattered and reconstructed differently. The artifice of the ‘digital’
nature of the virtual environment was revealed, reminding us of the presence of the artists’
dramaturgical voice.
Symbolically, the video poems mirrored the dominance of the human cognitive and
technological influence over ecology, and the human role in transforming and manipulating
the natural world. As an example, a forest was transformed into a digital inferno of fire-like
textures, and the experiencer was pressed down into the forest floor, while the trees
extended above her, burying her beneath the abstract fire. Experiencers communicated the
feelings of being buried and subsumed; some felt they were becoming fungus, some felt their
feet were physically pressing deeper into the sand.
As the poems transitioned into a final concluding environment, a soap bubble appeared on the
horizon, slowly growing in size until it swallowed the experiencer, translucently layering an
additional 360 Video of a cacophonous Zamboni truck resur- facing the ice of the sports hall.
The stark whiteness of the ice and the enormous size of the Zamboni visually jolted the
experiencer out of the previous dream into another past life of the hall, and yet it was possible
to look beyond these layers to perceive the ghost-like presence of the 3DCG representation.
Seconds later, the soap bubble disappeared and the experiencer was returned to the place
where they had started, but sometime in the hypothetical future. In this final 3DCG, the sports
hall felt abandoned, with windows and wall pieces missing. It was reclaimed by nature, as
raindrops fell through open sections of the roof and moss and grass recolonized the ground.

Page 10 of 20
One of the experiencers commented on how the sensation of sand under her feet belonged to
this environment, whereas it had felt dissonant at the beginning, aligning with her bodily
experience and arguing for ‘the primacy of the embodied experience in the end’.
We noticed through observations and interviews that the technical differences of the two
types of VR (3DCG and 360 Video) afford divergent modes of physical and mental engagement
with and within the virtual environment. Technically, 360 Video does not change its content
through kinaesthetic or tactile feedback by the experiencer, while real-time 3DCG does. This
means that with 360 Video in VR, the experiencer is mostly encouraged to look around and
possibly take time to mentally reflect, while physically being stationary, whereas real-time
3DCG affords physical movement and possibly some form of interaction with the virtual
environment. The hybridizing of both types of VR, 360 Video and real-time 3DCG, allowed for
new and diverse modes of engagement within the same experience.
Emergence of the Zamboni ‘video bubble’ and return to abandoned sports hall.
3D renders © Joris Weijdom and Paul Cegys.
Embodied design process
Working with complex digital technologies like VR, within the context of theatre and
performance, has an inevitable impact on the creative process due to its intrinsic
requirements, both in the phasing of the process and in its artistic outcome (Weijdom 2016).
Since the application of VR technologies within a performance context is relatively new, they
do not necessarily accommodate a creative process that incorporates embodied design
strategies. Even though the fields of interaction design and human-computer interaction (HCI)
offer specific know-how concerning the technical development of the experience, they do not
provide comprehensive tools for embodied improvisation as part of the collaborative creative
design process (Wendrich 2016). Furthermore, the production of artistic content for these
technologies has by and large come from film and game design, where the production
workflows differ significantly from those of theatre and live performance. Undoubtedly, the

Page 11 of 20
area of MR performance design can benefit from the fields of film and game design, especially
as VR struggles to gain recognition as an independent medium with its own disciplinary
knowledge, design strategies and concepts for analysis (Uricchio 2018). Additionally, we think
that Boenisch’s idea of theatre as a hyper-medium and Kattenbelt’s assertion of theatre’s
ability to incorporate all media (2008, 20) could be productive intermedial approaches for the
design, creation and study of VR and MR performances. Nonetheless, by introducing the Blue
Hour VR experience as a site-responsive MR installation in a performance design context, we
had to invent a new interdisciplinary design process, build technological tools for its facilitation
and question its impact on the dramaturgical experience. Resonating with the experimental
a
nd
explorative
f
r
a
me
of
the e
ncompassing
36
Q
i
n
stallatio
n
,
we chose to approach the design
of Blue Hour VR as an iterative experimental process that facilitated several embodied
improvisation sessions.
Screenshot of the real-time 3DCG reconstructed para-authentic sports hall. Screenshot © Joris Weijdom.
In preparation for the improvisation sessions, we built a digital 3D architectural
reconstruction of the sports hall, animated a few of the key events, gathered some 360-
degree test video and developed a system for kinaesthetic interactions with the 360 Video,
3D objects and environments. The physical setup was aligned with the simplified full-scale
physical version of the pool with a para-authentic representation of the pool in its complete
form within the digitally reconstructed sports hall in VR. The setup enabled a person to
physically move around in this MR space wearing a tethered HMD while being monitored by
the operator running the VR experience. The studio space was set up to facilitate direct
physical explorations with the full- scale VR prototype, reflective brainstorming sessions and
to afford simultaneous technical development.
Instead of working from a dramatic text, we chose to first map concepts based on inspi- ration
taken from the
36
Q
˚
installation,
its
physical
location
and
the experienti
al
possibil
ities
of VR
technology. For example, the 36 physical pools offered inspiration for the VR portals where
the experiencer would be given the choice to transport to other 360 Video worlds. Moreover,

Page 12 of 20
the particular architecture of the sports hall, especially its wooden overarching roof resembling
a ribcage structure, offered direct inspiration for the animated deconstruction of its para-
authentic counterpart in VR. This opening of the roof also triggered the idea of virtually moving
the experiencer outside of the building to oversee its structure while entering into the liminal
world modelled after blue hour. Based on our earlier exper- imentation, we wanted to create
a balance between physical action and mental contemplation, as divergent modes of
engagement. Often, interactive installations and games have ‘idle’ states of play, whereby the
participant triggers the next action or level in order to advance the experience. In contrast, the
design of Blue Hour VR included similar ‘idle’ states, but excluded the necessary trigger to
advance, so as to encourage the experiencer to ‘rest’ in a more contemplative state of being.
Operator live puppeteering (A) virtual environment (B) getting direct feedback from the experiencer
(C).
Photo © Joris Weijdom.
Based on ideas considering embodied interactions and a rough temporal structure, as a
succession of scenes, we defined the components of the key events and connected their
variables to physical controllers (mouse, keyboard and Musical Instrument Digital Interface
(MIDI) device) to be live controlled by an operator. For example, key animations like the
opening of the roof and deconstruction of the sports hall were connected to keys on the
keyboard that would trigger the animation when pressed.
Additionally, testing risky movements that could induce nausea, like the ascension of the
virtual position of the experiencer through the virtual environment, were performed by
manual mouse operation. More complex transformations were connected to the physical
knobs and sliders of the MIDI device. This enabled the operator to physically puppeteer the
timing of animations, positioning of the experiencer, and modification of different portal
actions, such as size, shape and proximity. Puppeteering in this way allowed for the testing of
complex cue-based sequences that could be modified in real-time based on the experiencer’s
behaviour and offered a more collaborative theatre-like rehearsal. In this way, the operator
became a co-performer mediating the experiencer’s virtual experience in real-time.
The improvisation sessions focused on using live performance to explore the dramaturgy and

Page 13 of 20
timing of scenes and the complex interactions and transitions of key events. As improvisation
technique demands,2 we deconditioned preconceived design ideas and assumptions about the
VR experience so as to cultivate sensitivity to the bottom-up sensations that came from the
physical body in response to the virtual input. Sensitivity to these inner sensations is
considered an essential source of information in the performing arts, in contrast to the field of
HCI, which generally focuses on the observation and analysis of external actions (Segura, Vidal,
and Rostami 2016). By incorporating a sensitivity to ‘the body’s awareness of the self’, without
the need for immediate expression through reflection in action, a less biased experiential space
allowed for new insights to emerge through what Núñez-Pacheco (2018) typifies as reflection
through inner-presence.
Team member sitting on pool ledge, reflecting through inner presence. Photo © Joris Weijdom.
This would enable the team member to let go of all preconceived ideas and assumptions
concerning the design and the idea of role-playing an imagined experiencer, while instead
focusing on real sensations in the present. Having the experiencer voice these sensations out
loud, without interpretation, allowed the team to associate them with either a visu- ally
recognizable physical action or a state of active perception without corporal movement.
After these improvisation sessions, the team gathered and reflected on the exercise, both
artistically and technically, to formulate dramaturgical pointers for improvement of existing
ideas or the generation of new ideas. Attending to both artistic and technological aspects, our
critical moments of reflection within this iterative process can be best typified as
technodramaturgy, which consists of ‘the interplay between traditional dramaturgies and the
innate, often concealed dramaturgies of technical systems themselves, whether software,

Page 14 of 20
hardware or mechanical’ (King 2018, 326). Finally, this reflection phase either resulted in
another intense improvisation session incorporating the agreed changes and questions or led
to a phase of further development of materials and computer programming.
Designing the Blue Hour VR experience through an iterative process that incorporated
embodied improvisation sessions created the possibility to utilize direct physical experience as
part of the design strategy. It allowed for new ideas to emerge, and offered a space to find
natural interaction schemes for complex interplay with the virtual environment. It afforded
the medial explorations of combining the two types of VR, 360 Video and real-time 3DCG, into
new hybrid forms, and it offered a space for the creative team to work together through a
shared experience.
The shifting boundaries of performance design in relation to VR experiences as a site-
responsive MR installation were explored in an interdisciplinary dialogue between the fields
of theatre, performance, film, game design and HCI. Furthermore, the discovery of potentially
new technodramaturgies for VR experiences necessitated exploration through embodied
improvisation. This allowed experiential qualities to emerge, unique to this new hyper-
medium, while composing the overall experience as ‘a process in which the invention is
performed while the performance is invented’ (Bertinetto 2011, 95).
Analysis
To imagine how dramaturgy and experience interwove in Blue Hour VR, we find it useful to
apply the terms intermedial texture and intermedial stratigraphy as put forward by Man-
cewicz (2014). We understand intermedial texture as an ephemeral tapestry of all of the digital
and non-digital media, bodily and textual elements, which, as an irreducible and complex
texture, deliver the experience and create the intermedial conceptualization in the observer’s
mind (Mancewicz 2014, 148). The concept of intermedial stratigraphy expands this notion
into time and space. We imagine intermedial stratigraphy as a
mental scaffolding, a
complex imaginary layering of meanings and phenomenological experiences. In
Mancewicz’s words, it is ‘an interplay of live and mediatized effects that leads to the
multiplication and layering of temporal and spatial perspectives’ (2014, 18). The concept
of an intermedial stratigraphy is particularly compelling when we try to grapple with
dramaturgies of non-unitary, intermedial, multi-layered mixed reality worlds.
The perceptions evoked by Blue Hour VR did not add up to a coherent and unitary
experience of space and time. Instead, in Maaike Bleeker’s words, ‘the perceptual and
cognitive engagement [was] … scaffolded … material and virtual worlds exist[ed]
simultaneously, partly side by side, partly overlapping, folding into or opening up to one
another’ (Bleeker, pers. comm.). The intermedial stratigraphy of Blue Hour VR was a very
dense entangled mesh of physical, perceptual and cognitive experiences. There were
multiple and mixed realities, and the in-betweens amid realities. The experience of Blue
Hour VR started within a virtual one-to-one representation of the sports hall. Experiences
were then transported to a reality beyond the real, the architecture of the virtual space
opened up onto illusory virtual transformation. Visual illusions overlapped the site-
responsive reality, and then immersed the experiencers in virtual reality 360
cinematographic captures of specific environments. Suspended and abstracted in time,
these environments embedded the experiencer in the hyper-reality of the natural world.

Page 15 of 20
Para- doxes ran thick in these layers. These environments, real but completely virtualized,
became invaded by digital manipulations and effects, which revealed their artifice; but,
of course, the algorithms that transgress and manipulate are the same as those that give
virtual reality its seeming ‘reality’. The experiencers never stood in a forest, they only
communed with its digital representation. As digitization encroached, the presence of the
artists was felt, as was the unfolding of a linear visual narrative. In the end, a post-human
futuristic sur-reality reclaimed former representations and returned the experiencer to
the transformed geography of the sports hall. Thrown somewhere into a future
dramaturgical time, but returned to the architecture of the place where the experience
had started, the experiencers were still rooted in sand. Sand, which gave the sensory input
for a phenomenological reality blended up into these various embodiments.
As intermedial performance enlivens, stages and amplifies hypermediacy, artists seek the
counterbalance of immediacy through immersion and interactivity, and through attention to
the haptic and phenomenological journey of the spectator (Lavender, in Chapple and
Kattenbelt 2006). Immersion in theatre strives to rekindle the awareness and engagement
of all of the senses, and to break the detachment, isolation and passivity of the audience
(Machon 2013).
Our conceptual decision to ground the experiencer in the haptic sensations of the ‘material
world’ derived from our striving for a holistic approach to using VR technology to design an
MR performative experience. Our position recognizes the centrality of the physical body in
concert with the perceptual mind in an interplay with the world, whether it be material or
virtual. This embedded unity of body and mind in the world is what Alva Noë describes as
being ‘at home in the world’ (2012, 5).
One of our experiencers, a six-year-old, put on the HMD then darted to and fro, exploring the
virtual environment very physically. She then abruptly called out to her mother: ‘Mama, do I
still exist?’ This child’s spontaneous, truthful, non-intellectualized remark
has inspired
questions about embodiment and presence. This little girl seemed to gain such a level of
immersion that she was no longer sure if her real self still existed. The virtual environment
was compelling enough that it activated her body to physically explore it. The place illusion
for the child seemed to become complete, and astonishingly existential.
As mentioned above, we found that many of the experiencers felt deeply connected to their
bodies despite there being no visual representation of their bodies. Often, they commented
that this was strange, but not destabilizing or off-putting. In fact, the deep sense of
embodiment that experiencers felt made us wonder what perceptual apparatuses they used
to seamlessly move between the virtual and the real. The body of the experiencer is ‘both
absent from the intermedial texture … and very much present as the locus of making sense
of the sensory impressions provided’ (Bleeker, pers. comm.). Through the enactment of
perceiving and making sense, ‘the intermedial stratigraphy unfolds from combinations of
visual, tactile and auditory stimuli’ (Bleeker, pers. comm.). Bleeker questions whether the
conceptualizations of meaning are embodied and embedded within the relationship between
the body and the environments.
With notable frequency, experiencers described and radiated a feeling of joy after emerging
from the experience. This was fascinating to us, as dramaturgically the video poems were
responding to the temporality of the human experience, ecological and social uncertainty, and

Page 16 of 20
the destructive power of the Anthropocene. While many experiencers emerged with a
meditative attitude as we had expected, many also emerged with a more exuberant feeling.
We presume that the feeling of joy was a result of their amazement and wonder at their sense
of embodiment within a virtual and illusory world, from their embodied journey beyond the
actual space into a virtual and poetic one. Experiencers seemed invigorated by their sense of
agency over their viewpoint, over their experience of virtual places that felt illusorily akin to
the embodied experience of a real place but within the safe space of the ‘theatrical frame’.
These feelings may have also arisen from the conceptual excitement of witnessing a new
frontier of technology. As the creators of Blue Hour VR, we felt a similar impetus of
excitement at imagining the role that VR and MR can play in engaging experiencers,
through embodiment and feelings of awe and wonder, in a fundamentally new way. This
is a productive way to think about how VR, with its incredible potential to ‘refresh our
perception’ (Kattenbelt 2008), can inject new momentum into theatre and performance,
which in turn can open up new dramaturgical potentials and scenographic opportunities.
We wonder, for example, about the potential of VR and MR to educate and transform our
understanding of social relations by enabling direct compelling experiences of
environments, places and perceptual and phenomenological intermedial representations
of the experiences of ‘others’.
3
We were struck by the experiencers’ overwhelming need to communicate and share the story
of their journey, and the act of recounting what they experienced seemed to be a necessary
part of bringing them back from the virtual. This may demonstrate the degree to which
experiencers felt ownership over their journeys. Being in relation with another human also
seemed to be a necessary bridge back from the absence of their virtual bodies.
Shauna Janssen commented on the affective estrangement related to the parameters of
time and space, which awakens ‘a desire for spatio-temporal “otherwises” whereby the
senses are negotiating the body as a site of intra-activity’ (Janssen, pers. comm.). Janssen’s
questions focused on the body as a site of participation, and how participation produces
meaning-making in VR. Blue Hour VR raises questions about the unprecedented
possibilities of intermedial embodiment within non-unitary spatial and temporal
environments stamped with the phenomenological sensory imprints of haptic experience.
The relevance of such new genres of experience is rife with significance for expanded
scenography and performance design within our hypermediated world.

Page 17 of 20
Experiencer in full immersion. Photo © Joris Weijdom.
Conclusion
Blue Hour VR, a site-responsive mixed reality performative installation, demonstrated our
fascination with the phenomenological experiences, visual perceptions and conceptualizations
of meaning that arose from the staged dialogue between physical and virtual environments,
and embodied experience. Blue Hour VR dealt with the cognitive acknowledgement of the
literal presence of its technological apparatus, and the bodily reactions to the fictional
presence of the virtual environment. We agree with Power’s conclusion that theatre’s strength
lies within its capability to perceptually stage the simultaneous- ness of the literal (or present)
with the fictional (or represented) and play with its inter-relationships and contradictions
(2008). It is through the staging, layering and merging of the fictional and literal environments

Page 18 of 20
that experiencers were able to perform multiple modes of simultaneous presence. Power
points out that ‘theatre’s manipulation of presence can be seen as a crucial asset at a time
when technologically produced representations are increasingly capable of concealing the
artificiality of their construction’ (2008, 205). Blue Hour VR’s highly performative staging
created the conditions in which to reflect on the complexities of our perception of reality and
its technological mediation.
Blue Hour VR demonstrated how low fidelity real-time 3D computer graphics and 360 Video
were applied in a hybrid design, and how the medium of VR was applied in a per- formative
context, to enliven a compelling and complex embodied mixed reality experience between
fictional and concrete environments. The design process of Blue Hour VR contributed findings
applicable to other MR design practices within a performative context. An embodied medium,
such as VR, needs an embodied design strategy that incorporates direct experience with inner
presence and dialogue with live puppeteered virtual events, especially when the medium itself
is yet to be fully understood. The application of complex technologies in a creative design
context requires a technodramaturgical approach, which acknowledges that both technology
and design influence one another.
Blue Hour VR is a compelling example of how the profound sense of embodiment, and
intermedial performance design, can expand the field of scenography and imbue the per-
formative act with new relations between creator(s) and experiencer(s). Conceptualizing
scenography as a performative act or ‘encounter’ (Hannah and Harsløf 2008, 15), rather than
simply one of communication, highlights the experiential and embodied nature of
scenographic experience and the dynamic interaction of bodies, environments and materials
that expanded scenography aims to generate (McKinney and Palmer 2017, 8). As a form
experiment, Blue Hour VR generated experiential insight for future MR performance designs.
Moreover, it explored how VR can be utilized in scenography and performance design as a tool
for social engagement with which the artist can access the experiencer in a new relationship,
in a new state of embodiment, and in a new form of presence.

Page 19 of 20
Notes
1.
Onboarding is a term used in game design to convey a basic level of knowledge needed to
navigate the designed experience. In Blue Hour VR, onboarding was an intentional performative
act led by the technical crew to assist the experiencer in transitioning between real and virtual
worlds, and in understanding how to use the technology and navigate the experience safely.
2.
We understand improvisation in the context of creativity and design as ‘the practice of com-
posing or inventing extemporaneously, through some kind of responsive and situationally-
dependent departure from preformed plans or expectation’ (Kang, Jackson, and Sengers 2018,
160:1).
3.
The scenographic process was influenced by the practices of relation scenography that evolved
out of The Home as part of the Digital Oral Histories for Reconciliation project. The Home is a VR
experience that attempts to create an immersive scenography with the aim of honouring the
principles of restorative justice (Roberts-Smith and Cegys 2019).
Acknowledgements
As the two creators of Blue Hour VR, we merge two perspectives within this article along the lines of our
collaborative process. Paul Cegys presents the phenomenological and visual dimensions of the Blue Hour
VR experience, and analyses the conceptualizations of meaning that arose from the complex intermedial
stratigraphy of the mixed reality experience. Joris Weijdom offers perspectives on embodiment and
presence in virtual and mixed reality, and discusses the hybrid and embodied technodramaturgical
design process of Blue Hour VR.
The Blue Hour VR experience was conceived and created by Paul Cegys and Joris Weijdom, com- posed
by Jonathan Cegys, programmed by Richard van der Lagemaat, with research and production support
from Jessica Bertrand. Thank you especially to Marketa Fantová and Jan Rolnik whose support was
central to the success of Blue Hour VR. We would also like to acknowledge the enormous effort it took
to create the PQ’s 36Q° 2019 project. The full list of collaborators can be found at
http://www.pq.cz/2018/03/27/36q-3/.
a
Notes on contributors
Paul Cegys’ (MSc) work merges multiple practices of performance creation and design, from theatre and
opera to site-responsive installation and intermedial VR/MR scenographies. Upholding his commitment
to ecological imperatives he merges his artistic work with his sustainability practice (MSc in Sustainability
Science and Environmental Studies, Lund University, Sweden). He is the digital scenographer for The
Home, a VR experience designed for the Digital Oral Histories for Reconciliation project and is on the
advisory board of the Toasterlab Mixed Reality Performance Atelier. He is on faculty at the University of
Waterloo in the Department of Communication Arts and is a PhD Candi- date in the School of Arts, Design
and Architecture at Aalto University. For more information about his work please visit
http://www.paulcegys.com
Joris Weijdom (MA) is a researcher and designer of mixed-reality experiences focusing on
interdisciplinary creative processes and performativity. He is a lecturer and researcher at the HKU
University of the Arts Utrecht where he founded the Media and Performance Laboratory (MAPLAB),
enabling from 2012 until 2015 practice-led artistic research on the intersection of performance, media
and technology. Currently he works as a researcher at the Professorship Performative Processes and
teaches at several BA and MA courses. As part of his PhD project Joris researches creative processes in
collaborative mixed reality environments (CMRE) in collaboration with University of Twente and Utrecht
University.

Page 20 of 20
References
Bakk, Á. K. 2019. “Performativity and Worldmaking.” Acta Universitatis Sapientiae, Film and Media
Studies 17 (1) (October 1): 169–178.
Bay-Cheng, Sarah, Chiel Kattenbelt, Andy Lavender, and Robin Nelson, eds. 2010. Mapping Intermediality
in Performance. Amsterdam: Amsterdam University Press.
Bertinetto, A. 2011. “Improvisation and Artistic Creativity.” Proceedings of the European Society for
Aesthetics 3: 81–103.
Biocca, Frank, and Mark R. Levy, eds. 1995. Communication in the Age of Virtual Reality. LEA’s
Communication Series. Hillsdale, NJ: L. Erlbaum Associates.
Chapple, Freda, and Chiel Kattenbelt, eds. 2006. Intermediality in Theatre and Performance. 1st ed.
Themes in Theatre 2. Amsterdam: Rodopi.
Deleuze, G., and F. Guattari. 1987. A Thousand Plateaus: Capitalism and Schizophrenia. Minneapolis:
University of Minnesota Press.
Hannah, Dorita, and Olav Harsløf, eds. 2008. Performance Design. Copenhagen: Museum Tusculanum.
Kang, L., S. J. Jackson, and P. Sengers. 2018. “Intermodulation: Improvisation and Collaborative Art
Practice for HCI.” In Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems, 160:1–160:13. CHI ‘18. New York: ACM.
Kattenbelt, C. 2008. “Intermediality in Theatre and Performance: Definitions, Perceptions and Medial
Relationships.” Culture, Language and Representation VI: 19–29.
Kilteni, K., R. Groten, and M. Slater. 2012. “The Sense of Embodiment in Virtual Reality.” Presence:
Teleoperators and Virtual Environments 21 (4): 373–387.
King, A. F. 2018. “Moving Masks and Mobile Monkeys: The Technodramaturgy of Augmented Reality
Puppets.” Theatre and Performance Design 4 (4): 324–341.
Lee, K. M. 2004. “Presence, Explicated.” Communication Theory 14 (1): 27–50.
Machon, J. 2013. Immersive Theatres: Intimacy and Immediacy in Contemporary Performance.
Basingstoke, Hampshire: Palgrave Macmillan.
Mancewicz, A. 2014. Intermedial Shakespeares on European Stages. Palgrave Studies in Performance
and Technology. London: Palgrave Macmillan UK.
McKinney, Joslin, and Scott Palmer, eds. 2017. Scenography Expanded: An Introduction to
Contemporary Performance Design. London: Bloomsbury Methuen Drama.
Merleau-Ponty, M. 1962. Phenomenology of Perception. London: Routledge. Milgram, P., and
F. Kishino. 1994. “A Taxonomy of Mixed Reality Visual Displays.” IEICE Transactions on
Information Systems E77-D, no. 12.
Noë, A. 2012. Varieties of Presence. Cambridge, MA: Harvard University Press.
Núñez-Pacheco, C. 2018. “Reflection through Inner Presence: A Sensitising Concept for Design.”
Multimodal Technologies and Interaction 2 (1): 5.
Power, C. 2008. Presence in Play. A Critique of Theories of Presence in the Theatre. Amsterdam: Rodopi.
Roberts-Smith, J., and P. Cegys. 2019. “Relational Scenography: Collaborative Scenographic Practices
for Restorative Justice in Virtual Reality.” Paper presented at the PQ Talks as part of the Prague
Quadrennial of Performance Design and Space, Prague, Czech Republic.
Segura, E. M., L. T. Vidal, and A. Rostami. 2016. “Bodystorming for Movement-Based Interaction
Design.” Human Technology 12 (2): 193–251.
Slater, M. 2018. “Immersion a nd the Illusion of Presence in Virtual Reality.” British Journal of Psychology
109 (3): 431–433.
Speicher, M., B. D. Hall, and M. Nebeling. 2019. “What Is Mixed Reality?” In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems, 537:1–537:15. CHI ‘19. New York: ACM.
Uricchio, W. 2018. VR: Between Hope, Hype, and Humbug. Los Angeles Review of Books. https://
lareviewofbooks.org/article/vr-between-hope-hype-and-humbug/
Weijdom, J. 2016. Mixed Reality and the Theatre of the Future. Brussels: IETM. https://www.ietm.org/
en/publications/fresh-perspectives-6-mixed-reality-and-the-theatre-of-the-future
Wendrich, R. 2016. “Hybrid Design Tools for Conceptual Design and Design Engineering Processes:
Bridging the Design Gap: Towards an Intuitive Design Tool.” PhD diss., University of Twente in
Enschede, Netherlands.