Content uploaded by Graham Wakefield
Author content
All content in this area was uploaded by Graham Wakefield on Jul 16, 2019
Content may be subject to copyright.
Conservation of Shadows: Shared Physicality Between Worlds
Haru (Hyunkyung) Ji1 and Graham Wakefield2
1OCAD University, 2York University
Toronto, Canada
hji@faculty.ocadu.ca, grrrwaaa@yorku.ca
Abstract
This article describes a site-specific interactive mixed reality
installation artwork involving a network of over a hundred
motor-actuated bells, projections upon a 4x6m bed of salt,
and a dual motion tracked virtual reality perspective
inhabited by artificial life and integrating real-time volume
capture. This work responds to very specific history of the
host venue as a former centre for disease control and reagent
storage, through a central conception of shadows as shared
physical images between visible and invisible worlds,
carried through with dual emphasis on functional and
contextual meaning in all components. Details of this
context, as well as the technical realization, are followed by
discussion of mixed reality art as a site-specific expression,
and directions for future development.
Keywords
Art Installation, Artificial Life, Embodiment, Interactive
Art, Mixed Reality, Physical Computing, Shadows, Site-
Specific, Spatial Sound, Virtual Reality
Introduction: site-specific mixed reality
Virtual reality (VR) is appealing in its promise of infinite
possibilities, and the sense of presence, or “being there”.
However this flexibility also puts it in a strange
relationship with exhibition spaces. When VR can be
viewed practically anywhere, and wherever it is, it can
present being somewhere else, what ties a work to a place
of an exhibition in particular, how can it avoid an inherent
sense of disconnection? We are excited by the potential of
Mixed Reality (MR) as a resolution that can bind virtual
spaces to specific physical places in both functional and
contextual ways. That is, rather than the virtuality
presenting escapism to independent worlds, it may connect
deeply to a particular place, ontologically rooted in site-
specific grounds, perhaps revealing expansive possibilities
within what is physically present. Not only in being there,
but also in being there. This question was foremost in our
conception and realization of the Conservation of Shadows
artwork (Figure 1), installed at SeMA Chang-go, a recently
acquired expansion of the Seoul Museum of Art within the
former grounds of the Korea Centre for Disease Control.
Figure 1. The Conservation of Shadows installation.
Seoul City remodeled this site of medicinal warehouses,
waste disposal facilities, animal testing labs, and reagent
storage, into a new Innovation Park, including youth and
community facilities as well as innovation hubs and the
gallery itself (see Figure 2). The artwork formed part of an
exhibition in late 2017 (“Requiem for Hybrid Life”,
organized by NMARA, a non-profit art-science association
directed by Kenny Kyungmi Kim) that responded to the
site’s specific history while also reflecting on a future
defined by biology and technology, in which distinctions
between machines and living beings are becoming blurred.
Figure 3. Pre-visualization of physical installation design.
Our first response to this site-specific condition
responded to the building itself: a red brick masonry
construction with wood truss ceiling dating from 1962. The
installation space is large (6m by 12m) and barn-like with
aged timber shelving and rafters (at 4-6m above the floor),
through which one can see the sky during daylight hours.
We felt that the space had a very charged atmosphere, not
only through its history, but also in its textures of dappled
light, quiet reverberation, temperature, humidity, and
subtle odors. It made us imagine unknown beings growing
fond of the wet texture of the old timber architecture, the
fragrance of sunshine smeared between cracks, and the
quietness of murmuring and whispering. To give life to
them, we determined to extend senses and mix realities
surrounded by softly ringing bells and the crunch of salt
underfoot as shadows of the new beings pass by, along
with a dual alternate perspective through head-mounted
display in which we may instead become the shadows
around which these beings play.
Our working method has always been to ensure all
components of a work inter-relate, with both functional
and contextual meaning [1]; what we achieved in this
installation is the extension of the principle more fully into
its physical realization. The floor of the room is laden with
a carpet of 330 kg of salt, covering an approximately four
by six meter area (the majority of the floor space), upon
which light is projected from above (see Figure 4). Motor-
actuated miniature bells surround the installation space,
hanging down from the rafters at different locations and
heights, varying in intensity and creating a variegated
spatial sonic experience. A projector mounted centrally at
around four meters above the ground casts a predominantly
white image covering the bed of salt, fading away at the
edges. Visitors walking on the salt thus inevitably cast
shadows, which mingle with projected shadows of
invisible creatures appearing as ghostly vortexes in the
space. By donning a head-mounted display visitors take on
an alternate perspective to witness another world co-
existing in superposition with our own. What first appeared
as flat shadows are unveiled to be the membranes of
comprehensible beings moving around and through our
now-shadowy bodies, and shaped by our movements and
gestures.
The physical duality of shadows
as a bridge between worlds
The heart of the piece is the character of shadows as a
physical connection between realms, both revealing and
concealing. Shadows are rich in functional and contextual
roles, deeply rooted in human vision processes,
psychological symbolism, and ancient myth. Yet, at the
same time, as mathematically conducive optical and
Figure 2. The original building taken over by Seoul Museum of Art, including the retained interior shelving for reagent storage.
geometric phenomena of linearity and functional
transformations between spaces, they grounded some of
the earliest science with chronological and cosmological
tools.
Shadows are two-dimensional projections that speak of a
three-dimensional reality, like the higher-dimensional
beings that strangely fragment, coalesce, and pass through
Flatland, incomprehensibly so to its occupants. [2, 3] As
detached images of our outer form, our shadows impart
less even than our reflections. Nevertheless they have
recurrent mythical relation to spirit and loss. For example,
the character Peter Schlemiel’s life among people is
undermined after a Faustian pact in which he sells his own
shadow, and he retreats instead into a life of dedicated
scientific naturalism [4]. Through shadows we can more
readily join with others and become other in shadow play,
puppetry, and theatre.
This spurred us to think of the physicality of shadows as
a liminal bridge suspended between superposed physical
and virtual space in both mathematical and mythological
senses, continuing a thread of our prior work blending
superposed physical and virtual space, perceptually and
conceptually, through mixed reality artificial life [5].
In inspiration, it echoes the subtle physicality of the
organisms moving from screen to body in Sugrue’s
Delicate Boundaries [6], the otherwise invisible
inhabitation of site-specific locations imagined in
Semiconductor’s Mini Epochs [7], and the narratives of
other “found” organisms occupying urban spaces in U-
Ram’s machinic sculptures [8]. We borrow the title of this
work from Korean author Yoon-ha Lee’s speculative
fiction, which opens with this intriguing anti-proposition:
“There is no such thing as conservation of
shadows. When light destroys shadows, darkness
does not gain in density elsewhere. When
shadows steal over earth and across the sky,
darkness is not diluted.” [9]
Although the work does not refer directly to her
reimagining of the Descent of Inanna with influences from
the video game Portal [10], there is nevertheless a related
sensibility of inter-spatial experience and the afterlife.
Inanna is an ancient Sumerian goddess known for
triggering Gilgamesh's grapple with mortality, and her
descent into and return from the shadowy underworld Kur,
where she is stripped of everything and passes through
death itself.
Details of the realization
From the first designs to the realization of the installation,
in both physical and virtual components, we focused on
maintaining a duality of functional and contextual
meaning.
Bed of Salt The choice of salt as material serves several
functional purposes. The near-white colour responds to
projection well, but more importantly the granular texture
does not take the projection coldly as an impassive screen.
The translucent granules diffuse the light, imparting a
material sense of depth and quenching pixelation. The salt
is heavy enough, and laid thickly enough, to take footprints
without dispersing too much or exposing the underlying
floor. Visitors nevertheless feel the grains shifting and
audibly crunching underfoot with each step as they
progress further into the environment.
This brings attention to the ground, literally the media
under our standing, and the shadows upon it, while also
subtly undermining sure footing. As a material, salt carries
meaning as a natural antibacterial/antiseptic preservative.
Within Korean shamanistic traditions salt also carries a
kind of enchantment. It is regarded as precious for its
ability purify space, purging or expelling unfaithful forces
and bad spirits.
Figure 4. (Above) Visitors’ shadows on the salt blend with those
of the otherwise invisible creatures surrounding them. (Below)
Close-up detail of projected shadows upon salt.
Bell-motor assembly network Functionally bells are well
suited for spatial sound, as they are rich in spectrum and
transients. Moreover the bells are not perfectly
manufactured, each with distinct imperfections. Together
this makes them very easy to localize, even when filled out
significantly through the warmly resonant reverberation of
the timber framework of the room.
Bells also have specific roles in Korean shamanism as
sounds that bridge between the living and dead, a way of
communication between the visible and invisible. Shaking
and rattling bells often accompany shamanist ceremonies
in Korea. Of the three kinds of shaman, the third kind is
one that is visited and becomes inhabited by small gods,
and who will no longer be subjects of their own lives. After
the ceremony of accepting a small god, such shamans have
the ability to help people. The first sign of this visitation is
that you hear bells ringing when nobody else does. In the
exhibition the bells do not jingle in a rhythmic or
celebratory way, but continually tremble and jostle against
each other, sometimes very softly, sometimes much more
insistently. Several visitors described it as “haunting”.
The bells are activated by a network of 12
microcontroller boards arranged in a 2x6 grid above the
exhibition space. The microcontroller boards1 are
customized Arduino Unos with a smaller form-factor and
built-in networking, designed to be used in large
addressable sensor network arrays. The entire network
including the motors are supplied power by PoE, and the
network is addressed from the installation PC by an FTDI
USB to serial ethernet converter. Each microcontroller
board provides six 5V analog outputs, each of which is
connected to an individual motor-bell assembly by
22AWG cable. Each assembly is two miniature bells and a
small 11000 RM 5VDC vibration motor, the same kind
that makes a cell phone vibrate, in direct contact (see
Figure 5). Motor-bell assemblies were positioned in pairs
for redundancy/robustness, with each microcontroller
effectively controlling three independent quadruplets of
bells that hung at heights of approximately 0.5m, 1.5m,
and 2.5m above the floor (over 150m of cable in total).
Thus although the installation utilized 144 individual bells
on 72 channels, with pairing and redundancy there were
effectively 36 independent 3D sound locations around the
installation space.
Varying the motor signal resulted in perceptually
smooth transitions between the gentlest sporadic tinkles to
continuous ringing to rather loud frenetic rattling. Thus,
rather than using a discrete event model of ringing bells
when virtual creatures intersect with them (collision-
based), we adjusted motor activation levels continuously
depending on a mapping of the distance and density of
living matter near to the location of the bell. These motor
1Using nD::Nodes with support from Professor Mark-David
Hosale, from the nD::StudioLab, York University.
www.ndstudiolab.com.
intensity updates were updated in the simulation and
distributed to all nodes in the microcontroller network at
rate of 60Hz.
Figure 5. Close-up photograph of a motor-bell assembly hung in
the installation space.
Simulation and Rendering The majority of the software
was authored in C++ and OpenGL/GLSL, running on a
small form factor Intel I7 PC with an Nvidia GTX 1080 Ti
GPU. The software renders two graphical views of the
same world: one shadow-render at 30 frames per second,
1920x1080 resolution, for the ceiling-mounted projector,
and one first-person view at 90 frames per second at
4096x4096 resolution per eye for the VR head mounted
display (HMD).
The projected image renders silhouettes of creatures
against a white background to simulate the shadows in the
installation. This is achieved by a shadow-render pass over
a frustum from the gallery floor plane to the projector
location above, with distance-dependent intensity and
sharpness. That is, creatures’ shadows are darker, crisper,
and scale-consistent when close to the ground, but much
larger, paler, and with softer penumbra when closer to the
projector above. With sunlight filtering through the rafters
by day, the shadows of the beings become more
pronounced by night (see Figure 6).
Figure 6. Visitors exploring the installation during the daytime
(above) and after sunset (below).
The virtual living beings form a multi-agent system of
up to four thousand members at a time, with awareness of
their environment implemented using vector density fields
and hashspace-encoded neighbour queries. These beings,
of around 25cm individual length, have varying
preferences for joining into multi-cellular superorganisms
that take on globular or snake-like undulating forms,
sometimes spanning many meters in size; recognizable as
living but not in a familiar way. Their behavioural and
structural preferences diffuse gradually through
populations as morphogens, sometimes differentiating, and
sometimes responding to the presence of the human
visitors around whom they dance.
VR Perspective For the VR HMD we used an HTC Vive,
expanding the effective tracking area to almost the entire
room using extension cables and a sync cable between the
Vive beacons. In literal contrast to the salt-projection, from
the HMD the world appears predominantly black.
Creatures are translucent, showing only surface reflections
from a virtual light source positioned in the same world-
location as the projector (Figures 7 & 9).
Figure 7. Superposed photograph and virtual world. Photograph
taken with motion-tracked camera, and virtual image rendered
simultaneously from camera’s perspective and location, to align
the physical and virtual spaces accurately. Although this “mixed”
view is never part of the installation experience itself, it may help
convey to the reader the nature of the system.
Before installation we performed a photogrammetric
capture of the room, registering & fusing a sequence of
RGBD captures into a 360-degree spatial cloud (see Figure
8). This cloud forms a barely-visible backdrop to the VR
experience, providing a subtle echo to root the virtual
space back to the physical. Some additional particles
would drift slowly upward, like bubbles, aligned with the
locations of dangling bells.
Figure 8. Photogrammetry of installation space.
Shadow-Volumes More importantly, all visitors to the
installation also appear in the virtual space, but as solid,
opaque shadow-volumes of a pure black void. These
shadow-volumes are effectively height maps derived from
the depth images captured by two RGB-D cameras
mounted above the installation space. The RGB-D cameras
were also mounted centrally, very close to the projector,
such that the shadow-volumes that visitors cast in the
virtual space appear cone-like and aligned to the same
location as the projector/light-source.
Figure 9. Screen capture of the VR perspective, with a shadow-
volume of another visitor occluding some of the undulating
aggregates of artificial life beings.
Figure 10. Visitor discovering the virtual shadow-volumes of
their own hands.
As shadows, visitors become stripped of all texture and
feature, existing only as moving presences whose forms
negate vision. That is, the shadow-volumes are rendered
black against a black background, and thus primarily
perceived by how they occlude the creatures inhabiting the
space (see Figure 9), and how they affect the creatures’
movements. The visitor wearing the headset thus discovers
their own body and hands (see Figure 10), but also
discovers the presence of other humans around them. This
presence is nevertheless paired with a somewhat eerie
sensation of absence; perhaps inversely akin to the feeling
of knowing when somebody else is with you in a dark
room, one that is simultaneously inhabited by others.
Figure 11. Visitors sharing physical and virtual space.
Discussion and future directions
Throughout this work we were conscious of mixed reality
technologies (VR, motion tracking, RGB-D, projection) as
site-specific methodologies. One of the comments most
frequently expressed by visitors concerned how well the
installation fit the space, sonically, visually, and
thematically. We also noted that although the VR headset
was in frequent use, it did not dominate visits to the
artwork, with no queues for it even during busy periods.
The artwork follows in an extended research-creation
project and series of interactive art installations by the
authors, surrounding humans with biologically-inspired
complex systems experienced in immersive mixed reality,
under the title “Artificial Nature” [11]. The invitation is to
become part of an world in superposition to ours that is
rich in living feedback networks, but not as its the central
subject. The host exhibition concerned itself with a living
environment that is increasingly saturated with
computational agency, that is also becoming more visually
and spatially aware, and in which there is no longer as
clear a divide between the perceptions of the body and
those of the spirit. Our natural response is an urge to give
life to mixed reality; to explore how we may coexist with
others in ways that are mutually more abundantly curious,
playful, and meaningfully rewarding.
The artwork has since been invited to several venues for
international exhibition in 2019. For these exhibitions we
are deepening this responsive/interactive relationship
between the human and the other beings in the space. In
particular we want to bridge the sense of cohabitation with
digital beings. We are exploring structural neuroevolution
for agents, and aim to combine this with curiosity-driven
behaviour through predictive/compression based reward
functions. The aim is toward experiences in which visitors
sense how they are perceived and related-to by other
beings that are clearly and playfully aware of them. In this
way we hope to challenge preconceptions that
computational systems are in opposition to human and
natural systems, while also conceiving how non-
anthropocentric and decentralized artificial ecosystems can
become more empathetic and conducive to sharing the
world.
Acknowledgements
With special thanks to Mark-David Hosale and students of
the Digital Media program at York University (Adiola
Palmer, Amir Bahador Rostami, Filiz Eryilmaz, and
Nicholas Abbruzzese) for invaluable help putting together
the microcontroller network and motor-bell assemblies.
This research was undertaken as part of the Vision:
Science to Applications program, thanks in part to funding
from the Canada First Research Excellence Fund.
References
[1] Ji, Haru and Wakefield, Graham. "Endogenous
Biologically Inspired Art of Complex Systems." Computer
Graphics and Applications 36, no. 1 (January, 2016): 16-
21. IEEE Computer Society.
[2] Abbott, Edwin A. Flatland: A romance of many
dimensions. OUP Oxford, 2006. (First published 1884).
[3] Sousanis, Nick. Unflattening. Harvard University Press,
2015.
[4] von Chamisso, Adelbert. Peter Schlemihl's Miraculous
Story. Camden House, 1993 (First published 1814).
[5] Wakefield, Graham and Ji, Haru. "Becoming-There:
Natural Presence in an Art of Artificial Ecologies." In
International Symposium on Ubiquitous Virtual Reality
(Daejeon, Republic of Korea), 11-14. IEEE Computer
Society, July, 2013.
[6] Sugrue, Chris, “Delicate Boundaries (2007)”, Personal
website, accessed November 15 2018,
http://csugrue.com/delicateboundaries.
[7] Jarman, Ruth and Gerhardt, Joe, “Mini-Epoch Series
(2003)”, Semiconductor website, accessed November 15,
2018,
http://semiconductorfilms.com/art/mini-epoch-series
[8] U-Ram, Choe. Artist’s website, accessed November 15,
2018, http://www.uram.net
[9] Lee, Yoon Ha. Conservation of Shadows. Gaithersburg,
MD: Prime Books, 2013.
[10] Lee, Yoon Ha. Personal website, accessed November
15, 2018, https://www.yoonhalee.com/?p=304
[11] Ji, Haru and Wakefield, Graham. "Biotopes
Computationnels (Computational Biotopes)" In Stream 04:
Les Paradoxes du vivant (The Paradoxes of the Living),
pp. 304-316 Philippe Chiambaretta Architecte, Paris,
November 2017.
Author Biographies
Artificial Nature is a research-creation project co-founded
by Haru Ji and Graham Wakefield in 2007. Haru Ji is a
media artist exploring the subject of life in art through
artificial life worldmaking. She holds a Ph.D. in Media
Arts and Technology from UCSB and is an assistant
professor in DPXA & the Digital Futures programs at
OCAD University in Toronto, Canada. Graham
Wakefield is an artist-researcher and software developer
exploring the liveness of computational media. As
Assistant Professor in Computational Arts and Canada
Research Chair in Interactive Visualization he directs the
Alice Lab at York University. Artificial Nature
installations have counted over forty exhibits across nine
countries, including festivals such as SIGGRAPH,
Microwave Hong Kong, and Digital Art Festival Taipei,
conferences such as ISEA, EvoWorkshops, and IEEE VIS,
venues including La Gaite Lyrique, ZKM, CAFA Beijing,
Seoul City Hall, MOXI and the AlloSphere Santa Barbara,
long with selection in the VIDA Art & Artificial Life
competition (2015) and the Kaleidoscope Virtual Reality
showcase (2017).