Content uploaded by Andreas Sieß
Author content
All content in this area was uploaded by Andreas Sieß on Feb 06, 2019
Content may be subject to copyright.
Worldmaking: Designing for Audience
Participation, Immersion and Interaction
in Virtual and Real Spaces
Andreas Siess1(B
), Daniel Hepperle1, Matthias W¨olfel1,
and Michael Johansson2
1Karlsruhe University of Applied Sciences, Moltkestr. 30, 76133 Karlsruhe, Germany
{andreas.siess,daniel.hepperle,matthias.wolfel}@hs-karlsruhe.de
2Kristianstad University, Elmetorpsv¨agen 15, 291 88 Kristianstad, Sweden
michael.johansson@hkr.se
Abstract. Artists often try to open up new experiences for people, chal-
lenging them to extend horizons and perception. This becomes partic-
ularly relevant when thinking about experiencing built environments:
Here, technologies like Cave Automatic Virtual Environments (CAVE)
or Head-Mounted Displays (HMD) can be used as a tool to offer richer
experiences to the audience in both art installations and exhibitions.
We have been developing several exhibitions tackling the challenges that
come with exhibiting in (semi -) public spaces: how do we engage visi-
tors in our exhibitions, what role do bystanders play and how can this
be considered in the development and design process? The exhibitions
were built in a chronological order (2015–2018) and increasing degree of
immersion and interaction. For exhibition one (“step-in/Ideal Spaces”),
we built a CAVE-like “tryptic” projection showing linear pre-rendered
videos of seven different built environments. In exhibition two (“fly-
over/Super Nubibus”) we build a replica of a hot-air-balloon and let
people experience architecture from birds eye view using a HMD. Exhi-
bition three (“cruise/Biketopia”) is also an immersive VR using a HMD,
but from a very different angle. Here we use a bike to let people actively
explore a space by regulating speed and direction of the bike. By using
the discreet method of observation, we ensured that the visitors were not
disturbed in their experience, which in turn would falsify our findings.
So we are able to compare and discuss these three approaches in regards
to the above mentioned criteria within this paper.
Keywords: Virtual reality ·Spatial perception ·Exhibition
(Semi-) public space ·Architecture ·Museum
1 Introduction
Prototyping and exploring worlds and environments for creating exhibitions with
and through technology we sometimes use consumer technology that shortens
c
ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2019
Published by Springer Nature Switzerland AG 2019. All Rights Reserved
A. L. Brooks et al. (Eds.): ArtsIT 2018/DLI 2018, LNICST 265, pp. 58–68, 2019.
https://doi.org/10.1007/978-3-030-06134-0_7
Worldmaking: Designing for Audience Participation 59
and speeds up our work process, and we can start sketching in hard- and software
already early in the conceptual phase of a project. Even consumer products today
come with a developing kit and widely open APIs which make them suitable
also for experiments and research. One risk although here is that one as an
artist, designer and/or researcher gets trapped inside of the presumed future
use of this technology, with all limitations that comes with that [9]. Relying
on “produced” technology we clearly see a need for a set of tactics to bring
our work in our own direction. In our recent practice based work and research,
that focuses on exhibition work that spans from 2015 and onwards, we have
returned to the idea that if we want to have an audience or a visitor experience
3D spaces of different sorts in an exhibition format, we need to find ways for the
visitor to enter, interact and experience these virtual worlds, without too much
former knowledge how to enter, navigate and perceive them in order to have
the visitor feel both present and immersed by them. Because there is always a
risk with new, not seen and experienced technologies that we have the visitor
exploring, the technology itself instead rather than the topic of our attention,
e.g. spatial constructions. Through our work we gained knowledge that many
forms of interaction takes hours of practice to learn and even longer time to
master, and therefore are not fit for use directly in an exhibition, but valuable
tools for us as artists, designers and technicians to help us create things that have
not existed before. In this line of work, we have learned that configuring space
via user participation and interaction is not easy, but crucial for an immersed
experience to take place. Throughout our work, developing, methods, tools and
processes we try to emphasize the importance of a multiperspective view of
space and its entities based on the idea to transcend merely scientific or artistic
approaches into a more comprehensive and immediate approach and working
practice, in which we try to use different forms of interaction and telepresence to
create a state of immersion. It is about symbolic objects and entireties (the issue
of “Gestalt”, not about mere construction and functions). To have the created
worlds to stand out as something believable (the issue of “representation”) and
how through thoughtfully designed user interaction can create an immersive
experience for the visitor.
2 Our Three Use Cases: Step in, Fly over and Cruise
The origins of the environments for this article: The used spaces are all derived
from the exhibition at the Biennale of Architecture in Venice 2016. Starting from
this basis, selected places (i.e. the shown worlds) were made accessible with new
media devices in order to explore how the spatial impression is changing as a
result of this new approach.
60 A. Siess et al.
Fig. 1. User’s perspective: (a) “step in” (b) “fly over” (c) “cruise” as WiP.
Fig. 2. Setup: (a) “step in” (b) “fly over” (c) “cruise”
2.1 Step in: The CAVE Installation
The installation1was shown from May to November as part of the Architectural
Biennale 2016 in Venice. The setup consisted of three projectors displaying a
seamless image onto a tryptic screen (total size: 3.60 ×6.0 m) (See: Fig. 2(a)).
The screen itself consisted of three canvases, each at an angle of 150◦to each
other. This CAVE-like installation—where also the user’s peripheral viewport
was covered—allowed each visitor to “dive” into the shown environments and
therefore enabled an immersive experience, and to create a bodily notion of that
featured space. On this projection screen, pre-rendered movies were shown. Due
to the fact that the camera movement was not bound to any (simulated) physical
or time constraints free movement in all dimensions was possible: The environ-
ments were shown from different perspectives, but always with a constant focal
length (35 mm). This approach was chosen in order to offer visitors a realistic
size estimation and a feeling of “being there” (dweller/pedestrian).
Findings: A: Size matters, being almost inside of a projection in a cave projec-
tion creates a presence of the spaces shown. B: Since every visitor individually
can approach the cave projection and also share the experience directly with
others. C: No technical barrier for the users to overcome. D: Immediateness: No
latency/lag and low-threshold for a majority of users.
1Created by ideal spaces working group 2016 (Matthias W¨olfel, Michael Johansson,
Daniel Hepperle, Andreas Siess, Ulrich Gehmann et al.).
Worldmaking: Designing for Audience Participation 61
2.2 Fly over: The Balloon Installation
For the ZKM—Center for Arts and Media in Karlsruhe, Germany we wanted to
set up an installation that is connected to the place where it is exhibited. Because
of its special kind of city planning (“fan city”) we already presented a version
of Karlsruhe in our CAVE Installation and therefore it was of interest for us to
make it accessible (here: Karlsruhe from 1834) from another perspective by rid-
ing over it via a balloon. In general spaced VR installations have some inherent
disadvantages for each individual visitor: The action space available to the user is
very limited and any danger of collision must be prevented in advance, e.g. by dis-
playing warnings (“chaperones”). How useful these interventions might be, they
strongly interrupt the immersive experience. Therefore, a concept is required that
plays creatively with the limitations of VR and simultaneously offers firm sup-
port in case of need. Hence we developed an installation that naturally and cred-
ibly shapes the playing space without being perceived as a limitation. We chose
to build a replica of a balloon-basket that is also mapped 1:1 in the virtual envi-
ronment (See: Figs. 1(b) and 2(b)). With this setup our installation features: 1.
birds-eye perspective for experiencing the special kind of Karlsruhe’s architecture
2. slow movements to reduce motion sickness 3. When wanting to ride/fly over a
city there currently is a rise in hardware that require users to strap themselves
onto it, which might be not perfectly suitable for (semi-) public space and also
for physically handicapped or introverted people (see [21]or[17]) and therefore
we thought a balloon might suit better. In addition, clinging to the balloons rail-
ing also reduces motion sickness and increases immersion [3]. To start the ride,
visitors had to enter the nacelle, which is also represented within the virtual envi-
ronment and pull down the rope (represented in physical and virtual space) to
start the ride/virtual burner (See: Fig. 2(b)). The ride took place on a pre-given
path with the possibility to change height by pulling down the rope to fuel the
burner. In the real environment, the burner was represented by spotlights which
radiated enough heat and was augmented with vibrations of the nacelle to foster
the illusion. Another trick to further improve the illusion of flying, was to add an
airstream using a fan. This, of course, is not correct from a physical point of view,
but fits well to most of the visitors (most of them who have not yet experienced a
real balloon ride).
Findings: A: One can increase immersion using elements that might not be
physically correct (wind in a balloon, vibration when pulling the trigger). B:
One can regulate viewing direction by implementing spatially located audio files.
C: Intentionally regulating the given space by using a physical restriction also
displayed in the virtual world (nacelle) helps people to orientate oneself within
the virtual reality. D: Although HMDs are becoming increasingly popular and
we added clear instructions next to the exhibition, museum guides still had to
attend the exhibition to help out the visitors.
62 A. Siess et al.
2.3 Cruise: The Bike Installation
The bike installation was shown to a broader public in an art/design per-
formance/exhibition in two cities (Kristianstad and Copenhagen). In this
art/design project with the bike we returned to Jeffrey Shaw’s original idea
from 1986, “Legible City” [18]. The Bike installation was one of five stations in
which the visitors could explore different ideas about urban development, here
in this installation we had the visitors to visit 15th century Milano to experience
a conceptual city space never realized. “As a space, [it] is constructed in such
a way that we always have the impression to see only a fragment, a more or
less small section of it—because at every meters x, the next structural element
can appear, ad infinitum” [5]. For this project we use a VirZoom bike with its
speed and direction sensors to explore two of the worlds from the 2016 exhibi-
tion. Here we want to see what happens when you take some worlds developed
for animation/film and translate that world into real time graphics (Unity3d).
What specific qualities get lost and which transfers well?
Findings: A: Direct immersion for the visitor when entering a virtual world
with a familiar navigational device such as the bike. They know how to direct
themselves almost immediately and there is a low threshold to learn how to nav-
igate oneself. B: Biking is not walking, therefore a space needs some adjustment:
this was a bit tricky since this world needs to be changed to fit navigation by
bicycle rather than by walking, for example have the bicycle go from one floor
to the next, using stairs could have been an option, but does not relate very
well to cruising—it will be a bumpy ride. We also wanted to keep the original
plan of the city and not introducing “modern” or “alien” elements into it. So to
keep the flow in constantly biking through Milano, we used ramps as discreet as
possible integrated in the original environment to move the visitor on it’s bicycle
in between the different levels of the city, and at the same time point the biking
experience for the visitor into new directions. C: The bike soon became an impor-
tant tool developing the worlds—iterating between modeling/texturing/lighting
and the experience of the changes made by bicycling in that area of attention.
D: It made us aware of the difference between seeing something on the screen
(editing) and experience it in full VR [19]. E: The bike itself is as we found out
also a camera rig and can be used for classical camera work to get camera data
out to other 3d programs but also to produce animation paths for Unity3d itself.
3 Thoughts on Audience Participation
One downside that comes with head mounted devices (HMD) in (semi-) public
spaces is, that they can only be used by one person at the time. In addition, while
experiencing the virtual world, the person who wears the HMD is almost com-
pletely isolated from his surrounding although other visitors can easily observe
him. This “voyeurism” behavior is quite similar to the one described as one of
the first parts of the audience funnel framework by Michelis and M¨uller 2011
[15]. Their framework is based on observations made in regards to interactions
Worldmaking: Designing for Audience Participation 63
with public displays, but while they argue that visitors either pass by the display
or view and react, we would like to add another category: The voyeur implicitly
does not want to experience the VR exhibition, but wants to passively observe
and find out what is happening. While in general, the term voyeurism has a
rather unpleasant notion, in this setting, this behaviour can lead to a honey-pot
effect, which then will bring other people to observe and interact with the current
user/voyeur or to try it by oneself. There are several possibilities one can come
up with to include bystanders into the installation (see Fig.3): 1. Directly dis-
play the virtual world onto a 2D display placed so that bystanders and voyeurs
can see it. 2. Include the actual user into the 3d world using a greenscreen as
described by Intel in 2017 [22]. 3. (Floor) projections [7,23]. 4. Project facial
expression on the front of the VR headset for bystanders to see it [11,13]. There
is also a lot to decide about on how to add auditory cues, but it would extend
the scope of this discussion2. In our current state, we are considering several
possibilities in engaging visitors to interact with the current user. For example:
should the visitor be able to see the virtual content on a 2D screen befo re he got
fully immersed into the virtual environment via HMD or should one try to only
make the content visible for participants after they went into the virtual reality
itself?
Fig. 3. Audience “funnel” for “Biketopia” VR exhibition.
4 Thoughts on Immersion
By using familiar or at least conventional devices, such as the CAVE, the balloon
and the bike, the users’ direct physical presence in the virtual environment rein-
2Kuutti 2014 discusses some of the different possibilities in (semi-) public spaces [10].
64 A. Siess et al.
forced the sense of physical presence of the virtual world and the interactivity
of the respective space or device used in our case studies enhance their sense of
immersion. We tried to lay the “focus on the activity, not the technology” and
made “the control mechanism obvious” [14] when designing the necessary tech-
nical part of our installations. Conversely, this means that immersion cannot be
designed directly, but that we have ensured that as many interfering factors as
possible are eliminated. As an artist/designer one should always be aware that
one never designs the actual immersed experience itself, but only the frame-
work wherein that experience can take place. Since the amount of immersion
is not easily measurable, we classify our three works by using the parameters
Slater and Wilbur came up with [20]. As one can see in Table 1the range of
sensor modalities (Extensiveness) used, varies. Also the idea on how visitors can
interact (Interactability) with the different installations is altered between the
exhibitions.
Table 1. Immersion in dependence on to the parameters defined in [20]
Cave Balloon Bike
Extensiveness Linear audio, video 3D spatial audio,
interactive visuals
(HMD)
3D spatial audio
interactive visuals
(HMD)
Matching None Vive tracking sensors Oculus rift sensors
Surroundness
(horizontal)
120◦110◦110◦
Vividness 30 fps
3600 ×1920 px
60 fps
2160 ×1200 px
30 fps
2160 ×1200 px
Interactability None Start, change height Start, change
direction, position,
velocity
Experience Multi person Single user
(observation possible)
Single user
(observation possible)
5 Thoughts on Interaction
On interaction we played safe and in each of our examples we deliberately used
well familiar forms of interaction by having the visitor, “step into”, “fly over” or
“cruise” our different environments. We wanted established forms that helped
the visitors more directly make the transition from the physical world on a device
they already used or at least seen before. Therefore interactivity and navigation
used in both the balloon and the bike was directly added to the experience
by a tight coupling between display, the movements of the user’s body (when
maneuvering the device) on which they made the transition from one world to
Worldmaking: Designing for Audience Participation 65
another. In the case of the CAVE example there were no interaction through any
technology, here we instead used the design of the physical space itself to have
the visitor just step into the worlds showed. Fundamentally, we can state that the
concept of affordance [6] from interaction design can also play to its strengths in
our context: Although it is ultimately just a sophisticated controller for a virtual
environment, the bicycle is clearly recognizable to every visitor. This simplicity
and accessibility effectively prevents fears of contact and takes little time to get
used to. The balloon can show these strengths even more effectively: The basket
restricts the possible degrees of freedom very effectively without actually being
perceived as a plain restriction or overruling. We believe that the museal context
demands this mixture of strict constraints and clear affordances to be successful.
As a designer it is our job to find concepts that integrate these principles into
a harmonious environment that plays creatively with these limitations and, at
best, uses them as framework for conceptual design.
6 Developing Visual Style and Aesthetics
Developing through iteration is common practice in many art/design disciplines
and also for our type of work in which we strive to facilitate an openness towards
what happens in-between the design cycles. Similar to the OODA-Loop [2]in
which one Observes, Orients, Decides and takes Action, in our case to challenge
the machine and software logic and our own limits and conventions, to produce
something that is both unexpected and valuable, and in the end will point out
possible new directions. So using this loop switching between the state of editing
(tweaking the parameters) and experience (being there) help us develop and put
forward qualities (that sometimes is produced by errors or wrongdoings) that
are hidden from the concept’s point of view and by doing so—have the concept
redeveloped itself through the results and experiences we achieve by iteration.
We try to explore what is being shadowed by the concept itself. Therefore we
do not formulate any detailed specification in relation to the concept before-
hand, or rather loosely “not photo real” or “do not use textures”. But at the
same time it is important for us to be able to control or at least understand
the in- and outputs of the development environment itself—from bits to spaces.
The shortcomings, errors and quirks are vital resources that suddenly can reveal
themselves as major feature with specific new qualities. When trying to follow the
design intention or concept, the materials/methods and prototypes themselves
and when we are “bending” [4] them, can produce qualities not known before-
hand. In our line of practice-based work for this to happen, the production needs
to reach a certain state of complexity, to be able for all of the possible parame-
ters to be explored thoroughly and have an impact on the details as well as the
whole. Therefore a production environment itself is crucial to facilitate practice
based research in the area of art and design, where one can game the rules of
play [9], otherwise there is a risk that art/design projects are just illustrations
of technology, and technology driven projects unreflected use conventions and
qualities from art. In the three projects we specifically address this in the visual
66 A. Siess et al.
styles we tried to develop. At present, one of the most important parameters to
benchmark an experience in VR is “realism”. But realism even technologically
advanced is still a cultural construction [12]. And realism in relation to digital
simulated ones, is not realism, it is reality seen by the camera lens, as Allen points
out “The intention of all technological systems developed since the beginning of
the 1950s has been towards reducing the spectators sense of their real world,
and replacing it with a fully believable artificial one” [1]. And even if realism
would be possible in a perfect replica, it is still a blunt showcase of technological
frameworks and not an independent aesthetic quality. If one really wanted to
depict physical realism in its original form, then a theoretically infinitely level
of detail would be necessary, which not only seems technically impossible, but
in particular also not desirable from an artistic perspective. So when designing
our environments in the first we have deliberately decided to work in favor of
this non realistic approach, to find a level that is believable rather then realistic.
For example we used the same light and shaders to develop a common style that
connected the seven disparate worlds from our first project with each other, to
find a reasonable level to avoid the uncanny valley—a mismatch or break in
presence where believability does not clash with falsehood [16]. And it is also
important to avoid that, “the virtual environment becomes less immersive since
they lose the interest and engagement of the user envision the reconciliation of
immersion and interactivity” [15]. During our iterative design process, a visual
aesthetic gradually emerged that portrayed the spatial situation in a very mini-
malistic, reduced yet consistent way. In the context of the “hunt for realism”, we
constructed just the opposite: a world that deviates completely from this ideal
and only shows details where they are indispensable. And that’s why we assume
these environments worked so well in our use case: because we found that con-
sistent and credible design is not tied to photorealism–although this hypothesis
needs empirical foundation in our future user tests.
7 Future Work
Currently we are actively working on Motopia—an environment designed by
Geoffrey Jellicoe in the 1960ies [8]—to enable this space for the VR-bike as well
as for a VR-car simulation. Since this conceptual environment was once meant to
be the city of motor-mobility we are interested if the 3D bike is able to tap these
potentials. For the Biketopia installation we are currently thinking about let-
ting the user take pictures from within the virtual environment and only present
those pictures to the audience (bystanders, voyeurs etc.). This could improve
conversation between the user and the bystanders in such a way, that they want
to ask more questions and maybe want to experience the world itself. In addi-
tion, the pictures might help the authors to get an idea of what the actual users
think might be the most interesting parts of their journey. We then can evaluate
this and improve our work based on this feedback. For the balloon flight we
implemented an Immersit Shaker System, which simulated the vibrations of a
balloon’s basket and ensured direct haptic feedback for pulling the drawstring.
Worldmaking: Designing for Audience Participation 67
We expect that the repetitive vibration that occurs when the (virtual) expan-
sion joints of the roadway are passed once again clearly emphasizes the spatial
impression as the underlying concept of Motopia as a modern era functional
city. In regards to our observations on audience participation, the next step will
be to monitor visitor-behaviour via top-down camera to generate heat-maps in
regards to where they are while they are in the balloon’s “sphere of activity” We
will continue to further investigate how space in different media is represented
via media technology and how it is used.
References
1. Allen, M.: Technology in Contemporary Hollywood Cinema. Taylor & Francis, New
York (2013)
2. Boyd, J.: Organic design for command and control (2005). https://www.
ausairpower.net/JRB/organic design.pdf. Accessed 15 July 2018
3. Carrozzino, M., Bergamasco, M.: Beyond virtual museums: experiencing immersive
virtual reality in real museums. J. Cult. Herit. 11, 452–458 (2010)
4. Eagleman, D., Brandt, A.: The Runaway Species: How Human Creativity Remakes
the World. Catapult, London (2017)
5. Gehmann, U.: Exhibition Venice 2016–ideal spaces (2016). https://www.
idealspaces.org/exhibition-venice-2016/#leonardodivinci. Accessed 29 Sept 2018
6. Hartson, R.: Cognitive, physical, sensory, and functional affordances in interaction
design. Behav. Inf. Technol. 22(5), 315–338 (2003)
7. Ishii, A., et al.: ReverseCAVE. In: ACM SIGGRAPH 2017. ACM Press (2017)
8. Jellicoe, G.: Motopia: A Study in the Evolution of Urban Landscape (1961)
9. Kajo, M., Johansson, M.: Common playground. In: Proceedings of Cast01 (2001)
10. Kuutti, J., Leiwo, J., Sepponen, R.E.: Local control of audio environment: a review
of methods and applications. Technologies 2(1), 31–53 (2014)
11. Kwatra, V., Frueh, C., Sud, A.: Headset “removal” for virtual and
mixed reality. https://ai.googleblog.com/2017/02/headset-removal-for-virtual-
and-mixed.html. Accessed 15 July 2018
12. Lister, M., Giddings, S., Dovey, J., Grant, I., Kelly, K.: New Media: A Critical
Introduction. Routledge, Abingdon (2010)
13. Mai, C., Rambold, L., Khamis, M.: TransparentHMD: revealing the HMD user’s
face to bystanders. In: Proceedings of the 16th International Conference on Mobile
and Ubiquitous Multimedia. MUM 2017, pp. 515–520. ACM (2017)
14. Maynes-Aminzade, D., Pausch, R., Seitz, S.: Techniques for interactive audience
participation. In: Proceedings of the 4th IEEE International Conference on Multi-
modal Interfaces, p. 15. IEEE Computer Society (2002)
15. Michelis, D., M¨uller, J.: The audience funnel: observations of gesture based inter-
action with multiple large displays in a city center. Int. J. HCI 27(6), 562–579
(2011)
16. Mori, M.: Bukimi no tani [the uncanny valley]. Energy 7, 33–35 (1970)
17. Revresh: Para parachute! http://revresh.com/paraparachute/. Accessed 15 July
2018
18. Shaw, J., Groeneveld, D.: Legible city. https://www.jeffreyshawcompendium.com/
portfolio/legible-city/. Accessed 15 Sept 2018
19. Sieß, A., H¨affner, N., W¨olfel, M.: Color preference differences between head
mounted displays and PC screens. IEEE (2018)
68 A. Siess et al.
20. Slater, M., Wilbur, S.: A framework for immersive virtual environments (FIVE):
speculations on the role of presence in virtual environments. Presence-Teleop. Virt.
Environ. 6(6), 603–616 (1997)
21. Somniacs: Birdly - the ultimate dream of flying. http://www.somniacs.co/.
Accessed 15 July 2018
22. Tyrrell, J., Bancroft, J., Gerald, M.: Sharing VR through green screen mixed reality
video (2017). https://software.intel.com/en-us/articles/sharing-vr-through-green-
screen-mixed-reality-video. Accessed 12 June 2018
23. Zenner, A., Kosmalla, F., Speicher, M., Daiber, F., Kr¨uger, A.: A projection-based
interface to involve semi-immersed users in substitutional realities. IEEE (2018)