Content uploaded by Robert Patz
Author content
All content in this area was uploaded by Robert Patz on Jul 31, 2019
Content may be subject to copyright.
Bodily Design Processes in Immersive Virtual Environments
Robert Patz1, Monika Grzymala and Christoph Gengnagel1
1Berlin University of the Arts, Hardenbergstraße 33, 10623 Berlin, Germany
robert-patz@udk-berlin.de
Abstract. Many of the sensory experiences of space, which were rather assigned to the realm of the vis-
ual, originate in other modes of perception and are integrated into the human understanding of its sur-
roundings. Computer Aided Architectural Design tools meanwhile focus on a design work only under
visual terms. With the help of Virtual Reality, among others, spatio-temporal and proprioceptive perceptu-
al aspects can be integrated into these processes. This paper undertakes a rapprochement between visual
arts, cultural theory, engineering and technology development. Design with all senses is here understood
as an attempt to integrate embodied knowledge and bodily interaction into digital design applications for
artists, architects and engineers.
Keywords: Computer Aided Architectural Design, conceptual design, Computational Creativity, Compu-
tational Design, form finding, Immersive Virtual Environments, Virtual Reality, Virtual Reality Aided
Architectural Designs
1 Introduction
In architecture today computer simulation and visualization are the dominant knowledge techniques and
design media. Here, geometry is the central resource of appropriating and projecting the world. Theorists
like Vilém Flusser or Jean Baudrillard have characterized and criticized the influential power of simula-
tion technologies (Flusser 1989, Baudrillard 2016). According to their perspectives, computational design
is simply there: designers remain motionless in front of their displays, working in a disembodied fashion
surrounded by a (geometrical) world of metaphors. The paradox is that in practice designers hardly ques-
tion these metaphors. In Computer Aided and Computational Design the decoupling of bodily-multisen-
sory sensations from design, as well as it's decoupling from motorically-conditioned thought processes—
which are presumed to be a factor for example in the process of producing sketches, drawings, or hand-
made models—can be seen problematic because the objective of architectural design processes is always
a bodily experienceable physical space. The raising questions are: What is the impact of this decoupling
on the understanding of the design object? And conversely: At which point in the context of digital work
do the designer’s multimodal experiences of space flow into the design object?
To answer this, we are testing out a theory formation on an empirical basis. The presented case
studies are the result of artistic-scientific collaborations: using the digital medium of Virtual Reality (VR),
the artist produced autonomous, immersive spatial drawings, which can be understood as architectural
spaces. For concrete interventions into the environments, bodily and hand movements were captured by
the computer and linked with control commands what made it possible to draw three-dimensionally in 1:1
scale, and to experience the drawing scene spatiotemporal. Formally, the resulting works operate inde-
pendently of a geometrical conception of space. Also, the VR works are a continuation of artist’s spatial
drawings and installations in the past, which she usually describes as “Raumzeichnung” (spatial drawing).
Literally translated Raumzeichnung means a space where a drawing is being created to define an own
topological space trough gestures and lines.
Proceeding parallel to the artistic work reciprocal observations and inquiries, between art and sci-
ence, took place. Relationships between image and visual perception, virtual space, bodily interaction,
and embodied cognition becoming evident in the process. Depicted through a description by the artist of
alternative conceptions of spaces, which are at least to some extent alien to architecture is an in-homoge-
nous perceptual space within an Immersive Virtual Environment (IVE). Indications were given that not
just mental states are expressed in the body; bodily states influence mental states as well, bodily postures,
voluntary movements, and spatial perceptions having an impact on spatial cognition, and hence on design
capacities. Remote from purposive design activity in architecture, there become instances of irritation of
the perceptual apparatus and the discrepancies they create between digital and ‘analog’ environments ex-
perienceable and describable. The overall results of the study indicate the path not only toward the further
development of digital design tools but rather toward a deeper understanding of the impact of digital de-
sign tools on architecture production. Through the presented artistic works an understanding of the rela-
tions between spatial intervention, that are not physical constraint, and physical architectonic space be-
come observable. The study exemplifies that perceiving is itself a bodily action, just like recognizing,
thinking and designing.
2 Context
Alongside purely verbal description, the tools, media, and procedures of architectural design consist of
drawings, sketches, graphic perspectival constructions, digital renderings, photographs and films, physical
and virtual models, calculations and simulations. The knowledge production in these practices is not un-
derstood comprehensively (cf. Hasenhütl 2013). However, where computation enters the picture, it can
bring most of these elements together. Computer interfaces not only make it possible to carry out tasks
that were formerly separated spatially and temporally, but also to offer comprehensive access to task-rele-
vant knowledge—the context of each design. For this reason, planning processes are undertaken today
almost exclusively using digital resources. Even where architects execute preliminary designs manually,
they nonetheless go on to produce digital architectural models on this basis, engaging in an iterative and
for the most part plural process of further development which allows a multiplicity of design decisions to
occur (cf. Hauser, 2013). Clearly, the computer has developed into a universal tool, one capable of com-
bining most of the functions of the above-named work processes (Uhlig 2012). Simulation has meanwhile
evolved into a dominant knowledge technique (cf. Gleininger and Vrachliotis 2008). Both visualization as
well as simulation is joined in computation. Here, geometry is a central resource of appropriating the
world (cf. Loukissas 2009).
2.1 Cultural context
Among others the works of Elisabeth Ströker and Erwin Panofsky can facilitate a de-subjectivization of
designed architectonic space, to reconstruct an originary perceptual space, i.e., on the basis of a non-ho-
mogenous special experience of antiquity, and its evolution toward the geometrical and infinite “system-
space” beginning in the early modern era. In considering the development of disegno, it becomes clear
how the art of the Baroque, for example, became deprived of a bodily access to the world. From this point
onward, and with a series of consequences, architecture was designed in exclusively visual terms. In this
development, designers abandoned the space they were designing. From a third-person, ‘divine’ perspec-
tive, they gaze downward on the people who experience the space, and design these people along with it,
with certain political implications.
With the help of Maurice Merleau-Ponty and Erwin Straus, it becomes possible not just to reca-
pitulate the ways in which science, from idealism to constructivism, has lost sight of the human body and
its movement through space as being fundamentally constitutive for consciousness. These thinkers as well
help to construct a counter-position. Within a point of view that still adheres to a dualism of body and
spirit, bodily memory and bodily knowledge are accorded significance for a start by thinkers like Roland
Barthes and Jean-François Lyotard. With Gilbert Ryle and Michael Polanyi, these categories become in-
tegral forms of knowledge, which—for example for Hans-Jörg Rheinberger and Richard Sennett—influ-
ence and condition processes of cognition as well as design. And inherent to these processes are, for ex-
ample, forms of attention, ambiguities, fuzziness, uncertainty, the concealed implications of depictions,
actions, etc. These modes of thinking and creating are familiar in traditional design techniques, like
sketching, drawing, and sculpting. Computer Aided and Computational Design tools lack these qualities.
Results are even in form of iterations of evolutional processes always exact in meaning of certainty. De-
pictions and actions are always explicit.—Through our work, we attempt to respond to this situation. To
be sure, there is nothing new in architecture about focusing attention on the human body; in the context of
digital design, however, this concern has remained relatively in the background up to this point.
2.1 Technological context
A verity of developments for involving an immersing the body in digital surroundings started in the mid
90s. To give a very brief overview, the Responsive Workbench for example is a multi user VR environ-
ment which allows two-handed direct manipulation of virtual 3D objects on a table-like display
(Agrawala et al. 1997, Cutler et al. 1997). With planeDesign, a VR tool for the concept phase of the de-
sign was developed. By moving surfaces, spatial situations are created and examined in the simplest way
(Donath and Regenbrecht 1999). There were also suggestions for the integration of sketching into VR:
The piloting project VR Sketchpad for example is a freehand drawing tool (Do 2001). In a simple way,
walls, columns and furniture are generated from diagrammatic sketches of spatial divisions. Another ex-
ample is the hybrid application immersive Drafted Virtual Reality (Dorta et al. 2008). The proposal inte-
grates a sketching function into an IVE too.
Through the breakthrough of VR as a mass medium we can find a variety of industrial ap-
plications today, such as Tilt Brush (by Google), Oculus Medium (by Oculus VR), Gravity Sketch (by
Gravity Sketch), or Mental Canvas (by Mental Canvas), which allow either to draw or sculpt in VR or to
interact with drawn contents in 3D. StructVR for example is an IVE designed to engage and educate users
via virtual physical interaction with digital structural systems which display immediate visual feedback on
structural deformations, internal forces and reactions (Quinn et al. 2018). Another more recent table-solu-
tion is Tangible Grasshopper, which integrates synchronous interaction with real and virtual objects of
e.g. urban development models (Plotnikov et al. 2016). Technologies like Infrared Cameras and body
capturing Systems enable the integration of the full designers body into an IVE (cf. Patz et al. 2018).
Meanwhile Augmented Reality (AR) systems allow the transition of digital modelings into real space.
One noteworthy and currently released application for engineers and designers in this context is Fologram
(by Fologram).—Looking at all these technologies, however, one question remains unanswered: How do
these technologies affect the design of spatial structures?
3 Experiment
As an experienced draftswomen and sculptor the Berlin-based German-Polish artist Monika Grzymala is
familiar with the corporeal dimension of hand drawing and bodily interacting in space. This without the
use of preliminary drafts, her works emerge from intuitive bodily impulses (Grzymala 2016). At times,
she allows herself to be guided by the resistant qualities e.g. of paper and tape as preferred materials. Her
ability to generate spatial structures has been deeply internalized. It is a question here of implicit knowl-
edge that has no need of guide lines or vanishing points. Enviable is Monika’s ability to reliably conceive
of the entire space, which at times stretches across many meters. Thought and action proceed in tandem
(Grzymala 2015). While holding the whole format in her head, she is able to retain and to link together a
multiplicity of settings in her mind’s eye. In the process, she entirely exhausts the manually accessible
space. Contrasts and a graphic density emerge by means of the differentiation of line types and line inter-
vals. Generated together with degrees of detail are effects of depth and brightness.—Would it then be
possible to transfer the above-described methodological qualities of her drawing processes into Virtual
Reality?
!
Figure 1: Spatial Drawings by Monika Grzymala in order of appearance: “Raumzeichnung Poyraz/Lo-
dos,” Arter Istanbul, 2015; “Raumzeichnung XYZ,” Summaria Lunn Gallery London, 2011; “Raumze-
ichnung Liquid,” Museum für Gegenwartskunst Siegen, 2015; “Raumzeichnung Bass,” Galerie Crone
Berlin, 2012 (courtesy of the artist).
3.1 Implementation
To answer this question we tested out digital, immersive drawing tools that made possible virtual spatial
drawings by tracing of physical interaction. This technology allows the computer to capture direct inter-
ventions in an IVE, which are accomplished through body and hand movements that are linked to control
commands. This allows the user to draw three dimensionally and freely in space, and in a second step, to
experience the resultant drawings spatially. The idea was to deploy Virtual Reality technology in order to
spatialize the world of Monika’s physical spatial drawings. The results of this method would allow view-
ers to enter her drawings, to wander or even fly through them. A three-dimensional, spherical image gen-
erated in this way would give viewers the sensation of standing at the center of one of her virtual land-
scapes.
To implement the case study, a computer system equipped with an AMD 8x 3.00 GHz processor,
32 GB of RAM and a 8 GB Nvidia GeForce GTX 1070 Graphics card was used. For operation in VR, an
HTC Vive VR headset was available. The device integrates acceleration sensors for measuring the body
and head position. The room-scale tracking technology detects movements while using two external in-
frared “lighthouses,” allowing users to move in 3D space and use motion-tracked handheld controllers to
interact with the environment. The exploratory behavior of users inside the IVE is limited to a natural
movement speed and individual’s eye level. As spatial drawing solution the software Tilt Brush was used,
which is a room-scale 3D VR painting application by Google (https://www.tiltbrush.com/). In this appli-
cation users are presented with a virtual palette from which they can select from a variety of brush types
and colors. Movement of the handheld controller in 3D space creates static or animated lines and planes
respectively brush strokes that follow in the IVE. For presentation four Oculus Go mobile VR Headsets
were available.
3.2 Evaluation
The quadripartite work entitled “Maze VR,” which was exhibited for the first time at Galerie Crone in
Berlin in November of 2018, serves to document that the above mentioned transfer of methodological
qualities of Monika’s drawing process is possible. Its development extended across a period of about a
year. One contributing factor that dominates the whole production process is, that dissociated, suspended
now in an infinite space of VR, she at first experienced sensations of vertigo, as she described. Produced
at the same time was a large series of ‘traditional’ drawings with ink on handmade paper which Monika
worked on in parallel through her experiences with Virtual Reality, so to say to ground herself in the
process of making despite the feeling of an allover dizziness. Although it sounds paradoxical, the element
of resistance resided in the boundlessness of this environment, not in the material or the interface. She
was compelled to create a basis environment for operating, to establish her own system of references,
which could serve as a point of departure for drawing her virtual structures. The result, once again, was
the bold stratification of architectural interventions. The sense of movement, the floating vortex, emerged
as though spontaneously.
What we know about space through our body and how we perceive it visually, are two insepara-
ble poles that belong together. The phenomenon of an ‘image’ is only a part of visual perception. The im-
age carrier (i.e., here, the boundless space in VR) and the image/drawing become one, they create their
own topological space. When communicating a spatial drawing or a multi-media installation as their cre-
ator, you don’t come around to consider the choreography of perception and how the viewer is going to be
guided or moved through this virtual landscape (cf. conversation btw. Ivana Wingham and Monika Grzy-
mala in: Wingham 2013). Later in the process of experimenting with the new medium this form of pre-
sentation of her art works Maze VR via Oculus Go deviated from artist’s holistic approach to a spatial
drawing, namely, the general idea of experiencing the world not only in a visual way, but with the whole
body. The Oculus goggles separate the viewer from other physical surrounding. However, like experi-
enced before, in the artistic process of making the virtual drawing through the sensation of vertigo, here
too the body seemed somehow blindfolded, or rather disconnected from the mind that actually perceives
the virtual art work. Drawn in the space, the idea of visceral, rhizomatic landscapes would interlace in all
perceivable dimensions given in the art work. With the beginning of a new collaboration at Berlin Open
Lab at Technical University of Berlin and Berlin University of the Arts the idea of working on interwoven
VR, AR and physical landscapes became part of the project concept.
!
Figure 2: The working artist.
The method of scientific work where consisting a participatory observation of all production and recep-
tion processes by the authors. In addition, there was an ongoing questioning with subsequent documenta-
tion of the dispute. The focus here was on the evaluation of the technical conditions, which are expressed
primarily through their effect on the artist’s spatial work. It was also important to observe the retroactive
effect of digital representations into real space, in particular on its perception by the artist as well as by
the recipients of the work. It is clear that many of the sensory experiences of space, which so far have
rather assigned to the realm of the visual by the arts and sciences, originate in other modes of perception
and are integrated together into the human understanding of the environment. Correspondingly, in design,
the non-visual modes of perception, like proprioceptive, haptic-tactile and auditory senses, associates with
interaction in space would have to be taken into account much more clearly. Updated perceptual theories
and design theories underline these so far less obvious conditions of spatial perception (cf. Pallasmaa
2009, Pallasmaa 2012). With the help of the tool VR spatio-temporal and proprioceptive perceptual as-
pects can be combined. In the above-mentioned prospective step, spatial-auditory perception is to be inte-
grated into the design processes.
!
Figure 3: “Maze VR One” by Monika Grzymala (courtesy of the artist).
!
Figure 4: “Maze VR Two” by Monika Grzymala (courtesy of the artist).
!
Figure 5: “Maze VR Three” by Monika Grzymala (courtesy of the artist).
!
Figure 6: “Maze VR Four” by Monika Grzymala (courtesy of the artist).
4 Conclusion
When we consider the results of the experiment, it is easy enough to imagine the effect of technology on
the body. In recapitulating this attempt, it becomes clear that a continuous interplay between tool and per-
ception is in play. Becoming accessible to experience now was the interdependency between spatial sen-
sations and the thought processes that were conditioned by the motoric aspect. While mental states were
expressed through the body, bodily states also influenced mental states. Bodily attitudes, independent
movements, and spatial sensations had an impact on the capacity to generate a virtual landscape. It ap-
peared as though perception itself was a form of bodily action, exactly like recognition, thinking, and
form-creation. The complexity of the creative process manifested in this work also serves as a mirror of
the complexity of that which is unleashed through our creative activity, and of that which continuously
takes form around it.
The rapprochement between visual arts, cultural theory, engineering and technology develop-
ment also took place through the development of a common vocabulary. Terms such as presence, at-
mosphere, affordability, responsiveness, and sensuality have significantly different meanings in the three
fields, but they play an extraordinarily important role in the material understanding of space, and thus in
design practice. Concrete artistic spatial works serve to illustrate the significance of multimodal, bodily
perceptions of material and space. An intrusion of knowledge gained in the design practice of the archi-
tecture is gained. At the same time the way to deeper theory formation should be opened.
Acknowledgements
Funding was provided by the DFG Research Training Group “Knowledge in the Arts” and the Institute of
Architecture and Urbanism (IAS) both at the Berlin University of the Arts.
References
1. Agrawala, M. et al., 1997: The Two-User Responsive Workbench. Support for Collaboration Through Individual
Views of a Shared Space, Proc. of the SIGGRAPH conference.
2. Baudrillard, J., 2016: Symbolic Exchange and Death. SAGE.
3. Cutler, L. D. et al., 1997: Two-Handed Direct Manipulation on the Responsive Workbench. Proc. of the Sympo-
sium on Interactive 3D Graphics.
4. Dorta, T. et al., 2008: The ideation gap. Hybrid tools, design flow and practice. Design Studies 29.2.
5. Do, E. Y.-L., 2001: VR sketchpad. In: Computer Aided Architectural Design Futures, Springer, pp. 161-172.
6. Flusser, V., 1989: Vom Unterworfenen zum Entwerfer von Gewohntem. In: Intelligent Building, Universität
Karlsruhe
7. Gleiniger, A.; Vrachliotis, G., 2008: Simulation: Präsentationstechnik und Erkenntnisinstrument. Basel:
Birkhäuser.
8. Grzymala, M.. 2015: The Making Of Forming Something New. Berlin: Galerie Crone.
9. Grzymala, M., 2016: Drawing Spatially – Raumzeichnung. Berlin: Hatje Cantz.
10. Hasenhütl, G., 2013: Politik und Poetik des Entwerfens – Kulturtechnik der Handzeichnung. Münster: LIT.
11. Hauser, S., 2013: Verfahren des Überschreitens. In: Ammon, S.; Froschauer, E. M. (eds.): Wissenschaft Entwer-
fen. Vom forschenden Entwerfen zur Entwurfsforschung der Architektur, Munich: Wilhelm Fink,
12. Loukissas, Y. A., 2009: Keepers of the Geometry. In: Turkle, S. (ed.): Simulation and Its Discontents, Cam-
bridge: MIT Press.
13. Pallasmaa, J., 2009: The Thinking Hand, Hoboken: Wiley.
14. Pallasmaa, J., 2012: The eyes of the skin: architecture and the senses, Hoboken: Wiley.
15. Patz, R. et al., 2018: Embodied Creativity in Computational Design. In: Mueller, C.; Adriaenssens, S. (eds.):
Proc. of the IASS Symposium, Boston.
16. Plotnikov, B. et al., 2016: Tangible Grasshopper. A method to combine physical models with generative, para-
metric tools. Proc. of the eCAADe.
17. Quinn, G. et al., 2018: StructVR Virtual Reality Structures. In: Mueller, C.; Adriaenssens, S. (eds.): Proc. of the
IASS Symposium, Boston.
18. Uhlig, F., 2012: Robinsons Pflug. In: Schmitz, T. H.; Groninger, H. (eds.): Werkzeug – Denkzeug. Manuelle
Intelligenz und Transmedialität kreativer Prozesse. Bielefeld: transcript.
19. Wingham, I., 2013: Mobility of the Line. Basel: Birkhäuser.
