ChapterPDF Available

Abstract and Figures

Many of the sensory experiences of space, which were rather assigned to the realm of the visual, originate in other modes of perception and are integrated into the human understanding of its surroundings. Computer Aided Architectural Design tools meanwhile focus on a design work only under visual terms. With the help of Virtual Reality, among others, spatio-temporal and proprioceptive perceptual aspects can be integrated into these processes. This paper undertakes a rapprochement between visual arts, cultural theory, engineering and technology development. Design with all senses is here understood as an attempt to integrate embodied knowledge and bodily interaction into digital design applications for artists, architects and engineers.
Content may be subject to copyright.
Bodily Design Processes in Immersive Virtual
Robert Patz
, Monika Grzymala, and Christoph Gengnagel
Berlin University of the Arts, Hardenbergstraße 33, 10623 Berlin, Germany
Abstract. Many of the sensory experiences of space, which were rather
assigned to the realm of the visual, originate in other modes of perception and
are integrated into the human understanding of its surroundings. Computer
Aided Architectural Design tools meanwhile focus on a design work only under
visual terms. With the help of Virtual Reality, among others, spatio-temporal
and proprioceptive perceptual aspects can be integrated into these processes.
This paper undertakes a rapprochement between visual arts, cultural theory,
engineering and technology development. Design with all senses is here
understood as an attempt to integrate embodied knowledge and bodily inter-
action into digital design applications for artists, architects and engineers.
Keywords: Computer Aided Architectural Design Conceptual design
Computational creativity Computational design Form nding
Immersive virtual environments Virtual Reality
Virtual reality aided architectural designs
1 Introduction
In architecture today computer simulation and visualization are the dominant knowl-
edge techniques and design media. Here, geometry is the central resource of appro-
priating and projecting the world. Theorists like Vilém Flusser or Jean Baudrillard have
characterized and criticized the inuential power of simulation technologies (Flusser
et al. 1989; Baudrillard 2016). According to their perspectives, computational design is
simply there: designers remain motionless in front of their displays, working in a
disembodied fashion surrounded by a (geometrical) world of metaphors. The paradox
is that in practice designers hardly question these metaphors. In Computer Aided and
Computational Design the decoupling of bodily-multisensory sensations from design,
as well as its decoupling from motorically-conditioned thought processeswhich are
presumed to be a factor for example in the process of producing sketches, drawings, or
handmade modelscan be seen problematic because the objective of architectural
design processes is always a bodily experienceable physical space. The raising ques-
tions are: What is the impact of this decoupling on the understanding of the design
object? And conversely: At which point in the context of digital work do the designers
multimodal experiences of space ow into the design object?
To answer this, we are testing out a theory formation on an empirical basis. The
presented case studies are the result of artistic-scientic collaborations: using the digital
©Springer Nature Switzerland AG 2020
C. Gengnagel et al. (Eds.): DMSB 2019, Impact: Design With All Senses, pp. 350359, 2020.
medium of Virtual Reality (VR), the artist produced autonomous, immersive spatial
drawings, which can be understood as architectural spaces. For concrete interventions
into the environments, bodily and hand movements were captured by the computer and
linked with control commands what made it possible to draw three-dimensionally in
1:1 scale, and to experience the drawing scene spatiotemporal. Formally, the resulting
works operate independently of a geometrical conception of space. Also, the VR works
are a continuation of artists spatial drawings and installations in the past, which she
usually describes as Raumzeichnung(spatial drawing). Literally translated
Raumzeichnung means a space where a drawing is being created to dene an own
topological space trough gestures and lines.
Proceeding parallel to the artistic work reciprocal observations and inquiries,
between art and science, took place. Relationships between image and visual percep-
tion, virtual space, bodily interaction, and embodied cognition becoming evident in the
process. Depicted through a description by the artist of alternative conceptions of
spaces, which are at least to some extent alien to architecture is an in-homogenous
perceptual space within an Immersive Virtual Environment (IVE). Indications were
given that not just mental states are expressed in the body; bodily states inuence
mental states as well, bodily postures, voluntary movements, and spatial perceptions
having an impact on spatial cognition, and hence on design capacities. Remote from
purposive design activity in architecture, there become instances of irritation of the
perceptual apparatus and the discrepancies they create between digital and analog
environments experienceable and describable. The overall results of the study indicate
the path not only toward the further development of digital design tools but rather
toward a deeper understanding of the impact of digital design tools on architecture
production. Through the presented artistic works an understanding of the relations
between spatial intervention, that are not physical constraint, and physical architectonic
space become observable. The study exemplies that perceiving is itself a bodily
action, just like recognizing, thinking and designing.
2 Context
Alongside purely verbal description, the tools, media, and procedures of architectural
design consist of drawings, sketches, graphic perspectival constructions, digital ren-
derings, photographs and lms, physical and virtual models, calculations and simula-
tions. The knowledge production in these practices is not understood comprehensively
(cf. Hasenhütl 2013). However, where computation enters the picture, it can bring most
of these elements together. Computer interfaces not only make it possible to carry out
tasks that were formerly separated spatially and temporally, but also to offer compre-
hensive access to task-relevant knowledgethe context of each design. For this reason,
planning processes are undertaken today almost exclusively using digital resources.
Even where architects execute preliminary designs manually, they nonetheless go on to
produce digital architectural models on this basis, engaging in an iterative and for the
most part plural process of further development which allows a multiplicity of design
decisions to occur (cf. Hauser 2013). Clearly, the computer has developed into a
universal tool, one capable of combining most of the functions of the above-named
Bodily Design Processes in Immersive Virtual Environments 351
work processes (Uhlig 2012). Simulation has meanwhile evolved into a dominant
knowledge technique (cf. Gleiniger and Vrachliotis 2008). Both visualization as well as
simulation is joined in computation. Here, geometry is a central resource of appro-
priating the world (cf. Loukissas 2009).
2.1 Cultural Context
Among others the works of Elisabeth Ströker and Erwin Panofsky can facilitate a de-
subjectivization of designed architectonic space, to reconstruct an originary perceptual
space, i.e., on the basis of a non-homogenous special experience of antiquity, and its
evolution toward the geometrical and innite system-spacebeginning in the early
modern era. In considering the development of disegno, it becomes clear how the art of
the Baroque, for example, became deprived of a bodily access to the world. From this
point onward, and with a series of consequences, architecture was designed in exclu-
sively visual terms. In this development, designers abandoned the space they were
designing. From a third-person, divineperspective, they gaze downward on the
people who experience the space, and design these people along with it, with certain
political implications.
With the help of Maurice Merleau-Ponty and Erwin Straus, it becomes possible not
just to recapitulate the ways in which science, from idealism to constructivism, has lost
sight of the human body and its movement through space as being fundamentally
constitutive for consciousness. These thinkers as well help to construct a counter-
position. Within a point of view that still adheres to a dualism of body and spirit, bodily
memory and bodily knowledge are accorded signicance for a start by thinkers like
Roland Barthes and Jean-François Lyotard. With Gilbert Ryle and Michael Polanyi,
these categories become integral forms of knowledge, whichfor example for Hans-
Jörg Rheinberger and Richard Sennettinuence and condition processes of cognition
as well as design. And inherent to these processes are, for example, forms of attention,
ambiguities, fuzziness, uncertainty, the concealed implications of depictions, actions,
etc. These modes of thinking and creating are familiar in traditional design techniques,
like sketching, drawing, and sculpting. Computer Aided and Computational Design
tools lack these qualities. Results are even in form of iterations of evolutional processes
always exact in meaning of certainty. Depictions and actions are always explicit.
Through our work, we attempt to respond to this situation. To be sure, there is nothing
new in architecture about focusing attention on the human body; in the context of
digital design, however, this concern has remained relatively in the background up to
this point.
2.2 Technological Context
A verity of developments for involving an immersing the body in digital surroundings
started in the mid 90s. To give a very brief overview, the Responsive Workbench for
example is a multi user VR environment which allows two-handed direct manipulation
of virtual 3D objects on a table-like display (Agrawala et al. 1997; Cutler et al. 1997).
With planeDesign, a VR tool for the concept phase of the design was developed. By
moving surfaces, spatial situations are created and examined in the simplest way
352 R. Patz et al.
(Donath and Regenbrecht 1999). There were also suggestions for the integration of
sketching into VR: The piloting project VR Sketchpad for example is a freehand
drawing tool (Do 2001). In a simple way, walls, columns and furniture are generated
from diagrammatic sketches of spatial divisions. Another example is the hybrid
application immersive Drafted Virtual Reality (Dorta et al. 2008). The proposal inte-
grates a sketching function into an IVE too.
Through the breakthrough of VR as a mass medium we can nd a variety of
industrial applications today, such as Tilt Brush (by Google), Oculus Medium (by
Oculus VR), Gravity Sketch (by Gravity Sketch), or Mental Canvas (by Mental
Canvas), which allow either to draw or sculpt in VR or to interact with drawn contents
in 3D. StructVR for example is an IVE designed to engage and educate users via virtual
physical interaction with digital structural systems which display immediate visual
feedback on structural deformations, internal forces and reactions (Quinn et al. 2018).
Another more recent table-solution is Tangible Grasshopper, which integrates syn-
chronous interaction with real and virtual objects of e.g. urban development models
(Plotnikov et al. 2016). Technologies like Infrared Cameras and body capturing Sys-
tems enable the integration of the full designers body into an IVE (cf. Patz et al. 2018).
Meanwhile Augmented Reality (AR) systems allow the transition of digital modelings
into real space. One noteworthy and currently released application for engineers and
designers in this context is Fologram (by Fologram).Looking at all these technolo-
gies, however, one question remains unanswered: How do these technologies affect the
design of spatial structures?
3 Experiment
As an experienced draftswomen and sculptor the Berlin-based German-Polish artist
Monika Grzymala is familiar with the corporeal dimension of hand drawing and bodily
interacting in space. This without the use of preliminary drafts, her works emerge from
intuitive bodily impulses (Grzymala 2016). At times, she allows herself to be guided by
the resistant qualities e.g. of paper and tape as preferred materials. Her ability to
generate spatial structures has been deeply internalized. It is a question here of implicit
knowledge that has no need of guide lines or vanishing points. Enviable is Monikas
ability to reliably conceive of the entire space, which at times stretches across many
meters. Thought and action proceed in tandem (Grzymala 2015). While holding the
whole format in her head, she is able to retain and to link together a multiplicity of
settings in her minds eye. In the process, she entirely exhausts the manually accessible
space. Contrasts and a graphic density emerge by means of the differentiation of line
types and line intervals. Generated together with degrees of detail are effects of depth
and brightness.Would it then be possible to transfer the above-described method-
ological qualities of her drawing processes into Virtual Reality? (Fig. 1).
Bodily Design Processes in Immersive Virtual Environments 353
3.1 Implementation
To answer this question we tested out digital, immersive drawing tools that made
possible virtual spatial drawings by tracing of physical interaction. This technology
allows the computer to capture direct interventions in an IVE, which are accomplished
through body and hand movements that are linked to control commands. This allows
the user to draw three dimensionally and freely in space, and in a second step, to
experience the resultant drawings spatially. The idea was to deploy Virtual Reality
technology in order to spatialize the world of Monikas physical spatial drawings. The
results of this method would allow viewers to enter her drawings, to wander or even y
through them. A three-dimensional, spherical image generated in this way would give
viewers the sensation of standing at the center of one of her virtual landscapes.
To implement the case study, a computer system equipped with an AMD 8x
3.00 GHz processor, 32 GB of RAM and a 8 GB Nvidia GeForce GTX 1070 Graphics
card was used. For operation in VR, an HTC Vive VR headset was available. The
device integrates acceleration sensors for measuring the body and head position. The
room-scale tracking technology detects movements while using two external infrared
lighthouses,allowing users to move in 3D space and use motion-tracked handheld
controllers to interact with the environment. The exploratory behavior of users inside
Fig. 1. Spatial Drawings by Monika Grzymala in order of appearance: Raumzeichnung
Poyraz/Lodos,Arter Istanbul, 2015; Raumzeichnung XYZ,Summaria Lunn Gallery London,
2011; Raumzeichnung Liquid,Museum für Gegenwartskunst Siegen, 2015; Raumzeichnung
Bass,Galerie Crone Berlin, 2012 (courtesy of the artist).
354 R. Patz et al.
the IVE is limited to a natural movement speed and individuals eye level. As spatial
drawing solution the software Tilt Brush was used, which is a room-scale 3D VR
painting application by Google ( In this application users
are presented with a virtual palette from which they can select from a variety of brush
types and colors. Movement of the handheld controller in 3D space creates static or
animated lines and planes respectively brush strokes that follow in the IVE. For pre-
sentation four Oculus Go mobile VR Headsets were available.
3.2 Evaluation
The quadripartite work entitled Maze VR,which was exhibited for the rst time at
Galerie Crone in Berlin in November of 2018, serves to document that the above
mentioned transfer of methodological qualities of Monikas drawing process is pos-
sible. Its development extended across a period of about a year. One contributing factor
that dominates the whole production process is, that dissociated, suspended now in an
innite space of VR, she at rst experienced sensations of vertigo, as she described.
Produced at the same time was a large series of traditionaldrawings with ink on
handmade paper which Monika worked on in parallel through her experiences with
Virtual Reality, so to say to ground herself in the process of making despite the feeling
of an allover dizziness. Although it sounds paradoxical, the element of resistance
resided in the boundlessness of this environment, not in the material or the interface.
She was compelled to create a basis environment for operating, to establish her own
system of references, which could serve as a point of departure for drawing her virtual
structures. The result, once again, was the bold stratication of architectural inter-
ventions. The sense of movement, the oating vortex, emerged as though
What we know about space through our body and how we perceive it visually, are
two inseparable poles that belong together. The phenomenon of an imageis only a
part of visual perception. The image carrier (i.e., here, the boundless space in VR) and
the image/drawing become one, they create their own topological space. When com-
municating a spatial drawing or a multi-media installation as their creator, you dont
come around to consider the choreography of perception and how the viewer is going
to be guided or moved through this virtual landscape (cf. conversation btw. Wingham
and Grzymala in: Wingham 2013). Later in the process of experimenting with the new
medium this form of presentation of her art works Maze VR via Oculus Go deviated
from artists holistic approach to a spatial drawing, namely, the general idea of
experiencing the world not only in a visual way, but with the whole body. The Oculus
goggles separate the viewer from other physical surrounding. However, like experi-
enced before, in the artistic process of making the virtual drawing through the sensation
of vertigo, here too the body seemed somehow blindfolded, or rather disconnected
from the mind that actually perceives the virtual art work. Drawn in the space, the idea
of visceral, rhizomatic landscapes would interlace in all perceivable dimensions given
in the art work. With the beginning of a new collaboration at Berlin Open Lab at
Technical University of Berlin and Berlin University of the Arts the idea of working on
interwoven VR, AR and physical landscapes became part of the project concept
(Fig. 2).
Bodily Design Processes in Immersive Virtual Environments 355
The method of scientic work where consisting a participatory observation of all
production and reception processes by the authors. In addition, there was an ongoing
questioning with subsequent documentation of the dispute. The focus here was on the
evaluation of the technical conditions, which are expressed primarily through their
effect on the artists spatial work. It was also important to observe the retroactive effect
of digital representations into real space, in particular on its perception by the artist as
well as by the recipients of the work. It is clear that many of the sensory experiences of
space, which so far have rather assigned to the realm of the visual by the arts and
sciences, originate in other modes of perception and are integrated together into the
human understanding of the environment. Correspondingly, in design, the non-visual
modes of perception, like proprioceptive, haptic-tactile and auditory senses, associates
with interaction in space would have to be taken into account much more clearly.
Updated perceptual theories and design theories underline these so far less obvious
conditions of spatial perception (cf. Pallasmaa 2009,2012). With the help of the tool
VR spatio-temporal and proprioceptive perceptual aspects can be combined. In the
above-mentioned prospective step, spatial-auditory perception is to be integrated into
the design processes (Figs. 3,4,5and 6).
Fig. 2. The working artist.
Fig. 3. Maze VR Oneby Monika Grzymala (courtesy of the artist).
356 R. Patz et al.
Fig. 4. Maze VR Twoby Monika Grzymala (courtesy of the artist).
Fig. 5. Maze VR Threeby Monika Grzymala (courtesy of the artist).
Fig. 6. Maze VR Fourby Monika Grzymala (courtesy of the artist).
Bodily Design Processes in Immersive Virtual Environments 357
4 Conclusion
When we consider the results of the experiment, it is easy enough to imagine the effect
of technology on the body. In recapitulating this attempt, it becomes clear that a
continuous interplay between tool and perception is in play. Becoming accessible to
experience now was the interdependency between spatial sensations and the thought
processes that were conditioned by the motoric aspect. While mental states were
expressed through the body, bodily states also inuenced mental states. Bodily atti-
tudes, independent movements, and spatial sensations had an impact on the capacity to
generate a virtual landscape. It appeared as though perception itself was a form of
bodily action, exactly like recognition, thinking, and form-creation. The complexity of
the creative process manifested in this work also serves as a mirror of the complexity of
that which is unleashed through our creative activity, and of that which continuously
takes form around it.
The rapprochement between visual arts, cultural theory, engineering and technol-
ogy development also took place through the development of a common vocabulary.
Terms such as presence, atmosphere, affordability, responsiveness, and sensuality have
signicantly different meanings in the three elds, but they play an extraordinarily
important role in the material understanding of space, and thus in design practice.
Concrete artistic spatial works serve to illustrate the signicance of multimodal, bodily
perceptions of material and space. An intrusion of knowledge gained in the design
practice of the architecture is gained. At the same time the way to deeper theory
formation should be opened.
Acknowledgements. Funding was provided by the DFG Research Training Group Knowledge
in the Artsand the Institute of Architecture and Urbanism (IAS) both at the Berlin University of
the Arts.
Agrawala, M., et al.: The two-user responsive workbench: support for collaboration through
individual views of a shared space. In: Proceedings of the SIGGRAPH Conference (1997)
Baudrillard, J.: Symbolic Exchange and Death. SAGE (2016)
Cutler, L.D., et al.: Two-handed direct manipulation on the responsive workbench. In:
Proceedings of the Symposium on Interactive 3D Graphics (1997)
Dorta, T., et al.: The ideation gap: hybrid tools, design ow and practice. Des. Stud. 29(2), 121
141 (2008)
Do, E.Y.-L.: VR sketchpad. In: Computer Aided Architectural Design Futures, pp. 161172.
Springer (2001)
Flusser, V.: Vom Unterworfenen zum Entwerfer von Gewohntem. In: Intelligent Building,
Universität Karlsruhe (1989)
Gleiniger, A., Vrachliotis, G.: Simulation: Präsentationstechnik und Erkenntnisinstrument.
Birkhäuser, Basel (2008)
Grzymala, M.: The Making of Forming Something New. Galerie Crone, Berlin (2015)
Grzymala, M.: Drawing Spatially Raumzeichnung. Hatje Cantz, Berlin (2016)
358 R. Patz et al.
Hasenhütl, G.: Politik und Poetik des Entwerfens Kulturtechnik der Handzeichnung. LIT,
Münster (2013)
Hauser, S.: Verfahren des Überschreitens. In: Ammon, S., Froschauer, E.M. (eds.) Wissenschaft
Entwerfen. Vom forschenden Entwerfen zur Entwurfsforschung der Architektur. Wilhelm
Fink, Munich (2013)
Loukissas, Y.A.: Keepers of the geometry. In: Turkle, S. (ed.) Simulation and Its Discontents.
MIT Press, Cambridge (2009)
Pallasmaa, J.: The Thinking Hand. Wiley, Hoboken (2009)
Pallasmaa, J.: The Eyes of the Skin: Architecture and the Senses. Wiley, Hoboken (2012)
Patz, R., et al.: Embodied creativity in computational design. In: Mueller, C., Adriaenssens, S.
(eds.) Proceedings of the IASS Symposium, Boston (2018)
Plotnikov, B. et al.: Tangible Grasshopper: a method to combine physical models with
generative, parametric tools. In: Proceedings of the eCAADe (2016)
Quinn, G., et al.: StructVR virtual reality structures. In: Mueller, C., Adriaenssens, S. (eds.)
Proceedings of the IASS Symposium, Boston (2018)
Uhlig, F.: Robinsons pug. In: Schmitz, T.H., Groninger, H. (eds.) Werkzeug Denkzeug.
Manuelle Intelligenz und Transmedialität kreativer Prozesse. Transcript, Bielefeld (2012)
Wingham, I.: Mobility of the Line. Birkhäuser, Basel (2013)
Bodily Design Processes in Immersive Virtual Environments 359
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
This paper presents Virtual Reality Structures (StructVR), an immersive virtual environment designed to engage, educate and entertain users via virtual physical interaction with digital structural systems which display immediate visual feedback on structural deformations, internal forces and reactions. Interaction with pre-set demos as well as a creative sandbox mode provide an environment which can help users significantly to develop their understanding of and intuition for structural behaviour and theory. The implications of this prototype which combines the geometric strengths of Rhino / Grasshopper, the simulation speed and precision of dynamic relaxation solver Kangaroo as well as the rendering and design capabilities of Unity are far reaching. This paper presents a proof-of-concept prototype and discusses its technical realisation.
Conference Paper
Full-text available
Digital design methods impact architectural structures. To contribute new impulses to Computational Design, we are focusing on a central obstacle in current digital design for architects and structural engineers , namely the suboptimal transformation of embodied three-dimensional imagination and knowledge of the designer into the digital work environment. Through a synthesis of findings from the fields of cognitive sciences, informatics and design sciences we propose a concept that deals with the question of how to integrate forms of embodied knowledge into digital design applications for architects and engineers. Using Virtual Reality (VR) and Mixed Reality (MR) technologies, as well as a full-body tracking system, we want to present a new way of modeling and perceiving digitally shaped models with this case study. A prototypical application makes it possible to experience the designers' body movements in virtual space. In comparison with digitally modeled objects differentiated insights about the users design decisions are gained. The use of the body to model space will show that (1) the advanced technologies of digital modeling can be used for a complex multi-sensory experience of space and the body itself, and (2) show how digital design applications can be enhanced by returning to an anthropomorphic scale. Our general approach is to identify exemplary ways out of a potential crisis in architecture and engineering related to the actual state of Computational Design, while expanding it into the field of Computational Creativity.
Conference Paper
Full-text available
The use of digital tools in the early, creative design process is the focus of an interdisciplinary teaching and research project. Starting from the question of how a seamless connection between physical and digital tools could be made possible, the proposed method tries to bridge the gap between both methodologies and provide intuitive, visual and collaborative design coupled with advanced, real time computer simulations. A design platform has been developed which supports a seamless connection between freely shaped physical models, GIS data and Grasshopper3D. The environment combines the reconstructed physical models with the digital one (surrounding buildings) and passes the information to a custom Grasshopper3D plug-in which serves as a link to existing and custom developed simulative tools. All simulations are performed and visualized in real time to support the intuitive and iterative design process.
Full-text available
This paper describes VR Sketchpad, a pen-based computing environment for inputting and locating 3D objects in a virtual world. Designer can use the transparency layers to quickly trace and extract any image underlay from other application software. The 3D scene generation has three levels of complexity: simple extrusion of any drawn lines of shapes (i.e., straight or curved wall and column extrusion), solid modelling from a given geometric object representation (spheres, cones and boxes), and complex configuration with objects from graphics library (furniture layout).
Full-text available
Ideation is still done with traditional analog manual tools because current computer interfaces are inconsistent with the needs of designers. The Hybrid Ideation Space (HIS) was developed to respond to this lack by augmenting analog tools with digital capabilities respecting the designer's needs for uninterrupted reflective conversation with the representation that should, in turn, enrich ideation. To assess ideation, we have developed the notion of Design Flow, which considers the pattern of multiple dimensions involved in ideation. Design practitioners testing the HIS showed that ideation was well supported in synchronous, individual or team settings.
Conference Paper
We have built a system that allows users to naturally manipulate virtual 3D models with both hands on the Responsive Workbench, a tabletop VR device. Our design is largely based upon Guiard's observations of how humans distribute work between the two hands in the real world. We show how to apply these principles for the workbenchenvironment and describe many issues encountered during the design. We first develop a framework for two-handed interaction and then explore a variety of two-handed 3D tools and interactive techniques. Related issues include how constraints are implemented and controlled by the two hands and how transitions between one-handedandtwo-handed tasks occur seemlessly. Informal observations of the system in practice show that users can perform navigation and manipulation tasks easily and with little training using the two-handed environment. One of our interesting findings was that users often performed two-handed manipulations by combining two otherwise independent o...