Conference PaperPDF Available

Abstract and Figures

Digital design methods impact architectural structures. To contribute new impulses to Computational Design, we are focusing on a central obstacle in current digital design for architects and structural engineers , namely the suboptimal transformation of embodied three-dimensional imagination and knowledge of the designer into the digital work environment. Through a synthesis of findings from the fields of cognitive sciences, informatics and design sciences we propose a concept that deals with the question of how to integrate forms of embodied knowledge into digital design applications for architects and engineers. Using Virtual Reality (VR) and Mixed Reality (MR) technologies, as well as a full-body tracking system, we want to present a new way of modeling and perceiving digitally shaped models with this case study. A prototypical application makes it possible to experience the designers' body movements in virtual space. In comparison with digitally modeled objects differentiated insights about the users design decisions are gained. The use of the body to model space will show that (1) the advanced technologies of digital modeling can be used for a complex multi-sensory experience of space and the body itself, and (2) show how digital design applications can be enhanced by returning to an anthropomorphic scale. Our general approach is to identify exemplary ways out of a potential crisis in architecture and engineering related to the actual state of Computational Design, while expanding it into the field of Computational Creativity.
Content may be subject to copyright.
Proceedings of the IASS Symposium 2018
Creativity in Structural Design
July 16-20, 2018, MIT, Boston, USA
Caitlin Mueller, Sigrid Adriaenssens (eds.)
Embodied Creativity in Computational Design
Robert PATZ*, Jörg BRINKMANNa, Christoph GENGNAGEL*
* Department for Structural Design and Technology [KET], Berlin University of the Arts
Hardenbergstrasse 33, 10623 Berlin, Germany
robert-patz@udk-berlin.de
a Faculty of Art and Design, Department for Media Environments, Bauhaus-University Weimar
Abstract
Digital design methods impact architectural structures. To contribute new impulses to Computational
Design, we are focusing on a central obstacle in current digital design for architects and structural en-
gineers, namely the suboptimal transformation of embodied three-dimensional imagination and
knowledge of the designer into the digital work environment. Through a synthesis of findings from the
fields of cognitive sciences, informatics and design sciences we propose a concept that deals with the
question of how to integrate forms of embodied knowledge into digital design applications for archi-
tects and engineers. Using Virtual Reality (VR) and Mixed Reality (MR) technologies, as well as a
full-body tracking system, we want to present a new way of modeling and perceiving digitally shaped
models with this case study. A prototypical application makes it possible to experience the designers'
body movements in virtual space. In comparison with digitally modeled objects differentiated insights
about the users design decisions are gained. The use of the body to model space will show that (1) the
advanced technologies of digital modeling can be used for a complex multi-sensory experience of
space and the body itself, and (2) show how digital design applications can be enhanced by returning
to an anthropomorphic scale. Our general approach is to identify exemplary ways out of a potential
crisis in architecture and engineering related to the actual state of Computational Design, while ex-
panding it into the field of Computational Creativity.
Keywords: Computer Aided Architectural Design, conceptual design, Computational Creativity, Computational Design, form
finding, Immersive Virtual Environments, Virtual Reality, Virtual Reality Aided Architectural Designs
1. Introduction
Technologies such as Computational Design and Building Information Modeling (BIM) have revolu-
tionized the design processes of engineers and architects. Today, building structures are to a large
extent completely digitally modeled and their physical behavior is simulated. Design decisions are
made on the computer. But both Computational Design and BIM get criticism by professionals, espe-
cially from the side of practical architecture. Because a technologically characterized functionalism
seems to be the only possible result of computational methods, and the technical optimum has become
the general axiom, there is a risk that architects will turn away from digital technologies. A technical
optimum is not necessarily related to a strong architectural experience. The human body and its inter-
actions within space have lost relevance for the design of architectural shapes and spaces (see also
Carpo [1]). The human body as a starting point, however, seems to be particularly important, because
every physical building structure is ultimately experienced in a bodily and multi-sensorial way.
Since the end of the twentieth century, a field of research has opened up among cognitive scientists,
which also has great significance for the design of space (Robinson and Pallasmaa [2]). In short, Em-
bodied Cognition theorizes that cognitive processes, which we usually consider to be located in the
brain, can occur outside the brain as well. The body influences how the brain works more strongly
than the brain actually influences the body. For example: When we walk, the brain may decide where
our feet are intending to go, but the way our feet work is determined, among other things, by the shape
Copyright © 2018 by Robert Patz, Jörg Brinkmann, Christoph Gengnagel
Published by the International Association for Shell and Spatial Structures (IASS) with permission.
Proceedings of the IASS Symposium 2018
Creativity in Structural Design
and sensations of the legs and feet themselves. The way the brain solves the task of walking is always
predetermined or conditioned by one’s individual physique. If we consider the interaction, it makes
sense to regard the body as part of our decision-making apparatus. The mind is therefore localized not
only in the brain, as postulated by a biological reductionism, but much more complex, namely a physi-
cal multisensory integration of body, brain and environment.
When taking a closer look at modern workspaces, it quickly becomes clear that today’s design and
planning take place almost exclusively on the computer as a highly efficiency-driven process. The
computer is a tool that combines all the functions of design and planning procedures from the work of
engineers and architects. Just as, for example, the smartphone can satisfy the majority of people's in-
formation and communication needs, the digital design and planning tools enable the completion of
once spatially separate and temporally sequential activities from one place simultaneously. This deve-
lopment has consequences, such as the unlearning of cultural techniques, such as handwriting or ma-
nual sketching are increasingly losing their importance. Connected to this is a loss of thought patterns
and thinking abilities, as psychology documents (e.g. James and Engelhardt [3] or Medwell and
Wray [4]). To put it bluntly, we seem to surrender to a form of immobility. We design and construct an
environment that we may not want to experience. Brains positioned in motionless bodies direct the
world from the location of their computer interfaces––what would be the pessimistic outlook.
With regard to the history of art and architecture, it becomes possible to get a different level of under-
standing about how the positioning of designers to space has developed. Thus a de-subjectification of
the designed architectural space could be reconstructed from an original space of perception based on
a non-homogeneous experience of space in the ancient world to a geometric, infinite “system space”
since the Early Modern period. Considering the development of disegno, it is more comprehensible
how physical access to the world is less relevant for the arts of the seventeenth century. Architecture is
from now on visually designed and constructed with a series of consequences. Although the human
body has served as a size reference for architectural proportions since the Renaissance, it is increasin-
gly being used as a “stylistic metaphor.”
At the same time, architects themselves begin to assign a kind of third-person perspective that is
slightly elevated outside the room (Pasqualini [5]). From this position we design space and project the
user’s spatial experience as well as their response to the design. The description and understanding of
the world from the supervision of a two-dimensional map or plan goes hand in hand. Movement in
space is predominantly visualized as a vector, not as a temporal interior experience. The analysis of a
social, emotional, or sensual performance appears to be imperfect in the simulation and analysis of
building structures (see also Turkle [6] or Gleiniger and Vrachliotis [7]). A variety of VR user studies
in architecture are similarly concerned with movement in space, which is a field we wish to remain
detached from in this particular case study.
2. Approach
The focus of our work is neither on the criticism of an artistic and scientific tradition, nor on arguing
for a loss of cultural techniques, but rather on an attempt to substantiate a hypothesis that new techno-
logies can be used to learn new cultural techniques, and can furthermore give meaning to the human
body. It is important to us to provide impulses to the field of Computational Design. Building on the
advanced tools of this field, VR and MR technologies are particularly promising for our purposes, as
they allow for the experience of Immersive Virtual Environments (IVEs). The interfaces respond to
users in such a way that they create the feeling of presence in a virtual environment (even the presence
of the environment itself), and at the same time enable the creation and processing of virtual objects.
It’s possible to virtually draw or model in 3D, and drawings or models can be spatially explored. For
specific intervention in the environment, body and hand movements are detected by the computer and
linked with commands.
The experience of one's own body combined with certainties about the proportions and distances of the
limbs, their position in space, or the relationship between one's body and other bodies these are all
measures of the spatial understanding of the environment and means of self-identification. In the work
of German architect Gottfried Semper (1803-1879), the idea of space emerges as a dress from an ana-
logy that Semper creates between the concepts of the “Wand” (wall) and the “Ge-Wand” (garment).
!2
Proceedings of the IASS Symposium 2018
Creativity in Structural Design
Two theoreticians, Heinrich Wölfflin (1853-1936) and August Schmarsow (1864-1945), expanded on
these reflections, and in the first half of the twentieth century, contributed to a transformation of the
ideal of the figurative representation of the human body as a stylistic metaphor for the correct propor-
tionality of architectural styles, towards an observant person immersed in space: Wölfflin with his
concept of "Raumgefühl" (sense of space) and Schmarsow with his idea of "Einfühlung" (empathy)
(Pasqualini [5]). One possible conclusion from considering these theories would be that we always
project parts of our body onto the environment to map ourselves and our environment in familiar phy-
siological terms (Stafford [8]). Considering the development of digital technologies, the question must
be: To what extent do we as computer users still rely on familiar physiological terms, or what is the
relevance of these terms in architectural design?
!
Figure 1: experience of the own body in the virtual environment
2.1. VR as a body experience
In this case study we describe a technological approach that aims to develop these physiological terms
even within architectural designs in virtual environments. Particularly in psychology, physical self-ex-
periences have been and are being investigated with the help of VR interfaces (see also Bohil et
al. [9]). In the field of trauma therapy, for example, dissociative bodily sensations and identifications
are a subject of investigation. For the field of design in architecture and engineering such methods
have rarely been used so far (e.g. Pasqualini [5]).
By using a marker-less full-body tracking system in combination with a VR headset it becomes possi-
ble to use the entire body to control the interface within an immersive virtual environment. The body is
captured from all sides by infrared video technology to record it and its movements. In the first step, a
virtual molding (shape shell) of the user’s body is created, which covers the entire surface of the body
like a “second skin.” Through bodily movements, the resulting form can be expanded in the next step.
If an arm is lifted, it pushes itself into the shape and tightens the form, whereby the original impression
is retained and enlarged. The more the person moves, the bigger the form becomes. In this way, the
user’s own movements can be recorded and experienced.
The user watches the changing shape shell in realtime from the inside, using their virtual reality head-
set. S/he perceives both the dimensions of their own body and the radius of the own limbs. The recor-
ding can be started and paused at any point, allowing it to leave the form inside the virtual environ-
!3
Proceedings of the IASS Symposium 2018
Creativity in Structural Design
ment. The resulting effect is an immediate understanding feedback of the radius or area of action of the
body. Additionally, the user is able to experience the back of their own body.
!
Figure 2: capturing body movements
2.2. One’s own body as a reference in one’s own design
Both the environment and the virtual form can be assigned to different visualization parameters, whe-
ther spatially contextless or contextualized. Without a spatial context, users are surrounded by a visual-
ly empty space, black, with no apparent horizon. The resulting body form is presented as an opaque,
white material, so that its entire extent is visible from every point. Contextualized as an alternative, a
concrete environment is displayed as a virtual model in a scale of 1:1. The virtual shape shell can be
viewed after the recording in immersive VR, as well as on the computer display. It can be spatially
contextualized during or after the process of forming, which means it can be placed in a virtual, archi-
tectural model, or presented as concrete design of the user’s own work. The key point is that the deve-
loped shape shell represents the body of the designing person. Our hypothesis is that the recognition of
the user’s own body leads to self-identification with the form.
The developed shape shell can be used as a permanently visible element in the design work represen-
ted as a CAD-Model. The three-dimensional form acts as a cognitively effective reference of a kinest-
hetic self-perception in the actual design. One’s own body is expanded into the virtual space while a
knowledge of the user’s own body is unfolded. The integration of perception processes as feelings of
self-identification, self-localization, embodiment, and action determine the experience of space (Pas-
qualini [5]). We assume that the projection of the user's own body, embodiment, and action through the
interface leads to a deeper understanding of the surrounding design.
If we understand tools as an extension of one’s own body and bodily functions, the virtual molding of
the user’s body and of its movements becomes a tool in itself (see also Schmitz and Groninger [10]).
The notion of presence of immersive environments and in immersive environments is extended to a
representational physical presence (see also Schuemie et al. [11]).
!
Figure 3: placing a three-dimensional body molding into a CAD-Model
!4
Proceedings of the IASS Symposium 2018
Creativity in Structural Design
!
Figure 4: Technical scheme
2.3. Implementation
To implement the case study, a computer system equipped with an AMD Ryzen 7 1700 8x 3.00GHz
processor, 32 gb of RAM and a 8GB MSI GeForce GTX 1070 Graphics card was used. For operation
in VR, an HTC VIVE VR headset was available. The device integrates acceleration sensors for measu-
ring the body and head position. The internal tracking detects movements of the full body while using
external infrared “lighthouses.” The exploratory behavior of users inside the virtual environment is
limited to a natural movement speed and individual’s eye level.
The full-body capturing was realized with a RealSense (by Intel) depth camera model D435. To get
access to streaming data, coming from RealSense, the RealSense SDK 2.0 (https://github.com /Intel-
RealSense/librealsense/releases/tag/v2.10.4) was used together with a Unity 3D Wrapper for RealSense
(https://github.com/IntelRealSense/librealsense/tree/development/wrappers/unity). The custom soft-
ware was built and written in C# in the cross-platform realtime render engine Unity 3D (by Unity
Technologies, https://unity3d.com).
!
Figure 5: General illustration of the interdisciplinary approach to digital design methods
3. Discussion and future refinements
With regard to the discipline of Interface Design, we understand the developed application as a sup-
plement to already existing spatial VR / MR drawing and modeling applications, such as Tilt Brush (by
!5
Proceedings of the IASS Symposium 2018
Creativity in Structural Design
Google), Oculus Medium (by Oculus VR), Gravity Sketch (by Gravity Sketch), or Mental Canvas (by
Mental Canvas). Aside from the obvious possibility to upscale the utility of our merged design tool
through increased computing power, we are planning to automate the workflow towards a plug-in for
the common design application Rhinoceros (by Robert McNeel & Associates) enabling the use by a
broad audience. The result is part of a set of planned tools that should be integrated into one applicati-
on respectively plugin, provided that they prove themselves worthy in the context of evaluations.
We are interested in both the cognitive mode of action and the creative use of the application as a de-
sign tool. In the next step, we plan to evaluate the resulting application. As part of a workshop, a co-
hort of art and architecture students will work with the application as subjects. The students will work
on concrete desings. The evaluation is carried out by means of a subsequent comparative survey. The
results should serve to improve the interface design, as well as to clarify the effectiveness of the appli-
cation itself. It will also provide insights into the effectiveness of different visualization parameters.
Within the framework of scientific and creative collaborations, we hope to use the resulting tool in
future for psychological studies, as well as in artistic-creative practice.
Acknowledgements
Funding was provided by the Department for Media Environments at the Faculty of Art and Design
(Prof. Ursula Damm) at Bauhaus-University Weimar, the DFG Research Training Group “Knowledge
in the Arts” and the Institute of Architecture and Urbanism (IAS) both at the Berlin University of the
Arts.
References
[1] M. Carpo, The Second Digital Turn: Design Beyond Intelligence, MIT Press, 2017.
[2] S. Robinson and J. Pallasmaa (eds.), Mind in Architecture: Neuroscience, Embodiment, and the
Future of Design, MIT Press, 2015.
[3] K. H. James and L. Engelhardt, “The effects of handwriting experience on functional brain deve-
lopment in pre-literate children.” Trends in neuroscience and education, vol. 1, pp. 32–42, 2012.
[4] J. Medwell and D. Wray, “Handwriting–A forgotten language skill.” Language and education, vol.
22, no. 1, pp. 34-47, 2008.
[5] I. Pasqualini, “Der architektonsiche Avatar: Multisensorische Aspekte in der Architektur,” In: Pro-
portionen und Wahrnehmung in Architektur und Städtebau, ed. by Andri Gerber, Tibor Joanelly,
and Oya Atalay Franck, Dietrich Reimer Verlag, 2017.
[6] S. Turkle, Simulation and its Discontents, MIT Press, 2009.
[7] A. Gleiniger and G. Vrachliotis, Simulation: Präsentationstechnik und Erkenntnisinstrument,
Birkhäuser, 2008.
[8] B. M. Stafford, “Neuronale Ästhetik: Auf dem Weg zu einer kognitiven Bildgeschichte,” In: Iconic
Turn, ed. by C. Maar and H. Burda, DuMont Verlag, 2004.
[9] C. J. Bohil, B. Alicea, and F. A. Biocca, “Virtual reality in neuroscience research and therapy,”
Nature reviews neuroscience, vol. 12, pp. 752, 2011.
[10] T. H. Schmitz and H. Groninger, Werkzeug – Denkzeug, Manuelle Intelligenz und Transmedialität
kreativer Prozesse, transcript, 2012.
[11] M. J. Schuemie, P. Van Der Straaten, M. Krijn, and C. A. P. G. Van Der Mast, “Research on pre-
sence in virtual reality: A survey.” CyberPsychology & Behavior 4, no. 2, pp. 183–201, 2001.
!6
... Technologies like Infrared Cameras and body capturing Systems enable the integration of the full designers body into an IVE (cf. Patz et al. 2018). Meanwhile Augmented Reality (AR) systems allow the transition of digital modelings into real space. ...
Chapter
Full-text available
Many of the sensory experiences of space, which were rather assigned to the realm of the visual, originate in other modes of perception and are integrated into the human understanding of its surroundings. Computer Aided Architectural Design tools meanwhile focus on a design work only under visual terms. With the help of Virtual Reality, among others, spatio-temporal and proprioceptive perceptual aspects can be integrated into these processes. This paper undertakes a rapprochement between visual arts, cultural theory, engineering and technology development. Design with all senses is here understood as an attempt to integrate embodied knowledge and bodily interaction into digital design applications for artists, architects and engineers.
... Technologies like Infrared Cameras and body capturing Systems enable the integration of the full designers body into an IVE (cf. Patz et al. 2018). Meanwhile Augmented Reality (AR) systems allow the transition of digital modelings into real space. ...
Conference Paper
Full-text available
Many of the sensory experiences of space, which were rather assigned to the realm of the visual , originate in other modes of perception and are integrated into the human understanding of its surroundings. Computer Aided Architectural Design tools meanwhile focus on a design work only under visual terms. With the help of Virtual Reality, among others, spatio-temporal and proprioceptive perceptual aspects can be integrated into these processes. This paper undertakes a rapprochement between visual arts, cultural theory, engineering and technology development. Design with all senses is here understood as an attempt to integrate embodied knowledge and bodily interaction into digital design applications for artists, architects and engineers.
Article
Full-text available
Virtual reality (VR) environments are increasingly being used by neuroscientists to simulate natural events and social interactions. VR creates interactive, multimodal sensory stimuli that offer unique advantages over other approaches to neuroscientific research and applications. VR's compatibility with imaging technologies such as functional MRI allows researchers to present multimodal stimuli with a high degree of ecological validity and control while recording changes in brain activity. Therapists, too, stand to gain from progress in VR technology, which provides a high degree of control over the therapeutic experience. Here we review the latest advances in VR technology and its applications in neuroscience research.
Article
Full-text available
Handwriting currently has a low status and profile in literacy education. This paper examines the situation of current handwriting pedagogy in England and considers why handwriting efficiency has been neglected. The paper goes on to identify a number of studies located in the domains of special needs and psychology which re-evaluate the role of handwriting efficiency. These studies suggest that handwriting is more than just motor skill and may make a very important contribution to children's composing of text. Existing research into the way handwriting efficiency affects composing suggests that further research, more appropriate assessment and focused intervention could all make a significant contribution to children's writing progress and might positively affect the progress of the many boys who struggle with writing throughout the primary school years.
Chapter
In "Proportions and Cognition in Architecture and Urban Design", practicing architects, historians, and theoreticians discuss the proportional systems that juxtapose aesthetic judgements, forms of practice, and human and social bodies with the norms and ideals that have resulted from these relationships. They retrace the history of these proportional systems, the expectations with which they were associated, how they were introduced into design, and how contemporary practice builds upon this tradition—or allows new interpretations to unfold.
Article
In an age of increasing technology, the possibility that typing on a keyboard will replace handwriting raises questions about the future usefulness of handwriting skills. Here we present evidence that brain activation during letter perception is influenced in different, important ways by previous handwriting of letters versus previous typing or tracing of those same letters. Preliterate, five-year old children printed, typed, or traced letters and shapes, then were shown images of these stimuli while undergoing functional MRI scanning. A previously documented ‘‘reading circuit’’ was recruited during letter perception only after handwriting—not after typing or tracing experience. These findings demonstrate that handwriting is important for the early recruitment in letter processing of brain regions known to underlie successful reading. Handwriting therefore may facilitate reading acquisition in young children.
Article
Virtual Reality (VR) is starting to be used in psychological therapy around the world. However, a thorough understanding of the reason why VR is effective and what effect it has on the human psyche is still missing. Most research on this subject is related to the concept of presence. This paper gives an up-to-date overview of research in this diverse field. It starts with the most prevailing definitions and theories on presence, most of which attribute special roles for the mental process of attention and for mental models of the virtual space. A review of the phenomena thought to be effected by presence shows that there is still a strong need for research on this subject because little conclusive evidence exists regarding the relationship between presence and phenoma such as emotional responses to virtual stimuli. An investigation shows there has been substantial research for developing methods for measuring presence and research regarding factors that contribute to presence. Knowledge of these contributing factors can play a vital role in development of new VR applications, but key knowledge elements in this area are still missing.
Simulation: Präsentationstechnik und Erkenntnisinstrument, Birkhäuser
  • A Gleiniger
  • G Vrachliotis
A. Gleiniger and G. Vrachliotis, Simulation: Präsentationstechnik und Erkenntnisinstrument, Birkhäuser, 2008.
Neuronale Ästhetik: Auf dem Weg zu einer kognitiven Bildgeschichte
  • B M Stafford
B. M. Stafford, "Neuronale Ästhetik: Auf dem Weg zu einer kognitiven Bildgeschichte," In: Iconic Turn, ed. by C. Maar and H. Burda, DuMont Verlag, 2004.