Content uploaded by Mauricio Loyola
Author content
All content in this area was uploaded by Mauricio Loyola on Oct 23, 2017
Content may be subject to copyright.
Performative Materiality
A DrawBot for Materializing Kinetic Human-Machine Interaction in
Architectural Space
Mercedes Peralta1, Mauricio Loyola2
1,2Princeton University
1,2{mperalta|mloyola}@princeton.edu
This paper presents an exploration of movement as a design material to evidence
human-machine interaction in an architectural space. An autonomous robotic
vehicle with environmental sensory capabilities interacts kinetically with people
by recognizing their emotional states from their body postures. A drawing device
installed in the vehicle leaves a trace on the floor as a material testimony to the
mutual dynamics. The complex yet surprisingly intuitive choreographic
interaction of the machine and its social and physical environment blurs the
boundaries between drawing, machine, and performance. In general, the project
conceptualizes movement as a design material, drawing as a performative action,
and social interaction as a physical force, all of which can be enhanced or
mediated by digital technologies to produce results with aesthetic value.
Keywords: Human-Machine Interaction, Drawing Machine, Performance
Design
INTRODUCTION
Automatic drawing machines are mechanical devices
that produce drawings by moving pens or other
drawing instruments along predefined paths or pat-
terns. Their captivating appeal lies as much in the
aesthetic qualities of the intricate outcomes as in
the rhythmic and hypnotizing movements generated
by the drawing process. While traditional drawing
machines -such as the popular Spirograph toy- rely
solely on harmonic movements driven by physical
forces, contemporary drawing machines take full ad-
vantage of electronics and computer technologies to
expand the range of movements, driving forces, and
outcomes. Digital technologies allow drawing ma-
chines to be responsive and interactive with their sur-
roundings, transforming the drawing process from
being a predetermined fixed result to being a nego-
tiated action between the machine and its interlocu-
tors. As the interaction increases in complexity, the
boundaries between artifact and performance be-
come more blurred.
Inspired by the beauty of traditional draw-
ing machines, and motivated by the opportunities
of contemporary human-machine interaction tech-
nologies, this project starts with the aim of produc-
ing a machine that uses the complexity of human be-
havior and initial environmental conditions as driving
forces to produce drawings in architectural spaces.
DESIGN TOOLS - PROGRAMS - Volume 1 - eCAADe 35 |611
Particularly, we present an autonomous robotic ve-
hicle capable of interacting kinetically with its so-
cial and physical environment in a mutually condi-
tioned relationship. The vehicle, equipped with mul-
tiple sensing capabilities, can detect the physical and
geometrical characteristics of its surrounding space,
and also communicate with humans by recognizing
their emotional states through their body postures.
Based on these environmental and social conditions,
the vehicle moves autonomously but responsively,
leaving behind a mark on the floor produced with
a drawing tool. Thus, there is a dialectic relation-
ship between the vehicle and the human in which
the movement of one affects the other, condition-
ing each other in an unintuitive way that forces both
to explore space. While the movement is futile, the
trace remains permanently imprinted on the archi-
tectural space as an aesthetic vestige of the choreo-
graphic dialectic of the performative experience. This
project re-conceptualizes movement as a design ma-
terial, drawing as a performative action, and social in-
teraction as a physical force, that all may be enhanced
or mediated by digital technologies to produce out-
comes with aesthetic value.
BACKGROUND
Re-thinking Drawing Machines
Automated drawing machines most likely originated
in the fifteenth century, motivated by the revolution-
ary advancements in mathematics, geometry, me-
chanics, and arts of the Renaissance. Examples range
from the simple ellipsographs and pantographs, to
the lesser known helicographs (for volutes and spi-
rals), antigraphs (for mirrored drawings), and cy-
clographs (for arcs and circles). Conchoidographs
and cycloidotropes (for roulettes) -the antecessors
of the famous Spirograph toy- and harmonographs
and pendulographs (for harmonic oscillatory move-
ments) were more complex apparatuses developed
soon after (Garcia, 2013). In general, these devices
are based on stationary mechanisms that follow pre-
defined cyclic movements determined by physical
forces, such as harmonic pendulums. Their graphic
outcome is highly determined by the mechanical be-
havior with little space for interaction with the hu-
man operator. In fact, the relationship between the
machine and the operators is static and unidirec-
tional, determined solely by the expert’s control over
the device.
Contemporary digital technologies offer an at-
tractive opportunity to revisit these devices and ex-
plore other drawing driving forces that transcend
the merely mechanistic. For example, Newswanger
(2012), Howsare (2012), and Clarholm (2014) devel-
oped devices based on traditional drawing machines,
but taking advantage of electric motors and com-
puter numerical control (CNC) systems. A slightly
more radical innovation can be observed in the
hanging devices designed by Noble (2011) or By-
noe (2013), who utilized Arduino boards and the full
range of contemporary electronics to create devices
capable to draw with fine precision, but at the same
time, to interact closely with their users.
A substantial different approach was taken by
a number of artists who explored the potentials
of wheeled robotic devices as drawing machines.
The vehicles are typically built with two wheeled
servomotors controlled by Raspberry Pi or Arduino
boards and attached pens or brushes. Some of them
move following a predefined scripted pattern (Pig-
ford 2013) and others incorporate sensors to react to
external conditions (Adenauer & Hass 2011). To us,
this added capability -reacting to environmental con-
ditions including people- is a revolutionary break-
point that completely redefines the idea of draw-
ing. By transforming the drawing machine into a
responsive device, the value of the drawing lies no
longer only in the aesthetic attributes of the out-
come, but also in its representational meaning as
physical record of an experience of interaction be-
tween the machine and its environment.
Kinetic Human-Machine Interaction
The terms Human-Machine Interaction (HMI) and
Human-Computer Interaction (HCI), were coined to
describe the study of how people communicate and
612 |eCAADe 35 - DESIGN TOOLS - PROGRAMS - Volume 1
interact with machines and computers (Card et al.
1983). As an extension, kinetic human-machine inter-
action refers specifically to systems for which move-
ment is the language of communication. In recent
years, this approach has attracted the interest of ar-
chitects, designers, and artists, who see an appeal-
ing opportunity for exploring the relationship be-
tween body, movement, and space in creative and
expressive ways (Fogtmann et al. 2008, Jensen 2007,
Larssen et al. 2007, Levisohn 2011, Loke and Robert-
son 2009, Moen 2007).
Current technology allows for a wide range of ki-
netic interaction, both gestural and full-body. A com-
mon approach is using optical technologies that cap-
ture light data (video, infrared, etc.), which is then an-
alyzed for differences in time that may indicatemove -
ment. An archetypal example is the Kinect, a motion
detection device originally intended as a peripheral
for Microsoft Xbox 360, but which is widely used in
interactive installations (Caon et al. 2011, Kang et
al. 2011, Shiratuddin and Wong 2011, Cheng et al.
2012). A different approach is the use of position
sensors (accelerometer, gyroscopes, tilt sensors, etc.)
to capture spatial data which is then analyzed for
patterns that may indicate movement. Depending
on the level of precision and complexity required for
the kinetic interaction, this approach can involve in-
stalling devices on the subject’s body (Buechley and
Eisenberg 2008).
In these projects, “machine” is not necessarily
understood as an independent device that is sepa-
rated from the human body, but more as a signif-
icant part of it, even if physically detached or re-
motely pulled through the agency of both. There
is always a co-dependence between the two and a
sense of confusion about the boundary. A number
of art projects and installations exemplify this ten-
sion. The works of Rebecca Horn -such as Unicorn
(1970-72) and Finger Gloves (1972)- could be exam-
ples of how body extensions and the environment
relate to each other through movement, equilibrium
and gravity. Michael Heizer with Circular Surface Pla-
nar Displacement (1969), instead, uses a motorcycle
to produce large scale modifications on the surface
of the desert. The human-machine interaction is still
present when looking at the traces in his series of
perfect-looking circles drawings: there is an embod-
ied mastered knowledge on the speed and traction
of the motorcycle and its turning radius through the
use of his own anatomy in order to achieve such con-
ditioned results.
METHODS
The implementation of the project can be under-
stood as the crossing between eighteenth cen-
tury drawing mechanical apparatuses based on har-
monic movements (Garcia 2013), Braitenberg’s au-
tonomous vehicles (1986), and wearable technolo-
gies for human-computer interaction based on ges-
tural detection (Hartman 2014).
The vehicle was built based on a standard kit
of a wheeled Arduino robot with two continuous-
rotation servos and a pivot ball. The basic move-
ments of the vehicle (forward, backward, and ro-
tation) were controlled through the relative speed
and rotational direction of the servos, previously cal-
ibrated to match microsecond values to actual RPMs.
Four environmental sensors were installed on the ve-
hicle: a photo-resistor on top, an infrared tracking
sensor on the bottom, an ultrasonic distance sensor
on front, and a magnetometer at its center. Addi-
tional turn on-off buttons and signaling LEDs were
also incorporated to ease its operation. For detecting
and tracking the movement of the human performer,
we designed a simple wearable Arduino-based sys-
tem with three tilt sensors (hands and head), a mag-
netometer, and a pressure sensor under a foot. This
combination of sensors was proven to be sufficient
to detect a wide range of position configurations and
movement patterns (see Figure 1). To communicate
the vehicle and the human, two radio-frequency (RF)
modules were installed as transceivers. A mini XY joy-
stick was also added as a safety measure to remotely
drive the vehicle in case of need. Two servo motors
installed on the back side of the vehicle were used
to control the position, orientation, and pressure of
DESIGN TOOLS - PROGRAMS - Volume 1 - eCAADe 35 |613
(various) drawing tools, with a drawing precision of
about 1% for linear movements and 2% for rotary
movements.
The operation of the vehicle was determined by a
series of predefined movements that were triggered
by different combinations of values of the postural
sensors in the human performer and of environmen-
tal sensors in the vehicle. Using psychological the-
ories of body language, sixteen basic body postures
were associated with presumed emotional states and
consequent specific movements in the vehicle. For
example, an aggressive posture in the human trig-
gered a backward movement in the vehicle. The
movements were not totally determined by the per-
son, but altered by environmental conditions. For
example, ambient light could act to indicate direc-
tion of movement, or floor color could indicate safe
rest areas. Thus, the final movement of the vehicle
was the result of a set of combinatorial rules. For ex-
ample, if the human performer held an inactive pos-
ture for too long, the vehicle could change the move-
ment rules in order to stimulate a new dynamic be-
tween them: perhaps the vehicle could escape from
the light, forcing the human performer to approach,
lean, and cast a shadow to keep it still. Programming
the vehicle was straightforward, considering the rel-
ative simplicity of the movement rules. The code was
implemented as a constant loop of two steps: first,
collecting measurements from all sensors, and then,
a series of mutually exclusive if clauses covering the
full range of movements.
We performed a series of three progressive pre-
liminary tests to evaluate and develop the function-
ality of the system (see Figure 2). Once operational, a
final pilot performance was planned and conducted.
A human performer provided with the body sensing
system but no knowledge of the vehicle’s behavior
was asked to interact freely and spontaneously in a
400 sq.ft. area which included a bench, two spot-
lights, and a 100 sq.ft square of white powder on the
floor. A sponge brush was attached to the vehicle as
“drawing” device.
Figure 1
combination of
sensors
Figure 2
preliminary tests
614 |eCAADe 35 - DESIGN TOOLS - PROGRAMS - Volume 1
Figure 3
performer at work
RESULTS AND DISCUSSION
The final performance showed that the vehicle in-
teracted dynamically with the human performer in
an unpredictable, but not random manner. At least
three “assumed emotional states” (surprise, fatigue
and aggressiveness) were effectively activated dur-
ing the exercise. The performer was able to decipher
the relationship between light, shadow and move-
ment affecting the drawing machine. She managed
to use her own movement as a control variable to
guide the robot to move towards her and/or over
the powdered area. Thus, the final “drawing” corre-
sponds to the traces and footprints on the floor left by
both the vehicle and the performer, a proof of a mu-
tually choreographed correlation between the ma-
chine, the human, and the environment.
The maturity of the final iteration of the draw-
ing is evidenced by the accumulation of synergy be-
tween the two performers after a considerable period
of time. The drawing left on the floor is more than
just a consequence of the brush sweeping the white
powder back and forth. It speaks for an aesthetically
unexpected representation of the combined efforts
between the person, the vehicle, and the contained
space. The drawing becomes a testimony of the spe-
cific moods of the performers (human, machine), but
also of the architectural environment they occupy.
However, it is also not a simple choreographic nota-
tion that allows the reproduction of the performance
-in the literal sense of choreography as the written
form of a sequence of movements (as in dance, sport
or military)- but is meant to evoke a sense of the
mood of the performance. Jagged lines may indi-
cate a stressful interaction, or circles may be sign of
ecstatic happiness. The drawings convey not only
movement, but also leave room for emotional inter-
pretations.
During the exercise, a different material with a
colored pigment was randomly added to alter the
performance outcome through a less binary oppo-
sition between the ground black background/floor
and the white powder. Although this decision did
DESIGN TOOLS - PROGRAMS - Volume 1 - eCAADe 35 |615
not alter the behavior of the vehicle, it added a dif-
ferent component that sustains a more provocative
collection of results.
This project raises interesting conceptual ques-
tions that can be framed in a more general ongo-
ing contemporary debate around the extensive lim-
its and thresholds of design by/on the human and
non-human (Colomina & Wigley 2016). Maybe the
most evident disciplinary question is the redefini-
tion of drawings as a performative action in space
that challenges the traditional notions of represen-
tation. Drawings, in architecture, often mediate be-
tween what the philosopher Nelson Goodman cat-
egorizes as autographic -arts where the presence of
the author is fundamental to the realization of the
work, like painting- and allographic -arts where the
works can be reproduced by different people multi-
ple times without the presence of the author, like the-
ater plays- (Allen, 2000). Architecture drawings are, in
fact, far more speculative and puzzling tools than the
plain result which becomes visible through different
forms of media. In this project, the tension between
the autographic and allographic is explicit: the draw-
ing is the result of a complex interaction not only
between the machine and the performer, but also
between the vehicle’s programmed movements and
the uncontrollable environmental factors. It is impos-
sible to clearly define whether the trace is more con-
trolled by the human, the environment, or the ma-
chine.
The project questions the aesthetic limitations
and potentials embedded within the idea of (ma-
chine) control. Is the human performer who is guid-
ing the vehicle by casting shadows over it? Or is the
vehicle and the environment conditions (i.e. the po-
sition of the lights in the room) that are causing the
user to move in a different way in order to get the
vehicle underneath the shadow? At some times, it
seemed that the vehicle governed all interaction, in
the same way as a little boy crawling freely leads his
mother behind him. Indirectly, the vehicle forces the
human performer to interact actively with the envi-
ronment, and consequently, to be aware of specific
details of the architectural context: the direction of
the lights, the color of the floor, the level changes. In-
directly, the work proposes an alternative form of art
production which could position notions of author-
ship in a more environment sensitive paradigm.
The empathetic relationship developed be-
tween the performer and the machine -although
both entangled in a socially awkward form of
communication- was a remarkable discovery that
we initially had not expected. It was not the indi-
vidual human body or its psychological reactions
what could explain variations in the line traces, but
the person’s empathy towards the machine’s sensing
capacity. This was clearly manifested through the
coexistence of responsiveness and incomprehension
between the performer and the machine. This is a re-
markable clue which could be developed further in
order to make the drawing and the perception over
the experience much richer.
Finally, this work also brings up questions of how
to position the outcome with regards to architecture
and art as close fields. In our view, it could be read as
mediating between an art and architecture project,
since it discusses the possibility of describing the re-
sults only as an aesthetic product but also as an spa-
tial intervention affecting human behavior. Yet, we
argue that this idea of solely an ‘aesthetic product’
could be strong enough to become the driving force
of an architectural investigation. Nevertheless, the
system of values through which we could judge the
experience is not the same that an art critic or archi-
tecture critic would adopt.
Figure 4
during the process
616 |eCAADe 35 - DESIGN TOOLS - PROGRAMS - Volume 1
Figure 5
final drawing
CONCLUSIONS
In this paper, we explored the use movement as a
design material to evidence human-machine interac-
tion in an architectural space. By using a drawing ve-
hicle with environmental and social sensory capabili-
ties, we produced abstract drawings on the floor that
are a confirmation of how the immersive experience
of basic emotional reactions and programmed sens-
ing can transform a particular space, environment or
setting into a relevant aesthetic material.
Our initial goal of building a drawing vehicle was
successful in terms of functionality. Still, a more ro-
bust machine would be needed in further experi-
ences with extreme environmental conditions (e.g.
liquids or non-smooth surfaces), large-scale territo-
rial contexts, or socially diverse environments.
The project lead us to re-think our own notions of
materiality, which could also be intangible as we see
it merely as movement. Indeed, we designed a flex-
ible material with which to explore actuation, sens-
ing and interface. Although at first it may seem that
digital technologies play a central role in this type of
architectural projects, we consider their position as
essentially instrumental, only as a means to generate
or mediate new spatial experiences, whose greater
value relies in the phenomenological quality of the
interactive experience. I n this sense, the resultsof the
indexed performative, social and physical aspects of
DESIGN TOOLS - PROGRAMS - Volume 1 - eCAADe 35 |617
the aesthetic product end up being almost entirely
subjective.
REFERENCES
Adenauer, J and Haas, M 2011 ’Emerging Col-
orspace’, http://sonicedevelopment.com/emerging-
colorspace/
Braitenberg, V 1986, Vehicles: Experiments in synthetic
psychology, MIT press
Bynoe, D 2013 ’Four Cable Drawing Machine’,
http://dbynoe.blogspot.com/2013/12/four-cable-
drawing-machine.html
Caon, M, Yue, Y, Tscherrig, J, Mugellini, E and Khaled, OA
2011 ’Context-aware 3d gesture interaction based
on multiple kinects.’, Proceedings of The First Inter-
national Conference on Ambient Computing, Applica-
tions, Services and Technologies
Card, SK, Newell, A and Moran, TP 1983, The psychology of
human-computer interaction, Lawrence Erlbaum As-
sociates
Cheng, L, Q, Sung, Cong, Y and Zhao, S 2012 ’De-
sign and implementation of human-robot interac-
tive demonstration system based on Kinect.’, Control
and Decision Conference (CCDC)
Clarholm, J 2014 ’Circlon Machine’, https://jenslabs.com/2014/06/15/the-
circlon-machine-and-the-pen-test/
Fogtmann, MH, Fritsch, J and Kortbek, KJ 2008 ’Kines-
thetic interaction: revealing the bodily potential
in interaction design’, Proceedings of the 20th Aus-
tralasian Conference on Computer-Human Interac-
tion: Designing for Habitus and Habitat
Garcia, P 2013 ’A Brief History of Drawing Machines,
Since 1425’, http://video.mit.edu/watch/pablo-
garcia-a-brief-history-of-drawing-machines-since-
1425-26876/
Howsare, R 2012 ’Drawing Apparatus’, http://roberthowsare.com/rational-
aesthetics/drawing-apparatus/
Jensen, MV 2007 ’A physical approach to tangible in-
teraction design’, Proceedings of the 1st international
conference on Tangible and embedded interaction
Kang, J, Seo, DJ and Jung, DS 2011, ’A study on the con-
trol method of 3-dimensional space application us-
ing Kinect system’, International Journal of Computer
Science and Network Security, 11(9), pp. 55-59
Larssen, AT, Robertson, T and Edwards, J 2007 ’Experien-
tial bodily knowing as a design (sens)-ability in inter-
action design’, Proceedings of the 3rd European Con-
ference on Design and Semantics of Form and Move-
ment
Levisohn, AM 2011 ’Designing for movement experi-
ence’, CHI’11 Extended Abstracts on Human Factors in
Computing Systems
Loke, L and Robertson, T 2009 ’Design representations
of moving bodies for interactive, motion-sensing
spaces.’, International Journal of Human-Computer
Studies
Moen, J 2007 ’From hand-held to body-worn: embod-
ied experiences of the design and use of a wearable
movement-based interaction concept.’, Proceedings
of the 1st international conference on Tangible and
embedded interaction
Newswanger, C 2012 ’Guilloche Drawing Machine’,
http://www.resonancestudio.com/craigsblog/?cat=20
Noble, S 2011 ’Polargraph’, http://www.polargraph.co.uk/
Pigford, AJ 2013 ’DrawBot’, http://ashleyjohnpigford.com/
Shiratuddin, MF and Wong, KW 2011 ’Non-contact multi-
hand gestures interaction techniques for architec-
tural design in a virtual environment.’, International
Conference on Information Technology and Multime-
dia (ICIM)
618 |eCAADe 35 - DESIGN TOOLS - PROGRAMS - Volume 1
