Movement Matters: How a Robot Becomes Body

Abstract

This paper explores movement and its capacity for meaning-making and eliciting affect in human-robot interaction. Bringing together creative robotics, dance and machine learning, our research project develops a novel relational approach that harnesses dancers' movement expertise to design a non-anthropomorphic robot, its potential to move and capacity to learn. The project challenges the common assumption that robots need to appear human or animal-like to enable people to form connections with them. Our performative body-mapping (PBM) approach, in contrast, embraces the difference of machinic embodiment and places movement and its connection-making, knowledge-generating potential at the center of our social encounters. The paper discusses the first stage of the project, in which we collaborated with dancers to study how movement propels the becoming-body of a robot, and outlines our embodied approach to machine learning, grounded in the robot's performative capacity.

Full text

Movement Ma�ers: How a Robot Becomes Body
Petra Gemeinboeck
Creative Robotics Lab
National Institute for Experimental Arts
Faculty of Art and Design
University of New South Wales
Sydney, UNSW, Australia
petra@unsw.edu.au
Rob Saunders
Design Lab
Faculty of Architecture, Design and Planning
The University of Sydney
Sydney, NSW, Australia
rob.saunders@sydney.edu.au
ABSTRACT
This paper explores movement and its capacity for meaning-making
and eliciting aect in human-robot interaction. Bringing together
creative robotics, dance and machine learning, our research project
develops a novel relational approach that harnesses dancers’ move-
ment expertise to design a non-anthropomorphic robot, its potential
to move and capacity to learn. The project challenges the common
assumption that robots need to appear human or animal-like to
enable people to form connections with them. Our performative
body-mapping (PBM) approach, in contrast, embraces the dierence
of machinic embodiment and places movement and its connection-
making, knowledge-generating potential at the center of our social
encounters. The paper discusses the rst stage of the project, in
which we collaborated with dancers to study how movement pro-
pels the becoming-body of a robot, and outlines our embodied
approach to machine learning, grounded in the robot’s performa-
tive capacity.
CCS CONCEPTS
•Computer systems organization →Robotics
;
•Computing
methodologies →
Learning from demonstrations; Neural networks;
KEYWORDS
Dance, kinesthetic empathy, movement, non-anthropomorphic robots,
machine learning, social robotics
ACM Reference format:
Petra Gemeinboeck and Rob Saunders. 2017. Movement Matters: How a
Robot Becomes Body. In Proceedings of MOCO ’17, London, United Kingdom,
June 28-30, 2017, 8 pages.
https://doi.org/10.1145/3077981.3078035
1 INTRODUCTION
In recent years, we have seen robots entering our everyday lives, in
the form of complex toys, ‘assistants’ in therapy, eldercare and edu-
cation, and ‘companions’ that oer entertainment at home. Hence,
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specic permission
and/or a fee. Request permissions from permissions@acm.org.
MOCO ’17, June 28-30, 2017, London, United Kingdom
©
2017 Copyright held by the owner/author(s). Publication rights licensed to Associa-
tion for Computing Machinery.
ACM ISBN 978-1-4503-5209-3/17/06.. .$15.00
https://doi.org/10.1145/3077981.3078035
robots are increasingly presented as ‘social actors’, designed to
assist and entertain humans in social environments [
4
,
8
,
24
]. As
robots are assigned social roles, which already exist in our society,
their design usually aims to t these previously human social tasks.
The majority of research in Social Robotics and Human-Robot
Interaction (HRI) thus focuses on anthropomorphic (humanoid)
and zoomorphic robots [
4
,
24
]. The underlying assumption is that
human– or pet–like appearance and behaviour helps us to form
meaningful connections with them.
HRI studies, however, consistently show that the more human-
like a robot appears, the more people expect it to also have human-
level cognitive and social capabilities, so that interacting with these
apparently humanlike machines often is frustrating and disappoint-
ing [
4
]. From a posthumanist viewpoint, this ambition to build
mechanical servants and companions in our own image promotes
not only humans making connections with machines but also elim-
inating human/machine dierence [
29
]. It could be argued that
robots mimicking humans or pets, often in cute, caricatured ways,
deliberately blur the dierence between organic and mechanical
bodies, and human and machine cognition, to elicit human invest-
ment based on supercial and often false social cues. Designs that
don’t rely on the familiarity of existing bodies, on the other hand,
allow for human-machine encounters that are not restricted by
“preconceptions, expectations or anthropomorphic projections ...
before any interactions have occurred” [4].
In this paper, we explore an alternative approach to robot design
and its capacities to learn and elicit responses, which aims to shift
the focus from representational qualities to the performativity of
human-machine congurations (see [
1
]). We believe that move-
ment and its connection-making, relational potential is key to the
becoming-body (bodying) of a robot and its capacity to relate to
other bodies and the world. Alluding to the generative capacity of
movement, according to Erin Manning, movement is bodying or
becoming-body, rather than “something the body does” [
17
]. Our
proposition is that it is movement from which the robot’s body,
with all its aective, intelligible qualities, emerges, rather than its
physical appearance. This approach opens up a much wider range
of possible robot morphologies and behaviours, based on machinic
forms of embodiment that don’t rely on mimicking familiar bodies.
Bringing together creative robotics, dance and machine learn-
ing, the project’s enactive approach harnesses dancers’ movement
expertise to design the robot’s non-anthropomorphic body, its po-
tential to move and capacity to learn. Our aim for working with
choreographers and dancers is not to render the robot more human
but rather to investigate how sociomaterial relations get produced
and activated and for the robot to learn about these bodily relations
Note: This is a final draft.
This paper has been published in the Proceedings of the 4th International Conference on Movement Computing
(MOCO '17), 28-30 June 2017, Goldsmiths, London. New York, NY: ACM Press

MOCO ’17, June 28-30, 2017, London, United Kingdom Petra Gemeinboeck and Rob Saunders
and movement qualities. Rather than understanding the robot as
a mechanical artefact, which requires to be implanted with social
qualities, this approach enacts the robot as a sociomaterial phenom-
enon by placing movement at the centre of the encounter.
2 PERFORMATIVE BODY MAPPING
This section discusses our core methodology, called Performative
Body Mapping (PBM), which harnesses dancers’ movement ex-
pertise to shape a robot and its ways of learning to move and
interact with the world. At the core of PBM is the development
of an autonomous robot with an abstract, non-organic form and
a capacity to learn how to move in ways that are unique to its
own machinic body, while ‘sensitive’ to the subtle movement quali-
ties it acquires from human dancers. The performative approach
comprises four stages: bodying, grounding, imitation, and impro-
visation. The project is still in progress, and in this paper we take
a closer look at the rst stage, the becoming-body of a robot: how
abstract, non-organic forms, when moving, take on a presence and
can elicit interesting responses. This will follow an outline of the
remaining three stages, which constitute our embodied approach
to machine learning.
2.1 Entangling Dancer and Machinic Form
The rst stage is concerned with the challenge to develop a robot’s
physical form in tandem with its movement capabilities. Commonly,
it is the robot’s functionality or social role that shape its physical
form, which inevitably manifests a number of assumptions about
its ways of moving and bodily relating to the world. All too often,
the robot’s physical form is reduced to a mobile container, allowing
its computational mind to process and interact with the world [
34
].
To avoid beginning with such an impoverished set of Cartesian as-
sumptions, form and movement (and learning, as we will see later)
need to be developed in concert. More so, based on our premise
that it is movement that ‘bodies’ the robot, rather than its physical
appearance, in this prototyping process, the form becomes a vehicle
to trace and study its movements. Our embodied process thus at-
tempts to resolve the ‘chicken-and-egg’ problem of not predening
the robot’s form, while still having ‘something’ to nd movements
with and learn from.
To ‘nd’ a robot’s movements and iteratively rene the robot’s
form, PBM involves a machine ‘costume’ or ‘prosthesis’, which is
inhabited and activated by a dancer. It is a wearable object, which
extends the dancer’s body and stands in for a potential machine
body, that is, the machine to be ‘bodied’. As discussed in the fol-
lowing sections, we experimented with a wide range of costumes/
prostheses, whose shape changed and evolved in response to what
kind of movements and bodily relations the dancer could activate.
The costume/prosthesis becomes thus the instrument for mapping
between the dierent embodiments of the human dancer and the
becoming-robot and, with it, their dierent movement capacities. It
allows (1) for the dancer to ‘feel into’ the machine’s form, and learn
to embody and move with it, and (2) for the robot to learn from
the dancer by imitating the recorded movements from the dancer,
disguised to mirror the robot’s embodiment.
From a technical viewpoint, the costume is a full-size, non-
mechanical prototype of a robot design in process. Involving the
bodily imagination [
7
] and kinesthetic empathy [
21
] of a choreogra-
pher and a dancer, however, it becomes an instrument for mapping
between two very dierent embodiments, and for the dancer to skil-
fully tune into this strange object to explore how it becomes-body
in movement. The dancer’s movements, in turn, are co-shaped by
the material forces and aordances of the machine costume, so that
its distinct movement qualities emerge from a material interdepen-
dence between the two. The use of costumes to co-shape dancers’
movements is not new. Oskar Schlemmer has designed geometric
costumes for his dramaturgical concept for Bauhaustänze [
2
], and
for his 1993 production of Tristan and Isolde, Heiner Müller asked
Yohji Yamamoto to design costumes for the singers “that would
impede on the movement they are used to” [
30
]. In PBM, we are
seeking a productive entanglement of the material potentials of
dancer and object, rather than impeding the dancer’s movement.
2.2 Materials, Forms and Forces
The rst research stage unfolded along 16 full-day workshops with
one choreographer and one or two dancers, and a number of ses-
sions, in which we reected on and evaluated the workshop results
with the movement experts. The rst seven workshops focused
on exploring and challenging our assumptions and preconceptions
with regards to possible machinic forms and movements. The sec-
ond series of workshops was concerned with converging our nd-
ings from the previous workshop days to develop extended pro-
totype costumes that would allow us to capture their movement.
While still exploratory and open, these later workshops took on a
more structured, systematic approach towards developing and nd-
ing movement ‘identities’ with selected forms, and the costumes/
prostheses’ movements were continuously recorded.
It is also worth noting here that we currently don’t have a spe-
cic social task in mind, which the robot should full, given that
any known social roles already bring with them a set of ‘do’s and
don’ts’. Rather, we explore how far we can push the relationship
between abstract, simple morphologies and their potential to elicit
empathic and aective responses (see Section 3). Particularly in the
early stages of our project, this open, exploratory approach allowed
us to experiment with a wide range of possible forms, materials,
movements, and dramaturgical scenarios without the constraint of
the robot design needing to full a specic purpose.
To experiment with and activate the machine costumes/prostheses,
we collaborate with dancers from the De Quincey Co. and its artistic
director and choreographer Tess de Quincey. De Quincey Co. [
3
]
trains in BodyWeather, a practice founded on Butoh dance, which
draws from both eastern and western dance, sports training, mar-
tial arts and theatre practice. BodyWeather practitioners are well
attuned to the challenging task of bodily thinking through ‘other’
body-forms. In Tess de Quincey’s words, “the whole point about
BodyWeather is to go beyond the biomechanics through images,
[that is] we recruit the biomechanics to nd ways to move, which
are not normally positioned as human movements” [6].
At the start of the project, we asked the dancers to inhabit a
wide range of materials, shapes and objects to narrow the scope
of possible paths. This included ltering out materials and forms
that, when activated, (1) either relied too much on the dancer’s own
morphology or whose structure and movements were so complex

Movement Ma�ers: How a Robot Becomes Body MOCO ’17, June 28-30, 2017, London, United Kingdom
that they were likely to be perceived as a spectacle, and (2) proved
dicult to successfully reconstruct as autonomously moving ma-
chines. It is worth noting here that we are not aiming to create a
machine spectacle, where people’s attention is focused on how the
machine looks like or what it ‘can do’. Similar to the issue of giving
lifelike characteristics to the robot, this would distract from our
aim to better understand how movement can produce and activate
connections and sensations that visual appearance alone cannot
(also see Section 3). Hence, our objective is to foreground movement
and how it ‘bodies’ [
17
], while eluding temptations to make analo-
gies to known or living ‘things’. The non-organic morphologies we
experimented with were simple abstract forms, similar to a blank
canvas and fullling our criteria of not having an obvious a front
or back, head or face, or limb-like structures. Another important
enabling constraint for the costume was that it can be reconstructed
as a mechanical prototype capable of moving on its own and able
to imitate the dancers’ movements (see 2.4).
Figure 1: Soft, textile costume, inhabited by Tess De Quincey.
©Petra Gemeinboeck
In the rst workshops we experimented with soft, textile struc-
tures, inhabited by the dancer, and surfaces with breglass ribs
to form architectural, parabolic shapes when bent, twisted and
pulled by the dancers. The relatively soft shapes, as shown in Fig. 1,
however, were too reliant on the dancer to give them a contour.
The architecture-inspired, textile shapes, supported by elastic ribs,
produced interesting evolutions of geometric volumes but didn’t
allow for smaller, subtler movements.
In following workshops we experimented with simple geometric
forms and material structures that could be transformed through the
dancer’s movements. It quickly became clear that the simpler the
form, the more our focus shifted towards the kinesthetic experience
produced by this dancer-structure entanglement, without being
distracted by many potentially moving parts. In the following, we
briey explore two of the most interesting objects, that is, simple
and yet surprisingly expressive when activated.
2.2.1 Spiral Tube. The rst object that we worked with in this
series of workshops was a 190cm-high, 50cm diameter spiral tube,
coated with a strong nylon textile, which acted like a relatively sti
spring, standing upright on its own but compressible to a height of
only 30cm. First the dancer explored the object’s materiality, seeing,
probing and feeling what it can do and learning to negotiate its
structural integrity. This included learning to move with the force
provided by the structure, rather than moving the structure. Soon
Figure 2: Spiral tube costume, inhabited by Kirsten Packham.
©Petra Gemeinboeck
the dancer (inside) began to improvise with the object, exploring
dierent movement shapes and playing with tension, based on the
feedback she received from the choreographer and the object itself.
The helical structure, shown in Fig. 2, allowed for simultaneous
contractions and expansions along the vertical axis of the object,
as well as being bent as to produce multiple dierently articulated
planes pivoted along its core. Both, exible and responsive, the
structure enabled the dancers to generate subtle movements, like a
teeter or twitch, which, together with more sustained movement
trajectories, produced a rich, expressive performance.
Figure 3: Box costume, inhabited by Linda Luke, tilted onto
one edge. ©Petra Gemeinboeck
2.2.2 Cardboard Box. This experiment involved the perhaps
most obvious simple, abstract form, yet not the most apparent in
terms of its evocative capacity—a box. At rst we asked the dancers
to inhabit a 150x55x45cm cardboard box, as shown in Fig. 3. The
stibox shape got immediately interesting when it balanced pre-
cariously on an edge or was tipped onto one corner by the dancer
inside. Confronting our notions of weight and gravity through tilt-
ing, swaying and teetering allowed for the box to loose its stability
and, with it, its ‘boxiness’. This transformation or making strange
and how it can open-up an object for becoming-body, taking on
its own presence and actively relating to its environment, is at the
very core of our methodology. In later sessions we used a cube
shape, rather than a tall regular prism, to move further away from
humanoid proportions. We also challenged the cube’s ‘boxiness’ by
adding concertinaed openings (see 2.3).
2.2.3 (Broken) Tetrahedron. As shown in Fig. 4, the regular tetra-
hedron has a 1m triangular base and 2m long upright edges. We

MOCO ’17, June 28-30, 2017, London, United Kingdom Petra Gemeinboeck and Rob Saunders
Figure 4: Tetrahedron prosthesis, built with PVC pipes,
tightly strung together.
wanted to bring back the elastic forces, which the dancers could
play with inside the spiral tube, and built the pyramid shape with
PVC pipes, tightly strung together with elastic rope. Using pipes
compressed through elastic rope allows for the shape to arise from
a continuous network of tension. Importantly for us, this elastic
tension allows for the object to maintain its shape but also for the
edges to be twisted and the vertices—being joints—to have some
play. Initially we had planned to coat the structure to produce a
closed object, which the dancers could inhabit, however in our rst
experiment one connection along one of the up-right edges broke.
Now the tetrahedron had a fth joint (Fig. 5). Rather than repairing
the fault, we were fascinated to nd the multitude of shapes we
could produce only by moving the joint at the bottom of the broken
edge. While this unexpected, emergent complexity counters our
aim to focus on very simple forms, the simplicity of the kinetics
that produces these transformations opened up a new pathway for
our study (also see Section 3). In the following section we take a
closer look at the potential for the becoming-body of this object.
2.3 Becoming-Body
As stated earlier, the form of the costume/prosthesis is not xed but
only provides a starting point for the iterative design and becoming-
body process. Once initial studies, such as discussed in 2.2, produce
interesting or unexpected results, the object is opened up, in a
way, to be expanded and rened. This process is informed by our
material observations, in-depth conversations with the choreog-
rapher and dancers, and reections on our decisions as well as
serendipitous events. The latter, e.g., the breaking of a component,
which opens-up another degree of freedom, or a material behaving
in unanticipated ways, played an important role in ‘nding’ and
rening the robot’s form. Interestingly, the decisions we make usu-
ally draw some sort of line, a boundary, which sets the direction
forward but also cuts oother potential pathways. Serendipitous
accidents, we found, work more like a small explosion, a sudden
release from set ideas and made assumptions, opening up new,
previously unseen pathways.
In this section we take a closer look at two modied costumes/
prostheses and at the dancers’ process of bodily negotiating their
materiality and the emerging transformation. It is the process of
the object becoming more than an object, that is, of it becoming
Figure 5: Tetrahedron prosthesis with one broken joint. ©Pe-
tra Gemeinboeck
an interesting (response-eliciting), aective body as it moves and
takes on a presence of its own.
We found that the costume/prosthesis becomes a body as soon
as the dancer enters it and begins to negotiate its material ten-
sions and forces and to ‘nd’ movements with them. Sometimes
these movements were relationally directed to the surrounds of the
dancer-costume entanglement or another object, while other times
they were directed inward, that is, they resulted purely from the
dancer exploring an internal image through the material qualities
and aordances of the costume/prosthesis. In one session, for in-
stance, Tess de Quincey asked the dancer, inhabiting the cardboard
box, to express a question mark. When the dancer responded to the
prompt, we witnessed the box performing a shape, seemingly posit-
ing layers of hesitation, inquiry and alertness along its movement
trajectory. Rather than a positing, to be precise, we experienced
the nding of a movement, starting owith a hesitating twist that
accelerated upwards with a slight inclination, before it came to
a sudden halt. This was not a visual representation of a question
mark, but rather the bodily processing of what a question mark
does. The box-becoming-body emerged from the “movement sub-
tleties and qualities, contrasts between tension and relaxation, and
between high degrees of physicality and absolute stillness” [28].
2.3.1 Cardboard Cube with Concertinaed Openings. The simple
cardboard cube promised to have an interesting bodying potential,
precisely because it was such a familiar, unassuming object, which
made witnessing it become more than a ‘box’ all the more surpris-
ing. This was conrmed when we expanded the cube shape with
concertinaed openings, as shown in Fig. 6. The idea was to see if the
expressivity of this simple object increased when we allowed the
shape to open up. To test this, we worked with a costume designer
to open up the cube’s four side faces and to reconnect them via
concertinaed paper membranes, spanned between each of the now
door-like open faces and the remaining cube ‘body’. Once inhabited
and activated, however, we found that this capacity to open-up, re-
congure and unfold the ‘box’ made it much harder to comprehend
and relate to the cube’s movements, in particular as it seemed to
overshadow any softer or fragile movement textures and rhythms
that the dancer produced.
There was an unexpected side eect, however, when the dancer
moved the cube without actively pushing-out or pulling-in the

Movement Ma�ers: How a Robot Becomes Body MOCO ’17, June 28-30, 2017, London, United Kingdom
Figure 6: Cardboard cube costume with concertinaed mem-
branes, inhabited by Kirsten Packham. ©Petra Gemein-
boeck
membranes. Then, any small slip or twist, sudden slide or tilt would
make the side faces quiver and wobble, as if each movement cre-
ated a wake (Fig. 7). This ‘secondary motion’, as it is referred to
in animation [
27
], extended the dancer’s movements, similarly to
the springy eects we saw in the spiral tube or the elastic tetra-
hedron. What is so interesting to us about this extra motion or
tension, is that it not only extends the object’s movements but also
performatively expands the material negotiation between dancer
and object. After all, as we discuss further in Section 3, the aim is
for the dancer not to control or puppeteer the movement of the
object but rather to develop movement with it, involved in a sort
of material feedback loop.
Figure 7: Cardboard cube costume, inhabited by Kirsten
Packham, playing with membranes’ secondary motion. ©Pe-
tra Gemeinboeck
2.3.2 Broken Tetrahedron. The dancers often talked about this
particular form as an extension of their body. Building the form out
of PVC pipes without giving it a surface meant that the dancers
could choose to work with it like a prosthesis, rather than a cos-
tume. The lightweight, open structure allowed them to easily move
between inside and outside, and thus also to approach and think
through the object from these dierent positions. Interestingly, as
they changed the location of their focus, they also used dierent
techniques to move with the form (also see Section 3). The slightly
broken tetrahedron proved a more interesting becoming-body than
the initially conceived, unbroken one. As a result of the broken
joint, the object does not only move and twist as much as the under-
lying elastic network permits but also recongures into a number
of shapes. Thanks to this continuous tension, holding the structure
together, as shown in Fig. 8, these transformations require only the
moving of one joint, either at the bottom of the broken edge or the
broken joint itself. Fig. 9–10 show some of the distinctly dierent
shapes (or bodies) that moving the single joint can produce. In later
sessions, we introduced broken joints in all three legs, so that the
structure no longer had a unique side or ‘face’ to it.
It is important to note that this structure’s transformability alone
is not rendering the object a more interesting potential body. On the
contrary, simply reconguring the structure produces intellectually
interesting and/or dynamic shapes, but we found that this built-in
‘cleverness’ doesn’t lend itself to becoming an aective body. The
rst engages us because we want to understand what it does and
how it does it, while the latter engages us in the form of sensations,
prior to formulating these questions. Sensations constitute, in Eliza-
beth Grosz’s words, a “zone of indeterminacy between subject and
object, the bloc that erupts from the encounter of the one with the
other” [
13
]. We are drawn in because of the way it moves, sustains
a tension, gently spaces a path or suddenly halts, etc., rather then
the shapes we can recognise (or not). Hence its presence emerges
from its movement, that is, the dierences in energy and texture
and how variations of tension and speed produce unpredictable yet
readable spacings and spatial relations. Erin Manning states that
“[w]hat dance gives us are techniques for distilling from the weave
of total movement a quality that composes a bodying in motion”
[
16
]. The structure’s transformability oered a seemingly endless
array of starting points for the dancers to play with, and hence for
the form to become more than object, to take on dierent identities.
Figure 8: Broken tetrahedron prosthesis, inhabited by Tess
De Quincey, moving only one joint. ©Petra Gemeinboeck
2.3.3 Motion Tracking. The motion of the activated costume/
prosthesis is tracked to (1) inform the model for a mechanical pro-
totype that resembles the costume/prosthesis and its capacities to
move as closely as possible, and (2) provide data for the machine
to learn from. Tracking the movement of the costume/prosthesis,
rather than the dancers’ movement, eectively allows us to isolate

MOCO ’17, June 28-30, 2017, London, United Kingdom Petra Gemeinboeck and Rob Saunders
the movements to be learned and performed by the robot, inde-
pendent of the dancers’ skeletal structure (also see 2.4.1). Whereas
the mechanical cube will require a separate internal mechanism to
imitate the movements of the costume–dancer entanglement, the
tetrahedron structure, once extended with motorized joints, already
embodies its movement mechanism.
Tracking the Cube. We recorded the cube’s movements using a
video-based motion tracking system and, attaching several coloured
targets to the cube’s surface, as can be seen in Fig. 6 and 7. Acti-
vated by a dancer inside, the cube was recorded using two HD
cameras arranged to ensure that all sides of the cube, except the
base, were captured. The video recordings were analysed using
motion tracking software and the resulting tracked 3D points were
used to animate a model of the cube using Maya and custom scripts,
which was then exported as a log le with the x, y, z position and
the yaw, pitch, roll angles of the cube, together with the angle of
each side face, when pushed open. Each of these samples becomes
an input (in the form of a vector) to the machine learning system,
whose task will be to learn how these vectors change over time.
Tracking the Tetrahedron. The movements of the (broken) tetra-
hedron were recorded by augmenting the construction with in-
struments to measure the angle that the broken leg of the tetra-
hedron makes with the oor. This was achieved by attaching the
tetrahedron’s base to the oor and using two Dynamixel MX-64T
servomotors, arranged as a pan-tilt unit (Fig. 8–10), to measure the
angles produced by the upright leg in relation to the xed base.
Given the orientation of the leg, we can then determine the posi-
tion of the ‘knee’ of the broken leg. The geometric and physical
constraints on the tetrahedron are such that a 3D simulation of
the complete system can be reconstructed. This simulation is then
used to determine the positions of any motors needed to imitate
the recorded movements, e.g., pan and tilt units for all three legs.
Like above, these tracked angles will serve as an input variable (in
the form of a vector) to the machine learning system.
2.4 Learning To Move Based on the Robot’s
Unique Embodiment
Recognising and tapping into the dierence of the machine’s em-
bodiment and how it can elicit new relations is at the very core of
our project. Rather than looking at the robot’s body as a mobile
container, we developed our machine learning approach in tandem
with the robot’s embodiment and capacity to move. To explore
this interdependency in more detail, the following outlines the rst
three machine learning phases, grounding, imitation and improvisa-
tion. Later learning phases will engage choreographers and dancers
to develop performance scenarios for the machine to learn and
improvise in more complex sociomaterial environments beyond
the lab. Our goal is not to replicate the movements we captured in
the workshops but rather for the machine learning system to learn
these captured movement qualities as a series of implicit biases,
such that it can generate new improvisations using the same biases.
2.4.1 Grounding. In the grounding phase, the robot learns how
it can move in relation to its environment through trial-and-error
to ground its movements and relations and any future learning in
its own specic embodiment [
20
]. This approach contrasts common
approaches in social robotics, in which the robot’s control system
and its body are still considered separate, so that the articial ner-
vous system operates “largely independent of the body it is carried
out in” [
33
]. It deploys the developmental robotics [
19
] method of
‘motor babbling’ [
22
], which allows for the robot to ‘discover’ its
own body and possible kinesthetic relations in response to envi-
ronmental aordances. Through this active self-exploration, the
robot gradually generates a body map, which is unique to its own
material body and intricately couples it with the control system,
developed in response to the body’s capacity to move. This body
map will allow the robot to learn and improvise movements later
on, without requiring them to be programmed ‘into it’.
2.4.2 Imitation. In the imitation phase, the robot learns to imi-
tate the movements of its dancer-activated costume twin, as closely
as its own body map allows. Imitation learning is the most common
type of social learning in Human-Robot Interaction (HRI) and is
generally used to teach robots humanlike skills and behaviours. The
challenge of this embodied form of learning arises from having to
map/translate between the two very dierent embodiments of hu-
man and machine, which results in a machine-learning problem, the
well-known correspondence problem [
5
]. Our Performative Body
Mapping method ooads the morphological mapping onto the
dancer, as she learns to move with the costume/prosthesis, avoid-
ing the need for complex data-mapping between radically dierent
bodies. Rather, the tracked data from the costumes/prostheses es-
sentially allows the robot to learn from movement data of its own
mirror image. As it learns to imitate the costume’s movements,
the goal is for the robot to learn the constraints that produce the
movement qualities and subtleties, which emerged from the dancer-
costume enmeshment. Hence, rather than only learning a specic
set of movements, the robot gradually learns patterns of movement,
that is, “the systematic way patterns are structured, sequenced, and
related to one another” [
28
], based on its own machinic body sense.
2.4.3 Improvisation. Finally, in the improvisation phase, the ro-
bot learns to adapt its previously learned movement patterns to
invent new movements, with feedback from the choreographer.
Drawing on methods from computational creativity [
23
], the ma-
chine learns to play with the movement material given to develop
movements that are unique to its own machinic body and its rela-
tions to the environment.
3 DISCUSSION
This section discusses the embodied nature of our PBM method
and how it enables kinesthetic empathy. The concept of kinesthetic
empathy, as we refer to it, is concerned with the body’s sensitivity
to and connectedness with other bodies (incl. non-organic) and its
environment. At rst we take a closer look at the dancers’ kines-
thetic dialogue with the costume/prosthesis and how it facilitates
the ‘bodying’ of the robot. This is followed by a brief consideration
of future audiences based on our inherent kinesthetic abilities to
form connections with other bodies, human and nonhuman.
Core to PBM is our belief that the experience of embodiment and
materiality is essential to produce kinesthetic empathy, albeit an ob-
ject’s movements, behaviours and expressions of intent can also be
designed and studied using screen-based software tools. Animation,
for instance, has a long history of animating live-less shapes and

Movement Ma�ers: How a Robot Becomes Body MOCO ’17, June 28-30, 2017, London, United Kingdom
objects and imbuing them with behaviours, disposition and intent.
Similarly to the abstract forms we worked with in our workshops,
these objects can be surprisingly simple, as demonstrated in the
classic examples of Chuck Jones’s The Dot and the Line (1965) and
John Lasseter’s Luxo Jr. (1986), which features two desk lamp char-
acters. Not surprisingly, software-based animation techniques have
also been used to simulate robots’ behaviours or to design robotic
‘characters’ [
8
,
31
,
32
]. Human-robot interaction, however, is usu-
ally concerned with embodied encounters of humans and machines.
In contrast to primarily software-based AI applications, robots have
a ‘body’ through which they perceive, interact with and recongure
the world, enabling them to share our social spaces in embodied
ways. The relational, embodied nature of dance, can teach us about
the empathic potential of kinesthetic experience [
10
,
26
]—more so
than animation’s visual, storytelling-focused medium, we believe.
As briey discussed in Section 2.3, the dancers ‘nd’ move-
ments or expressions by articulating intentionality in and through
the dancer’s body (and traced by the costume/prosthesis), rather
than dening poses [
15
]. PBM’s strange, non-anthropomorphic
robot/prosthesis, essentially, introduces an additional source for
the dancers’ inspiration and specic material forces to explore and
‘nd’ movements with. The unique kinesthetic experience that is
produced from this material entanglement then becomes a unique
set of constraints and biases for the robot to learn from. Rather than
moving the costume/prosthesis, dancers quickly learned to move
with the strange morphology, and its inherent material aordances.
Figure 9: Broken tetrahedron prosthesis, inhabited by Tess
De Quincey. ©Petra Gemeinboeck
With the ‘broken tetrahedron’ and its open pipe structure, we
introduced a new variant of the PBM method, which allowed the
dancers to move between inside and outside. Interestingly, this
choice opened up further insights into the capacity of the cos-
tumeâĂŞdancer entanglement with regards to ‘nding’ movements.
Previously, with closed objects such as the spiral tube and the cube,
the material entanglement and process of bodying often extended
beyond the physical connes of the costume/prosthesis, as the
dancer required the choreographer’s external view to negotiate the
costume-becoming-body. In contrast, the open, ‘broken tetrahe-
dron’ oers dancers not only a material experience from the inside;
they can choose to inhabit the structure or position themselves out-
side to recongure the structure. It was interesting to witness how
this dierence in positioning themselves also changed the location
of their focus and made them use dierent techniques and imagery
to move with the form. While the dancers were able to extend their
bodies (and images) to the structure, even when positioned outside
the object (Fig. 10), their bodies clearly got more entangled with
the material structure when inhabiting the structure, as shown in
Fig. 9. According to the dancers, they relied more on their visual
sense, when positioned outside, to initiate and explore movement
patterns, which made them vulnerable to attempting to control
the structure’s movement [
12
]. Whereas from inside the structure,
they used dierent body congurations and intensities to feel into,
reshape and move with and, essentially, negotiate the structure
[
12
]. This observation armed two core assumptions, which our
PBM approach builds on: (1) the signicance of a physical robot
costume/prosthesis, which can be bodily thought with, and (2) the
potential of this costume/prosthesis to be bodily inhabited and thus
bodily negotiated, rather than controlled. This bodily thinking with
external forces and other bodies is, we believe, a powerful example
of kinesthetic empathy.
Figure 10: Broken tetrahedron prosthesis, recon�gured by
Kirsten Packham. ©Petra Gemeinboeck
While we have not yet reached the project stage, in which we
study audiences’ experience of our robots, the success of our kines-
thetic approach will not only rely on the dancers’ empathic experi-
ence with these strange, other bodies but also the empathic response
of non-expert audiences. Here again, we believe that movement and
the dynamic, material interplay of forces are key. In Maxine Sheets-
Johnstone’s words, “[w]e literally discover ourselves in movement”
[
25
], and we make sense of the world and other bodies based on
our kinesthetic understanding and sensibilities. With regards to
the aective potential of movement, there is much research on a
moving body’s capacity to resonate with the observer [
9
,
14
,
28
],
arguing that observed movement literally moves and bodily aects
us [
9
]. This aective potential is also central to the interdisciplinary
concept of kinesthetic empathy, exploring our innate capacity to
kinesthetically experience other bodies. It is “a movement across
and between bodies, which âĂę can have aective impact with
potential to change modes of perception and ways of knowing”
[
21
]. This powerful relational capacity has also been studied in
interactions with objects and environments [
18
,
21
]. Empathy with
nonhuman ‘things’ is aligned with anthropologist Alfred Gell’s
view that “it does not matter, in ascribing ‘social agent’ status, what

MOCO ’17, June 28-30, 2017, London, United Kingdom Petra Gemeinboeck and Rob Saunders
a thing (or a person) ‘is’ in itself; what matters is where it stands in a
network of social relations” [
11
]. This suggests that social capacity
is not restricted to familiar physical attributes, but arises from a
body’s capacity to relate to its social environment.
We are yet to develop autonomously moving mechanical proto-
types and evaluate the kinesthetic experience of non-expert au-
diences. In this rst research stage, our evaluation focused on
the kinesthetic experience of our movement experts, reected on
and continuously probed in in-depth discussions, which routinely
sparked new pathways to explore. This includes the validation of
the methodology itself, and while by no means complete, our work-
shop results have established the signicant potential of materially
entangling a dancer and robot body-to-become for bodily thinking
and nding movement with across human and nonhuman domains.
4 PRELIMINARY CONCLUSIONS
This is an ongoing research project, and the next stage will involve
implementing the machine learning and autonomously moving me-
chanical prototypes and to evaluate their kinesthetic performance
in public settings involving non-expert audiences. The workshops
to date have explored the becoming-body of robotic forms through
an iterative process of prototype construction and embodied explo-
rations of the kinesthetic potential of machine costumes/prostheses
by choreographers and dancers.
The exploratory nature of the bodying workshops, a core com-
ponent of our PBM method, has permitted the discovery of unan-
ticipated aspects that will drive the design of our prototype robots
and their movement potential. The serendipitous discovery of the
surprisingly aective potential of a tetrahedron with a single bro-
ken leg has opened up new design possibilities for simpler robotic
prototypes. The ability of secondary motion to amplify the subtle
movements of dancers will be explored in the design of machine
learning systems to determine if grounded robot control systems
are able to exploit secondary motion, not under direct motor control,
to increase the aective potential.
The results of the workshops supported our proposition that the
robot design process can be eectively opened up to movement
experts in ways that allow them to bodily engage with possible
robotic forms and explore their kinesthetic potential. Exploiting
the tacit knowledge of movement experts, the design process tran-
scends the production of geometric or life-like forms to become a
process of bodying that is grounded in kinesthetic experience.
ACKNOWLEDGMENTS
This research is supported under the Australian Research Council’s
Discovery Projects funding scheme (project number DP160104706).
REFERENCES
[1]
K. Barad. 2003. Posthumanist Performativity: Toward an Understanding of How
Matter Comes to Matter. Signs: Journal of Women in Culture and Society 28, 3
(2003), 801–831.
[2]
J. Birringer. 2013. Bauhaus, constructivism, performance. PAJ 35, 2 (2013), 39–52.
[3] De Quincey Co. (????).
[4]
Kirsten Dautenhahn. 2013. Human–Robot Interaction. In Encyclopedia of Human-
Computer Interaction (2nd ed.), Mads Soegaard and Rikke Friis Dam (Eds.). Inter-
action Design Foundation, Aarhus.
[5]
K. Dautenhahn, C. L. Nehaniv, and A. Alissandrakis. 2003. Learning by experience
from others. In Adaptivity and Learning, R. Kühn, R. Menzel, W. Menzel, U.Ratsch,
M. Richter, and I. O. Stamatescu (Eds.). Springer, Berlin, 217–421.
[6] T. De Quincey. 2015. Video recording. (26 March 2015). unpublished.
[7]
S. DeLahunta, G. Clarke, and P. Barnard. 2012. A conversation about chore-
ographic thinking tools. Journal of Dance & Somatic Practices 3, 1–2 (2012),
243–59.
[8]
T. Fong, I. Nourbakhsh, and K. Dautenhahn. 2003. A survey of socially interactive
robots. Robotics and Autonomous Systems 42 (2003), 143–166.
[9]
S. L. Foster. 2008. Movement’s Contagion: The Kinesthetic Impact of Perfor-
mance. In The Cambridge Companion to Performance Studies, Tracy C. Davis
(Ed.). Cambridge University Press, Cambridge, 46–59.
[10]
S. L. Foster. 2010. Choreographing empathy: Kinesthesia in performance. Routledge,
London.
[11]
A. Gell. 1998. Art and agency: An anthropological theory. Oxford University Press,
Oxford.
[12]
P. Gemeinboeck and T. De Quincey. 2017. In conversation, Video recording. (31
January 2017). unpublished.
[13]
E. Grosz. 2008. Chaos, Territory, Art: Deleuze and the Framing of the Earth.
Columbia University Press, New York.
[14]
I. Hagendoorn. 2004. Some Speculative Hypotheses about the Nature and Per-
ception of Dance and Choreography. Journal of Consciousness Studies 11 (2004),
79–110.
[15]
J. Lasseter. 2001. Tricks to animating characters with a computer. ACM Siggraph
Computer Graphics 35, 2 (2001), 45–47.
[16]
Erin Manning. 2013. Always More Than One: Individuation’s Dance. Duke
University Press, Durham.
[17]
E. Manning and B. Massumi. 2013. Just Like That: William Forsythe, Between
Movement and Language. In Touching and to Be Touched. Kinesthesia and Em-
pathy in Dance and Movement, G. Brandstetter, G. Egert, and S. Zubarik (Eds.).
DeGruyter, Berlin, 35–62.
[18]
J. McKinney. 2012. Empathy and Exchange: Audience Experiences of Scenogra-
phy. In Kinesthetic empathy in creative and cultural practices, D. Reynolds and
M. Reason (Eds.). Intellect Books, 219–236.
[19]
P.-Y. Oudeyer. 2010. On the impact of robotics in behavioral and cognitive sci-
ences: from insect navigation to human cognitive development. IEEE Transactions
on Autonomous Mental Development 2, 1 (2010), 2–16.
[20]
R. Pfeifer and J. Bongard. 2007. How The Body Shapes The Way We Think: A New
View Of Intelligence. MI T Press, Cambridge.
[21]
D. Reynolds. 2012. Kinesthetic Engagement: Embodied Responses and Intersub-
jectivity: Introduction. In Kinesthetic empathy in creative and cultural practices,
D. Reynolds and M. Reason (Eds.). Intellect Books, London, 87–90.
[22] R. Saegusa, G. Metta, G. Sandini, and S. Sakka. 2008. Active motor babbling for
sensorimotor learning. In Proceedings of the IEEE International Conference on
Robotics and Biomimetics. 794–9.
[23]
R. Saunders. 2012. Towards autonomous creative systems: A computational
approach. Cognitive Computation 4, 3 (Special issue on ‘Computational Creativity,
Intelligence and Autonomy’ 2012), 216–25.
[24]
N. Sharkey and A. Sharkey. 2010. Living with robots. In Close Engagements With
Arti�cial Companions, Y. Wilks (Ed.). John Benjamins, Amsterdam, 245–56.
[25]
M. Sheets-Johnstone. 2011. The primacy of movement. John Benjamins Publishing,
Amsterdam and Philadelphia.
[26]
M. Sheets-Johnstone. 2012. From movement to dance. Phenomenology and the
Cognitive Sciences 11, 1 (2012), 39–57.
[27]
J. Stacey and L. Suchman. 2012. Animation and Automation: The liveliness and
labours of bodies and machines. Body and Society 18, 1 (2012), 1–46.
[28]
C. Stevens and S. McKechnie. 2005. Thinking in action: thought made visible in
contemporary dance. Cognitive Processing 6 (2005), 243–252.
[29] L. Suchman. 2011. Subject Objects. Feminist Theory 12, 2 (2011), 119–45.
[30]
S. Suschke. 2003. Müller macht Theater: Zehn Inszenierungen und ein Epilog.
Theater der Zeit, Berlin.
[31]
L. Takayama, D. Dooley, and W. Ju. 2011. Expressing Thought: Improving Robot
Readability with Animation Principles. In Proceedings of Human-Robot Interaction
Conference: HRI 2011. Lausanne, 69–76.
[32]
A. Van Breemen. 2004. Bringing robots to life: Applying principles of animation
to robots. In Proceedings of the Workshop on Shaping Human-Robot Interaction:
Understanding the Social Aspects of Intelligent Robotic Products, CHI 2004. Vienna,
5–8.
[33]
T. Ziemke. 2001. The Construction of ‘Reality’ in the Robot: Constructivist Per-
spectives on Situated Articial Intelligence and Adaptive Robotics. Foundations
of Science 6, 1-3 (Special issue on ‘The Impact of Radical Constructivism on
Science’ 2001), 163–233.
[34]
T. Ziemke. 2016. The body of knowledge: On the role of the living body in
grounding embodied cognition. BioSystems 148 (2016), 4–11.
View publication statsView publication stats