Autonomous Agents as
Robotics, Evolution and Art Lab
IT University of Copenhagen
November 5, 2016
Hi and welcome. My talk today is entitled "Autonomous Agents as Artistic
Collaborators" and in it,I'm going to explore the role of scale in my work as an artist
and engineer, in which I use autonomous robotic agents to drive the creation of
The work that I do tends to be physical and sculptural in nature. Engineering is in
many ways focused on the ability to manipulate and control space and therefore one
of my central interests is in ceding that control and attempting to capture the
uncontrollability of the world in my work. In that sense, I am interested in systems
that are generative and emergent. My work asks the question: What can we make
when we give up absolute control?
Letting go of an absolute level of control, however, does not mean in my case, giving
way to pure randomness.I am interested in collaboration with the robotic agents that
I create. Because those agents are often small in relation to the patterns that they
form as a whole, they have an innate relationship with the notion of scale, which
leads to the central question that I would like to address within the framework of this
Is it possible to exert
creative direction on the
emergence of macro-scale
patterns from the actions of
Is it possible to exert creative direction on the emergence of macro-scale patterns
from the actions of micro-scale autonomous agents?
Put another way: How, in the design of autonomous robotic agents, can we help to
define the collective patterns of behaviour that they will ultimately create? This
question is important to my artwork because it helps to identify the creative mode of
the work. But it is also important in a larger context. Many of the technological
systems that are coming into use today could be characterized in much the same way
as my artwork, simply on a larger scale. Consider the network of smart home
thermostats that have the potential to create large-scale patterns of energy
consumption through their independent actions. In a social context, we can view
ourselves as the micro-scale agents and ask the question of how our collective
actions create society's patterns and structures.
If we can begin to examine those larger questions through the process of play and
creation in an artistic realm, perhaps we can at the same time begin to understand
the larger challenges we face.
I want to begin this exploration by introducing a project that I have been working
intermittently over the last few years. I'll use that project to guide the discussion
through two of the questions that were proposed as part of this panel.
1. A portmanteau of inorganic and
2. A silicon and copper-based
creature displaying some
characteristics of living, organic
Inorganisms is a workshop that models the process of building diverse,
interconnected entities. The creative task is modelled on the operations of
functioning ecosystems. Participants are asked to design and construct an entity
made of inorganic materials that models an organic being, a self-contained object
exhibiting the ability to sense and act upon the world. They are asked to develop its
context: Where does their inorganism live? What are its wants? What are its needs?
What can it give to the world and to others? Just as importantly, they are told that
their inorganism does not exist in a vacuum. They must interface with other
inorganisms, matching their desires and capabilities. The result --- if the participants
are successful --- is an interconnected ecosystem of inorganisms. The actions of one
trigger the reactions of others and responses cascade through the ecosystem.
Patterns of behaviour emerge and shift as people move through the ecosystem and
add their own stimuli to the fold. Through experience --- the experience of designing
the individual inorganisms, but also the experience of the inorganism ecosystem as a
whole --- participants learn about the types of distributed, connected design
practices that can begin to address wicked global problems.
Each inorganisms has a single mode of perception and a single mode of action.
Perception is limited to the ability to sense light or sound and action is similarly
limited to the ability to create light and sound. This effectively creates four species of
Inorganisms, defined by why they can sense and how they can respond. The
perceptual modes are purposefully chosen as being ones which people are able to
engage as well, so that human visitors to the ecosystem of Inorganisms become
actors in their own right in the context of the system as whole.
To get a better sense,I'd like to show you a short video of this particular Inorganism,
named "Sound Chameleon".
How are notions of
meaningful to creativity
at social and planetary
The first question that I'd like to address from the panel proposal is: How are notions
of "emergence" or "agent-structure" meaningful to creativity at social and planetary
I would like to revise that question a little to add the miniature scale to our
While recognizing that there is a whole discourse about the nature of agent-structure
relationships, I'd like to begin roughly from Giddens' notion of structuration, whereby
agents and structures cyclically affect each other. Structuration is dynamic,
recognizing that agents and structures are in constant flux. But that movement is
often minimal in a structural sense. Most often, it represents a change of structural
parameters, rather than a wholesale change in the form of the structure. The notion
of structure points to the resilience of a system, and the metaphor inherent in the
term structure points to something that is solid, long-lasting.
Agent | Structure
Because structure depends on the combined actions of agents, it is difficult for any
one agent to affect the structure of a system on their own. Being the time that it is, it
seems appropriate to use American politics as a reasonable example of this
phenomenon. For most people in the country, the ability to shift the political system
is quite limited. In a structure defined by the agency of over 300 million people, that
is to be expected. Of course, some people exert outsized influence on the system:
Presidents, wealthy campaign donors, media personalities, and others. Even so,
witness the difficulty that those agents have had in creating relatively minor shifts in
the structure of American life.
Photo by Brian Gatwicke (https://www.flickr.com/photos/briangratwicke/) CC-BY
However, this analysis is performed at the physical scale of human society. Now
imagine that the structure that we are examining is a colony of leafcutter ants. Ant
colonies are highly structured systems in which each ant agent plays a role in the
formation of the system structure, while the structure of the colony dictates much of
each ants behaviour. Leafcutter ants in particular have a complex society where some
ants are tasked with foraging, while others farm fungi using the foraged leaves as a
growing medium. How does a human agent fit into this system? I would argue that in
this system a human actor could not actually be considered an agent. Our power to
influence its structure, for example by the introduction of a fungicide, is too great for
us to be considered agents in the traditional sense.
The same logic applies on a planetary scale. Humans actors can be viewed as agents
within the structure of Earth, but larger scale actors such as celestial bodies cannot.
The power of the Sun or a particularly large asteroid to impact the agent-structure
relationship of Earth is simply too large to fit into our traditional notions of
structuration. Celestial bodies therefore cannot be considered agents when viewing
the planet as a system. The role of an actor as either an agent within a structured
system or as something outside the system can be understood to be a function of
Where does emergence fit into this analysis? Emergence can be roughly described as
process by which the patterns arise in a complex system devoid of any central
Photo by Mali Mish (https://www.flickr.com/photos/danlin/) CC-BY-NC-ND
Its canonical examples include the flocking behaviour of starlings and the
schooling behaviour of fish, as well as the formation of snowflakes from water
droplets and formation of tornadoes and hurricanes. In all of these examples, we see
the formation of macro-scale patterns that result from the actions of micro-scale
actors. Described in this way, emergence can be viewed as the process by which
structure arises from the behaviour of agents.
Stephanie Forrest describes this behaviour in her book"Emergent Computation". "In
these systems interesting global behaviour emerges from many local interactions.
When the emergent behaviour is also a computation, we refer to the system as
emergent computation... Three important and overlapping themes that exhibit
emergent computation are self-organization, collective phenomena, and cooperative
behaviour (absence of any centralized control)."
That notion of the lack of centralized control brings us back to the inorganisms
project. Here, the potential for emergent behaviour is rather clear. What is less
well-defined is the role of the human actors in the project.
If we return, for a moment to the consideration of scale and notions of agency and
structure, Inorganisms operate on a very human scale and in fact explicitly include
humans as agents in their ecosystems. At the same time, the act of creating the
inorganisms gives a human actor a much larger scale influence on the final ecosystem
than we would afford a general agent within the system. Does this render the artist
as a non-agent during the creative process?
Perhaps, in retrospect, this is part of the reason that I decided to conceive of this
project as a series of workshops in which participants create the Inorganisms, as
opposed to a project in which I alone create a set of autonomous agents. In asking
others to help create the Inorganisms, I further remove my own creative influence
from the system, placing me if not quite back into the realm of an agent, then close
How does "diversity"
at the micro-scale
relate to creativity at
The second question proposed for this panel is whether "diversity" at the micro-scale
relates to creativity at the meso-scale, and if so, how? To examine this, let's look at
the effects of diversity on the emergent result of a system.
A complete absence of diversity
The extreme opposite of diversity would be complete uniformity.
If we take this box to be our system, we can imagine every pixel being an individual
autonomous agent. In this system, the agents' behaviour is to select their pixel shade.
If the agents are uniform, we might imagine that the resulting system looks
something like the box on the screen. A uniform blob of grey agents.
Uniformity at the micro-scale produces uniformity at the meso scale. Assuming, for a
moment, that we are discussing a creativity that is complex and nuanced, creative
results at a meso-scale will not emerge from a uniform set of agents at the
A pure diversity, 10x10
Leaping to the opposite end of the spectrum, what would result from pure diversity?
If we conceive diversity as a question of "how different is each agent from all of the
other agent?" then pure diversity could be considered to be set of actors operating
In this simulation, each pixel in the 10x10 grid has selected randomly to appear black
or white. The resulting pattern is interesting at least. We could consider this to
appear somewhere on a creative scale in terms of the texture that it generates. It has
the vague appearance of a Rorsarch ink blot, or an abstract painting of some sort.
But what of the role of scale?
A pure diversity, 100x100
Here, we've zoomed out by a factor of 10. There are still interesting local features
present and, if I were to try to quantify creativity, which I don't advocate doing in the
least, I would give this roughly the same score as the previous image.
A pure diversity, 1000x1000
But as we step back another factor of 10, to a 1000x1000 grid, our randomness
begins to look remarkably uniform.
A pure diversity, 10000x10000
In fact, as we get to a 10000x10000 grid,our random system appears almost
identical to the uniform one from a few slides ago. Randomness, or pure diversity, is
actually quite uniform as we move further and further away from it. That result, in
fact, is the basis of statistics.
Randomness produces lots of diversity at a scale close to that of the random agents,
but as one zooms out, randomness becomes uniform. As we change scales, our
perception of random agents becomes averaged and what was once diverse becomes
a sea of similarity. If we can conceive of creative systems as those which are complex
and ever-shifting, then we can understand that creativity as arising from systems
that, as Philip Galanter and Murray Gell-Mann put it, "exhibit a dynamic tension
between order and disorder". In their view, both highly non-diverse (ordered) and
highly diverse (disordered) systems are simple.
What we seek, is a complex diversity that holds across many scales. I would call this a
sort of fractal diversity.
To find diversity that holds across scales, we can look to biology. Diversity in a
biological sense consists of diversity contained within hierarchies of relationships, in
the kind of fractal pattern that I mentioned previously. Living beings can be roughly
placed into groups that seem internally similar and externally dissimilar, but those
groups can once again seem remarkably diverse if one focuses on any one group.
In the context of the Inorganisms project, this notion of fractal diversity plays out in
the most minimal of ways. Inorganisms are effectively limited to four distinct species,
given their abilities to either see or hear, and create either visual or auditory
stimulus. There is a minimal sort of diversity in the physical form of the Inorganisms
but there is also an inherent discrete grouping. The grouping defined which species
can respond to another --- only species that can hear can respond to auditory
stimulus, for example. But, there is also individual variation within the groups as each
Inorganism is programmed individually and therefore responds differently to the
same stimulus as another Inorganism within its species. This means that the
propagation of stimulus through the environment is directional and
source-dependent, and the combinations of sequences of events within the space are
Where does this leave us? I propose we return to the original questions:
Is it possible to exert creative direction on the emergence of macro-scale patterns
from the actions of micro-scale autonomous agents? It seems as though how we
exert this creative direction depends partly on our understanding of emergence,
agent-structure, diversity and the effect of scale on those factors.
How are notions of "emergence" or "agent-structure" meaningful to creativity at
microscopic, social and planetary scales? The proposal here is that the role of an
actor as"agent" depends on their scale in relation to the systems and that larger scale
actors can directly influence the structure of a system, while smaller scale actors can
act as agents within it.The idea of emergence relates to the process by which
structure arises from agent actions, as opposed to structure which arises from
non-agent action, which would be non-emergent.
How does "diversity"at the micro-scale relate to creativity at the meso-scale? As for
diversity, it alone is not enough to determine creativity at a meso-scale. There must
be a recursive, fractal relationship that creates diversity at many scales in order to
create complex creativity at a larger scale.