The intangible material of interactive art: agency, behavior and emergence

Abstract

This paper presents a conceptual analysis of some of the basic notions for the practice of interactive art and the relations among them. A sound understanding of these notions is essential for the creation of the aesthetics of artistically behaving systems. Interactivity, agency, behavior and emergence are presented as the building blocks of this practice, understanding that they are at least as important as the materials that physically instantiate the pieces and installations that constitute the body of interactive art. Interactivity is defined and confronted to the metaphor of the conversation and to the idea of designing interactive systems with artistic purposes. The notions of agency, behavior and performativity are reviewed through the reading of Andrew Pickering’s account for the ontology of Cybernetics and in relation to interactive art practices. Finally, the concept of emergence and Peter Cariani’s emergence-relative-to-a-model are presented as a theoretical framework with which the analysis and creation of unexpected and non pre-designed behaviors in interactive systems can be based.

Full text

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
artnodes
http://artnodes.uoc.edu
Universitat Oberta de Catalunya
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
Abstract
This paper presents a conceptual analysis of some of the basic notions for the practice of
interactive art and the relations among them. A sound understanding of these notions is
essential for the creation of the aesthetics of artistically behaving systems. Interactivity, agency,
behavior and emergence are presented as the building blocks of this practice, understanding
that they are at least as important as the materials that physically instantiate the pieces and
installations that constitute the body of interactive art. Interactivity is defined and confronted to
the metaphor of the conversation and to the idea of designing interactive systems with artistic
purposes. The notions of agency, behavior and performativity are reviewed through the reading
of Andrew Pickering’s account for the ontology of Cybernetics and in relation to interactive art
practices. Finally, the concept of emergence and Peter Cariani’s emergence-relative-to-a-model
are presented as a theoretical framework with which the analysis and creation of unexpected
and non pre-designed behaviors in interactive systems can be based.
Keywords
interactivity, behavior, agency, aesthetics, emergence, design, interactive art
43
CC
Submission date: August, 2015
Accepted date: October, 2015
Published in: November, 2015
ARTICLE
The intangible material of
interactive art: agency, behavior
and emergence*
NODE: “ART MATTERS II”
Joan Soler-Adillon
Universitat Pompeu Fabra
Joan Soler-Adillon
2015 by FUOC
CC
* This work is part of the Project “Active Audiences and Journalism. Interactivity, Web Integration and Findability of Journalistic Information”. CSO2012-
39518-C04-02. National Plan for R+D+i, Spanish Ministry of Economy and Competitiveness.

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
44
CC Joan Soler-Adillon
2015 by FUOC
CC
El material intangible del arte interactivo: agencia,
comportamiento y emergencia
Resumen
Este artículo presenta un análisis conceptual de algunas de las nociones básicas para la práctica
del arte interactivo y las relaciones entre estas. Un buen conocimiento de estas nociones es
fundamental para la creación de la estética de los sistemas con un comportamiento artístico.
La interactividad, la agencia, el comportamiento y la emergencia se presentan como los pilares
fundamentales de esta práctica, entendiendo que son como mínimo tan importantes como
los materiales que físicamente instancian los componentes e instalaciones que constituyen
el cuerpo del arte interactivo. La interactividad se define y confronta con la metáfora de la
conversación y la idea del diseño de sistemas interactivos con fines artísticos. Las nociones de
agencia, comportamiento y performatividad se revisan a través de la lectura de la interpretación
de Andrew Pickering para la ontología de la cibernética y en relación con las prácticas artísticas
interactivas. Por último, el concepto de emergencia y la emergencia relativa a un modelo de
Peter Cariani se presentan como un marco teorético en el que se puede basar el análisis y la
creación de comportamientos inesperados y no diseñados previamente en sistemas interactivos.
Palabras clave
interactividad, comportamiento, entidad, estética, emergencia, diseño, arte interactivo
1. Introduction
This paper presents a conceptual analysis of some of the basic
notions for the practice of interactive art and of its relation. A sound
understanding of these notions is essential for the creation of the
aesthetics of artistically behaving systems. Interactivity, agency,
behavior and emergence are presented as the building blocks of
this practice, understanding that they are at least as important as
the materials that physically instantiate the pieces and installations
that constitute its body of work.
Interactive art materializes in computer screens, installation
spaces, robotic devices and the like. The interactor (the one who
dialogues with the piece) relates with it through sensor interfaces:
keyboard, mouse, proximity sensors, customized buttons, etc. and
is addressed through actuators: loudspeakers, screens, motors,
etc. All of these materially instantiated elements are certainly a
part of the interactive experience, and it is through them that the
artistic experience is brought about. In addition, more often than
not, interactive art pieces are characterized through these elements:
categories such as net.art, installation art or robotic art are examples
of this.
However, the choosing of components and the designed interfaces
is only a part of the creative endeavor of interactive art. Arguably, what
is essential to this practice is precisely that which is not physically
instantiated, even in robotic pieces. Agency and behavior, which
form the basis of performativity, are instead the defining and very
central aspects of interactive art. They are the intangible material of
interactive art. In this context, the concept of emergence appears
as an essential piece in order to understand the possibility of non
pre-defined behaviors to appear in such pieces.
2. Interactivity
There is a body of literature that has addressed the definition of
interactivity in relation to both general and computer media, mostly
since the mid-late 1990s. The general consensus is that there are
three main approaches to defining the concept from the media studies
point of view, depending on whether the defining efforts situate the
focus of the definition on the structure of media, on the user, or on
the process of communication that develops between them (Downes
and McMillan, 2000; Kiousis, 2002; Quiring, 2009; Mechant, 2012;
Weber, Behr and DeMartino, 2014).
A first group of authors is that which focuses on interactivity as
a characteristic of media (Durlak, 1987; Baecker and Buxton, 1988;
Wilson, 1994; Jensen, 1998; 1999; 2008; Sundar, 2004; Lee, Park
and Jin, 2006). The center of interest within these approaches is on
how media is structured in order to afford interactivity: what it offers
to the potential user and how, so that interaction can take place.
Interactivity is often presented in these cases as a continuum, and
different media are placed in different points along that continuum
(from less interactive to more interactive media). The types of activities

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
45
CC Joan Soler-Adillon
2015 by FUOC
CC
that each of these media offer to users is also read within these
parameters. On the other end, some authors place the focus of study
on the user, mainly around the concept of perceived interaction (Wu
1999; Liu and Shrum, 2002; Leiner and Quiring, 2008; Quiring 2009).
What is most important here is what interactivity means for the person
that uses the interactive system and how he or she relates to media
on its terms. Interactivity here is an information-based process that is
relevant in terms of individual perception (Newhagen, 2004). Finally,
there is a third group of authors that center their approach on the
communicational process that interactivity represents (Kiousis, 2002;
Rafaeli, 1988; Rafaeli and Sundweeks, 1997; Crawford, 2003; Noble
2009; Penny 2000; 2011; Green, 2010). The focus in these cases is
on the process that develops as the interactive system and the user
act and react to one another.
I have defined interactivity as “a series of related actions between
two or more agents where (1) at least one of them is an artificial
system that (2) processes its responses according to a behavior
specified by design and (3) takes into account some of the previous
actions executed by them” (Soler-Adillon, 2015). This definition aims
not at trying to solve the everlasting discussion on how to define the
term but at clarifying the concept in the context of interactive art.
The definition addresses the three main aspects that should be taken
into account when trying to frame such a complex concept within a
particular context:1 first, specify the system under analysis (ie, are
we analyzing the computer as a whole, for instance, or a particular
piece of software?); second, analyze what the system is capable of
doing (ie, how it responds to its environment); third, following one
of the classic definitions of the term (Rafaeli, 1988), discern if these
responses are merely reactive or are a case of interactivity in terms
of acknowledgement of already performed actions.
This definition corresponds to the third category mentioned above.
It aims at clarifying the communicational process of interactivity, and
it understands it in terms of design and functionality, not of subjective
experience by the user or interactor. The point of view for the analysis
is on how the systems are designed in order to respond to the users’
actions in a way that is read as interactive; on the process of dialogue
that is created between the behaving system and the human interactor
and not on the subjective experience of the latter.
Within this discussion, the metaphor of the conversation is
useful: the dialogue between the human and the machine that
inspired Cybernetics and that is the foundation of Human-Computer
Interaction. After all, the idea is that computers exchange information
with users, as the back and forth of messages advances. User does,
computer reacts, then user does again in response to this reaction,
and so on and so forth. This conversation can be understood in a
very simplistic manner, in terms of the control of systems where
the artificial system responds to the commands of a human user, or
with a more ambitious interpretation of the metaphor, which aims at
generating, through the interactive system, a series of meaningful
exchanges.
Gordon Pask’s Conversation Theory is an important effort to
theorize this (Pask, 1975; 1976; 1996), although it is a general
theory not specifically centered on man/machine conversation. Pask’s
writings on the subject find its context in his interest in learning
devices. Conversation is in his work presented – and interpreted by
latter authors – as an advanced form of interaction by systems that
have the ability to learn. More recently, Paul Pangaro has elaborated
on Conversation Theory in relation to social media and design
(Pangaro, 2009; Dubberly and Pangaro, 2009) and to relevant models
of interactivity (Dubberly, Pangaro and Haque, 2009). Following Pask,
Pangaro and colleagues understand that conversing systems are
a sophisticated assemblage of (second-order cybernetic) learning
systems that feed on and react to each other. This conversation is,
according to them, the most elaborate form of interaction.
In an effort to differentiate systems that are merely reactive
from those that afford interactivity, Dubberly, Panagaro and Haque
distinguish among static, dynamic-reactive, and dynamic-interactive
systems. The first are systems that “cannot act and thus have little
or no meaningful effect on their environment (a chair, for example)”.
Dynamic systems do have the ability to act and relate to their
environment. Within those, they distinguish between systems that react
(they have a linear relation between system activation and response)
and systems that interact. The categorization of interactive systems
is further elaborated and, as said above, they present conversation
as the most sophisticated form of interaction (Dubberly, Pangaro
and Haque, 2009). A similar categorization of passive, reactive and
interactive systems was presented in (Soler-Adillon, 2012).
3. Designing artistically behaving artifacts
When designing interactive artistic artifacts, the approach to
interactivity differs from that of designing functional and efficient
digital systems. In the latter, system’s interfaces should be easy
to use, robust, consistent with existing standards, etc. and users
should know what to expect from the system and how to achieve
their goals. The interactivity of interactive art, on the other hand,
needs not to be constrained by these principles, as it fundamentally
differs from functional interaction in its intentions. This dichotomy
has been characterized, from the point of view of the creator, as
the ‘design for efficiency’ and ‘design for motivation’ approaches
to creating interactive media (Ribas, 2001). Along the same lines,
1. Jens F. Jensen labeled interactivity as a multi-discursive concept (Jensen, 1998), following Tim O’Sullivan and colleagues’ definition, which states that this kind
of concepts are, precisely, those the meaning of which changes depending on the context in which they are used (O’Sullivan et al., 1994).

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
46
CC Joan Soler-Adillon
2015 by FUOC
CC
Simon Penny has elaborated the notion of poetic interaction, which is
at the heart of his theorization of the aesthetics of behavior (Penny,
2000; 2012; 2015). The basic idea is that when designing interactive
artistic artifacts, the type of interactivity that one is seeking to create
is very different from the one in the case of functional interaction.
Poetic interaction is the kind of interactivity that develops within
a system formed by an interactor and a behaving system that has
been designed with artistic purposes. The aesthetics of behavior
refer to the analysis of such type of devices and the experiences
they afford.
In respect to functional systems, aesthetically behaving systems
move towards unpredictability. Systems are neither necessarily easy
to control in terms of how the interface is presented, nor are they
necessarily predictable in their behavior. The idea is that they do so
without falling into what is perceived by the interactor as complete
random behaviors. Otherwise they become uninteresting in terms
of interactivity, since when that is the case all sense of dialog or
control is lost. Thus, artistic interaction is situated in the search for
a point of equilibrium between what is predictable and what is not.
According to Simon Penny it “should not be predictably instrumental,
but should generate behavior which exists in the liminal territory
between perceived predictability and perceived randomness, a zone
of surprise, of poetry” (Penny, 2011). And it is in this liminal territory
that it seeks to create conversation. Moving away from the linear
responsive model of interaction, as described by Dubberly, Pangaro
and Haque in (2009), conversation is here a more elaborated mode of
relation to the artificial system: “artistic interaction [can be conceived]
as an ongoing conversation between system and user rather than the
conventional (Pavlovian) stimulus and response model” (Penny, 2000).
Poetic interaction lives in the realm of art and experimentation. In
it, interactivity and behavior become aesthetic concerns and this has a
series of implications in this type of art creation and on the discourses
around it. In terms of understanding how interactivity is designed
and understood, this approach is to be interpreted as differentiated
from functional interaction. The objective of poetic interaction is to
experiment with the possibilities of interactivity. As noted by Krueger
when describing his early video tracking experiments, interactivity
becomes here a central aesthetic concern (Krueger, 1989) and, as
such, becomes a fundamental part of the artistic interest in these
pieces, although not necessarily in an exclusive manner.
The methods of designing artifacts for poetic interaction may
coincide with those of functional interaction, and some of the design
principles that apply to the latter apply to the former. However, since
the objectives are fundamentally different when designing such
systems, their interpretation has to be made from a different point
of view. These experiences and the processes that develop as the
interactor and the system act and react to each other shall be read
as any other aesthetic experience: a dance between expectations
and surprise where the interactive artifact seeks to engage the user
into exploring and experimenting with it; a dance of agency, to use
Andrew Pickering’s terms (1995). The artistic interactive system is
not conceived as a tool. Instead of being a system to help a user
perform a task efficiently, in this paradigm it exists on its own right,
as an agent – a behaving entity – that will drive the experience of
the visitor to the piece.
Thus, creating artistic interactive systems means designing
performative artifacts that will relate in a meaningful way to an
interactor. This means that there are mainly two focuses of interest
in terms of design: interface and behavior. Within the system formed by
interactor and performative artifact, the first is the point of connection
between them. In accordance, its design is crucial in establishing
this relation. The artist creating the artifact knows that the interactor
relates to his or her environment by doing things with it. According to
this approach, the idea is that he or she will acquire the necessary
knowledge to relate to the interactive object through this embodied
doing, not through an objective abstract examination of it. This means
that, as opposed to the design of a website from the point of view of
usability, for instance, the design approach must take into account
the actions of the interactor at all levels. The analysis of the artifact’s
behavior comes through the doing things with it in relating to its
interfaces (using the buttons, moving in front of it, etc.).
Behavior is the essential element in defining the artifact, and the
interfaces are the means of establishing the relation to it. All other
aspects of the piece are accessory to behavior. This doesn’t mean
they are not important, but external appearance, for instance, is here
one step below in aesthetic relevance. Indeed, interactive art pieces
can be analyzed aesthetically from the point of view of appearance
or in relation to the technology of the time. But according to the
approach proposed here, it is in terms of performativity, of behavior,
that are most relevant. Since they aim at relating to their visitors in
a meaningful way, and are designed to exploit such relations and to
provoke exploration of behavior on the part of the human interactor,
they are aesthetically relevant mostly in terms of interactive behavior.
4. Agency, behavior and performativity
An agent is a behaving entity. Andrew Pickering has described agency
as a basic building block of the influence that an entity can have on
its environment. Put simply, agency is about “doing things in the
world” (Pickering, 2002), a notion that he elaborates in detail in his
work (Pickering, 1995; 2002; 2010). Other accounts take agency
further into the ontological and metaphysical, and place it at the very
center of the configuration of matter and reality. According to Barad,
agency cannot be an attribute of things as it ontologically precedes
the things themselves (Barrad, 2007, p. 178). In any case, arguably in
terms of designing interactivity, it is sufficient to state that an agent
is a non-passive entity. That is, one that either responds reactively or

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
47
CC Joan Soler-Adillon
2015 by FUOC
CC
interactively to the actions that it is able to perceive from whomever
is relating to it.
The creation of agents with artistic purposes (agents as artworks)
has been studied by Simon Penny (2000). An agentic artwork,
according to Penny, is not just one that does things in responding
to its surroundings. It can be understood as a cultural actor in its
own right. These agents are designed and built within a cultural
tradition, and in a particular social context. And it is when they are
understood and read as such, and not abstracted from their social
and technological context, that they can be better understood.
Penny’s own Petit Mal is a paradigmatic example of this approach.
This robotic piece intentionally eschews all anthropomorphic or
zoomorphic appearance, but its behavior is infallibly attributed to
some sort of animal-like entity, as it hesitantly moves towards and
away from the visitors in a constant loop of recognition, retreat
and forgetfulness. The materialization of the piece is crucial, of
course. Penny acknowledges that it did have the right size so as
not to be scary, but at the same time not to be regarded as a sort
of pet robot (Penny, 2015). However, what defines Petit Mal as
much as its material instantiation is its behavior. Penny’s naming
it after the disease that informs how it responds to its environment
is no coincidence here. The defining trait of the piece is its agency;
one that performatively relates to the visitors in a neat example of
embodied interaction.
With these type of pieces, then, the aesthetic focus on interactivity
is a step away from the object and a step closer towards its relation
to the interactor. The work is neither just the behaving system nor its
behavior, but it too encompasses the situated actions of the visitor,
the embodied ‘contextualized doing’ of whomever interacts with the
system, much in accordance to the performative ontology that Andrew
Pickering describes as the basis of Cybernetics. As Simon Penny puts
it, in connection to ideas of embodied interaction and enaction, “the
lesson of performativity is that the doing of the action by the subject
in the context of the work is what constitutes the experience of the
work. It is less the destination, or chain of destinations, and more the
temporal process which constitutes the experience” (Penny, 2011).
As said, an important issue here is with what Pickering has
identified as the performative ontology of Cybernetics, which
is arguably extensible to interactive art practices in general. The
basic idea is that, in opposition to the representational ontology of
modern science, Cybernetics is based on a performative ontology: “a
decentered perspective that is concerned with agency – doing things
in the world – and with the emergent interplay of human and material
agency” (Pickering, 2002). Within this context, Cybernetics appears as
a paradigmatic discipline of the ‘performative idiom’. The discipline
as a whole “is all about this shift from epistemology to ontology, from
representation to performativity, agency and emergence, not in the
analysis of science but within the body of science itself” (Pickering,
1995).
Thus, the performative idiom, according to Pickering, is an ontology
in its own right, and a nonmodern one in the sense that it differentiates
itself from the reductionist approach of the scientific method (the
modern ontology). In contrast to Bruno Latour’s characterization of
modernity as being determined by the dualism of people and things
(Latour, 1993), the cybernetic approach, in which this frontier would
be blurred, “stages for us a nonmodern ontology, in which people and
things are not so different at all” (Pickering 2010, p. 18). As he has
explained (Pickering, 2008; 2010), this ontology allows Cybernetics
to propose an image of the world that is performative rather than
representational. He calls it the ‘ontological theater’, a nonmodern
understanding of the world and our relationship with it that implies
“a vision of knowledge as ‘part of’ performance rather than as an
external controller of it” (Pickering 2010, p. 25).
Here the idea of the black box is useful to understand how
this is brought into practice. According to Pickering, Cybernetics
proposes a theory of knowledge that is largely built up through a
performative relationship with what we can understand as black
boxes. Rather than being about control in a classical sense, “the
entire task of Cybernetics was to figure out how to get along in
a world that was not enframable, that could not be subjugated to
human designs – how to build machines and construct systems
that could adapt performatively to whatever happened to come
their way” (Pickering 2010, pp. 30-31).
In this context, he argues how Cybernetics assumes in fact an
ontology that, contrary to the reductionist approach, involves a certain
degree of unknowability, as it “tries to address the problematic of
Figure 1. Petit Mal. © Simon Penny.

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
48
CC Joan Soler-Adillon
2015 by FUOC
CC
getting along performatively with systems that can always surprise
us” (Pickering 2010, p. 23). These are what Stafford Beer labeled
as exceedingly complex systems. That is, systems which are,
unlike simple and merely complex systems, neither predictable nor
susceptible to be treated by the methods of modern science and
engineering. Exceedingly complex systems, like the interior of the
black boxes, are unknowable. They are too complex to be grasped
representationally, and they change over time, so that future behavior
cannot be anticipated trough current knowledge.
So if Cybernetics was about that, as Pickering argues, it was
not so much about control, as some literature has portrayed it, as it
was about the study of the conditions under which the interaction
among different parts of technology could result in these complex,
unanticipated patterns of behavior. Some of the devices built by the
cybernetic practitioners were clearly built in order to experiment in this
direction, and were in this respect anticipative of interactive art. Ross
Ashby’s Homeostat was a machine whose sole purpose was to exist
performatively. Once set in motion a homeostat unit would connect
to others and seek its own equilibrium, with no other goal than that.
Norbert Wiener defined it as the “brilliant idea of the unpurposeful
random mechanism which seeks for its own purpose through a
process of learning” and qualified it as “one of the great philosophical
contributions of the present day” (Wiener, 1989, p. 38). Grey Walters
tortoises, which behaved much similarly to the contemporary Roomba
robots – certainly not by chance –, also represent an example of
such practices. These avant-la-lettre bottom-up robotic devices were
capable of surprising its own designer, Walter, who described them
as having a ‘remarkably unpredictable’ behavior (Walter, 1950). As
Owen Holland has noted, they were a clear anticipation of Artificial Life
(Holland, 1997; 2003). These two pieces, along with Gordon Pask’s
Musicolour and Colloquy of Mobiles, suffice to exemplify the early
Cybernetic’s remarkable anticipation of interactive art.
5. Emergence
The element of surprise, of impossibility of a full anticipation that
Pickering attributes to Cybernetics, and that can be expanded to
interactive art in general, clearly resonates with the idea of emergence,
a concept that is often explained with the idea of a whole being ‘more’
than just the sum of its parts; of being irreducible to the mere addition
of its constituting elements. These explanations are usually articulated
in terms of different levels of complexity where parts or agents are the
constituents of a larger systemic level (the whole) in which emergent
phenomena appear. What is emergent are properties and behaviors
that can be observed at the system level, and they have a degree of
complexity that, as it is argued, cannot be accounted for by a simple
addition of the properties and behaviors of the parts.
In the academic literature the concept of emergence appears
as related to two different and not necessarily related phenomena:
self-organization and the appearance of novelty (Soler-Adillon and
Penny, 2014). Many authors present it as the result of multiple local
interactions among agents within a system that produce observable
patterns at the system level. These patterns are emergent phenomena
in the sense that they could not be understood, nor anticipated, through
the analysis of the elements and their behaviors in isolation (Langton,
1988; Holland, 1998; Bedau, 2008). For other authors, emergent
phenomena are related to fundamental novelty and, thus, to creativity.
For them, emergence is synonymous to the appearance of new
functions or behaviors in a known system (Steels, 1995; Cariani, 2012).
The typical examples of emergence are systems that exhibit
complex behaviors from a relatively small set of simple rules and
behaviors. Ant or termite colonies and their social complexity, the human
mind understood as a product of the interconnectivity of neurons in the
brain, chemical clocks, traffic jams or cellular automata are some of
the most cited examples. When related to novelty, emergence is often
used when referring to learning systems or adaptive devices.
Prominent among the authors that are concerned with emergence
as a generator of novelty and, thus, as creativity, Peter Cariani has
elaborated the theory known as emergence-relative-to-a-model (eg
Cariani 1992; 2009; 2011; 2012). Cariani articulates a discourse
that aims at identifying emergence as novelty in a given system
in a way that can be scientifically communicated. His approach is
concerned with how new functions can appear in systems or devices
that perceive and act on their environment. The basic idea is that
this newness can only be accounted for scientifically if, first, the
observer of the system defines its states and state-transitions by
creating a model of it. Once this is done, these observations are
used to make predictions on the futures states of the system. In
this context, emergence occurs whenever unanticipated behaviors,
states or functions appear: “emergence is the appearance of novel
entities that in one sense or another could not have been predicted
from what came before” (Cariani, 2009).
Figure 2. Walter’s tortoises. © Bristol Robotics Laboratory.

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
49
CC Joan Soler-Adillon
2015 by FUOC
CC
Cariani bases his modeling on three aspects: how the system
reads and acts on its environment (semantics); how it decides how
to act according to these readings (syntactics); and how it evaluates
the actions performed according to its goals (pragmatics). All these
actions are performed according to the basic building blocks of what
the system can operate with: the primitives, in Cariani’s terminology.
Within this framework, he identifies two ways in which emergence
can occur. The first is Combinatoric Emergence, which consists in the
appearance of new system functions through new combinations of the
primitives with which the system operates (eg, genetic algorithms).
The second is Creative Emergence, which is the appearance of new
functions trough the introduction of new primitives in the computations.
This second type of emergence, equivalent to the introduction of a new
sensory organ in an animal species through the course of evolution,
is extremely rare in artificial systems, and in fact Cariani identifies
in his literature only one case: Gordon Pask’s electrochemical ‘ear’
(Cariani, 1993; 2012). Despite these difficulties in the case of artificial
systems, however, Cariani opens a door to mixed computer-human
systems (ie, interactive systems) to be generators of this latter kind
of emergence (Cariani, 2009; 2012).
The relationship between emergence and interactive art can be
exemplified by the idea of the unpredictable black box (Soler-Adillon,
2011). In the prototypical engineering black box, the researcher can
figure out the mappings between inputs and outputs; i.e. he or she
must be able to develop a protocol to map what goes in and what
comes out of the box (Ashby, 1957). In contrast, unpredictable black
boxes would be devices that appear as black boxes even to their own
creator; systems in which the relation of inputs and outputs is not
fully foreseeable, and the inside of which is not only unknown but
unknowable. Thus, the only possible relation with these unpredictable
black boxes is the continuous performative act of interacting with
them. Not even the eventual opening of the box (or zooming in into
the system) would solve the problem of the unpredictability that they
would have up to some degree.
Ideally, the result of the unpredictable black box is not mere
randomness but something that, although coherent with the general
behaviors of the system, was not explicitly built in it by its designer –
something that lives in the liminal territory of poetic interaction and
can be qualified as emergent. As often happens in interactive art (eg
in generative art and especially in Artificial Live Art), it is sought by
the artist to create systems or processes that exceed his or her own
expectations. The idea is to do so not through some blind trial and
error, but through the creation of the conditions where emergent
phenomena can occur. This creative effort can be conceptualized with
the idea of emergence as generation of novelty (Soler-Adillon, 2015).
In terms of designing interactive artifacts potentially capable
of exhibiting emergent behavior, Cariani’s emergence-relative-to-
a-model is arguably the most solid framework to approach the
challenging task. Designing such devices means facing the apparent
paradox of designing something that is, in principle, impossible to
be designed. However, designing the possibilities for emergence to
appear is indeed possible, and it has been the goal of a series of
practitioners, mostly in relation to Artificial Life Art.
A successful example of such pieces is Ruairi Glynn’s Performative
Ecologies (Glynn, 2008), a series of robots that learn how to best
attract the gaze of the visitors to the installation space by ‘dancing’
in front of them. As they perform their moves, they calculate (through
facial recognition) how much of the attention of a visitor these
moves are capable of attracting. Then, using genetic algorithms, the
individual robots will create new dance moves based on those that
were successful within the group. With this, eventually the robots
should be capable of attracting more and more of the users’ attention
as they learn over time to do so. The new behaviors that the robots
learn and perform, and that were not pre-designed by the artist, can
be regarded as a case of emergence – particularly, of combinatoric
emergence according to Cariani’s categorization (Soler-Adillon, 2015).
Figure 3. Performative ecologies. © Ruairi Glynn.
6. Conclusions
Interactive art is about behaving entities, agents, performatively
relating to their environment, which includes the interactor (visitor,
user, etc.). The theorization and practices of early Cybernetics,
through what Pickering has labeled the performative idiom, can
be a good theoretical framework to understand how these ideas
relate to each other. The embodied doing of the interactor with the
interactive systems, and the connection to ideas of unpredictability
and emergence offer an interesting arena for the analysis and
creation of interactive art pieces. According to this, the guidelines
of design for emergent behavior are possible to produce. Cariani’s
emergence-relative-to-a-model and the theorization of emergence as
self-organization and as generation of novelty constitute a theoretical
framework within which this task can be undertaken.

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
50
CC Joan Soler-Adillon
2015 by FUOC
CC
However, any aesthetic discussion on this area requires a sound
conceptual analysis; a clarification of terms that helps avoid the
ambiguities of discussing interactivity and emergence without a clear
statement of what is exactly meant by such multi-discursive concepts.
In this respect, the proposed definition of interactivity, along with the
theorization of emergence as self-organization and as generation of
novelty (Soler-Adillon, 2015) represents an effort to contribute to the
elaboration of the aesthetics of artistically behaving systems.
References
ASHBY, W. R. (1957). An Introduction to Cybernetics. New York: Wiley.
BAECKER, R. M.; BUXTON, W. A. S. (1988). Readings in Human-
Computer Interaction: A Multidisciplinary Approach. San Mateo,
CA: Kaufmann.
BEDAU, M. A. (2008). “Downward Causation and Autonomy in Weak
Emergence”. In: M. A. BEDAU, P. HUMPHREYS (eds.). Emergence.
Contemporary Readings in Philosophy and Science. Cambridge,
Massachussetts: MIT Press, pp. 155–188. <http://dx.doi.
org/10.7551/mitpress/9780262026215.003.0010>
CARIANI, P. (1992). “Emergence and artificial life”. In: C. G. LANGTON,
C. TAYLOR, J. D. FARMER, S. RASMUSSEN (eds.). Artificial Life II
Reading, Massachussetts: Addison-Wesley Publishing Company,
pp. 775–797.
CARIANI, P. (1993). “To evolve an ear: epistemological implications
of Gordon Pask’s electrochemical devices”. Systems Research,
vol. 10, no. 3, pp. 19–33. <http://dx.doi.org/10.1002/
sres.3850100305>
CARIANI, P. (2009). “Strategies for Creating New Informational
Primitives in Minds and Machines”. Proceedings of the Dagstuhl
Seminar on Computational Creativity. Wadern, pp. 1–13.
<http://drops.dagstuhl.de/opus/volltexte/2009/2192/pdf/09291.
CarianiPeter.Paper.2192.pdf>
CARIANI, P. (2011). “The Semiotics of Percept-Action Systems”.
International Journal of Signs and Semiotic Systems, vol. 1, no.
1, pp. 1–17. <http://dx.doi.org/10.4018/ijsss.2011010101>.
CARIANI, P. (2012). “Creating New Informational Primitives in Minds
and Machines”. In: J. MCCORMACK and M. D’INVERNO (eds.).
Computers and Creativity. New York: Springer, pp. 395–430.
CRAWFORD, C. (2003). Chris Crawford on Game Design. San Francisco:
New Riders.
DOWNES, E. J.; McMILLAN, S. J. (2000). “Defining Interactivity:
A Qualitative Identification of Key Dimensions”. New
Media & Society, vol. 2, no. 2, pp. 157–179. <http://doi.
org/10.1177/14614440022225751>
DUBBERLY, H., & PANGARO, P. (2009). “What is conversation, and how
can we design for it?”. Interactions Magazine (ACM), vol. XVI, no. 4.
<http://www.dubberly.com/articles/what-is-conversation.html>
DUBBERLY, H.; PANGARO, P.; HA QUE, U. (2009). “What is interaction? Are
there different types?”. Interactions Magazine (ACM), vol. XVI, no. 1.
<http://www.dubberly.com/articles/what-is-interaction.html>
DURLAK, J. T. (1987). “A Typology for Interactive Media”. In: M. L.
MCLAUGHLIN (ed.). Communication Yearboook. Newbury Park,
CA: Sage, vol. 10, pp. 743–757.
GLYNN, R. (2008). “Conversational Environments Revisted”. 19th
European Meeting of Cybernetics & Systems Research. Viena.
GREEN, J. (2010). “Interactivity and Agency in Real Time Systems”.
Soft Borders Conference & Festival Proceedings. São Paulo, Brazil:
Centro Universitário Belas Artes de São Paulo. Pp. 1–5.
HOLLAND, J. (1998). Emergence. From Chaos to Order. New York:
Basic Books.
HOLLAND, O. (1997). “Grey Walter: The Pioneer of Real Artificial Life”.
In: C. G. LANGTON and K. SHIMOHARA (eds.). Artificial Life V:
Proceedings of the Fifth International Workshop on the Synthesis
and Simulation of Living Systems. Cambridge, Massachussetts:
MIT Press. Pp. 34–41.
HOLLAND, O. (2003). “The first biologically inspired robots”. Robotica, no. 21,
pp. 351–363. <http://dx.doi.org/10.1017/S0263574703004971>.
JENSEN, J. F. (1998). “Interactivity” Tracking a New Concept in Media
and Communication Studies”. Media and Communication Studies,
Nordicon R, no. 1, pp 185–204.
JENSEN, J. F. (1999). “The Concept of “Interactivity” in Interactive
Television”. In: J. F. JENSEN and C. TOSCAN (eds.). Interactive
Television: TV of the Future or the Future of TV?. Aalborg: Aalborg
Universitetsforlag, pp. 25–66.
KIOUSIS, S. (2002). “Interactivity: A concept explication”. New Media
& Society, vol. 4, no. 3, pp. 355–383. <http://dx.doi.org/10.1177
/146144480200400303>.
KRUEGER, M. (1991). Artificial Reality II. New York: Addison-Wesley
Professional.
LANGTON, C. G. (1988). “Artificial life”. In: C. G. LANGTON (ed.).
Artificial Life: Proceedings of an Interdisciplinary Workshop on
the Synthesis and Simulation of Living Systems. Boston: Addison-
Wesley Publishing Company. Pp. 1–47. <http://library.iscpif.fr/
files/CSSS2010-20100805-Beslon-lecture4.pdf.>
LATOUR, B. (1993). We Have Never Been Modern. Cambridge,
Massachussetts: Harvard University Press.
LEE, K. M.; PARK, N.; JIN, S.-A. (2006). “Narrative and interactivity in
computer games”. In: P. VORDERER and J. BRYANT (eds.). Playing
video games: Motives, responses, and consequences. Nahwah,
NJ: Erlbaum, pp. 259–274.
LEINER, D. J.; QUIRING, O. (2008). “What Interactivity Means to the User
Essential Insights into and a Scale for Perceived Interactivity”. Journal
of Computer-Mediated Communication, vol. 14, no. 1, pp. 127–
155. <http://dx.doi.org/10.1111/j.1083-6101.2008.01434.x>
LIU, Y.; SHRUM, L. J. (2002). “What Is Interactivity and Is It Always Such
a Good Thing? Implications of Definition, Person, and Situation

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
http://artnodes.uoc.edu The intangible material of interactive art…
artnodes
Universitat Oberta de Catalunya
51
CC Joan Soler-Adillon
2015 by FUOC
CC
for the Influence of Interactivity on Advertising Effectiveness”.
Journal of Advertising, vol. 31, no. 4, pp. 53–64. <http://dx.doi.
org/10.1080/00913367.2002.10673685>
MECHANT, P. (2012). “An illustrated framework for the analysis of
Web2.0 interactivity”. Contemporary Social Science, vol. 7, no. 3,
pp. 263–281. <http://doi.org/10.1080/21582041.2012.716524>
NEWHAGEN, J. E. (2004). “Interactivity, Dynamic Symbol Processing,
and the Emergence of Content in Human Communication”. The
Information Society, vol. 20, no. 5, pp. 395–400. <http://doi.
org/10.1080/01972240490508108>
NOBLE, J. (2009). Programming Interactivity. A Designer’s Guide
to Processing, Arduino, and OpenFramework. Sebastopol, CA:
O’Reilly Media.
PANGARO, P. (2009). “How Can I Put That? Applying Cybernetics to
“Conversational Media”. In: Proceedings of the Annual Meeting
of the American Society for Cybernetics, pp. 1–16.
PASK, G. (1975). Conversation, Cognition and Learning. A Cybernetic
Theory and Methodology. New York: Elsevier.
PASK, G. (1976). Conversation Theory: Applications in Education and
Epistemology. New York: Elsevier.
PASK, G. (1996). “Heinz von Foerster’s Self-Organisation, the
Progenitor of Conversation and Interaction Theories”. Systems
Research, vol. 13, no. 3, pp. 349–362. <http://dx.doi.
org/10.1002/(SICI)1099-1735(199609)13:3<349::AID-SRES1
03>3.0.CO;2-G>
PENNY, S. (2000). “Agents as Artworks: and Agent Design as Artistic
Practice”. In: K. DAUTHENHAHN (ed.). Human Cognition and Social
Agent Technology. Amsterdam, Netherlands: John Benjamins
Publishing Company, pp. 395–414. <http://simonpenny.net/texts/
agentdesign.html> <http://dx.doi.org/10.1075/aicr.19.18pen>
PENNY, S. (2011). “Towards a Performative Aesthetics of
Interactivity”. Fiberculture, no. 19, pp. 72–108. <http://nineteen.
fibreculturejournal.org/fcj-132-towards-a-performative-
aesthetics-of-interactivity/>
PENNY, S. (forthcoming). “Between rocks and hard places – Robotics
and Art, Computationalism and Embodiment”. In: D. HERATH, C.
KROOS, and S. STELARC (eds.). Robots and Art: Exploring an
Unlikely Symbiosis. Springer.
PICKERING, A. (1995). The Mangle of Practice: Time, Agency, and
Science. Chicago: The University of Chicago Press. <http://dx.doi.
org/10.7208/chicago/9780226668253.001.0001>
PICKERING, A. (2002). “Cybernetics and the Mangle: Ashby, Beer and
Pask”. Social Studies of Science, vol. 32, no. 3, pp. 413–437.
<http://dx.doi.org/10.1177/0306312702032003003>
PICKERING, A. (2008). “Ontological Theatre. Gordon Pask, Cybernetics,
and the Arts”. Cybernetis And Human Knowing, vol. 14, no. 4, pp.
43–57. <http://dx.doi.org/10.1177/1461444809336511>
PICKERING, A. (2010). The Cybernetic Brain. Sketches of Another
Future. Chicago: The University of Chicago Press.
QUIRING, O. (2009). “What do users associate with “interactivity”?: A
qualitative study on user schemata”. New Media & Society, vol. 11,
no. 6, pp. 899–920. <http://doi.org/10.1177/1461444809336511>
RAFAELI, S. (1988). “Interactivity: From new media to communication”.
Sage Annual Review of Communication Research: Advancing
Communication Science, no. 16, pp. 110–134.
RAFAELI, S., SUNDWEEKS, F. (1997). “Networked Interactivity”. Journal
of Computer-Mediated Communication, vol. 2, no. 4, pp. 0–0.
<http://dx.doi.org/10.1111/j.1083-6101.1997.tb00201.x>
RIBAS, J. I. (2001). “Difusió cultural i comunicació audiovisual
interactiva”. Temes de Disseny, no. 18, pp. 170–204. <http://
tdd.elisava.net/coleccion/18/ribas/view?set_language=ca>
SOLER-ADILLON, J. (2011). “Creating Black Boxes: Emergence
and Interactive Art”. In: Proceedings of the 17th International
Conference on Electronic Art. <http://isea2011.sabanciuniv.edu/
paper/creating-black-boxes-emergence-interactive-art>
SOLER-ADILLON, J. (2012). Principios de diseño de interacción
para sistemas interactivos. <http://repositori.upf.edu/
handle/10230/21513>
SOLER-ADILLON, J. (2015). Emergence as Self-Organization and as
Generation of Novelty. A Framework for Understanding Emergence
in the Context of Interactive Art. Universitat Pompeu Fabra.
SOLER-ADILLON, J.; PENNY, S. (2014). Self-organization and novelty:
pre-configurations of emergence in early British Cybernetics. In:
Norbert Wiener in the 21st Century (21CW), 2014 IEEE Conference
on. Boston, MA: IEEE, pp. 1-8. <http://doi.org/10.1109/
NORBERT.2014.6893946>
STEELS, L. (1995). “The Artificial Life Roots of Artificial Intelligence”.
In: C. G. LANGTON (ed.). Artificial life. An Overview1. Cambridge,
Massachussetts: MIT Press, pp. 75-110. <http://dx.doi.
org/10.1080/01972240490508072>
SUNDAR, S. S. (2004). “Theorizing Interactivity’s Effects”.
The Information Society, vol. 20, no. 5, pp. 385–389.
<http://doi.org/10.1080/01972240490508072>
WALTER, W. G. (1950). “An Imitation of Life”. Scientific American,
vol. 182, no. 5, pp. 42–45. <http://dx.doi.org/10.1038/
scientificamerican0550-42>
WEBER, R., BEHR, K.-M., DEMARTINO, C. (2014). “Measuring
Interactivity in Video Games”. Communication Methods and
Measures, vol. 8, no. 2, pp. 79–115. <http://doi.org/10.1080/19
312458.2013.873778>
WIENER, N. (1989). The Human Use of Human Beings. Cybernetics
and Society. London: Free Association Books.
WILSON, S. (1994). “The aestheteics and practice of designing
interactive computer events”. SIGGRAPH ’94 Visual Proceedings.
ACM Press.
WU, G. (1999). “Perceived interactivity and attitude toward web sites”.
Proceedings of the 1999 Conference of the American Academy
of Advertising.

A UOC scientific e-journalArtnodes, no. 16 (2015) I ISSN 1695-5951
2015 by FUOC
CC
E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY
2015 by FUOC
CC
Sergio Martínez Luna
FUOC, 2015
CC
52
CC
http://artnodes.uoc.edu
artnodes
Universitat Oberta de Catalunya
The intangible material of interactive art…
Joan Soler-Adillon
2015 by FUOC
CC
Joan Soler-Adillon
Universitat Pompeu Fabra
http://orcid.org/0000-0002-7959-4192
joan.soler@upf.edu
Universitat Pompeu Fabra
Campus de la Comunicació-Poblenou
Roc Boronat, 138 Desp. 52.803
08018 Barcelona
Graduate in Philosophy from the Universitat Autònoma of
Barcelona, Master’s degree from New York University’s Interactive
Telecommunications Program (Tisch School of the Arts) and Official
Master’s Degree in Cognitive Systems and Interactive Media from
Universitat Pompeu Fabra. He holds a PhD in Social Communication from
Universitat Pompeu Fabra, awarded for his thesis entitled “Emergence
as Self-Organization and as Generation of Novelty. A Framework for
Understanding Emergence in the Context of Interactive Art”. At UPF he is
a lecturer and researcher in the Interactive Communication Area of the
Communication Department since 2006 and member of the DIGIDOC
research group since 2009. He has directed the Digital Video Graduate
Program, and lectured at the Digital Art’s Master and Online Digital
Documentation Master. He has also lectured in the Computer Science
undergraduate degree and in Universidad San Francisco de Quito, in
Ecuador. He has participated in interactive installation, performance and
video projects, which have been shown in New York and Barcelona, as
well as several workshops and conferences. His research is centered
on the concept of emergence and its relation to interactive media and
art, and also on the analysis and creation of interactive documentaries
and experimental interactive work.
CV
Recommended citation
SOLER-ADILLON, JOAN (2015). “The intangible material of interactive art: agency, behavior and
emergence”. In: Pau ALSINA and Ana RODRÍGUEZ GRANELL (coord.). “Art Matters II”. Artnodes.
no. 16, pp. 43-52. UOC [Accessed: dd/mm/yy].
<http://journals.uoc.edu/ojs/index.php/artnodes/article/view/n16-soler/n16-soler-pdf-en>
<http://dx.doi.org/10.7238/a.v0i16.2744>
This article is – unless indicated otherwise – covered by the Creative Commons Spain Attribution 3.0
licence. You may copy, distribute, transmit and adapt the work, provided you attribute it (authorship,
journal name, publisher) in the manner specified by the author(s) or licensor(s). The full text of the
licence can be consulted here: http://creativecommons.org/licenses/by/3.0/es/deed.en.