Content uploaded by Fabio Morreale
Author content
All content in this area was uploaded by Fabio Morreale on Jan 25, 2017
Content may be subject to copyright.
RENEGOTIATING RESPONSIBILITIES IN
HUMAN-COMPUTER ENSEMBLES
Fabio Morreale
Raul Masu
Centre for Digital Music, EECS
Queen Mary University of London
f.morreale@qmul.ac.uk
interAction Lab
University of Trento
raul.masu@unitn.it
Figure 1. Chimney, the software used as a probe to reflect on new scenarios of human-machine collaborations.
ABSTRACT
This paper proposes discussions for the (re)negotiation
of responsibilities in performances that involve a collab-
oration between human and computer agents. Most of
current research is human-hierarchical, leaving the ma-
chine the role of a mere tool at performer’s service. In
this paper a more balanced distribution of responsibilities
between the two agent is proposed. Chimney, a software
developed by one of the authors, is proposed as a design
probe to reflect on this topic. Chimney allows the com-
poser to control only the musical material of a piece,
leaving its evolution to an algorithmic agent whose deci-
sions are unpredictable. This redistribution of responsi-
bilities results in a compositional shift that causes the
roles of the composer / performer to overlap. The impli-
cations for the experience of the performer operating
Chimney are also discussed.
1. INTRODUCTION
At the dawn of the last century, the digital revolution of-
fered musicians and researchers the possibility to explore
new forms of musical creativity using computers. Be-
sides offering unprecedented interaction possibilities
thanks to novel instruments and interfaces, musicians
started experimenting with new compositional strategies
by detaching human responsibilities to the compositional
process, which were partially or completely delegated to
computers. Such strategies had been extensively ex-
plored by George Lewis, one of the pioneer musical
pieces that “emerges from a nonhuman intelligence” [1].
Copyright: © 2016 Morreale an d Masu. This is an open-access article
distributed under the terms of the Creative Commons Attribution License
3.0 Unported, which permits unrestricted use, distribution, and repro-
duction in any medium, provided the original author and source are cre-
dited.
In his work, he explored performance interactions be-
tween improvising musicians and computer programs
that create new music material at performance time.
The work described in this paper keeps up with the
encounter between human and algorithmic agents to col-
laboratively operate to create new music. In particular,
we propose discussions about the role of the machine in
such hybrid ensemble. Traditionally, computer agents
have been relegated as tools to serve the performer. We
here advocate a more equal redistribution of the roles of
the two agents. Chimney, a software that shares the con-
trol over the performance between the musician and the
computer, is presented as a probe for discussion. In
Chimney the musician-part of the control over the com-
position is reduced to (i) selecting the musical material
to be played during the execution; and (ii) deciding their
likelihood of being played temporally closer to each
other.
To emphasise the non-hierarchical nature of the en-
semble, a visual identity was assigned to the computer
agent. An algorithmic random walker unsystematically
roams throughout the screen: as it encounters an object,
the sound associated to that specific object starts playing
with a sound level which is proportional to its distance to
the walker. The human agent can only decide the objects
to be placed on the canvas, their position, and their size.
Under these conditions, the musician cannot organise a
temporal structure, which is entirely controlled by the al-
gorithmic agent.
Such renegotiation of responsibilities results in two
principal compositional shifts: (i) the music is no longer
organised according to the phraseological-temporal
structure; (ii) the sound objects become the focus of the
composition. As a consequence, the system fosters the
musician to elaborate new compositional strategies.
It is worth remarking that the scope of interest of this
paper lies beyond music and art domains. Chimney be-
longs to a recent corpus of work of the authors which
aims at as well as other works from the authors [4] [5]
that encompass reflections on philosophical and cultural
concerns. Our pieces are probes that pose questions and
speculate about possible future scenarios of computa-
tional art. For instance, what would be the consequences
of (partially) delegating artistic creation to an autono-
mous agent? How would the role of art and the artist
change?
The remainder of the paper is structured as follows.
The next section presents the technical details of Chim-
ney; Section 3 describes the computational shift that a
composer needs to face when utilising this system; Sec-
tion 4 presents Alinearity, a piece written by one of the
authors for Chimney and a trumpet; Section 5 discusses
more insights about the experience of the trumpet player
that interacted with the system.
2. CHIMNEY
Chimney is a multi-platform open source application de-
veloped with Processing1. Quite an efforts was made to
1 https://processing.org
build a strong visual identity to the algorithm for two rea-
sons: (i) to provide the algorithmic agent a character,
thus emphasising the absence of hierarchy between the
two agents; (ii) to help the audience understand the per-
formance. The visual interface is composed of a canvas
that displays the status of the algorithmic walker and the
sonic material.
2.1 The random walker: the fly
The canvas is initially blank with the exception of a par-
ticle - the random walker - that moves throughout the
screen. The idea was to give the walker a behaviour that
recalled that of a fly. For this reason, we based the fly on
an adapted version of the Perlin Noise [6]. The Perlin
Noise is a random generator function originally devel-
oped to produce natural looking textures on computer
generated surfaces. This function produces a more natu-
ral, harmonic succession of numbers than that of pure
random functions.
To the original Perlin Noise function we added a few
features to match a our requirements. In particular, the fly
had:
1. to cover the entire space of the canvas;
2. to be independent of the presence or absence of
other objects in the canvas;
3. to have non-deterministic movements: the path
taken by the fly cannot be predicted nor controlled
in any possibly ways by the musician.
Figure 2 shows a possible path taken by the fly over a
period of two minutes.
Figure 2. A possible trajectory of the fly.
2.2 The compositional material: sound circles
The performer can interact with the canvas by adding
sound sources to it. These sound sources are displayed as
circles (Figure 1), whose placement and radius can be
controlled by the user interacting with the mouse. Once
positioned in the canvas, the performer can interact with
it by changing its size, by repositioning it and by deleting
it. Every time a new circle is added to the canvas, the
system sets it in idle state and mutes it. As the fly enters
a circle, the sound connected to that circle increases. The
maximum level is reached when the fly is at the centre of
the circle.
The sound sources are made of pre-recorded excerpts
stored as audio files. The user can access these files
through a sidebar that lists all the sound files contained
in a specific folder (Figure 1, on the left). Once selected
a new sound source, the next circle to be positioned in
the canvas will be associated to that sound source. A col-
our code helps identifying what sound each circle is as-
sociated to.
3. COMPOSITIONAL SHIFT
The theoretical ideas that lie behind the compositional
approach to Chimney was based on the early approach to
composition and improvisation of John Cage. In particu-
lar, during his first improvisation period (following the
Feisst’s analysis of Cages’ work [2]), Cage categorised
compositional process as a fourfold activity composed
of: structure, material, method, and form. The structure
is the temporal division of the sections, the material con-
sists of the sound objects (including noise and silence),
the method is the “note to note procedure”, and the form
is the “morphological line of the sound-continuity”. Fol-
lowing Cage’s opinion, material and methods can be both
improvised and composed. Form cannot be composed
but only improvised, as it results from the sum of the
other activities. Finally, structure is the only element that
cannot be improvised.
Having control on the structure allows a composer to
be in charge of the evolution of the piece. In fact, he can
move back and forth on the music timeline. Sarath for-
mulated this concept proposing the term expanding tem-
porality of a compositional process [7]. Given that the
composer does not work at performance time, the musi-
cal decisions are not directly influenced by the previous
events. The composer can freeze time, re-think passages,
and anticipate future musical phrases. As a result, the de-
velopment of the piece is based not only on previous
events, but also on the anticipation of the material to be
presented at successive moments. A different kind of
temporality is the inner directed temporality [7], a con-
cept proposed to describe the evolution typical of im-
provisation. When improvising, a musician cannot pre-
cisely anticipate how the music will develop. Thus, im-
provisers can mostly focus their attention on the present,
with little attention to the near past and future.
Chimney offers the composer the possibility to rede-
fine the edge of composing and improvising. Given that
the temporal evolution of the piece is not under his or her
control, the structure can no longer be composed. Chim-
ney restrains human control over the development of the
piece as the composer cannot step back and forth and re-
compose a particular event. As the music is shaped at run
time, the performance itself represents the final compo-
sitional process. As a consequence, performing with
Chimney requires an inner directed temporality ap-
proach. However, at the same, it is not pure improvisa-
tion: despite the freedom to take real time decisions the
performer cannot improvise the actual musical material.
The musical material itself is the element that bounds
performance to composition. The musical gesture, lim-
ited to organising pre-composed music, is composed out-
2 https://cycling74.com/products/max/
side before the performance, thus it is closer to the con-
cept of expanding temporality. The musical material be-
comes the very core of the piece and can be seen as a
meta composition of the music, which ultimately gener-
ates at performance time.
The method touches both the compositional and the
improvisational aspects of music making. It is involved
both in the creation of the sound sources and the strate-
gies adopted to place them on the canvas. The form and
the structure are both determined during the perfor-
mance, and consequently are improvised elements.
4. ALINEARITY
This section exemplifies the innovative aspects of Chim-
ney by presenting Alinearity, a piece of music composed
for Chimney by one of the authors. Alinearity was com-
posed to provide the performer with the necessary degree
of freedom to properly interact with Chimney while
maintaining a degree of harmonic coherence. The piece
can be performed as a solo piece for Chimney or as a duo
for Chimney and a melodic instrument. Alinearity was
performed in 2015 at the International Society for Impro-
vised Music Festival (Château-d'Oex, Switzerland) in a
version for Chimney and 10 string electric guitar and in
2016 at the International Conference on the Design of
Cooperative Systems 2016 (Trento, Italy) for Chimney
and Trumpet.
4.1 Description of the piece
The most important aspect of the compositional process
adopted for Alinearity is the creation of sound objects as
the main elements of the piece. The musical form is not
determined by the development of musical material mov-
ing toward a specific section. Thus, there are no cadences
and the material itself does not evolve. The development
of the piece is based on the overlapping of different
sound excerpts that create different situations. The musi-
cal form is shaped by the very transit of the fly through
different cluster of sounds (the circles in the canvas).
This transit can be gradual or sudden, depending on the
behaviour or the fly.
The musical material was composed to be congruous
and at the same time to open a wide range of possible
combinations that guarantee coherence and expressive-
ness to the piece. It consists of 20 short monodic musical
excerpts, each with a timbral, melodic, harmonic, and
rhythmic value pre-recorded using a synthesiser made
with Max-Msp2. The synthesiser is controlled with stand-
ard MIDI messages.
4.2 Harmony
The harmony of the excerpts is organised as a politonal-
ity over two tonality centres: Gmin and C#. The note of
the excerpts can either belong to one specific scale or to
both scales (C, which enharmonically corresponds to B#,
F#, and Bb, which enharmonically corresponds to A#).
More specifically, the musical excerpts are harmonically
organised as follows:
• 7 excerpts belonging to Gmin;
• 7 excerpts belonging to C#;
• 6 excerpts belonging to both tones.
Their combination can create situations with distinct key
notes, polytonal harmonies, or situations with a modal
appearance that lacks a clear tonic centre.
4.3 Melody and rhythm
From a melodic point of view, the excerpts are clustered
into three sets. They can be long pedal notes, very short
patterns, and short themes:
• 6 long pedal notes;
• 6 short melodies;
• 8 short patterns.
The harmony of the six long pedal note belongs to both
tonal centres. The short patterns have sharper rhythm and
can create particularly complex polyrhythmic structures.
4.4 Timbre and register
The excerpts spread across five octaves. Long notes have
low register and short notes have high register. As a con-
sequence, the six long pedal notes are the lowest register,
the six short melodies lays in the medium register, and
the eight short patterns have the highest register.
The core of the synthesiser is a bank of eight resonant
band pass filters, which processes white noise in input.
The central frequencies of the filters are composed of a
fundamental tone and its seven higher harmonic partials.
The resulting sound can be accurately manipulated by in-
teracting with its nosiness and brightness, as well as its
envelope. Lower notes have a slow transients and a high
level of nosiness. Higher notes have fast transients and
low nosiness.
5. EXAMINING EXPERIENCES
Alinearity can be performed as a solo interface or in a
duo with a melodic instrument. This sections analyses the
different experiences of the musicians who (a) performs
Chimney, and (b) plays the melodic instrument playing
along Chimney.
5.1 Performing Chimney
Given that Chimney has been mainly operated by one of
the authors, the following discussion is based on self-
analysis of the experience. When using Chimney, com-
position and performance are superimposed. The emer-
gence of this hybrid composer-performer was thoroughly
described by Vallis [8][9]. The implications of this su-
perimposition on the iteration strategies adopted during
the performance are substantial. Four main gestures, or
approaches to interaction, with Chimney emerged.
Placing excerpts and spectating the movements of the fly
In this first approach, the performer does not consider the
movements of the fly at all. The gesture is guided by his
knowledge of the musical material, thus by placing the
excerpts on the canvas. The activities of the human per-
former have no effects on the musical output, who is rel-
egated to being a spectator of the performer. Depending
3 An excerpt of the exhibition can be found at
https://youtu.be/8-zVa1tN3Ak
on the movements of the fly, the actions of the musician
can also fail to influence the music at all as the fly could
potentially never reach the circles. From a musical per-
spective, this approach is the closest approach to tradi-
tional composition. Indeed, it operates in an expanded
temporality, and all the performers’ activities result in
long term effects. This approach is typically adopted at
the beginning of a performance.
Fostering a gradual passage to new situations
In this approach, the performer positions in the canvas a
limited amount of excerpts and follows the movement of
the fly. As a consequence, the control is more balanced
between the human and the algorithm. The performer fol-
lows the changes introduced by the transit of the fly to a
new circle by adding or subtracting elements from the
canvas. This approach reduces the time spanning to a
more localised present. The performer can focus on the
previous events and on the changings that are occurring.
The time window could be fairly wide but always centred
on the present.
Following the fly to increase music complexity
In this approach, the performer closely follows the move-
ment of the fly by adding circles to its trajectory. The hu-
man has the highest control over the composition, thus
the stochastic aspect is consequently less influent. As the
performer gesture are closely related to the movement of
the fly the temporarily is more focused on the present,
similarly to the inner directed case earlier described
(Section 3). This approach is normally adopted to create
local musical climax and results in a complex structure.
Such complexity results from the number of overlapping
excerpts that is achieved by adding several circles that
follow the fly. The outcome is a much more complex tex-
ture articulated by a bigger number of musical objects.
Furthermore, the music becomes richer and the quantity
of rest decreases.
Removing all the musical excerpts
Chimney consents to delete all the excerpts that are visi-
ble on the canvas at once. In this case, the musical gesture
is similar to what typically occurs with traditional instru-
ments: to one action corresponds one musical outcome.
The temporality is focused on the very time of the key
pressing, but is influenced by recent developments in the
music. This kind of interaction normally occurs only
once during a performance at the end of the last climax.
The sudden rest that results from this gesture resolves the
tension and gently goes towards the end of the piece.
When Alinearity was performed as a duo piece for Chim-
ney and Trumpet at COOP 2016 in Trento (Figure 3)3,
peculiar human-human interactions occurred. The shift-
ing among the different approaches was determined both
by the personal taste on the musical output as well as by
the input received by the other musician. The trumpeter
sometimes proposes shifting that did not follow the
movement of the fly. The operator of Chimney - one of
the authors of the paper - who was controlling Chimney
was thereby required to pay careful attention on the trum-
pet. In these case, gradual transits to new situations were
not determined by the fly movements but rather by the
other musician.
Figure 3. A trumpeter improvising along to Chimney.
5.2 Improvising along to Chimney
This section describes the trumpeter experience of play-
ing along to Chimney. The trumpeter had no score to fol-
low and was instructed about how the musical excerpts
were composed. He was also given some time rehearsing
the piece to become familiar with the software and the
musical material.
After the concert, he was asked to elaborate on his
experience of playing along to Chimney. The objective
was to understand his musical strategies and to collect
reflections about the interaction among the three agents
(the trumpeter, Chimney, and the performer of Chim-
ney).
The trumpeter reported that understanding all the de-
tails of the music played by Chimney was particularly
demanding. In particular, as opposed to traditional im-
provisation, he did not feel free to propose new musical
material. He was always answering to Chimney pro-
posals. Each note, or phrase, was indeed primarily based
on listening. As a result, his gestures were mostly about
imitating, completing, or opposing what Chimney pro-
posed.
From a musical point of view, the trumpeter was
mostly influenced by harmony and rhythms proposed by
Chimney. The approaches he adopted can be clustered
into three main categories:
• using exactly the same notes of the excerpts;
• using the same scale of the excerpts;
• moving gradually from one scale to the other.
The rhythm could be influenced by the single pattern or
by the overlapping of more than one. In the first case, the
excerpts themselves determined the rhythmical result. In
the second case, he was instead mostly influenced by the
movement of the fly. With respect to the form, the trum-
peter tended to follow the output of Chimney, enhancing
his rhythm and loudness coherently with the overlapping
of more excerpts.
When prompted to discuss his relation with the two
other agents, the trumpeter reported that he experienced
two distinct interactions. He perceived to be in closer
contact with Chimney as he felt he was duet-ting with it.
The performer controlling Chimney was only the access
to ask something to the computer agent. Only in a few
cases the main connection was between the two humans.
This was particularly the case when the performer was
following the fly or when he cleared the canvas. In these
cases, Chimney was considered closer to a musical in-
strument.
He also commented that he would have preferred a
higher complexity in the music generated by Chimney.
In those situations, the trumpeter compensated for this
perceived deficiency by increasing the rhythm complex-
ity or the loudness of his instrument. To conclude, the
inherent limitation of Chimney of having a limited mate-
rial and being non-responsive fostered the trumpeter to
find novel musical strategies.
6. CONCLUSIONS
Chimney was mainly developed as a design probe to re-
flect on the consequences of delegating part of the com-
positional process to an algorithmic agent. In this con-
text, a number of negotiations between the human and
the computer agent have to take place. The implication
of such negotiations are evident from a musical perspec-
tive and from the performer’s experience alike. With re-
spect to the musical perspective, the focus of composi-
tional effort shifts to the interaction strategies more than
the form, and a number of compositional decisions are
left to performance time. As a consequence, the distinc-
tion between composition and improvisation, which is
neat in traditional performances, becomes blurred.
The disruption of built-in hierarchies of human-
leads-computer-follows also caused a series of implica-
tions in the experience of the performer who is control-
ling Chimney and of other musicians that play along to
it. Whereas some of these implications can be attributed
to the non-responsive nature of the system, others are at-
tributable to the distinctive character of the interaction.
In particular, both Chimney performer and the musician
that improvises along to it, are withhold the possibility to
control all the aspects of the performance. Freed by a pre-
cise control on musical structure and declined the possi-
bilities to have veto buttons, they need to surrender a
more balanced collaboration with the algorithmic coun-
terpart.
7. ACKNOWLEDGEMENTS
We are grateful to Riccardo Terrin for having performed
Alinearity, and to the organisers of the COOP and ISIM
conference.
8. REFERENCES
[1] D. Charles. For the Birds. Boston: Marion Boyars,
1981.
[2] S. M. Feisst. John Cage and Improvisation: An Un-
resolved Relationship. Musical Improvisation: Art,
Education, and Society: 38-51. 2009.
[3] G. E. Lewis. “Interacting with Latter-Day Musical
Automata”. Contemporary Music Review 18(3):
99–112, 1999
[4] R. Masu, A. Conci, Z. Menestrina, F. Morreale, A.
De Angeli. “Beatfield: An Open-Meaning Audio-
visual Exploration”, 2016
[5] F. Morreale. “Generative Everything 2083”. In Pro-
ceedings of the XVIII Generative Art Conference,
2015
[6] K. Perlin. "An image synthesizer." ACM Siggraph
Computer Graphics 19, no. 3: 287-296, 1985
[7] Sarath, Ed. "A new look at improvisation." Journal
of Music Theory (1996): 1-38.
[8] O. S. Vallis, Contemporary approaches to live
computer music: the evolution of the performer
composer. Victoria University of Wellington, 2013.
[9] O. S. Vallis, A. Kapur. "Community-based design:
The democratization of musical interface
construction." Leonardo music journal 21, 2011:
29-34, 2011.
