Content uploaded by Charles Patrick Martin
Author content
All content in this area was uploaded by Charles Patrick Martin on Dec 08, 2015
Content may be subject to copyright.
CREATING MOBILE COMPUTER MUSIC FOR PERCUSSIONISTS:
SNOW MUSIC
Charles Martin
cpm@charlesmartin.com.au
Luleå University of Technology
Department of Arts, Communication
and Education
ABSTRACT
This paper traces the development of an Apple iPad
and iPhone based musical instrument, Snow Music, and
its use in performances by Ensemble Evolution, a
percussion group formed by the author and two other
percussionists.
This artistic research is motivated by a desire to use
computer based instruments along with percussion in a
group with non-programmer performers. The aims were
to develop an elegant, portable, and flexible computer
instrument that is accessible to the group and to find out
what opportunities it enables in performance practice.
A project was undertaken to develop Snow Music,
using Pure Data and libpd, and to collaboratively
compose a musical work. The process was documented
with video recordings and analysed using ethnographic
methods.
The resulting setup was extremely portable and
convenient for rehearsal and performance. A design that
emphasised “percussive” interaction with the instrument
and an improvised performance practice contributed to a
collaborative development cycle. Analysis of
performances gave insight into the limitations and
affordances of mobile computer music devices.
1. INTRODUCTION
Starting in 2010, I co-founded a percussion group,
Ensemble Evolution1 in Piteå, Sweden with Jacob
Remington and Maria Finkelmeier. The goal of the
group was to explore projects in composition and
improvisation. As a percussionist and computer
musician, one of my goals was to integrate computer
based instruments into this group. As the other members
of the group had no experience with computer music it
became clear to me that I needed new computer music
tools to complement percussion instruments in my work
as a music creator, collaborator and performer.
Inspired by recent developments in computer music
on mobile devices, I decided to use Apple’s iPad and
iPhone, as platforms for computer music in Ensemble
Evolution. I thought that these devices could work better
in this situation that the laptop based computer systems I
had used previously.
1 Performance and other information about Ensemble Evolution
can be found at http://www.ensemble-evolution.com
There are several previous works that inspired this
research. Schiemer and Havryliv’s work, Pocket
Gamelan [6], exploits the portability of mobile devices
by having non-expert performers swing the devices on a
cord in performance. Tanaka’s four-hand iPhone
performances demonstrate the affordances of sensors
built into mobile devices through a collaborative
improvised performance practice [7].
For Ensemble Evolution, it was important that the
mobile devices were portable enough to easily fit into
our percussion setups and to take on tour. The interesting
new interfaces for musical interaction, such as touch
screens, could be useful for engaging non-programmer
percussionists in performance.
The research project presented in this paper was
designed to study the process of producing a new iOS2
based instrument to be used alongside percussion
instruments. The project also studied the performance
practice that emerged from collaboratively creating a
musical work using this instrument. Over the course of
the project, both the musical work and the iOS app were
called Snow Music.
This project is part of a larger study that is the topic of
my master’s thesis Mobile Computer Music for
Percussionists, completed in June 2012 at Luleå
University of Technology [3]. In addition to the project
described in this paper, the thesis includes a study of a
mobile computer music system for vibraphone and an
examination of the current state of research into mobile
computer music.
2. RESEARCH QUESTIONS
The goal of the project was to address four research
questions about mobile computer music:
1. Heaviness. Can computer music setups be
made more simple, elegant and convenient
using mobile devices?
2. Shareability. How can mobile computer music
instruments be made accessible to a non-
programmer percussion ensemble, and what
creative processes can be used to explore
them?
2 iOS is the operating system that runs on Apple’s iPhone, iPad
and iPod Touch. It is possible to produce “universal” iOS apps
that can run on all of these devices.
3. Playability. How can the affordances of mobile
music devices be used to create playable
instruments for percussionists?
4. Performance practice. What new performance
practices are enabled or demanded when
complementing acoustic percussion
instruments with mobile music devices?
3. RESEARCH METHOD
The research questions of this project were addressed
qualitatively through ethnographic3 analysis of the
process of developing, rehearsing and performing the
collaborative work with Ensemble Evolution. These
processes were documented through written notes and
video recordings. Analysis of the video documentation
was made through a process of coding, following
techniques described in Krüger’s Ethnography in the
Performing Arts [2]. In this process, key-concepts are
chosen and assigned to events occurring in the video.
Events can then be grouped by key-concept to make
connections between the various rehearsals and
performances. The process for coding verbal information
is well understood, conversations can be transcribed and
then coded easily in this form. Performances were coded
following Östersjö’s method of working directly from
the video [4], using time codes to identify events.
The artistic investigation was pursued experimentally
with the choice of hardware and software elements
informed by current trends in computer music research.
3.1. libpd: Pure Data in iOS
The iOS-based instrument for this project was developed
as a native iOS app, using libpd [1], a general project to
port Pure Data [5] to a number of computing platforms.
libpd is an effort to separate the audio synthesis parts of
Pure Data from its graphical programming interface.
Pure Data can then be used as a synthesis library for
other applications.
In this project, libpd allowed the musical elements of
the app to be composed in the familiar environment of
Pure Data while the user interface was created in
objective-C using XCode.
4. RESULTS
In order to capture the creative process in developing
Snow Music, every rehearsal, and a series of initial
public performances was video recorded. The rehearsals
took place in February 2012 in Piteå, Sweden, while the
performances occurred in March 2012 in Australia.
3 Ethnography is a qualitative research method for studying
cultural phenomena. The researcher conducts fieldwork to
collect notes, audio and video recordings, and images relating to
the phenomena, an active and subjective method. “The open-
ended nature of the ethnographic approach is particularly
suitable for active discovery and exploration” (Krüger [2]).
Analysis of the data can be an iterative process with multiple
phases of field work and analysis to refine the research question.
Conclusions are drawn inductively from the data gathered.
Over the first three rehearsals, I introduced the Snow
Music app to the other performers. The app was designed
to give the performers “percussive” control of field
recordings from the harsh winter in Piteå. The early
rehearsals were marked by changing ideas about the
musical content of the work and discovery of how to
play the iOS devices.
In the fourth rehearsal, it was agreed that the
performance would be a free improvisation, Snow Music,
motivated by the computer sounds. Each player would
use an iOS device and a keyboard percussion instrument.
The iOS devices were each connected directly to a
powered loudspeaker.
There were three public performances of the work in
Australia, all including collaborations with performers
outside of Ensemble Evolution. The first was at
Melbourne Conservatory and included Nat Grant, a
percussionist and computer musician. The second, in
Serial Space, Sydney, took the form of a group
improvisation with members of Nomad Percussion. The
final performance was part of an iPad music workshop
for high school students at Canberra Grammar School.
4.1. The Snow Music app
The final iOS app for Snow Music has two main
functions. The first is a touch interface that creates snow
sounds when the screen is tapped or swiped with a
finger. The dynamic of tapped sounds can be controlled
by the distance of a tap from the centre of the screen.
The dynamics of swiped sounds corresponds to the speed
of the swipe. The second function was a set of three
background soundscapes, algorithmic processes that
produce phrases of notes in free rhythm. The three
soundscapes are of bells, cymbals, and a swooshing,
wind-like, snow sound. The bells were set to use notes
only from the C aeolian scale while the cymbals and
snow sounds were of indeterminate but changing pitch.
The overall dynamic of the sounds could be controlled
using the iOS devices’ hardware volume controls. In the
first rehearsal, only a prototype of the snow sounds was
available. All sounds had been developed by the fourth
rehearsal.
4.2. Rehearsing and Developing Snow Music
4.2.1. Problems
A number of problems came up repeatedly over the
rehearsal process. Most of the technical problems related
to situations where an interaction with the app did not
align with the player’s expectation. In the first rehearsal,
Maria remarked that “the interaction was throwing me
off a bit”. Managing the expected volume of the iOS
devices through the loudspeakers was also a problem.
The musical problems were related to the clarity of
expression that was possible with the app. Initially, it
was difficult to express clear rhythms with the iOS
devices and our early attempts to play precise, repetitive
rhythms with the snow sounds were not very successful.
Also, the sounds initially produced by the iOS devices
had a lack of timbral clarity. Jacob commented that the
app felt “limited” and its available sounds just “sounded
too similar” in rehearsal two.
4.2.2. Solutions
Fortunately, these problems could be addressed. In some
cases, all that was needed was some more explanation
from me, as my ideas about programming the interaction
between the performers’ gestures on the touch screen
and the sound output were rapidly evolving.
The interactive concept came down to using
“percussion-y technique”. Common techniques for using
hands to play percussion instruments – taps, finger rolls,
scraping and rubbing – also work on the iOS devices’
touch screens. Another simple explanation was to use the
hardware volume control on the devices to adjust for
unexpected volume levels. Finding these techniques was
an important part of developing a performance practice
with the iOS devices.
The musical problems were also addressed through
updates to the app made in between rehearsals in
response to suggestions and inspiration from the group.
For example, the group suggested using more mimetic
sounds from Piteå. In response, I added the sound of
footsteps in snow and the wind that blows around the
student housing.
4.3. Performing Snow Music
To understand the performance practice that we
developed for Snow Music, three representative
performances from different stages of the project were
chosen and coded. These were: Our first4 attempt at
using the iOS app with three keyboard percussion
instruments; a free improvisation5 in the fourth rehearsal
with a guest, Matteo Spano, playing guitar (see Figure
1); and lastly, a public concert6 in Melbourne with a
guest artist, Nat Grant, performing on laptop. The results
can be categorized as concerning techniques for playing
the iOS devices, musical affordances and musical
applications of the devices in performance.
Figure 1. Ensemble Evolution and Matteo Spano rehearsing
Snow Music.
4 First Snow Music performance with mallet percussion
instruments: http://youtu.be/W091aaKSIU4
5 Snow Music performance along with Matteo Spano:
http://youtu.be/MWicLWySxdI
6 Snow Music performance at Melbourne Conservatory along with
Nat Grant: http://youtu.be/Vz5aQxc_jlE
4.3.1. Techniques
Since all of the interactions for the Snow Music app
occurred through the touch screen we needed to use skin
contact to activate it. Although it is possible to hold
mallets in our hands touch the devices with the heel of
our hands, or the little finger, we generally played the
iOS devices without simultaneously trying to hold
mallets. This encouraged us to also play the keyboard
percussion instruments with our fingers, an intimate, but
striking sound. It also turned out to be possible to hold
mallets in one hand and play the iOS device with the
other.
One of the limitations of the iOS app was in
expressing clear rhythm. The performers adopted several
techniques to address this issue. One strategy was using
“finger rolls” to express continuous rather than singular
sounds. Other experiments included exaggerated swipes
across the whole iPad screen.
4.3.2. Musical affordances
The three background sound generators (bells, cymbals
and “snow-wind”) were added to the app in response to
the discussions in rehearsals. It is clear from the
performance videos that the musical affordances of these
sounds inspired their use during our improvisation. The
bell sound was by far the most used. The clear tonal
centre and texture cut through other sounds in our
improvisation. The group responded to these affordances
by using the sound to trigger sections of tonal
improvisation in free rhythm.
The other background sounds, cymbals and snow
wind, were not used as frequently. Where they did
appear, their musical function was to create a dense
texture that would support various styles of
improvisation. The Melbourne performance had a much
denser texture due to the fourth player on laptop and as a
result, these sounds were not activated as frequently.
4.3.3. Musical application
Over the rehearsal process, the sounds we could create in
the Snow Music app came to define the improvised piece
we were developing. Even in the earliest performances,
we started the piece with a “snow collage” by tapping
the screens. In the later performances, this element
became better defined. The sounds of footsteps in snow
were strongly mimetic and gave a distinctive theme to
the piece.
It is interesting to see how the Snow Music app was
used to change the intensity of the improvisation. While
the opening “snow collage” built intensity, the app was
mostly used to decrease intensity. My use of the app at
the end of the Melbourne performance triggered a
closing “snow collage”.
The ability to present a “new idea” to the ensemble is
critical in free improvisation. Significantly, the “bells”
background sound was useful for driving the
improvisation into a particular direction.
5. CONCLUSIONS
Overall, the Snow Music project was successful in
introducing computer instruments to Ensemble Evolution
and resulted in extremely satisfying performance results.
The investigation uncovered conclusions to address all
four research questions, which are briefly summarized
here and discussed in more detail in my thesis [3].
5.1. Heaviness
The setup for Snow Music was extremely simple using
only the three iOS devices and audio cables. In our
rehearsal studio we used three powered speakers for
performing while on tour we used a compact mixer to
distribute the sound to two loudspeakers.
The light weight of the equipment and short set up
time greatly contributed to the agility of our rehearsal
process and performances on tour. The whole
ensemble’s computer music setup (not counting
loudspeakers) could fit in one backpack.
5.2. Shareability
Snow Music was successful in putting computer based
musical instruments in the hands of non-programmers.
The performances demonstrate all the performers using
the iOS devices confidently and musically. I received an
unexpected level of input and engagement from Jacob
and Maria during the rehearsals. Suggestions from the
group directly connected to many of the technical and
musical problems with the app. Importantly, the input
from the group helped me to focus my programming in
between rehearsals to address the most pressing musical
issues.
5.3. Playability
The interactive focus in Snow Music was the touch
screens of the iOS devices. The rehearsal process
allowed us to uncover some of the problems encountered
when playing these instruments. The most frustrating
moments in the rehearsals were caused by a disconnect
between gesture and result. These issues were able to be
resolved through experience and fine tuning the app.
5.4. Performance Practice
The Snow Music project resulted in an investigation into
free improvisation performance practice combining iOS
devices and acoustic instruments. The group discovered
that the field recording based sounds from the Snow
Music app could define and motivate the piece and
helped to give structure and meaning to our
improvisation.
Analysis of the performances makes some of the
constraints of the Snow Music app clear. Although many
of the sounds are distinctive and allow a degree of
precise playing, some were subtle and mainly textural.
There were limits to the extent that these sounds could
be used in a precise rhythmic context. While this is
probably not “bad”, it is important to understand such
limits and decide what impact they have on the
compositional process.
6. FUTURE DIRECTIONS
The rapid, collaborative development process that was
achieved for Snow Music can serve as a model for future
projects. While musical interactions with the app had
limitations, it may be that techniques can be found to
address these. Future projects could also benefit from an
investigation into how mobile music apps could be used
to contribute “new ideas” in an improvised performance.
7. ACKNOWLEDGEMENTS
This research was supported by the Department of Arts,
Communication and Education at Luleå University of
Technology as part of my master’s studies there. My
thesis was supervised by Stefan Östersjö and Anders
Åstrand.
8. REFERENCES
[1] Brinkmann, P., Kirn, P., Lawler, R., McCormick,
C., Roth, M., & Steiner, H.-C. “Embedding Pure
Data with libpd,” in Pure Data Convention.
Weimar: Faculty of Media, Bauhaus-Universität
Weimar, 2011.
[2] Krüger, S. Ethnography in the Performing Arts: A
Student Guide. 2008, Lancaster, UK: Palatine.
[3] Martin, C. Mobile Computer Music for
Percussionists. 2012, Master’s thesis, Department
of Arts, Communication and Education, Luleå
University of Technology.
[4] Östersjö, S. SHUT UP ‘N’ PLAY! Negotiating the
Musical Work. 2008, Doctoral dissertation, Malmö
School of Music, Lund University.
[5] Puckette, M. 1996. “Pure Data: Another Integrated
Computer Music Environment,” in Proceedings of
the Second Intercollege Computer Music Concerts,
Tachikawa, Tokyo, Japan: Kunitachi College of
Music, pp. 37-41.
[6] Schiemer, G., & Havryliv, M. “Pocket Gamelan:
Interactive Mobile Music Performance,” in
Proceedings of Mobility Conference 2007: The 4th
International Conference on Mobile Technology,
Applications and Systems: IS-CHI 2007, Singapore:
Research Publishing, 2007, pp. 716-719.
[7] Tanaka, A. “Mapping out instruments, affordances,
and mobiles,” in Proceedings of the International
Conference on New Interfaces for Musical
Expression, Sydney, Australia: University of
Technology Sydney, 2010, pp. 88-93.
