ArticlePDF Available

Gesture Sonification. An Interaction Design Approach to an Artistic Research Case

Authors:
Article

Gesture Sonification. An Interaction Design Approach to an Artistic Research Case

Abstract and Figures

INTRODUCTION: The subject of this paper is a phenomenological study of cognitively plausible relationships between gesture and sound mediated by technologies. In this discussion, the term ‘gesture’ indicates a movement of the body consciously performed and able to express or communicate something. The expression ‘cognitively plausible’ refers to an interactive sound response that is immediate, continuously varying and enactively coherent with the generating (sonified) gesture. OBJECTIVES: The objectives of this paper are two. The first is an artistic exploration of gesture sonifications in various contexts. The second is rather methodological, i.e., to provide a possible general paradigm for Artistic Research (AR). METHODS: More in detail, the AR phenomenological approach is modeled according to an Interaction Design (IxD) research paradigm. RESULTS: A number of case studies of gesture sonification are presented and discussed according to the above methodological framework. CONCLUSION: We claim that the introduction of such methodological framework was successful in terms of providing robust guidelines for our research and for a clear and structured presentation of its results.
Content may be subject to copyright.
EAI Endorsed Transactions
on Creative Technologies Research Article
1
Gesture Sonification.
An Interaction Design Approach to an Artistic Research
Case
P. Polotti1 and M. Goina2
1Department of New Musical Languages - Conservatory of Music G. Tartini, Trieste, Italy
2Fondazione Teatro Lirico Giuseppe Verdi, Trieste, Italy
Abstract
INTRODUCTION: The subject of this paper is a phenomenological study of cognitively plausible2 relationships between
gesture and sound mediated by technologies. In this discussion, the term gesture indicates a movement of the body
consciously performed and able to express or communicate something. The expression cognitively plausiblerefers to an
interactive sound response that is immediate, continuously varying and enactively3 coherent with the generating (sonified)
gesture.
OBJECTIVES: The objectives of this paper are two. The first is an artistic exploration of gesture sonifications4 in various
contexts. The second is rather methodological, i.e., to provide a possible general paradigm for Artistic Research5 (AR).
METHODS: More in detail, the AR phenomenological approach is modeled according to an Interaction Design (IxD)
research paradigm.
RESULTS: A number of case studies of gesture sonification are presented and discussed according to the above
methodological framework.
CONCLUSION: We claim that the introduction of such methodological framework was successful in terms of providing
robust guidelines for our research and for a clear and structured presentation of its results.
Keywords: sonic interaction design, basic design, artistic research, sonification, embodiment, enaction.
Received on 24 November 2020, accepted on 03 March 2021, published on 19 March 2021
Copyright © 2021 P. Polotti et al., licensed to EAI. This is an open access article distributed under the terms of the Creative
Commons Attribution license, which permits unlimited use, distribution and reproduction in any medium so long as the original
work is properly cited.
doi: 10.4108/eai.19-3-2021.169036
1 Corresponding author. Email: pietro.polotti@conts.it
2 The concept of “cognitive plausibility” is here freely used, and does not refer in a strict sense to the term employed by psychologists to assess the accuracy
with which models are believed to represent human cognition. It is rather borrowed by other disciplines such as linguistics [1] or geography [2].
3 By enaction, we mean here rapid, incremental and reversible human operations with immediate perceptual feedback [3], [4].
4 For the definition and the concepts around the term ‘sonification’ we refer to [5].
5 About the definition and scope of AR see [6], [7], [8] and the Vienna Declaration published on June 26. 2020 and available at
https://cultureactioneurope.org/news/vienna-declaration-on-artistic-research/ (consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
2
1. Introduction
In general, we view art as a means to explore and propose
new perspectives for comprehending the world. We also
think that artistic investigation can parallel scientific and
humanitarian research in terms of construction of evidence,
case studies, counterexamples and critique of questioned
ideas. In our case, we take into consideration the field of
Sonic Interaction Design (SID) [9], [10], adopting the
concept of interaction with sound as a possible reference
point for cooperation between art, science and technology.
We also believe that embracing design methodologies in art
could be an effective way to foster a dialogue and a mutual
exchange between artistic and scientific researchers.
1.1. HCI Epistemological Revolution
In order to face the new challenges posed by an Information
and Communication Technology (ICT) society, the
necessity of promoting a wide-ranging interdisciplinary
research effort from the humanities to the natural sciences
has clearly emerged over the past decades [11], [12]. The
epistemological revolution introduced by Human Computer
Interaction (HCI) [13] and Artificial Intelligence (AI) [14]
provides examples of this. When technology was employed
only within the scope of the industrial revolution in the
epistemological framework of a positivist philosophy of
science [15], the complexity of human psychology, social
behavior and culture was oversimplified, if not neglected.
On the contrary, with fast-evolving digital means and the
computer undergoing what Brey calls the passage from an
epistemic role to an ontic role [13], it becomes necessary to
conceive a technological development according to
culturally, psychologically and anthropologically ergonomic
criteria. Although, AI is gaining importance in interaction
with sound and music as well [16], in our work, AI is not
involved at all. The focus of this paper is on HCI
body/perception aspects rather than on brain/intelligence
issues.
In this sense, it is also remarkable as the name of a
discipline born in a computer science framework, such as
HCI, is gradually changing its denomination into Interaction
Design (IxD), dropping the term machine and introducing
the concept of design with all its ambiguities [17], [18]. In
particular, IxD can be thought of as a discipline, which is in
middle of natural sciences, humanities, technology and art.
As Stolterman says “interaction design research has for
some decades developed theoretical approaches, methods,
tools, and techniques aimed at supporting interaction
designers in their practice” and “many of them have
intellectual roots in other academic areas, such as science,
engineering, social science, humanities, and in the
traditional art and design disciplines” [19].
In the debate within the HCI/IxD community, one of the
questions is if a quantitative validation of the results could
even be omitted, since in some cases it is rather meaningless
due to the kind of studies and subjects involved in the
discipline [20]. By pursuing the definition of general, simple
models and the reproducibility of results as a guarantee of
objectivity, reductionism shows its relativity as soon as the
object of study and its environment involve human factors
either psychological, social or cultural.
An example of a qualitative research approach in SID can
be found in a comprehensive work by Frauenberger and
Stockman [21]. In that paper, the authors propose to
consider design pattern analysis as a point of reference for
the discipline, introducing a method based on the definition
of context spaces explored by pattern mining. They are
aware of the pros and cons of a pattern-based approach, and
the context is meant as an organizing substrate. This would
provide designers facing new problems with useful
references to already existing IxD cases and promote the
growth of the discipline by building upon previous
knowledge in the form of qualitative descriptions and
analysis.
1.2. Artistic Research Practices for a
Sustainable Relationship with Technology
HCI investigates technology at the point of contact with
human complexity, i.e., with the non univocal nature of
human thoughts, emotions and behaviors. As a consequence,
HCI becomes a natural laboratory for an epistemological
revolution, and, at the same time, provides the challenging
playground for the development of a technology devoted to
humans (and not the opposite). The goal is to go beyond the
“computer metaphor and the related Cartesian mind-body
dualism [that] have resulted in a fairly mechanical
comprehension of the human being using a technical
device” [22], in order to develop a technology, which has to
be meaningful, ergonomic, and sustainable from many
points of view: physical, psychological, social, cultural,
ethical, and environmental.
As already claimed, we believe that art can play the role
of a laboratory for developing compelling examples and
new perspectives of comprehension of the world. A kind of
research shortcut, following the path of intuition and
creativity instead of that of strictly logic thinking. In
particular, we believe that Interactive Art (IA) is an
important actor in the development of a sustainable
relationship with technology. IA can constitute the
workbench for a free experimentation with conceiving,
employing, analyzing, interpreting, and critiquing
technology in a complementary and synergic way with
respect to IxD and HCI.
Given the relevance of AR and IA, it is then useful to
pose a methodological question: could design theory act as
point of reference for an artist that works with digital
technologies? We believe that the interconnection between
art and design can be reciprocal, i.e., a designerly approach
can be methodologically fruitful if adopted in IA (and art in
general), and facilitate a dialogue between the two fields.
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
3
2. A Design-Oriented Artistic Research
Paradigm
As a first step, we have to reconsider the value and, in some
cases, the necessity of a reductionist strategy. Some sort of
simplification has to be taken into account, in order to face
questions and problems that are complex and multifaceted.
We take as point of reference the methodological approach
of basic design as developed in post-Bauhaus design schools
(Ulm, Yale, Chicago). The main principle of basic design is
to identify fundamental categories of problems that can be
investigated by means of the proposition of design exercises
with well-defined objectives and constraints around specific
themes [23]. Basic design methods consist of analyzing
actions, extracting what Findeli [24] calls interaction
gestalts, i.e., elementary interactions and general primitives,
and designing exercises around each of them (see also [25],
[26]).
Our first claim is that making art inspired by a basic
design approach, can be a fruitful strategy for art as well.
Fundamental problems can be dealt with by numerous
alternative solutions, which in turn can be compared to
reveal and to combine multiple critical points of view.
Second, the principle of cyclic iteration of i) proposed
solutions, ii) evaluation and iii) redefinition of the solutions
according to the evaluation results can provide a strong
reference point to establish an artistic practice structured and
biased towards a systematic investigation of a problem.
Third, the fundamental design principle of going through
rapid sketching and/or realization of mock-ups offers a
powerful operating paradigm to face the unavoidable
rapidity of technological evolution. Since technology is ever
changing, it is problematic to base on it an interactive
system that is durable and repeatable. However, an
interactive artwork can be considered independent from any
particular technology, if sketches and mock-ups can play the
role of the ‘score’ of the artwork. Fourth, a design approach
involves teamwork as a connate praxis: the artwork becomes
a product of a team used to sharing ideas, plans and goals.
Working in a team on an artistic project in a design fashion
can be likened to the activity of a scientific research group,
contrary to the model of an artist as an isolated creator.
Nowadays, interactive artists are increasingly trained to
work with other people. This means a shared capacity
among interactive artists to work synergistically with people
having disciplinary skills, methodologies and goals different
from theirs.
The framework of an artistic practice inspired by a design
methodology is shown in the diagram in Figure 1. In the
artistic case, the starting idea is not the outcome of
brainstorming around consumer/stakeholder needs and/or
requirements, but a free artistic investigation. In the same
way, the validation/evaluation phase by means of user-tests
is substituted by the rehearsals and discussions with the
performers or by the observation of the reactions of the
visitors to a system exhibited in public spaces (a sort of
qualitative user test), without any quantitative evaluation as
usually required in product design.
Figure 1. A design-oriented AR methodological
framework. The multiple arrows indicate how an initial
idea is developed in different (alternative) ways and
generates different artworks.
3. Gesture Sonification from an Artistic
Research Point of View
In this section, we discuss the application of the previously
introduced methodological framework to an artistic research
project on gesture sonification. In general, we talk about
gesture rather than body movement, since we are concerned
about the expressiveness and the meaning that a gesture
gives. As already mentioned, we embrace a definition of
gesture as a body movement able to consciously or
unconsciously express or communicate something. Gesture
is a concept that one can extend also to the sonic world and
to the musical domain: musical events considered in their
parametric dimensions (pitch, duration, dynamics and
timbre) can be described and characterized according to the
temporal directionality of those parameters. One of the first
artworks on human gesture controlling digitally generated
sounds in a continuous and expressive way was A Very
Nervous System the pioneering interactive artwork by David
Rokeby (1986-1990)6.
3.1. The Design-Oriented AR Methodological
Framework Applied to Gesture Sonification
We look at design in two different ways, as argued in this
and in the next subsection. The first reference to design
regards the application of the methodology (Figure 1) to the
investigation of interactive systems used for creating
embodied and enactively coherent relationship between
gesture and ICT generated sound. As depicted in Figure 2,
we tackled the problem by defining four scenarios ranging
from an artistic and professional context to an everyday one
that can be experienced by anyone (see the alternatives in
Figure 2).
6 http://www.davidrokeby.com/vns.html (consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
4
Figure 2. Gesture sonification: an artistic research project approached in an interaction design fashion. The
names of the prototypes are the titles of the artworks discussed in detail in the section entitled Case Studies.
The four rectangles correspond to the four iteratively cycling steps or Figure 1.
Each scenario produced multiple prototypes. The
whole structure of the research is illustrated and
documented in a website7
The first scenario was developed into two sub-
scenarios, one related to the contemporary dance and the
other to contemporary circus denoted as Cirque Nouveau
and based almost solely on human skills and cross-
fertilized by other performing arts [27]. The cyclic
iterations leading to the refinement of the artistic
7 http://www.visualsonic.eu/production.html
(consulted 20.02.2021).
prototypes were mainly carried out through discussions
among the authors and the performers during the
developing phase, as well as by considering the response
of the audience, the critique by colleagues after the public
performances and the analysis of the audiovisual
documentation.
The second scenario gave birth to two versions of the
same prototype, and the refinements were due to a)
discussions among the authors with the collaborating
professional dancer or with occasional non-professional
users, and b) critique by colleagues after public
performances in international conferences.
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
5
The prototypes of the third scenario were in the form of
a public artwork, and they were the most complex, since
they also involved visual aspects. The first version was
iteratively refined according to the users' comments
during many exposure of the system to small audiences in
a portable form using a laptop and a graphic tablet as
interface. Finally, a second version was presented in the
form of a public artwork with a large video projection in
an open space allowing people to move freely and
expressively, while performing their task.
The two prototypes of the fourth scenario, with the
second being an evolution of the first, were based on the
discussions between the authors, the comments of the
users during the exhibitions and the analysis of the
audiovisual documentation after the exhibitions. The
collection and reorganization of video documentation
provided a fundamental tool for qualitative data analysis.
Even at a rudimentary level, the video documentation was
the basis for building up a corpus of compelling examples
and evidence. Each prototype will be discussed in detail in
Section 4.
3.2. Use of Elementary Gestalts: Klee’s and
Bauhaus Legacy
The second reference to design is given by Paul Klee's
work as educator during the period he was part of the
Staatliches Bauhaus project in the Weimar Republic of
Germany. A common idea of Bauhaus was the existence
of a universal, not figurative visual language, often
compared to the universal language of music [28]. Being
mainly a painter, Klee represented a paradigm of
collaboration between design and artistic practices. In
1927, Klee wrote the Pedagogical Sketchbook that was
adopted at the Bauhaus as a fundamental reference for the
module Design Theory [29]. In the Sketchbook, Klee
traced a didactic route and, in parallel to this, he
introduced the general tenets of his own artistic
investigation. For instance, Klee illustrates the conversion
of a dot from a “static element” into “linear dynamics”.
Sybil Moholy Nagy, a historian of art and architecture,
wrote in her preface to the Sketchbook how the line, being
a progression of dots, “walks, circumscribes, creates
passive-blank and active filled planes” (see Figures 3, 4
and 5).
Figure 3. A moving dot generates an active line. The
movement is circular. Drawings inspired by the
Klee’s original (1965).
Figure 4. A second example of moving dot that
generates an active line. The movement is again
circular.
Drawings inspired by the Klee’s original (1965).
Figure 5. A third example of a moving dot that
generates an active line. In this case, the movement
is piece-wise straight. Drawings inspired by the
Klee’s original (1965).
In the same way as Klee started from a dot, which
moving, creates lines and planes, we started thinking of
gestures as generated by sequences of punctual positions,
which formed configurations that grew in complexity in
different degrees. Initially, the approach was essentially
abstract. The aim was to create an interface, which
produced sounds controlled by means of gesture analysis
and recognition, and where gestures were disaggregated
into elementary trajectory segments. Elementary sounds
were coherently defined and employed for the sonification
of different elementary gesture gestalts. We were
interested in the emergence of basic gestalts, i.e.,
elementary perceptual/expressive units, composed by
gestures and sounds. Acknowledging that any connection
between gesture and technologically mediated sound is
arbitrary, the aim was to explore ways to build new
cognitively effective and meaningful relationships
between them by means of appropriate mappings. Our
initial idea was to determine a few elementary gesture
trajectories and map each of them onto a well-defined set
of sounds.
From a practical point of view, the first mobility agent
we used, corresponding to Klee’s dot, was a hand-held
light. Indeed, the movements of a light (hand-held by the
performer and optically sampled by a camera) were the
dot source forming lines and curves in a 2D space and
controlling the sound generation. The beginning, thus,
was a punctual position in space, e.g., a hand able to
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
6
generate gestures that were kinetically analyzed in terms
of trajectory elements, in order to produce convenient
sound responses.
As a first step, we segmented gestures taking into
account a limited number of components, according to
linear and curvilinear distinctions as in Klee’s examples
of Figures 3, 4, and 5. For the gesture sonification, we
used a series of sounds matching the segment categories.
The selection of the different sounds, in fact, was
controlled by means of these basic geometric principles.
Such a setup was employed in two preliminary artworks
for a dancer and a computer presented in Trieste, Italy, at
Sala Tripcovich in 2008 and in Graz, Austria, at Theater
im Palais in 20098.
3.3 Everyday and Choreographic Gesture
Sonification. The EGGS System
At a certain point in the development of the new system
for gesture sonification, we decided to call it, Elementary
Gestalts for Gesture Sonification (EGGS) [30]. EGGS
provides a system for gesture sonification, based on
configurable and redefinable elementary mappings
between gesture and sound that can be combined and
articulated with a high degree of freedom. The system is
versatile and has proven useful in performing arts,
interactive dance and public interactive installations.
Quite different from other gesture-sound mapping
strategies [31], [32], [33], we have chosen an elementary
kinematic gesture analysis, which is abstract and suitable
for dealing with general categories. In this sense, our
approach also differs from the affective analysis of
gesture adopted in other works on gesture sonification
[34], [35], [36]. Furthermore our work does not concern
gestures in musical interactions [37], [38], since sound
outputs are considered as a sonic representation of non-
musical gestures.
As explained before, we initially regarded gestures as
kinematic trajectories generated by moving points, with
the point being a marked hand, or elbow or knee, and we
established some segmentation rules based on simple
geometric criteria. The EGGS system, in fact, processes
the visual data of a moving punctual source with a
trajectory tracking routine, which returns different indexes
matched to different trajectory categories. We expanded
the system from the two initial basic categories, straight
and circular, to five categories: i) circular clockwise ii)
circular counter-clockwise iii) straight iv) direction
inversion and v) still. Additional motion parameters were
taken into account for the purpose of making the sound
feedback more perceptually coherent with gesture
evolution. These parameters are vector and scalar
velocity, vector and scalar acceleration, and absolute
position. Such attributes were necessary to go beyond an
abstract geometric approach and gain expressivity and
8 See the two last videos at http://visualsonic.eu/performance.html
(consulted 20.02.2021).
dynamic coherence between gestures and sonic feedback.
The very first simple (however effective) ideas were to
map velocity and acceleration onto sound level, and the
absolute position onto the pitch (high position meaning
high pitch and, vice-versa, low position, low pitch).
Sometimes, the absolute position was mapped onto some
audio effects by varying the timbre, mainly along the
vertical axis. Furthermore, vector quantities allowed us to
discriminate among different inclinations with respect to
the vertical axis. In some of the latest versions of the
system, we considered also angular velocity and
acceleration, spanning all of the Dynamic Interaction
Primitives (DIP) introduced in Virtual Reality during the
last decade [39].
A challenge we soon faced during the early
experiments with professional performers was that EGGS
entails learnability issues (concerning apprenticeship in
New Interfaces for Musical Expression - NIME9 research,
see [40]). However, after many iterative refinements, the
system was developed into a more immediate version,
easily usable by anybody since it could be setup so as to
output meaningful sonifications of any arbitrary and
ordinary gesture. The user-friendly issue was fundamental
in public installations that foresaw the involvement of an
audience. Even visitors with no specific training could
have a stimulating, enjoyable and satisfying experience
during the sonic-embodied interaction with the public
installations. We avoided any complex and non-
immediate mappings, such as those obtainable by storing
and recalling past gestures, i.e., by introducing memory in
the system [41].
Provided these founding criteria, the investigation on
sonification as well as the system development followed
the design principle of realizing multiple alternative
versions of the same idea through different artworks (the
prototypes). Also, similar to a design practice, where new
prototypes of a certain version depend on a cycling way
on a critical analysis of the previous results, we produced
subsequent artworks of the same scenario. The two
strategies allow us, respectively, to broaden and to refine
our knowledge about the main theme: the effectiveness of
sound as representation and continuous feedback of
expressive gestures in a multimodal sense. Here,
multimodality has to be considered both from the
audience point of view (watching the dancer, while
listening to the sound produced by the gestures) and from
the performer/user point of view (listening to the sound in
a proprioceptive way).
9 https://www.nime.org/ (consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
7
4. Case Studies
In this section, we analyze in detail the four scenarios and
the related cases corresponding to the prototypes of Figure
2. In the following, we designate the exercises as cases.
Scenario 1: Professional Performance
In case of an artistic context, when working with
professional performers, we first introduce them to the
system saying that sound is treated as a consequence and a
representation of the choreographic gesture and its
expressive content. We could conceive of the EGGS
system as a choreophone, in that the performers do not
follow a musical piece, do not conduct a musical piece,
and do not even create any music. Rather, they ‘listen’ to
their gestures, enactively managing and adjusting their
choreographic actions according to the sonic responses of
the system. Thus, sound is an auditory after-effect and an
embodied continuous feedback, able to augment the
proprioception of the performer, and in no way separable
from the gesture itself.
In the form of an exercise in a basic design fashion, the
work with professional performers can be formulated as
follows:
Case 1
Theme: Continuous sonic feedback for two
independent arms moving in a kinesthetic sphere.
Objectives:
Overcome the separation of music and dance
(sound and movement) in professional dance.
Enhance the performer’s proprioception.
Develop the idea of 'listening to gestures'.
Constraints: The sonic feedback should be
continuous, ecological sound-based [42], intuitive and
trajectory-dependent in order to provide a variety of
gesture-sound gestalts.
From this case, we created an interactive performance
for solo dancer entitled Swish ‘n’ Break (SnB)10. In SnB,
the performer held one light per hand. Each light worked
as a marker and defined the gesture. A gesture was, thus,
processed by a light tracking routine able to discriminate
between different categories of trajectories, such as
previously mentioned: straight, inversion, circular
clockwise (CW), circular counter-clockwise (CCW), as
well as different inclinations with respect to the horizontal
plane, and different curvature angles. In all of the
artworks, still corresponded to silence. Technically, the
distinction between different trajectories (gesture gestalts)
was achieved by evaluating the angle variations of the
segments connecting three consecutive pairs of detected
10 See the first video at http://visualsonic.eu/performance.html (consulted
20.02.2021).
points, which corresponded to the centripetal acceleration
(see Figures 6 and 7). A variation close to zero mapped to
a linear trajectory, while a bigger variation mapped to one
of the curvilinear categories according to the sign of the
variation.
Figure 6. Trajectory categorization criteria.
An example of small angle variation Δ designating a
straight gesture gestalt. Depending on the system
frame rate, P(tn) is the detected light position at the
discrete time tn.
Figure 7. Trajectory categorization criteria.
An example of large angle variation Δ designating a
CW curvilinear gesture gestalt. Depending on the
system frame rate, P(tn) is the detected light position
at the discrete time tn.
The coupling of the gesture dynamics with the sound
dynamics and other sound parameters added a further and
fundamental expressive layer. In Tables 1 and 2, we show
some examples of mapping strategies. According to
performer feedback after a number of rehearsals, the
outcomes were deemed immediately expressive as well as
potentially elaborate, and the system offered and elicited
unexpected solutions in terms of sound-gesture
relationships.
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
8
Table 1. Mapping examples between trajectory
gestalts and sound families.
Type of trajectory
Sonification
still
silence
diagonal (0 45 degrees)
small stick swish
vertical (45-135 degrees)
rope swish
diagonal (135-180 degrees)
wire swish
Table 2. Another mapping example between
trajectory gestalts and sound families.
Type of trajectory
Sonification
still
silence
CW
water
CCW
air
straight
fire
inversion
hammer strike on a wall
SnB was presented for the first time at the SMC 2010
conference [43]. The performance was conceived as a
controlled improvisation on a predefined score of sounds
and gestures. All of the sounds were drawn from the
Freesound11 project. A set of keywords were established
in advance, in order to retrieve and define a number of
sound families. More in detail, following a programmatic
compositional approach, the keywords determined the
three sections: 1) Swish, 2) Air Water Fire Earth
[the basic elements of nature] and 3) Break. The central
section 2), the most elaborate, from a sonic point of view,
was developed in terms of a gradual change from an
outdoor natural soundscape to an artificial indoor one. In
each section, we opted for different sound-trajectory
mappings (see Tables 1 and 2 illustrating the mapping of
the first and the second sections, respectively). Moreover,
within fixed constraints, the laptop performers could
modulate the sound timbre as well as other parameters of
the system responsiveness, thus creating a dialogue with
the dancer.
The final output was a mostly predefined sonic-
choreographic score, agreed upon by the three authors
after many rehearsals. During the rehearsals, the dancer
explored the potentialities of the system sonic feedback
and selected, adjusted and refined the gesture accordingly.
11 www.freesound.org (consulted 20.02.2021).
Reciprocally, the dancer learned how to manage and
exploit the sound materials through the gesture. Any
decision about the development and refinement of both
the performance and the system, were agreed upon by all
of the three through discussions and experimentation
during the rehearsals. Typically, different selections,
combinations and concatenations of gesture-sound
mappings were tested, aiming at an optimization of the
final result, in accordance with the creative methodology
previously discussed and illustrated in Figure 2.
According to basic design practices, the same theme
with a different set of constraints may make perfect sense
as a different case. This happens in Case 2, where we
changed the constraints concerning the trajectory
dependence.
Case 2
Constraints: With respect to Case 1, we waived the
distinction between straight, CW and CCW
trajectories (while maintaining the discrimination
among the inclinations) in order to explore a less
geometrically constrained gesture and a simplified
gesture-sound mapping.
Case 2 resulted in a performance entitled un-pLugged
pLoden, a Ligament Lento. The performance was in the
same style as SnB and was presented at the International
Computer Music Conference ICMC-2012 in Ljubljana12.
The sounds utilized in un-pLugged pLoden were
retrieved from the above mentioned Freesound project
using the keywords slowdown, decreasing and braking.
As in SnB, the keywords defined the global structure of
the artwork. Its form, in fact, was characterized by a
constant and gradual deceleration beginning with
explosive sounds produced by large decelerating engines,
then decelerating train rumbles, next car braking sounds,
finally faint bike brake whistles.
The next two cases were the outcome of
experimentation in a contemporary circus performing
context.
Case 3
Theme: Discrete sonic feedback for four independent
limb movements in a free space.
Objectives:
Overcome the separation of sound and gesture in
acrobatic exercises exercise in a professional
contemporary circus context.
Enhance the acrobat's proprioception.
Develop the idea of 'listening to gestures'.
Constraints: The sonic feedback should be discrete,
speech-based, intuitive and (occasionally) limb-
dependent.
12 See the program online:
http://www.irzu.org/files/ICMC2012_DailySchedule.pdf p. 87
(consulted 20.02.2021), and the second video at
http://visualsonic.eu/performance.html (consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
9
We started to work with professional contemporary
circus artists for the first time in 2013. The research was
funded by a grant of the Lerici Foundation of Stockholm,
within the Gynoïdes Project and in the context of a
collaboration between the KTH Royal Institute of
Technology and the DOCH University of Dance and
Circus in Stockholm [44]. After the first test with circus
artists of different disciplines, we decided to move to
Wireless Inertial Measurement Units (WIMU) for motion
tracking purposes, relinquishing the optical system
previously employed. In fact, it was clear that light bulbs
used as optical markers as in dance performances (Cases 1
and 2) were not convenient in the circus environment.
Glass fragility is a severe limitation. In circus practice,
movements can be very broad and powerful, and include
most body parts touching the ground or tools. Bulbs break
easily, becoming useless as optical markers and potential
health hazards. Another good reason for using a WIMU-
based motion tracking system instead of a single camera-
based optical one was the complete freedom of
movements around the stage for the circus artist, and the
independence from lighting conditions. Also, the
experimentation with WIMUs led to the introduction of a
new category of elementary gesture in the EGGS
repertoire, rotation around one axis, a movement whose
velocity value was given directly by the gyroscope of the
WIMU.
The first WIMU-based circus performance was called
CyborgAcrobat13. In this performance, the artist, a free
body acrobat, performed stylized and mechanical gestures
sonified by recorded excerpts of her own voice reading an
ironic text. Four WIMUs were placed on the each limb of
the performer (see Figure 8), and an audio fragment
(elementary sonic gestalt) of a pre-recorded text was
triggered each time the rotation velocity value on one axis
of one of the WIMUs exceeded a certain threshold. The
text was composed and recorded by the performer herself,
and then segmented into single words or longer phrases,
the constrain being that every single audio text unit had to
be understandable. During the performance, these
elementary audio fragments could be selected in three
different modalities: normal order, random order, and one
that considered only four predefined words triggered by
four predefined gestures, one for each limb. During the
performance, the three modalities were changed several
times by the laptop performer, following a choreographic
score. Tracked gestures were forearm extensions, and leg
flexions and rotations. Compared to Case 1 and 2, where
the 2D projections of straight and curvilinear trajectories
in the camera visual field were considered, gestures in this
case were invariant in the 3D space, because their rotation
axes were integral to the WIMUs.
After the performance, we collected the impressions
from the audience and had discussions with the performer
in order to evaluate the outcome. Our conclusion was that
using a discrete sonic feedback was a good solution
13 See the third video at http://visualsonic.eu/performance.html
(consulted 20.02.2021).
specifically for audio text. According to the performer,
this kind of gesture-controlled triggering of speech
fragments was effective, simple, clear and successful in
generating sonically complex and interesting solutions.
Several comments from the audience confirmed that this
kind of sonification was effective, because the gesture-
sound relationships were understandable and enjoyable.
Additionally, the speech fragments enhanced the
expressivity by adding a semantic layer to the multimodal
perception and to the cognitive aspects involved in the
performance.
Figure 8. The acrobat in action (Case 3).
Case 4
Theme: Continuous sonic feedback for tool-motion
sonifications.
Objectives:
Overcome the separation of sound and gesture in
acrobatic exercises, in a professional contemporary
circus context.
Enhance the acrobat's proprioception and
responsive listening.
Constraints: The sonic feedback should be
continuous, ecological sound-based, intuitive.
Although the Case 3 outcome was positive, after initial
experimentation with Case 4, the following basic design
exercise, it soon became clear that a discrete, speech-
based sonic feedback would not be the optimal solution in
any circus discipline. Case 4, titled Sonified Wheel14, was
different in various aspects: the artist, sensors-free, used a
tool, a Cyr Wheel. The Cyr Wheel is a large metal wheel,
in which performers can stand forming a single unit with
the wheel and roll around the stage [44]. Two WIMUs
were fixed on the internal side of the Cyr Wheel, (see
Figure 9). The sonic feedback was chosen to be
continuous and based on everyday sounds. In
correspondence with the wheel shape and motion, we
decided to use a continuous and cyclic sonic feedback,
such as that of sound loops. The loops were triggered by
14 See the fourth video at http://visualsonic.eu/performance.html
(consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
10
the lateral and longitudinal orientation of the wheel, and
the playback velocity was modified by the rotation
velocity at the triggering times. During the performance,
different types of sounds were used by the laptop
performer following a predetermined score. In a range that
went from human to mechanical sounds, the sound
sequence was: human breath, wind, old scanner, techno
loop and industrial loop. All the sounds were taken from
the already mentioned Freesound project.
Differently from the other cases treated in this article,
sounds were controlled by the motion of a tool, and not
directly by body movements. Indeed, this was an object
motion sonification and not a gesture sonification.
However, every movement of the Cyr Wheel depended on
the artist's gestures, when held, and from the Cyr Wheel
inertia, when released, so that this can be regarded as a
gesture-mediated sonification.
Based on the public performances, discussions with the
performer and several comments from the audience, we
can conclude that the sonification adopted in Case 4 was
expressively effective, but the gesture-sound relationships
were less clear to the audience than in the other cases.
This was probably due to the complexity of some of the
sonic loops, resulting in sonic feedback in some parts less
adherent to the motion. For the same reason, a very
careful listening is required to the performer, in order to
be able to responsively control the sonic feedback, and
hence Case 4 can be considered a good exercise for sound
exploration and a success for enhancing the performer
enactive listening.
Figure 9. The acrobat in action (Case 4).
Scenario 2: Professional and Non
Professional Disco-Club Dance
The second scenario considered a different kind of
performance/dance, which could also involve non-
professional dancers, for example, people having fun in
clubs.
Case 5
Theme: Continuous and discrete sonic feedback for
two independent arms moving in a kinesthetic sphere.
Objectives:
Overcome the separation of music and dance
(sound and movement) for trained club dancers.
Enhance the dancer's proprioception.
Introduce a flavor of electroacoustic music in
disco club culture.
Constraints: The feedback should be discrete,
musical, intuitive, and multimodal (sonic and visual).
From this case, we created a performance, entitled
Body Jockey15 that was presented in a concert at the
international conference NIME 2011 in Oslo [45]. In Body
Jockey, the idea was to introduce embodiment in
Electronic Dance Music (EDM). The hardware setup was
a readaptation of that of SnB. The software for the sound
control was completely rewritten. The Freesound project
was once more the source of the major part of the sounds.
During the performance, the dancer triggered sounds
through movements (see Figure 10), while the two
electronic performers acted as a DJ and a VJ, changing
sound and video mappings, respectively. In Oslo, the
visual part consisted of a graphical representation of the
dancer’s gestures projected on a big screen at the back of
the stage. The aim was to obtain a multimodal experience
both for the dancer and the audience. As in the previous
cases, the purpose was also to generate a tight relationship
between the laptop performers and the dancer’s actions.
The overall structure of the performance was fixed. On
the other hand, within each of its sections, the musical and
visual discourse was developed by means of controlled
improvisation. Sounds, graphics, mappings and beat
changed in every section. The first section was based on a
regular beat and percussive sounds. Trajectory inversion
was the elementary gesture gestalt employed to trigger
sounds. The main rhythmic patterns were based on the
usual EDM even meters. The volume of each sound was
constant, while the dynamic changes were obtained by
varying the sound event density. The central section was
more free-style and based on sustained sounds that were
not beat constrained, somewhat in a SnB fashion. In fact,
the elementary gesture gestalts illustrated in Case 1, such
as the straight and circular trajectories, were employed to
modulate the sound parameters. The final section was a
reprise of the first one in an EDM style.
The ultimate goal was to produce an enhanced disco-
club event, where body, music and video provided a
multimodal, however indivisible, experience. By means of
the gesture sonification and visualization, the dancer
seemed to embody the music and, according to the
commentaries gathered after the performance, this was
clearly perceived also by the audience.
15 See the first video at http://visualsonic.eu/discoclub.html (consulted
20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
11
Figure 10. The performer in action (Case 5).
Case 6
Theme: Discrete sonic feedback for two independent
arm movements in a kinesthetic sphere.
Objectives:
Overcome the separation of music and dance
(sound and movement) in contemporary untrained
club dancing.
Enhance the dancer's proprioception.
Constraints: The feedback should be discrete,
musical sound-based, intuitive.
Case 6 was mainly the product of the critique of
various colleagues after the performance at NIME 2011.
The resulting performance was Body Jockey-2, a
completely re-composed version of that presented in Oslo.
The video feedback was dropped, since it was judged
redundant and distracted from the musical contribution of
the choreographic action. Also, the hybrid music style,
mixing EDM and Electroacoustic music, that
characterized the first version of Body Jockey was
critiqued as non-suitable for untrained club dancing. Thus,
Body Jockey-2 was conceived in a pure EDM style and it
was presented at the international conference ICMC-SMC
2014 in Athens16.
We tested the system with non-professional users too,
who gave univocally very positive feedback. Even if the
tests were not performed in a real club, everybody judged
the experience as exciting and involving.
16 See the second video at http://visualsonic.eu/discoclub.html (consulted
20.02.2021).
Scenario 3: Public, Pseudo-Artistic and
Multimodal
Case 7
Theme: Continuous sonic and visual feedback for
hand-drawing movements on a surface.
Objectives:
Fuse proprioception, sonic and visual enactive
feedback into a unique self-representation.
Develop the idea of 'listening to gestures'.
Constraints: The sonic and visual feedback should
be continuous, intuitive and trajectory dependent; the
visual feedback should also be abstract.
The outcome of this scenario was Visual Sonic
Enaction (VSE)17, an interactive and multimodal
installation aimed at creating audiovisual representations
of the gestural expressivity of interacting visitors. VSE
was initially presented as a system for painting on a
computer screen guided by sounds. In fact, sound
stimulated and drove the movements of the visitor’s hand
using a wireless pencil on an external tablet interface, thus
producing an embodied, multimodal and continuous
feedback to the hand gesture. Indeed, sound worked as the
pivot component of the visual, auditory and
proprioceptive elements of VSE.
Three groups of basic sounds and three families of
elementary graphic icons were selected and used for the
sonification and visualization of two fundamental
categories of gesture gestalts: circular and straight. For
each visual-sonic drawing, the user could employ only
one sonic group. On the other hand, within a single
drawing, the user could change graphic families at any
moment and any number of times. The three graphic
families were schematically depicted in the bottom-right
corner of the VSE canvas shown in Figure 11, while the
icons in the top-right corner represented the three sound
groups: i) metallic and tinkling samples, ii) low pitch
Frequency Modulation (FM) synthesis-generated sounds,
and iii) crystal tinning sounds synthesized by means of the
Sound Design Toolkit (SDT) physics-based sound models
[46].
The three groups of sounds were experimented with
individually in three separate visual-sonic drawings. Only
when the user saved an existing drawing and started the
next one from scratch, was it then possible to proceed to
the next sound group. In VSE, we implemented various
types of mappings, more or less modulated in timbre and
other parameters in correspondence to the gesture
classification. Some mappings were based on a distinction
between straight and circular trajectories, while others
depended continuously on the trajectory curvature angle.
Similar mappings were applied to the graphic parameters.
This resulted in a significant differentiation among the
nine possible combinations of sonic and graphic families.
17 http://visualsonic.eu/publicart.html (consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
12
When presenting VSE, users were told that the aim was
not to paint. Rather, what appeared on the computer
screen was a visualization of their hand gesture
expressivity. In this way, the visual feedback enactively
guided the visitors’ gesture differently according to the
different graphic families. Employing distinct graphic
families was a crucial strategy for allowing people to
distinguish between achieving self-awareness of a gesture
and the mere act of painting. In addition, it uncoupled a
graphic family from a particular sonic group. The same
gesture, in fact, could produce completely different,
however coherent, graphic feedback, given a certain sonic
feedback. Indeed, according to Michel Chion’s definition
of audio-vision [47], any association of audio and moving
images creates a complex and independent object
belonging to a third multimodal dimension, i.e., a kind of
perceptual and cognitive ‘vector product’. Moreover, in
VSE, a concurrent production of sounds and graphics as a
consequence of a single gesture analysis gave an
impression of coherence in the arbitrary juxtaposition of
basic sonic and graphic families unified by the same body
action.
VSE was proposed to different users in a studio
environment and it was refined by means of many cyclic
iterations of observations of the users’ behavior,
discussions with the users after the experience as well as
discussions among the authors.
Case 8
Theme: Continuous sonic and visual feedback for one
arm movements in a kinesthetic sphere.
Objectives:
Fuse proprioception, sonic and visual enactive
feedback into a unique self-representation.
Develop the idea of 'listening to gestures'.
Constraints: The sonic and visual feedback should be
continuous, intuitive and trajectory dependent; the
visual feedback should also be abstract.
Case 8 represents an extension of Case 7. The
system was installed in a public location, where a video
art exhibition was taking place. It was introduced to the
audience by means of a graffiti painting metaphor: we let
the visitors freely ‘painton a wide wall using an electric
torch, which simulated a spray can. The same graphic and
sound materials and algorithms were adopted from Case
7. The torch light was detected by a camera as in Case 1
and 2. The pictures on the wall were video projections
from the computer. Moreover, the users could freely
choose the different graphics by shaking three colored
interactive bottles available on a small table beside the
painting area. The bottles were augmented by means of
wireless accelerometers. The interacting user wore
wireless headphones, which provided a more intimate and
immersive experience. At the end of the experience, the
visitors could take away their abstract visual-sonic ‘self
portrait’ in the form of an audio-video file containing their
gestural expressivity. Furthermore, bystanders could
watch and listen to the bodily expressivity of the
interactive visitor both during the event and afterwards, on
the website, where the visual-sonic self-portraits were
uploaded. Among the audience, not everybody but many
were able to understand very quickly the aim of VSE and
strove to listen and watch their gesture as represented by
the various sonic/graphic combinations. Some were able
to reach a good coherence among all of the aspects
involved (see, for instance, the outcomes by Serena18).
Figure 11 reports the graphical part of one of the visual-
sonic self-portrait.
Figure 11. The final image of a visual-sonic self-
portrait. On the right side, the icons represent the
sound groups (top) and the graphics families
(bottom).
The video documentation is fundamental for the
analysis of the visitors’ responses. On the website, we
uploaded a short excerpt of some of the users’
performances. For example, it is interesting to notice the
case of Fabiola, a professional painter, who first hesitated
and painted with small gestures, and then, when she
discovered the sonification effect, her gestures became
broader and more embodied.
Scenario 4: Everyday Life
We explored the main theme of this study also in an
everyday context. The particular subject we investigated
was the sonification of gait expressiveness. As Marcel
Mauss [48] has already observed, gestures such as
standing, sitting, or walking are important vehicles of
communication. With this in mind, we formulated the
following two case studies.
Case 9
Theme: Continuous sonic feedback for two
independent legs walking or running along a straight
path.
18http://visualsonic.eu/serena.html (consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
13
Objectives:
Fuse proprioception and sonic feedback into a
unique gait self-representation.
Enhance ecological sound listening awareness.
Develop the idea of 'listening to everyday
gestures'.
Constraints: The feedback should be continuous,
ecological sound-based, intuitive.
This scenario was firstly explored by means of a public
interactive installation titled Sonic Walking (SW)19 and
presented at the European Night of Researchers in Trieste,
Italy, in 2010. With respect to VSE, we shifted the
attention from a creative movement to an everyday action
and from the upper limbs to the lower ones. In fact,
visitors walked along a linear path in an ordinary space.
The objective was the gait sonification through ecological
sounds. More precisely, the sounds were related to the
basic elements of nature, i.e., air, fire, earth and water. In
particular, we chose the sound of a strong wind, the sound
of a roaring fire, and, for the earth, the sound of a rain
stick containing grains. For the water, we chose two
different sounds: gentle waves on the seaside and an
underwater sound. The audience experimented with the
five sounds in a pre-established sequence: water, earth,
fire, air, underwater.
When introducing the system to the visitors, they were
told that they would listen to their walking and that their
walking would first patter in water, then crunch in the
sand, crackle in the fire, blow like the wind and, finally,
submerge deep underwater. The interacting visitors could
move along an approximately nine-meter-long track,
wearing two small lamps fixed to the exterior sides of
their knees. In this way, every one of the two cameras
positioned on the two sides of the track could separately
detect each of the lights. Furthermore, the interacting
visitors used wireless headphones, which provided a more
intimate and immersive experience of their body
movements, represented by a continuous sonic feedback.
The sounds were also amplified by four loudspeakers
positioned at the far ends of the track, so that the
bystanders could listen to the gait sonification of the
interacting visitor. Differently from the previous
scenarios, in Sonic Walking there was no trajectory
analysis, and the sonic feedback was driven only by the
dynamic characteristics of the gait.
The emotional response of the visitors is well
represented in the video documentation: one adult woman
was almost dancing, while a child was almost scared by
the fire sound. On the other hand, a second child was
clearly engaged in the ear-driven action of swimming
underwater. It is significant that we were not able to
engage any adult man among the hundreds of visitors to
the event: only women and children of both sexes.
19 See the first video at http://visualsonic.eu/everydaylife.html (consulted
20.02.2021).
Case 10
Theme: Continuous sonic feedback for two
independent legs of two people moving in a free
area.
Objectives:
Fuse proprioception and sonic feedback into a
unique gait self-representation.
Enhance ecological sound listening awareness.
Elicit social behaviors and non-verbal
communication through sound and gesture
expressiveness.
Develop the idea of 'listening to everyday
gestures'.
Constraints: The feedback should be continuous,
ecological sound-based, intuitive.
Case 10 was presented at the international FKL
symposium on soundscape, in Florence in 2011. It
represents a further development of SW after discussion
between the authors. This time, the public installation
involved two visitors at a time, engaging sonic dialogues
through walking. The socialization potentialities of the
tool are well described by the short video20 that shows
two users employing the system for the first time.
5. Discussion
During the past years, AR has become an important topic
at different levels throughout academia. Different
stakeholders in politics and academia are enhancing the
role of creativity in research by means of funding
strategies and policy objectives. In the last two decades,
many university of the arts have opened throughout
Europe. Often, this was a direct outcome of political
strategies that encouraged aggregations and collaboration
among partners/faculties/stakeholders from different
artistic fields, ranging from visual arts to music and
extending in some cases to design. One of the main
challenges was, and is currently, to define research
paradigms that involve practitioners, and develop PhD
programs endowed with proper assessment criteria,
procedures and standards21.
One of the main points of reference for AR is the
widely discussed concept of Practice-Based Research
(PBR). PBR refers to not only arts but also design, and
other disciplines. In the creative arts, PBR has been
present in academic contexts for over 35 years [49]. Many
questions and issues emerged from scholarly debates,
leading to different definitions for ‘practice’ used in
research, e.g., PBR, Practice-led research, Practice as
research (PaR), and creative arts PhD [50]. According to
Candy and Edmonds [49] PBR concerns the production of
20 See the second video at http://visualsonic.eu/everydaylife.html
(consulted 20.02.2021).
21 http://www.sharenetwork.eu/artistic-research-overview
(consulted 20.02.2021).
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
14
creative artifacts that contribute to general knowledge,
while Practice-led research “leads primarily to new
understanding about practice”.
The research methodology introduced in this paper can
apply to research in artistic practice, where the main goal
is the development of art for art's sake. However, as
argued in the introduction of this paper, our model aims at
artistic production as a means of investigating the world,
in parallel with scientific research. Art as a means of
producing evidence and questioning of facts. The aim of
AR is not to establish facts or test and confirm theories as
in scientific research rather it is intended to 'generate' new
'facts' that can help to interpret, to change perspectives
and to understand facts that are objects of systematic
scientific investigation. We think that imagination is the
common ground/asset that allows the expansion of human
awareness and knowledge both in the sciences and the
arts. In this sense, we argue that a cross-fertilization is
possible if the different 'imaginaries'/imaginations are able
to communicate and dialogue. A common language or at
least clear translation parameters between the different
languages/disciplines and methodologies needs to be
agreed. This paper goes in this direction.
Furthermore, our model, based on an IxD paradigm,
provides a general framework for this common language.
This methodological framework is not a particular product
of a specific project (as in [49]) but a conceptual paradigm
that paraphrases the epistemic structure of an
interdisciplinary field such as design. Additionally, it
represents a research paradigm that goes beyond the
circumscribed scope of PhD program development and
proposes to artists a conceptual framework for planning
and reflecting on their artworks beyond academic
contexts. As demonstrated in the ten cases discussed in
this paper, this paradigm provided a fundamental
guideline for the development of the research path that we
followed in our artistic investigation of gesture
sonification. For this reason, we argue that the ten cases
generated, in Candy and Edmonds words, “new
knowledge that can be shared and scrutinized” [49].
Another important point is the idea of artistic
production as teamwork, in opposition to the paradigm of
a single or main author. We believe that the role of artistic
production can be redefined as a shared activity, involving
common authorship, i.e., art as an outcome of multiple
minds.
Finally, we admit one of the limitations of this paper is
that it lacks effective and structured assessment
procedures. In the ten cases, the assessment was
embryonic and not systematic. No structured collection
and interpretation of output data according to some
predefined experimental protocol was performed.
Nonetheless, we argue that the discussions with peers
during conferences and the public exposure to the cases
were of extreme importance, in order to stimulate debate
and reflection within our team, and to direct each
following step of the research. Furthermore, artistic
research frameworks discussed by Candy and Edmonds
[49], though single project-related, presented detailed
assessment procedures and results. The definition of
assessment procedures could be a further development
and refinement of the methodological framework
introduced in this paper. A recent interesting example of
evaluation procedures in the context of AR was proposed
in ARCAA (Actors, Roles, Contexts, Activities, and
Artefacts) [51], based on thematic analysis borrowed from
the qualitative research methodologies adopted in
psychology, However, we should keep in mind what
Vanlee and Ysebaert said about, the importance of
allowing an assessment culture to emerge from
practitioners themselves, instead of imposing ill-suited
methods borrowed from established scientific evaluation
models” [52].
6. Conclusions
In this paper, we presented the results of several years of
work on gesture sonification. Gesture sonification was
defined as expressive body movement in space,
interactively mediated and conditioned by ICT generated
sound. The subject was investigated from many points of
view and in many contexts ranging from professional
artistic performance to everyday scenarios in the form of
public art installations. In some cases, we added a
multimodal dimension, taking into consideration gesture
visualization.
The resulting ten case studies were organized in an AR
methodological framework that takes as the point of
reference the theory and methodologies of IxD. We claim
that the introduction of such a methodological framework
was successful in terms of providing solid guidelines to
our AR work and its results organization and presentation.
The AR framework is considered the main result of this
paper.
Acknowledgements.
We are thankful to the Servizio università, ricerca e
innovazione of the Regione Friuli Venezia Giulia and the
Fondazione Cassa di Risparmio di Trieste, who supported
our work.
References
[1] Keller, F. Cognitively Plausible Models of Human
Language Processing. In: Proceedings of the 48th
Annual Meeting of the Association for Computational
Linguistics: Short Papers. July 1116; Uppsala,
Sweden; 2010. 60–67.
[2] Fabrikant, S. I., and Skupin, A. Cognitively Plausible
Information Visualization. In: J. Dykes, A.M.
MacEachren and M.-J. Kraak editors. Exploring
Geovisualization. Amsterdam: Elsevier Science, 2005.
667-90.
[3] Stewart, J., Gapenne, O., and Di Paolo, E. A. (editors).
Enaction. Toward a New Paradigm for Cognitive
Science. Cambridge, MA: MIT press, 2010.
EAI Endorsed Transactions on
Creative Technologies
Online First
Gesture Sonification. An Interaction Design Approach to an Artistic Research Case
15
[4] Read, C., and Szokolszky, A. Ecological Psychology
and Enactivism: Perceptually-Guided Action vs.
Sensation-Based Enaction. Frontiers in Psychology.
2020; 11:1270.
[5] Hermann, T., Hunt, A., and Neuhoff, J. G. (editors).
The Sonification Handbook. Berlin: Logos Publishing
House, 2011.
[6] Hannula, M., Suoranta, J., Vaden, T. Artistic Research
-Theories, Methods and Practices. Espoo, Finland:
Cosmoprint Oy, 2005.
[7] Sullivan, G. Research Acts in Art Practice. Studies in
Art Educarion. 2006; 48(1): 19-35.
[8] Candy, L. The Creative Reflective Practitioner:
Research Through Making and Practice. Abingdon,
UK: Taylor Francis, 2020.
[9] Rocchesso, D. Explorations in Sonic Interaction
Design. Berlin: Logos Publishing House, 2011.
[10] Franinović, K., and Serafin, S. (editors). Sonic
Interaction Design. Harvard, MA: MIT Press, 2013.
[11] Löwgren, J., Stolterman, E. Thoughtful Interaction
Design: A Design Perspective on Information
Technology. Cambridge, MA: MIT Press, 2004.
[12] Bannon, L. Reimagining HCI: Toward a More
Human-Centered Perspective. Interactions. 2011;
15(4):50-57.
[13] Brey, P. The Epistemology and Ontology of Human-
Computer Interaction. Minds and Machines. 2005;
15(3-4):383-398.
[14] Walton, D. Computational dialectic and rhetorical
invention. AI & Society. 2011; 26:317.
[15] Comte, A. A General View of Positivism, [Bridges,
J.H. (transl.)]. London, Trubner and Co, 1865.
[16] Miranda E. R. (editor). Handbook of Artificial
Intelligence for Music: Foundations, Advanced
Approaches, and Developments for Creativity.
London: Springer, 2021.
[17] Ralph, P., and Wand, Y. A Proposal for a Formal
Definition of the Design Concept. Design
Requirements Engineering: A Ten-Year Perspective.
Lecture Notes in Business Information Processing.
2009; 14(2):103-136.
[18] Krippendorff, K. The Semantic Turn; A New
Foundation for Design. Boca Ratan, London, New
York: Taylor & Francis CRC, 2006.
[19] Stolterman, E. The Nature of Design Practice and
Implications for Interaction Design Research.
International Journal of Design. 2008; 2(1):55-65.
[20] Rogers, Y. New Theoretical Approaches for Human-
Computer Interaction. Annual Review of Information
Science and Technology. 2004; 38:87-143.
[21] Frauenberger, C., and Stockman, T. Auditory display
design: An investigation of a design pattern approach.
International Journal of Human Computer Studies.
2009; 67(11):907922.
[22] Pirhonen, A. Gestures in Human-Computer Interaction
Just Another Modality? In: S. Kopp and I.
Wachsmuth editors. Gesture in Embodied
Communication and Human-Computer Interaction.
Heidelberg: Springer Verlag Berlin; 2010. 281288.
[23] Lupton, E., Phillips, J. C. Graphic Design: The New
Basics. Princeton: Architectural Press, 2008.
[24] Findeli, A. Rethinking design education for the 21st
century: Theoretical, methodological, and ethical
discussion. Design Issues. 2001; 17(1):5–17.
[25] Franinović, K. Basic interaction design for sonic
artefacts in everyday contexts. Focused - Current
Design Research Projects and Methods. 4th Swiss
Design Network Symposium; May 30-31; Bern: Swiss
Design Network; 2008. 95–112.
[26] Franinović, K. Toward basic interaction design.
Elisava Temes de Disseny Journal. Retrieved
November 20, 2020, on
https://www.raco.cat/index.php/Temes/article/view/13
6817. Barcelona: Elisava, 2009.
[27] Albrecht, E. The New American Circus. Gainesville:
University Press of Florida, 1995.
[28] Kennedy, A. Bauhaus. London: Flame Tree
Publishing, 2006.
[29] Klee, P. Pedagogical Sketchbook, [transl. Sibyl
Moholy-Nagy]. New York: Frederick A. Praeger,
1965.
[30] Goina, M., and Polotti, P. Elementary Gestalts for
Gesture Sonification. In: Proceedings of the 2008
International Conference on New Interfaces for
Musical Expression (NIME08); 2008; Genova, Italy.
150–153.
[31] Bevilacqua, F., Müller, R. and Schnell, N. MnM: a
Max/MSP mapping toolbox. In: Proceedings of the
2005 International Conference on New Interfaces for
Musical Expression, (NIME05); May 26-28;
Vancouver, Canada; 2005. 85-88.
[32] Van Nort, D. and Wanderley, M. The LoM Mapping
Toolbox for Max/MSP/Jitter. In: Proceedings of the
2006 International Computer Music Conference
ICMC 06; November 6 - 11; New Orleans, LA: 2006.
397-400.
[33] Visi, F., and Tanaka, A. Interactive Machine Learning
of Musical Gesture. Accepted manuscript, to appear as
a chapter in: E. R. Miranda editor. Handbook of
Artificial Intelligence for Music: Foundations,
Advanced Approaches, and Developments for
Creativity. Cham: Springer Nature, 2021.
[34] Camurri, A., Lagerlöf, I., and Volpe, G. Recognizing
emotion from dance movement: comparison of
spectator recognition and automated techniques.
International Journal of Human-Computer Studies.
2003; 59(1):213-225.
[35] Landry, S. G., Ryan, J. D., and Jeon, M. Design Issues
and Considerations for Dance-Based Sonification. In:
Proceedings of the 20th International Conference on
Auditory Display (ICAD-2014); June 22-25; New
York: 2014. 35-38.
[36] Giomi, A. Somatic Sonification in Dance
Performances. From the Artistic to the Perceptual and
Back. In: Proceedings of the 7th International
Conference on Movement and Computing (MOCO
’20). July 1517; Jersey City/ Virtual, NJ: 2020.
[37] Godøy, R. I., and Leman, M. editors. Musical
gestures: Sound, movement, and meaning. New York:
Routledge, 2010.
[38] Godøy, R. I., Song, M., Nymoen, K., Haugen, M. R. ,
and & Jensenius, A. R. Exploring Sound-Motion
Similarity in Musical Experience. Journal of New
Music Research. 2016; 45(3): 210-222.
[39] Richard, P., Naud, M., Inglese, F. X., Richard, E.
Modeling Dynamic Interaction in Virtual
Environments and the Evaluation of Dynamic Virtual
Fixtures. Presence: Teleoperators and Virtual
Environments. 2012; 21(3):321–337.
[40] Jordà S. Digital Instruments and Players: Part I –
Efficiency and Apprenticeship. In: Proceedings of the
2004 International Conference on New Interfaces for
EAI Endorsed Transactions on
Creative Technologies
Online First
P. Polotti and M. Goina
16
Musical Expression (NIME04), Hamamatsu, Japan:
2004. 5963.
[41] Jeon, M., Landry, S., Ryan, J. D., and Walker, J. W.
Technologies Expand Aesthetic Dimensions:
Visualization and Sonification of Embodied Penwald
Drawings. In: Arts and Technology Fourth
International Conference, ArtsIT 2014; November 10-
12; Istanbul, Turkey: LNICST; 2015; 145:69–76.
[42] Gaver, W. What in the World do we Hear? An
Ecological Approach to Auditory Event Perception.
Journal of Ecological Psychology. 1993; 5(1):129.
[43] Goina, M., Polotti, P. and Taylor, S. Swish & Break
Geschlagene-Natur. In: Concert around Freesound,
SMC 2010, 7th Sound and Music Computing
Conference; July 22, 2010; Barcelona, Spain:
Universitat Pompeu Fabra, Sala Polivalent.
[44] Elblaus, L., Goina, M., Robitaille, M. A., and Bresin,
R. Modes of Sonic Interaction in Circus: Three Proofs
of Concept. In: Proceedings of Sound and Music
Computing Conference; September 14-20; Athens,
Greece: ICMA; 2014. 1698-1706.
[45] Taylor, S., Goina, M. and Polotti, P. Body Jockey. In:
Musical Program of the International Conference on
New Interfaces for Musical Expression (NIME2011);
May 30 - June 1, 2011; Oslo, Norway: Norwegian
Academy of Music Lindeman Hall.
[46] Delle Monache, S., Polotti, P., and Rocchesso, D. A
Toolkit for Explorations in Sonic Interaction Design.
In: Proceedings of Audiomostly '10; 2010; Pitea,
Sweden. 713.
[47] Chion, M. Audio-Vision -- Sound on Screen. New
York: Columbia University Press; 1994 [Original
edition: L’audio-vision. Son et image au cinéma.
Paris: Editions Nathan; 1990].
[48] Mauss, M. Les techniques du corps. Journal de
psychologie normal et patholigique. 1935;
XXXII:271-93.
[49] Candy, L., and Edmonds, E. Practice-Based Research
in the Creative Arts, Foundations and Futures from the
Front Line. Leonardo. 2018; 51(1): 6369.
[50] Liggett, S. Positioning the Arts in the Research
Process: Perspectives from Higher Education. In: R.
Earnshaw, S. Liggett, P. Excell and D. Thalmann
editors. Technology, Design and the Arts
Opportunities and Challenges. London: Springer
Series on Cultural Computing, 2020. 9-21.
[51] Masu, R., Correia, N. N., Jurgens, S., Feitsch, J., and
Romão, T. Designing interactive sonic artefacts for
dance performance: an ecological approach. In:
Proceedings of the 15th International Conference on
Audio Mostly. September, 14-17; Graz. Austria: Audio
Mostly; 2020. 122-129.
[52] Vanlee, F., and Ysebaert, W. Disclosing and
evaluating artistic research. Journal of Data and
Information Science. 2019; 4(3): 35–54.
EAI Endorsed Transactions on
Creative Technologies
Online First
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
In this paper, we propose to consider the sonic interactions that occurs in a dance performance from an ecological perspective. In particular, we suggest using the conceptual models of artefact ecology and design space. As a case study, we present a work developed during a two weeks artistic residency in collaboration between a sound designer, one choreographer, and two dancers. During the residency both an interactive sound artefact based on a motion capture system, and a dance performance were developed. We present the ecology of an interactive sound artefact developed for the dance performance, with the objective to analyse how the ecology of multiple actors relate themselves to the interactive artefact.
Article
Full-text available
Ecological Psychology and Enactivism both challenge representationist cognitive science, but the two approaches have only begun to engage in dialogue. Further conceptual clarification is required in which differences are as important as common ground. This paper enters the dialogue by focusing on important differences. After a brief account of the parallel histories of Ecological Psychology and Enactivism, we cover incompatibility between them regarding their theories of sensation and perception. First, we show how and why in ecological theory perception is, crutially, not based on sensation. We elucidate this idea by examining the biological roots of work in the two fields, concentrating on Gibson and Varela and Maturana. We expound an ecological critique of any sensation based approach to perception by detailing two topics: classic retinal image theories and perception in single-celled organisms. The second main point emphasizes the importance of the idea of organism-environment mutuality and its difference from structural coupling of sensations and motor behavior. We point out how ecological—phenomenological methods of inquiry grow out of mutualism and compare Gibson's idea of visual kinesthesis to Merleau-Ponty's idea of the lived body. Third, we conclude that Ecological Psychology and varieties of Enactivism are laying down different paths to pursue related goals. Thus, convergence of Ecological Psychology and Enactivism is not possible given their conflicting assumptions, but cross-fertilization is possible and desirable.
Chapter
Full-text available
Research in the visual arts has contributed to the creation of environments that involve cross-disciplinary, multidisciplinary and interdisciplinary or transdisciplinary projects in departments within and across universities. An overview is provided of the historical context of doctoral awards in the arts with a definition of the terms practice-based, practice-led, and practice as research discussed. It articulates the challenges when acquiring explicit and exact knowledge alongside more subjective approaches that utilize tacit knowledge from artistic practice in research projects. Drawing on examples from art practice and doctoral students work, it analyzes objective, subjective, empirical, and hermeneutic paradigms, as described by Pierre Bourdieu, which can combine empirical approaches and individual understandings to re-enforce our understandings of the world.
Article
Full-text available
Purpose This study expands on the results of a stakeholder-driven research project on quality indicators and output assessment of art and design research in Flanders—the Northern, Dutch-speaking region of Belgium. Herein, it emphasizes the value of arts & design output registration as a modality to articulate the disciplinary demarcations of art and design research. Design/methodology/approach The particularity of art and design research in Flanders is first analyzed and compared to international examples. Hereafter, the results of the stakeholder-driven project on the creation of indicators for arts & design research output assessment are discussed. Findings The findings accentuate the importance of allowing an assessment culture to emerge from practitioners themselves, instead of imposing ill-suited methods borrowed from established scientific evaluation models (Biggs & Karlsson, 2011)—notwithstanding the practical difficulties it generates. They point to the potential of stakeholder-driven approaches for artistic research, which benefits from constructing a shared metadiscourse among its practitioners regarding the continuities and discontinuities between “artistic” and “traditional” research, and the communal goals and values that guide its knowledge production (Biggs & Karlsson, 2011; Hellström, 2010; Ysebaert & Martens, 2018). Research limitation The central limitation of the study is that it focuses exclusively on the “Architecture & Design” panel of the project, and does not account for intra-disciplinary complexities in output assessment. Practical implications The goal of the research project is to create a robust assessment system for arts & design research in Flanders, which may later guide similar international projects. Originality/value This study is currently the only one to consider the productive potential of (collaborative) PRFSs for artistic research.
Article
Full-text available
This article explores the subject of practice-based research, its application in the creative arts and its role in generating new forms of knowledge in the context of the PhD. Our aim is to provide more clarity about the nature of practice-based research, the approach we advocate and how it contributes to new knowledge that can be shared and scrutinized in a form that is both accessible and rich in its representation of the full scope of creative arts research. We draw on examples spanning over 35 years of experience in supervising interdisciplinary PhD research programs in the arts, design and digital media.
Article
Full-text available
People tend to perceive many and also salient similarities between musical sound and body motion in musical experience, as can be seen in countless situations of music performance or listening to music, and as has been documented by a number of studies in the past couple of decades. The so-called motor theory of perception has claimed that these similarity relationships are deeply rooted in human cognitive faculties, and that people perceive and make sense of what they hear by mentally simulating the body motion thought to be involved in the making of sound. In this paper, we survey some basic theories of sound-motion similarity in music, and in particular the motor theory perspective. We also present findings regarding sound-motion similarity in musical performance, in dance, in so-called sound-tracing (the spontaneous body motions people produce in tandem with musical sound), and in sonification, all in view of providing a broad basis for understanding sound-motion similarity in music.