Conference PaperPDF Available

Unconstrained and constrained embodied interaction with audiovisual feedback during vocal performance

Authors:

Abstract

This paper presents the work on unconstrained and constrained embodied interaction with live audiovisual processing parameters during singing. Building upon the concept of affordance and embodiment and adopting a User-Centred Design approach, two case studies were realised. The first case study in a context where a performer is free to move and interact with the MyoSpat interactive system for live sound processing parameters (unconstrained interaction); and, the second in a performative situation where the musician is limited by a played instrument or interface (constrained interaction). The interaction design solution proposed in the two case studies was welcomed by the performers; its potential and limitation allowed invited the exploration of new gestures-sound relationships.
SIIDS 2020 Sound, Image and Interaction Design Symposium
Unconstrained and constrained
embodied interaction with audiovisual feedback
during vocal performance
Balandino Di Donato
University of Leicester
balandino.didonato@leicester.ac.uk
ABSTRACT
This paper presents the work on unconstrained and constrained embodied interaction with live audiovisual
processing parameters during singing. Building upon the concept of affordance and embodiment and adopting a
User-Centred Design approach, two case studies were realised. The first case study in a context where a
performer is free to move and interact with the MyoSpat interactive system for live sound processing parameters
(unconstrained interaction); and, the second in a performative situation where the musician is limited by a
played instrument or interface (constrained interaction). The interaction design solution proposed in the two case
studies was welcomed by the performers; its potential and limitation allowed invited the exploration of new
gestures-sound relationships.
INTRODUCTION
Performers demonstrate a high level of interactivity afforded by the ability to physically control musical nuance
of the instrument. Similar self-evident interactivity is necessary when using interfaces for performance and
production. While this is essential, it is not always the case since current technology might not establish such a
level of interactivity and direct interaction with sound. In designing modes of interaction with an interface,
various factors should be considered: performance’s genre, instruments, available sensing technology, required
computational power, and the interface’s affordances (Tanaka et al., 2012). It is the latter aspect that is
particularly relevant to this work. Godøy (2010) demonstrates that music can invite certain gestures that are
often encouraged by sound’s timbral and dynamic qualities, the affordance of musical sound. In their study,
Caramiaux et al. (2011) concluded that gestures can be evoked by the sound’s causality. Although these sound-
tracing studies identify important information about gesture-sound relationships, they did not consider the
potential bias of the gesture tracking device’s affordance on the sound-tracing exercise. Tanaka et al. (2012)
explored gestural affordance with sound in relation to different devices, concluding that the combination of
input devices and output feedback leads to the construction of highly sophisticated affordances, which require
an equally complex construct of affordance. To reduce this complexity, I refer to the idea described by (Leman,
2007), that interfaces and modes of interactions should put the user in an ideal position where the interface itself
disappears. It is necessary to specify that here the word disappear does not aim to promote the idea of an
invisible system, but it refers to the interaction between the human and the machine (Dourish, 2001). Thus, the
musician’s and the audience’s perception should not focus on the mechanics of interaction. Nevertheless, “a
musical communicative channel should be established that is catalysed by the modes of interaction, but not
hindered by them(Tanaka and Knapp, 2002). Interaction design is, therefore, a vital aspect of attaining musical
satisfaction. Body movements during performance enhance the music experience, and the embodiment of the
interaction with sound plays an important role to facilitate the encoding of expressive gestures into sounds, and
the decoding of sounds into expressive gestures (Leman and Maes, 2015). Building on the concept of
affordance of musical sound, and principles of embodiment and natural interaction, in this paper, I present
modes of interaction with sonic and visual in two different contexts reflecting the constraints posed by the
technology during music performance.
UNCONSTRAINED AND CONSTRAINED EMBODIED INTERACTION
This paper proposes modes of embodied interaction in singing performance in two contexts: unconstrained and
constrained. In the former, the performer is free to move the body freely; and in the latter, the performer’s
movements are restricted by an instrument or interface. For this research, it was realised a custom interactive
system named MyoSpat (Di Donato et al., 2017).
Modes of interaction and MyoSpat were informed by research conducted adopting a User-Centred Design
SIIDS 2020 Sound, Image and Interaction Design Symposium
(UCD) approach (Norman and Draper, 1986). The UCD cycle included a first phase of an in-depth review of
body movements during the singing and the iterative work with the performers, which then led to defining the
interaction design, the realisation of MyoSpat, the performance of VoicErutseG by Vittoriana De Amicis, and
the composition and performance of Music Gesture Beatbox by Grace Savage.
Case Study 1: Unconstrained Interaction
In the first case study, I explored modes of interaction in a situation in which the singer can move her whole
body freely, without any technological constraint. This is defined as unconstrained interaction. In this context,
Vittoriana De Amicis, performed VoicErutseG (Figure 1) (De Amicis and Di Donato, 2015a; De Amicis and Di
Donato, 2015b) a performance with no restriction of movement, if not the choreographic indications of the
already composed music. VoicErutseG explores the possibility of extending existing works in the vocal solo
repertoire through gesture-controlled sound spatialisation, paying careful attention not to disrupt the
choreographic aspects and required vocal techniques of the original works. The performance aimed to inform
and demonstrate the implementation of a natural and embodied interaction design for controlling the spatial
position of the vocal sound, considering factors that might interfere with aspects of the performance and the
concentration of the performer. Taking into account all aims and considerations, the interaction design consisted
in a pointing gesture to establish the spatial position of the sound in the listening area (see video performance1).
Figure 1 Vittoriana De Amicis performing VoiceErutseG.
Case Study 2: Constrained Interaction
In the second case study, the interaction design is realised in a context where the movements of the performers
were partially constrained by a musical instrument or interface during the performance. I defined this
constrained interaction. This mode of interaction was explored in a real-world scenario through the work of
Grace Savage, who composed and performed Music Gesture Beatbox using MyoSpat and a loop-station (Savage
and Di Donato, 2016). Technological constraints were determined by the use of a hand-held microphone and a
loop-station, and the MyoSpat interactive audiovisual system. In addition to performative aspects, the
interaction design was realised taking into account the musicians’ creative process of composing and
performing. The interaction design consisted in the (i) pointing gesture to control sound spatialisation
parameters and lighting effects, (ii) the rising hand gesture to process the voice with a band-pass filter and video
colour filtering (Figure 2), (iii) the throwing gesture to generate visual feedback and sound spatialisation
trajectories, and (iv) the beat gestures speed to control the speed in which videos were played (see video
performance2). These gestures were drafted and used by Savage in Music Gesture Beatbox to address issues that
arise when composing a piece around technological constraints, and the need of freedom to explore and embody
audiovisual processes.
1 VoicErutseG video: https://www.youtube.com/playlist?list=PL60BfNS9JlwuSitKCKrYVRhcXsW6gbG5g
2 Music Gesture Beatbox video: https://youtu.be/DRFqCXpvfW0
SIIDS 2020 Sound, Image and Interaction Design Symposium
Figure 2 Grace Savage performing Music Gesture Beatbox.
RESULTS AND CONCLUSION
The work presented in this paper aimed at the creation of modes of interaction that would result in embodied
vocal performance, and that would not disrupt the execution of the performance nor the creative process that led
to its realisation. Main findings from observations and post-interviews with performers showed that designing
interactions that allow freedom of movement in the control sound processing parameters, can support the use of
a system in any context, regardless choreography elements imposed by a score. Thus, it allows performers to
focus on the performance rather than learning the use of an interactive system. The use of a constrained
interaction design coupled with a sophisticated technological ecosystem, required a bigger effort from the
performer and reduced the number of choreographic possibilities. However, it was observed that this was not a
major issue; Savage took advantage of the required learning curve, during which she builds a habituated
mental model of its constraints (Magnusson, 2010), and found the virtuosity within the interaction design and
technological constraints.
In the future, the interaction design in a constrained and unconstrained context will be explored in relation to the
idea of Human-Sound Interaction (Di Donato et al., 2020), which focuses on our diversity in perceiving sound,
its affordances, and consequently how this affects our interaction with live audiovisual processes.
REFERENCES
Caramiaux, B., Susini, P., Bianco, T., et al. (2011). Gestural Embodiment of Environmental Sounds: an
Experimental Study. In Proceedings of the International Conference on New Interfaces for Musical
Expression, NIME ’11, pages 144–148, Oslo, Norway
De Amicis, V. and Di Donato, B. (2015a). VoicErutseG. Frontiers Festival. Birmingham, United Kingdom, 17
March.
De Amicis, V. and Di Donato, B. (2015b). VoicErutseG. EmuFest. Rome, Italy, 7 October.
Di Donato, B., Dewey, C., and Michailidis, T. (2020) Human-Sound Interaction: Towards a Human-Centred
Sonic Interaction Design approach. In 7th International Conference on Movement and Computing
(MOCO ’20), July 1517, 2020, Jersey City/ Virtual, NJ, USA.ACM, NewYork, NY, USA, 4 pages.
DOI: 10.1145/3401956.3404233
Di Donato, B., Dooley, J., Hockman, J., Bullock, J., and Hall, S. (2017) MyoSpat: a hand-gesture controlled
system for sound and light projections manipulation. Proceedings of the International Computer Music
Conference (ICMC), Shanghai, China
Dourish, P. (2001). Where The Action Is: The Foundations of Embodied Interaction. MIT Press
Godøy, R. I. (2010). Gestural affordances of musical sound. In Godøy, R. I. and Leman, M., editors, Musical
Gestures: Sound, Movement, and Meaning, pages 115–137. Routledge, New York, New York.
SIIDS 2020 Sound, Image and Interaction Design Symposium
Leman, M. (2007). Embodied Music Cognition and Mediation Technology. The MIT Press.
Leman, M. and Maes, P.-J. (2015). The role of embodiment in the perception of music. Empirical Musicology
Review, 9(3-4):236246.
Magnusson, T. (2010). Designing constraints: Composing and performing with digital musical systems.
Computer Music Journal, 34(4):6273.
Norman, D. A. and Draper, S. W. (1986). User-Centered System Design: New Perspectives on Human-
computer Interaction. Lawrence Erlbaum Associates, Hillsdale, New Jersey.
Savage, G. and Di Donato, B. (2016). Music Gesture Beatbox. Music Tech Fest. Berlin, Germany, 29 May.
Tanaka, A., Altavilla, A., and Spowage, N. (2012). Gestural Musical Affordances. In Proceedings of the 9th
International Conference on Sound and Music Computing, Copenhagen, Denmark.
Tanaka, A. and Knapp, R. B. (2002). Multimodal Interaction in Music Using the Electromyogram and Relative
Position Sensing. Proceedings of the 2002 Conference on New Interfaces for Musical Expression, NIME
’02, pages 1–6, Dublin, Ireland
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
We present MyoSpat, an interactive system that enables performers to control sound and light projections through hand-gestures. MyoSpat is designed and developed using the Myo armband as an input device and Pure Data (Pd) as an audiovisual engine. The system is built upon human-computer interaction (HCI) principles; specifically, tangible computing and embodied, sonic and music interaction design (MiXD). This paper covers a description of the system and its audiovisual feedback design. Finally, we evaluate the system and its potential use in exploring embodied , sonic and music interaction principles in different multimedia contexts.
Article
Full-text available
Since its breakthrough at the beginning of the twenty-first century, the embodied music cognition theory has inspired empirical research on the role of action-perception couplings in musical activities. The integration of novel technologies and analysis methods inspired empirical research advancing knowledge regarding the role of embodiment in music perception and musical signification processes. In this paper, we present recent and on-going research in the field of embodied music cognition, with a focus on studies conducted at IPEM, the research laboratory in systematic musicology at Ghent University, Belgium. Attention is devoted to encoding/decoding principles underlying musical expressiveness, synchronization and entrainment, and action-based effects on music perception. The discussed empirical findings demonstrate that embodiment is only one component in an interconnected network of sensory, motor, affective, and cognitive systems involved in music perception. Currently, these findings drive the embodiment theory towards a more dynamical approach in which the interaction between various internal processes and the external environment are of central importance. Additionally, this approach envisions practical outcomes in the field of music affect research, wellbeing, healing, sports, music engineering, and brain studies.
Conference Paper
Full-text available
This paper is a comparative study of gestural interaction with musical sound, designed to gain insight into the notion of musical affordance on interactive music systems. We conducted an interview base user study trialing three accelerometer based devices, an iPhone, a Wii-mote, and an Axivity Wax prototype, with four kinds of musical sound, including percussion, stringed instruments, and voice recordings. The accelerometers from the devices were mapped to computer based sound synthesis parame- ters. By using consistent mappings across different source sounds, and performing them from the three different devices, users experienced forms of physical, sonic, and cultural affordance, that combine to form what we term musical affordance.
Article
Full-text available
Currently, a musician working with digital technology is faced with a panoply of musical tools that can be roughly characterized by a split between ready-made music production software on the one hand, and audio-programming environments such as SuperCollider, CSound, Pure Data, Max/MSP, ChucK, or Audiomulch (to name but a few) on the other. Problems with the former lie in the conceptual and compositional constraints imposed upon users by software tools that clearly define the scope of available musical expressions. It is for this reason that many musicians, determined to fight the fossilization of music into stylistic boxes, often choose to work with programming environments that allow for more extensive experimentation. However, problems here include the practically infinite expressive scope of the environment, sometimes resulting in a creative paralysis or in the frequent symptom of a musician-turned-engineer. Consequently, a common strategy can be detected, defined here as that of designing constraints, where the instrument designer, the composer, or the performer (a distinction often irrelevant in these systems; see Drummond 2009) devises a relatively high-level system of constraints, encapsulating a defined space for potential expression, whether of compositional or gestural nature. This article engages with this situation by exploring the Human–Computer Interaction (HCI) concepts of affordances, constraints ,a ndmapping. We examine how computational systems of musical expression always involve the establishment of a particular stratum that provides certain affordances to the musician, while concurrently posing important constraints. The article studies how constraints can serve as sources for creative explorations. For this purpose, three systems are analyzed: the mLog, Phalanger, and the ixi lang. All these systems are instantiations that proscribe complexity in favor
Conference Paper
Full-text available
This paper describes a technique of multimodal, multichannel control of electronic musical devices using two control methodologies, the Electromyogram (EMG) and relative position sensing. Requirements for the application of multimodal interaction theory in the musical domain are discussed. We introduce the concept of bidirectional complementarity to characterize the relationship between the component sensing technologies. Each control can be used independently, but together they are mutually complementary. This reveals a fundamental difference from orthogonal systems. The creation of a concert piece based on this system is given as example.
Gestural Embodiment of Environmental Sounds: an Experimental Study
  • B Caramiaux
  • P Susini
  • T Bianco
Caramiaux, B., Susini, P., Bianco, T., et al. (2011). Gestural Embodiment of Environmental Sounds: an Experimental Study. In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME '11, pages 144-148, Oslo, Norway
VoicErutseG. Frontiers Festival
  • V De Amicis
  • Di Donato
De Amicis, V. and Di Donato, B. (2015a). VoicErutseG. Frontiers Festival. Birmingham, United Kingdom, 17 March.
  • V De Amicis
  • Di Donato
De Amicis, V. and Di Donato, B. (2015b). VoicErutseG. EmuFest. Rome, Italy, 7 October.