Conference PaperPDF Available

MyoSpat: A hand-gesture controlled system for sound and light projections manipulation

Authors:

Abstract and Figures

We present MyoSpat, an interactive system that enables performers to control sound and light projections through hand-gestures. MyoSpat is designed and developed using the Myo armband as an input device and Pure Data (Pd) as an audiovisual engine. The system is built upon human-computer interaction (HCI) principles; specifically, tangible computing and embodied, sonic and music interaction design (MiXD). This paper covers a description of the system and its audiovisual feedback design. Finally, we evaluate the system and its potential use in exploring embodied , sonic and music interaction principles in different multimedia contexts.
Content may be subject to copyright.
335
HEARING THE SELF
MyoSpat: A hand-gesture controlled system for sound and light projections
manipulation
Balandino Di Donato
Integra Lab
Birmingham Conservatoire
balandino@integra.io
James Dooley
Integra Lab
Birmingham Conservatoire
james@integra.io
Jason Hockman
DMT Lab
Birmingham City University
jason.hockman@bcu.ac.uk
Jamie Bullock
jamie@jamiebullock.com
Simon Hall
Birmingham Conservatoire
simon.hall@bcu.ac.uk
ABSTRACT
We present MyoSpat, an interactive system that enables
performers to control sound and light projections through
hand-gestures. MyoSpat is designed and developed using
the Myo armband as an input device and Pure Data (Pd) as
an audio-visual engine. The system is built upon human-
computer interaction (HCI) principles; specifically, tan-
gible computing and embodied, sonic and music interac-
tion design (MiXD). This paper covers a description of the
system and its audio-visual feedback design. Finally, we
evaluate the system and its potential use in exploring em-
bodied, sonic and music interaction principles in different
multimedia contexts.
1. INTRODUCTION
As noted by McNutt [1], performing with technology
requires the development of new skills and flexibilities
often at odds with musical techniques, with potential
negative ‘disruptive’ effects. Our hypothesis is that a
gesture-controlled electronic interaction system, in partic-
ular MyoSpat, can move towards overcoming the ‘disrup-
tive’, ‘highly complex’ nature of live electronic process-
ing experienced by many performers, providing them with
an opportunity for new expressive ideas. Bullock et al.
[2] identify that the lack of familiarity with highly com-
plex systems can cause divergence between the performer
and technology, which adversely affects the performer’s
experience. This can create a disassociation between the
performer’s gesture and the sonic result. Lippe [3] states
the importance of allowing musicians’ expressivity to ex-
tend to their control over any electronic part of a perfor-
mance. Consequently, musicians must be able to interact
confidently with technology in order to present a musi-
cal and expressive performance. With MyoSpat, we un-
derline the importance of embodying music [4], allowing
the performers’ musical expression to be extended by their
gestural control over any electronic part in performance.
Copyright: c
2016 Balandino Di Donato et al. This is an open-access
article distributed under the terms of the Creative Commons Attribution
License 3.0 Unported, which permits unrestricted use, distribution, and
reproduction in any medium, provided the original author and source are
credited.
Visual feedback can enhance the gesture-sound relation-
ship, playing an important role in guiding the user’s actions
while using the system [5] and strengthening our percep-
tion of the auditory feedback [6]. MyoSpat affords musi-
cians a greater sense of control over sound through a direct
connection between movement and audio-visual feedback,
whilst making the newly learnt as intuitive and comple-
mentary to instrumental technique as possible.
Motion tracking allows complex physical movements to
be effectively captured and mapped to audio-visual re-
sponses. However, tracking both body motion and biodata,
such as muscle activity, can create stronger action-sound
relationships [7], leading to a better understanding of the
dynamics and mechanisms embedded in these actions [8].
Over the last two decades many systems combining both
motion with myograph data have emerged [9, 10]. Fur-
ther developments over the past two years have seen the
inclusion of the Myo armband in this area of work, demon-
strating its reliability and appropriateness as an expressive
gestural controller for musical applications [11, 12, 13].
The remainder of this paper is structured as follows: Sec-
tion 2 outlines the MyoSpat system. Section 3 present our
evaluation methodology and results are provided in Section
4. Conclusion and future works are presented in Section 5.
2. THE SYSTEM
MyoSpat is a interactive system that allows performers
to manipulate sound and light projections through hand-
gestures in a musical performance. Developed through an
iterative design cycle, MyoSpat’s design utilises context-
based, activity-centred and emphatic design approaches:
interactions between users and mediating tools are posi-
tioned within the motives, community, rules, history and
culture of those users [14, 15].
The current version of the system (outlined in Figures 1
and 3) uses (i) the Myo armband as an input device to track
hand-gestures; (ii) Myo Mapper 1to extract and convert
data from the Myo into Open Sound Control (OSC) mes-
sages; (iii) Wekinator 2for the gesture recognition process;
(iv) Pure Data (Pd) for the audio-visual signal elaboration
and (v) Arduino for converting serial data into DMX sig-
nals. The simplicity of MyoSpat’s audio engine allows the
1http://www.balandinodidonato.com/myomapper/
2http://www.wekinator.org/
336 2017 ICMC/EMW
user to use a stereophonic or quadraphonic audio system
without making changes to the code.



 
 


Figure 1. MyoSpat system structure.
2.1 Input device
We chose the Myo armband after carrying out a survey
of commercial devices capable of reliably and wirelessly
monitoring motion and muscle activity, without restricting
or creating conflict with the movements required during in-
strumental performance. Due to a lack of reliable software
to easily map and manage Myo data through a GUI, we
developed the Myo Mapper software. Myo Mapper also
features calibration and scaling functions in order to fa-
cilitate data management during musical performance; the
calibration function helping to overcome issues linked to
Myo’s data drift [7].
2.2 Interaction design
MyoSpat’s interaction design (IxD) draws on mimetic the-
ories, embodied simulations [16] and metaphorical actions
[17], directly connecting the audio-visual feedback pro-
duced by gestural interaction. MyoSpat approaches sound
manipulation in a way that considers sound as a tangible
entity to be grasped and shaped through continuous and
tactile interactions [18].
The system’s IxD was developed around three main ob-
jectives: to create a gesture vocabulary that enables in-
teraction with the system through meaningful gestures for
both performer and audience; to produce a clear, strong
relationship between gestures and audio-visual feedback;
and to enable musicians to use the system through natu-
ral interactions. The term natural here refers to the con-
textualised interaction, conditioned from previous knowl-
edge, with physical and virtual objects [19, 20]. In this
specific case we refer to previously acquired instrumental
techniques.
MyoSpat allows the user to wear the Myo armband on
either the left or right arm and retain the same level of
interaction with the system. The following explanation
of MyoSpat’s interaction design considers the user to be
wearing the Myo on the left arm.
The IxD comprises five activation gestures and one mod-
ulating gesture, allowing the sound and light to be manip-
ulated in six different ways.
LEGEND:
Area 1, Clean Sound
Area 2, Reverb
Area 3, Pitch shift
Lower Extend Lower
Clean
Clean
Clean
Clean
FRONT
LEFT
RIGHT
SPAT
SPAT
SPAT
SPAT
Figure 2. MyoSpat interactive areas.
The clean gesture is performed by orienting the arm to-
wards the front of the body and/or inwards towards the
chest (Figure 2, area 1). It allows users to obtain a clean
sound and to set the lighting system colour to white.
The extend gesture is performed by orienting the arm out-
wards (Fugure 2, area 2), allowing users to process the
sound through a long reverb with fixed parameters and to
set the lights’ colour to blue.
The lower gesture is performed by lowering the arm to-
wards the ground (Figure 2, area 3). It enables the user to
pitch shift the sound one octave lower, setting the lighting
colour to green.
The crumpling gestures are performed by rapidly moving
the fingers or performing a similar movement that engages
the forearm’s muscles and generates fluctuations in EMG
data, as taken from previous experiments with sound de-
sign in mixed realities [21]. The sonic response allows the
audio signal to pass through amplitude modulation (AM)
and delay effects connected in series. AM gain and delay
time are controlled by a direct mapping of Myo’s EMG
data.
The throwing gesture involves a rapid movement of the
arm as if throwing an object, enabling the user to spatialise
the sound through an automated trajectory. Duration and
direction of the trajectory are respectively driven by direct
mapping of Myo’s EMG mean absolute value (MAV) [22],
Myo yaw value and the moment at which the gesture is
recognised by Wekinator. The brightness of each light ad-
justs relative to the spatial position of the sound as a con-
sequence of the generated audio trajectory. The throwing
gesture is inspired from previous works on approaches to
visualising the spatial position of ‘sound-objects’ [23].
The waving gesture allows the user to pan the sound
around the room; the spatial location of the sound is es-
tablished through a direct mapping of the yaw data. The
brightness of each light is adjusted relative to the spatial
position of the sound.
The relationship between sound and gesture is built upon
the metaphor and mimetic theories [24] embedded in the
movements performed when the hand moves towards each
of the three areas, and not the pose assumed by the hand
once it has reached one of these areas. When performing
the extend gesture, users move their arm outwards, thus
extending the area that the body covers within the space.
We try to represent the expansion of the body within the 3D
space extending the sound with a long reverb. We associate
337
HEARING THE SELF
the lower gesture with a pitch shift one octave lower, as a
result of lowering a part the arm.
2.3 Gesture recognition
The gesture recognition process was implemented using
Wekinator. The clean,extend,lower and crumpling ges-
ture were recognised using the Neural Network Model,
which is an implementation of Weka’s 3Multilayer Per-
ceptron class. These four models were trained using Myo’s
yaw, pitch and EMG MAV data. These models were
trained with 5476 samples. The clean,extend,lower and
crumpling gesture models were built using respectively 1,
1, 2 and 3 hidden layers.
We observed that the throwing gesture is better recog-
nised through a gestural data analysis over time. For this
reason, the relative model was created using dynamic time
warping. It was trained using Myo’s acceleration and gyro
data. This was trained with 50 samples.
2.4 Audio and light projection processing
The Pure Data (Pd) programming environment was used
to develop the engine behind audio manipulations and
light projections, with data sent from Wekinator and Myo
data changing the parameters of each component. The Pd
patches used in Integra Live’s 4Pitch Shifter and Reverb
modules were implemented in the MyoSpat Pd patch to
control pitch shift and reverb respectively. MyoSpat’s spa-
tialiser has three components: gain control, a high-pass fil-
ter and a delay line. Gain factor, filter cut-off frequency,
and delay time are controlled by a direct mapping of the
Myo yaw value. Data from the throwing gesture model
triggers an automated sound spatialisation trajectory. A di-
rect mapping of the Myo EMG data determines the dura-
tion for which this effect is applied to the audio signal.
Data from Wekinator controls the light colour, with the
Myo yaw value independently adjusting the brightness of
each light depending on the sound’s spatial placement. Pd
then rescales the Myo and Wekinator data, transmitting it
via serial connection to an Arduino with a Tinkerkit DMX
Master Shield. A perfect white light was not obtained due
to the low resolution of the DMX lights used in the user
study (Section 3); higher resolution lights capable of pro-
ducing a pure white light may improve visual feedback.
The audio system was formed using a quadraphonic au-
dio system of four Genelecs 8050A speakers. The lighting
consisted of four RGB LED light sources placed next to
each speaker.
3. EVALUATION
The gesture recognition evaluation was carried out adopt-
ing a direct evaluation approach [25].
A user study was conducted to evaluate the gesture recog-
nition models, reliability, learnability, as well as the partic-
ipants’ perception of the audio-visual feedback and their
user experience, and the ease of use. 11 participants took
part in the study: 9 musicians and 2 non-musicians. The
musicians were 1 harpist, 2 keyboardists, 2 saxophon-
ists, 1 percussionist (snare drum), 1 singer, 2 guitarists,
3http://www.cs.waikato.ac.nz/ml/weka/
4http://integra.io/integralive/
Arduino +
DMX Shield
Lighting
System
Signal router
Wekinator & Myo data Audio In
Reverb Pitch Shifter
Signal router
SPAT
Loudspeaker System
AM
DELAY
Wekinator Output
Myo EMGs
Myo Yaw
Myo Yaw,
Wekinator Output
Trajectory
Generator
Myo Yaw, EMGs
Wekinator Output
Figure 3. System implementation.
1 engineer and 1 visual artist. Of the musicians 6 were
conservatoire-trained, with most actively using or having
used technology in their professional practice.
The user study was divided into four stages: (i)
structured interview, (ii) training, (iii) qualitative eval-
uation/interview and (iv) user experience questionnaire
(UEQ) [26]. Participants were first asked questions about
their musical background and experience with technology
through a structured interview. During the training stage
participants were instructed about the MyoSpat system,
and the interaction design and audio-visual feedback they
should expect to experience. This was followed by a
practice period lasting for a maximum of 10 minutes.
After this training period, a qualitative evaluation was
conducted. Participants were asked to perform each of
the recognised gestures 5 times with visual feedback and
5 times without. An audio file containing the sound of
water flowing was used during this part of the study, with
participants manipulating it with the recognised gestures.
With every successful gesture and pose recognised by
the system, we asked the participants if they positively
perceived a change in the audio manipulation mapped to
each gesture. Direct evaluation of the second machine
learning model was also conducted. This was done by
tracking how many times out of 5 Wekinator correctly
recognised the gestures. During the qualitative evaluation
data about the system and each user’s experience was
gathered through an informal, semi-structured interview.
Lastly, participants were asked to complete a UEQ.
4. RESULTS
The clean,extend and lower gesture were recognised with
an accuracy of 100%; the crumpling gesture achieved 97%
accuracy and the throwing was recognised only 75% of the
time (Figure 5).
Data from the interview and the qualitative evaluation
were collated and statistically analysed; the UEQ Analy-
sis Tool was used to tabulate and analyse this data [26]. A
338 2017 ICMC/EMW
Grounded Theory approach, also used in [27], was adopted
to analyse qualitative data gathered through the informal
semi-structured interview.
Results from the qualitative evaluation, indicate that the
MyoSpat system is easy to learn, and with practice can
adapt to the instrumental technique required for the instru-
ments used during the test. MyoSpat’s flexibility in allow-
ing the user to wear the Myo armband on either the right
or left arm partly facilitates this. Participants felt confident
using the system, with an average training time of 04’55”
(min 2’30”, max 10’18”). We believe that this is a very
short time for learning how to use a new interactive system
to control audio-visual feedback. However, when com-
paring data from the UEQ collected from the user study
and from a workshop at Cardiff Metropolitan University,
where 13 participants used the system for an average of
2’30” each, it emerged that goal-directed tasks were more
difficult to perform, suggesting that improvising with the
system offers a better user experience than composing. At-
tractiveness and hedonic qualities scored similarly on the
UEQ, as can be seen in Figure 4.
0
0.5
1
1.5
2
2.5
3
Attractiveness Pragmatic Quality Hedoni c qualit y
User St udy Workshop
Figure 4. User experience questionnaire results from data collected dur-
ing the user study and a workshop delivered using MyoSpat.
As discussed in [28], audio feedback can be meaningful
to the user for exploring the system when fulfilling sound-
oriented tasks. However, the completion of the task is im-
proved when using visual feedback and the IxD-auditory
feedback relationship becomes stronger.
Most participants initially interacted with the system
through the crumpling gesture. When using it to manip-
ulate a pre-recorded audio file containing a flowing wa-
ter sound, their interaction appeared natural and embod-
ied. The lower gesture created an underwater sound, re-
sulting in participants interacting with it as if they were
submerging their hand in a tub filled with water. Other ges-
tures included splashing and swirling water, 5demonstrat-
ing MyoSpat’s potential to explore embodied interaction
with sonic objects, in line with similar research [29, 28].
When using their instruments with the system, these em-
bodied gestures were further explored by creating vibrato
and tremolo effects through the periodic waving of the
hand similar to the vibrato technique on a string instru-
ment; this in turn modulated MyoSpat’s amplitude mod-
ulation and delay effects. Interestingly, the same gesture
was performed not only by guitarists but also by keyboard
and saxophone players. 6This way of interacting with the
system was non-previously considered to us. As stated in
[30], it is clear that non-obvious gestures and the sound
5https://vimeo.com/209717708
6https://vimeo.com/202150793
such interactions produce [31] are relevant for informing
the designing natural and embodied music interactions.
All participants considered the lower,clean and waving
gestures easy to learn and perform, being intuitive, natural
and highly related to the audio manipulations associated
with each gesture. Participants considered the relationship
between the throwing gesture and its related audio-visual
response to be the weakest. This is most likely attributed to
a lower accuracy (75%) of the gesture recognition model
by the machine learning algorithm. Only in the case of
one participant, the throwing gesture was recognised cor-
rectly by the system and perceived correctly by the par-
ticipant 100% of the time. Participants were able to per-
ceive the throwing gesture’s audio response better and use
it creatively, when interacting with the system using their
instrument instead of the flowing water audio file. 7De-
tailed data about gesture recognition performance and the
participants’ perception of the audio-visual feedback are
reported in Figure 5.
60
65
70
75
80
85
90
95
100
Clean Low er Extend Weaving Crumpling Throwing
Gesture Recognition Accuracy (%) Participant's perception (%) AVG ( %)
Figure 5. Gesture recognition and participants’ perception of the audio-
visual feedback perception, calculated in percentages.
The degree of freedom imposed by the physical require-
ments of playing a musical instrument, coupled with the
movement limitations imposed by the hand gestures of
the interaction design, allows performers to explore new
postures and ancillary gestures. 6 participants interpreted
these limitations as potentially having a restrictive effect
on their musical performance, and as a result may impose
certain body postures and instrumental techniques.
Participant 1 (guitar player) said, “when you play a fast
passage you contract the muscles, and you might not want
to trigger the delay effect. In that case you don’t have con-
trol of the effect. To play that kind style (musical style)
might be difficult. But the more you practice you will be
able to play.
Participant 9 (guitar player) said, “when you are doing
bar chords, the muscles naturally tense. So you cannot
really control the effect. The more you go down (closer to
the guitar’s bridge), the muscles tense a lot more and more,
so you cannot control the effect at all.
At the same time, some of the participants interpreted
these restrictions as something with which they could ex-
plore new creative possibilities as well. Participant 11
(saxophone player) said, ”When the fingers are moving, it
causes certain things to happen, and I’m one who doesn’t
7https://vimeo.com/209730610
339
HEARING THE SELF
move much, so in this case the system imposes a perform-
ing style that you don’t necessarily want, but exploring the
three spaces is very interesting. However, I would like to
have a smoother transition between the effects, for the rea-
son that I like to get to the spaces in between. Because
the three spaces are so well defined, apart from the drop
in pitch, the reverb, the delay and the trajectories there is
not much. But it’s a good sign because you can easily un-
derstand and control what you want to do. But by the way
I perform, the places in between would be the interesting
spaces that I want to get to, but I can’t. My approach to
electronic systems is to have something very limited, which
then gets incorporated with other things. The saxophone
is very limited, it’s a metal tube, and actually it does one
thing, part of the fun is to find out where the boundaries
are. With three effects it’s enough, there is interesting stuff
there and you can also start to test where the edges are,
where breaking points are, to test the misbehaviour.
Many gestures and poses used by participants to trig-
ger audio manipulations were not anticipated by the au-
thors, demonstrating the system’s potential to allow new
ways to approach bodily exploration during musical per-
formance. Participants felt that visual feedback from the
light projections helped to guide and improve the accu-
racy of their interaction with the system, improving their
ability to identify the audio manipulations associated with
each gesture. It also provided an extra layer of interaction
to explore, enhancing the level of immersiveness partici-
pants felt. Though visual feedback was noted to enhance
the user experience, it also had the potential to distract the
user, shifting their focus from auditory to visual.
MyoSpat was considered by all participants in the user
study as an engaging and stimulating creative tool for
musical performance. Despite UEQ results suggesting
MyoSpat offers a better user experience when impro-
vising, one participant used the system to compose The
Wood and the Water, a work for harp and electronics [32].
Although a pre-composed work, the composer did note
that recreating each performance exactly was not possible,
highlighting that an improvisatory element existed in the
work. These types of “misbehaviours” of the system also
allow a reciprocal sonic interaction between the performer
and the machine, making the live electronics as interactive
electronics within an compositional context [33]. A video
of participants improvising using MyoSpat can be found
here. 8In addition to musical performance applications,
one participant (amateur dancer) saw a potential use of
the system to recognise dance movements and map them
to audio-visual responses. 9Interestingly, systems for
extending other form of interactive dance performances
have been recently developed [34, 35].
5. CONCLUSIONS
We have presented MyoSpat, an interactive hand-gesture
controlled system for creative audio manipulation in mu-
sical performance. Machine learning was successfully in-
corporated to recognise a number of physical gestures, en-
abling audio-visual manipulations to be mapped to each
8https://vimeo.com/205202681
9https://vimeo.com/202118370
one of them. Results from the user study demonstrate that
the system can be used for improvisation and composition
[32], allowing users to explore a novel range of embod-
ied physical interactions during the music making process.
Results demonstrate that the use of a prosthetic gestural
controller that does not restrict the user’s movements has
potential applications beyond purely musical ones.
Suggestions from user study participants form the basis
for future work on MyoSpat. These include developing
(i) stronger audio-visual elaborations, (ii) smoother transi-
tions between audio and visual effects, (iii) a wider palette
of gestures and (iv) ways to manipulate sound and light
projections, (v) better panning curves, and (vi) a more co-
herent colour mapping of the lighting system. One partic-
ipant suggested using the primary colours red, green and
blue for the three areas. Another participant, who also
composed and performed using MyoSpat, requested the
development of a MyoSpat standalone application, allow-
ing any performer to use MyoSpat without any previous
knowledge.
Acknowledgments
We acknowledge Dr. Rebecca Fiebrink (Goldsmiths, Uni-
versity of London) for her advice in conducting a reli-
able machine learning model evaluation. We also thank
Eleanor Turner (Birmingham Conservatoire) for practis-
ing and composing using MyoSpat, and Lamberto Coccioli
(Birmingham Conservatoire) his contribution.
6. REFERENCES
[1] E. McNutt, “Performing electroacoustic music: a
wider view of interactivity,” vol. 8, no. 03, 2004, pp.
297–304.
[2] J. Bullock, L. Coccioli, J. Dooley, and T. Michailidis,
“Live Electronics in Practice: Approaches to training
professional performers,” vol. 18, no. 02, 2013, pp.
170–177.
[3] C. Lippe, “Real-Time Interactive Digital Signal Pro-
cessing: A View of Computer Music, vol. 20, no. 4,
1996, p. 21.
[4] A. Cox, “Embodying Music: Principles of the Mimetic
Hypothesis,” vol. 12, no. 2, 1996.
[5] L. Vainio, R. Ellis, and M. Tucker, “The role of visual
attention in action priming,” vol. 60, no. 2, 2007, pp.
241–261.
[6] A. I. Goller, L. J. Otten, and J. Ward, “Seeing Sounds
and Hearing Colors: An Event-related Potential Study
of Auditory–Visual Synesthesia,” vol. 21, no. 10, 2009,
pp. 1869–1881.
[7] K. Nymoen, M. R. Haugen, and A. R. Jensenius,
“MuMYO: Evaluating and Exploring the MYO Arm-
band for Musical Interaction,” in Proceedings of Inter-
national Conference on New Interfaces for Musical Ex-
pression, Baton Rouge, LA, USA, 2015.
[8] B. Caramiaux, M. Donnarumma, and A. Tanaka,
“Understanding Gesture Expressivity through Muscle
Sensing,” vol. 21, no. 6, 2015, pp. 1–26.
340 2017 ICMC/EMW
[9] G. Dubost and A. Tanaka, “A Wireless, Network-based
Biosensor Interface for Music,” 2002.
[10] A. Tanaka and R. B. Knapp, “Multimodal Interaction
in Music Using the Electromyogram and Relative Po-
sition Sensing,” in Proceedings of the 2002 conference
on New Interfaces for Musical Expression, Singapore,
Singapore, 2002, pp. 1–6.
[11] C. Benson, B. Manaris, S. Stoudenmier, and T. Ward,
“SoundMorpheus: A Myoelectric-Sensor Based Inter-
face for Sound Spatialization and Shaping,” in Pro-
ceedings of International Conference on New Inter-
faces for Musical Expression, Brisbane, Australia,
2016, pp. 332–337.
[12] M. Weber and M. Kuhn, “KONTRAKTION: Sonifica-
tion of Metagestures with electromyographic Signals,”
in the Audio Mostly 2016. New York, New York,
USA: ACM Press, 2016, pp. 132–138.
[13] J. C. Vasquez, K. Tahiroglu, and J. Kildal, “Motivic
Through-Composition Applied to a Network of Intelli-
gent Agents,” in Proceedings of the International Com-
puter Music Conference, Utrecht, Netherlands, 2016.
[14] G. Gay and H. Hembrooke, Activity-Centered Design.
An Ecological Approach to Designing Smart Tools and
Usable Systems. The MIT Press, 2004.
[15] P. Dourish, Where the Action is: The Foundations
of Embodied Interaction, illustrated, reprint ed., ser.
Bradford books. The MIT Press, 2004.
[16] R. W. Gibbs, “Artistic understanding as embodied sim-
ulation,” vol. 36, no. 02, 2013, pp. 143–144.
[17] N. Schnell and F. Bevilacqua, “Engaging with
Recorded Sound Materials Through Metaphorical Ac-
tions,” vol. 35, no. 4-5, 2017, pp. 379–401.
[18] Y. Visell, R. Murray-Smith, S. Brewster, and
J. Williamson, “Continuous auditory and tactile inter-
action design,” in Sonic Interaction Design, K. Frani-
novic and S. Serafin, Eds. The MIT Press, 2013, pp.
77–124.
[19] F. Kistler, B. Endrass, I. Damian, C. T. Dang, and
E. Andr´
e, “Natural interaction with culturally adaptive
virtual characters,” vol. 6, no. 1-2, 2012, pp. 39–47.
[20] B. Leibe, T. Starner, W. Ribarsky, Z. Wartell, D. Krum,
B. Singletary, and L. Hodges, “The Perceptive Work-
bench: toward spontaneous and natural interaction in
semi-immersive virtual environments,” in Proceedings
IEEE Virtual Reality 2000, 2000, pp. 13–20.
[21] B. Di Donato and J. Bullock, “gSPAT: Live sound spa-
tialisation using gestural control,” in Student Think-
Thank at the 21st International Conference on Auditory
Display, Graz, Austria, 2015, pp. 7–8.
[22] Z. Arief, I. A. Sulistijono, and R. A. Ardiansyah,
“Comparison of Five Time Series EMG Features Ex-
tractions Using Myo Armband,” in International Elec-
tronics Symposium, 2015, pp. 11–14.
[23] J. Bullock and B. Di Donato, “Approaches to Visualiz-
ing thee Spatial Position of ’Sound-objects’,” in Elec-
tronic Visualisation and the Arts (EVA), London, UK,
2016, pp. 15–22.
[24] P. Wigham and C. Boehm, “Exploiting Mimetic The-
ory for Instrument Design,” in Proceedings of Interna-
tional Computer Music Conference, Utrecht, Nether-
lands, 2016.
[25] R. Fiebrink, P. R. Cook, and D. Trueman, “Human
model evaluation in interactive supervised learning,” in
the 2011 annual conference. New York, New York,
USA: ACM Press, 2011, pp. 147–10.
[26] B. Laugwitz, T. Held, and M. Schrepp, “Construction
and evaluation of a user experience questionnaire,” in
Symposium of the Austrian HCI and Usability Engi-
neering Group. Springer, 2008, pp. 63–76.
[27] A. Johnston, “Beyond Evaluation: Linking Practice
and Theory in New Musical Interface Design, in In-
ternational Conference on New Interfaces for Musical
Expression, Oslo, Norway, 2011, pp. 280–284.
[28] E. O. Boyer, L. Vandervoorde, F. Bevilacqua, and
S. Hanneton, “Touching Sounds: Audio Virtual Sur-
faces,” in IEEE 2nd VR Workshop on Sonic Interac-
tions for Virtual Environments, 2015.
[29] B. Di Donato, “Balandino Di Donato: Using the XTH
Sense to Create Virtual Water Games,” in XTH Sense
Blog (Accessed: 17/03/2017, https:// goo.gl/Bh7WQe),
2016.
[30] M. M. Wanderley, “Non-obvious performer gestures
in instrumental music,” in International Gesture Work-
shop. London, UK: Springer, 1999, pp. 37–48.
[31] ——, “Gestural control of music,” in International
Workshop Human Supervision and Control in Engi-
neering and Music. Citeseer, 2001, pp. 632–644.
[32] E. Turner, “The Wood and The Water,” in Premier
at Birmingham Harp Day (https://goo.gl/ NtVKJA),
Birmingham, UK, 2017.
[33] A. M. Gioti, “From live to interactive electronics. Sym-
biosis: a study on sonic human-computer synergy,” in
Proceedings of the International Computer Music Con-
ference. University of Music and Performing Arts
Graz, Austria, 2016, pp. 572–578.
[34] J. Jaimovich, “Emovere: Designing Sound Interactions
for Biosignals and Dancers,” in Proceedings of Inter-
national Conference on New Interfaces for Musical Ex-
pression, Brisbane, Australia, 2016, pp. 316–320.
[35] S. Webber, M. Harrop, J. Downs, T. Cox, N. Wouters,
and A. V. Moere, “Everybody Dance Now: Tensions
between Participation and Performance in Interactive
Public Installations,” in Proceedings of the Annual
Meeting of the Australian Special Interest Group for
Computer Human Interaction. ACM, 2015, pp. 284–
288.
... Tanaka et al. [17] MPNN and HHMM 4 All gestures user-designed in a workshop Dalmazzo and Ramirez [24] DT and HMM 4 4 fingers (left hand, index to ring fingers) Ruan et al. [25] MPNN 3 3 fingers (left hand -thumb, index, middle) Di Donato et al. [26] MPNN and DTW 6 5 activation gestures and 1 modulating gesture (left arm) Erdem et al. [27] RNN 6 6 gestures (exercises on guitar playing -both arms) Dalmazzo and Ramírez [28] HHMM and DT 7 Violin bowing techniques Literature within the scientific community has been integral when informing how we should use complex EMG information from available GIs. In particular, Arief et al. [29] showed that the mean absolute value (MAV) feature extraction method is one of the most efficient when working with EMG data. ...
... Recent studies using Wekinator to investigate how IML can be used to drive sophisticated interactive gestural systems, in music, have seldom used several ML models to do so [17,26]; instead, they focus on the usability of a few models to drive such a system. This means that an ML model is used to achieve a particular research aim but optimal model choice is uninformed. ...
Article
Full-text available
Interactive music uses wearable sensors (i.e., gestural interfaces—GIs) and biometric datasets to reinvent traditional human–computer interaction and enhance music composition. In recent years, machine learning (ML) has been important for the artform. This is because ML helps process complex biometric datasets from GIs when predicting musical actions (termed performance gestures). ML allows musicians to create novel interactions with digital media. Wekinator is a popular ML software amongst artists, allowing users to train models through demonstration. It is built on the Waikato Environment for Knowledge Analysis (WEKA) framework, which is used to build supervised predictive models. Previous research has used biometric data from GIs to train specific ML models. However, previous research does not inform optimum ML model choice, within music, or compare model performance. Wekinator offers several ML models. Thus, we used Wekinator and the Myo armband GI and study three performance gestures for piano practice to solve this problem. Using these, we trained all models in Wekinator and investigated their accuracy, how gesture representation affects model accuracy and if optimisation can arise. Results show that neural networks are the strongest continuous classifiers, mapping behaviour differs amongst continuous models, optimisation can occur and gesture representation disparately affects model mapping behaviour; impacting music practice.
... Therefore, the more familiar they are with hand exploration, the better their ability to recognize the environment. There exist quite a few studies on hand gestures for sound control [11][12][13][14][15], which, however, were not applied to the welfare of the visually impaired. Withagen et al. [16] think that the visually disabled can built three-dimensional images in their mind in the same way as normal people do. ...
... Since its commercial release in 2015, the Myo has proven to be popular for performative interactions among electronic artists, musicians, and dancers (e.g., [6,1,2]). However, in our work with the Myo, enthusiasm has been tempered by two technical limitations: First, Thalmic Labs' Myo Connect, which is often used by third-party software, does not support Linux, so it cannot be used on embedded platforms such as Raspberry Pi and Beaglebone. ...
Preprint
Full-text available
This paper describes the process of developing a standstill performance work using the Myo gesture control armband and the Bela embedded computing platform. The combination of Myo and Bela allows a portable and extensible version of the standstill performance concept while introducing muscle tension as an additional control parameter. We describe the technical details of our setup and introduce Myo-to-Bela and Myo-to-OSC software bridges that assist with prototyping compositions using the Myo controller.
... In the former, the performer is free to move the body freely; and in the latter, the performer's movements are restricted by an instrument or interface. For this research, it was realised a custom interactive system named MyoSpat (Di Donato et al., 2017). Modes of interaction and MyoSpat were informed by research conducted adopting a User-Centred Design (UCD) approach (Norman and Draper, 1986). ...
Conference Paper
Full-text available
This paper presents the work on unconstrained and constrained embodied interaction with live audiovisual processing parameters during singing. Building upon the concept of affordance and embodiment and adopting a User-Centred Design approach, two case studies were realised. The first case study in a context where a performer is free to move and interact with the MyoSpat interactive system for live sound processing parameters (unconstrained interaction); and, the second in a performative situation where the musician is limited by a played instrument or interface (constrained interaction). The interaction design solution proposed in the two case studies was welcomed by the performers; its potential and limitation allowed invited the exploration of new gestures-sound relationships.
... Likewise, Tanaka et al. [6] found that participants were able to associate the embedded connection between controller generated affordance and the produced gesture-sound relationship. The potential of visual interaction between sound and movement is similarly investigated in [7,8]. ...
Conference Paper
Full-text available
Mechanical vibrations have typically been used in the performance domain within feedback systems to inform musicians of system states or as communication channels between performers. In this paper, we propose the additional taxonomic category of vibrational excitation of musical instruments for sound generation. To explore the variety of possibilities associated with this extended taxon-omy, we present the Oktopus, a multipurpose wireless system capable of motorised vibrational excitation. The system can receive up to eight inputs and generates vibrations as outputs through eight motors that can be positioned accordingly to produce a wide range of sounds from an excited instrument. We demonstrate the usefulness of the proposed system and extended taxonomy through the development and performance of Live Mechanics, a composition for piano and interactive electronics.
... Since its commercial release in 2015, the Myo has proven to be popular for performative interactions among electronic artists, musicians, and dancers (e.g., [6,1,2]). However, in our work with the Myo, enthusiasm has been tempered by two technical limitations: First, Thalmic Labs' Myo Connect, which is often used by third-party software, does not support Linux, so it cannot be used on embedded platforms such as Raspberry Pi and Beaglebone. ...
Conference Paper
Full-text available
This paper describes the process of developing a standstill performance work using the Myo gesture control armband and the Bela embedded computing platform. The combination of Myo and Bela allows a portable and extensible version of the standstill performance concept while introducing muscle tension as an additional control parameter. We describe the technical details of our setup and introduce Myo-to-Bela and Myo-to-OSC software bridges that assist with prototyping compositions using the Myo controller.
Chapter
Hand Gesture Recognition systems using electromyography sensors in conjunction with data from inertial measurement units are currently largely used for musical interfaces. However, bracelets are susceptible to displacements causing a decrease in the accuracy when they are used in such applications. In this study, a hand gesture recognition model applied to a musical interface has been tested using two different commercial armbands, Myo and GForce. Both armbands use the same pre-trained gesture recognition model and same hand gestures are recognized. We evaluate the robustness of the pre-trained model and the reached accuracy correcting the displacement of the sensors. The test performed to evaluate the system shows a classification accuracy of 94.33% and 90.70% respectively considering the same pre-trained model. The accuracy results obtained with both sensors are similar which evidences the robustness of the tested model and the importance of correcting the displacements.
Conference Paper
Full-text available
MyoSpat is an interactive audiovisual system that aims to augment musical performances by empowering musicians and allowing them to directly manipulate sound and light through hand gestures. We present the second iteration of the system that draws from the research findings to emerge from an evaluation of the first system [1]. MyoSpat 2 is designed and developed using the Myo gesture control armband as input device and Pure Data as gesture recognition and audiovisual engine. The system is informed by human-computer interaction (HCI) principles: tangible computing and embodied, sonic and music interaction design (MiXD). This paper reports a description of the system and its audiovisual feedback design. We present an evaluation of the system, its potential use in different multi-media contexts and in exploring embodied, sonic and music interaction principles.
Conference Paper
Full-text available
We present an innovative sound spatialization and shaping interface, called SoundMorpheus, which allows the placement of sounds in space, as well as the altering of sound characteristics, via arm movements that resemble those of a conductor. The interface displays sounds (or their attributes) to the user, who reaches for them with one or both hands, grabs them, and gently or forcefully sends them around in space, in a 360° circle. The system combines MIDI and traditional instruments with one or more myoelectric sensors. These components may be physically collocated or distributed in various locales connected via the Internet. This system also supports the performance of acousmatic and electronic music, enabling performances where the traditionally central mixing board, need not be touched at all (or minimally touched for calibration). Finally, the system may facilitate the recording of a visual score of a performance, which can be stored for later playback and additional manipulation. We present three projects that utilize SoundMorpheus and demonstrate its capabilities and potential.
Conference Paper
Full-text available
This prospective study concerning the perception of audio virtual surfaces (AVSs) was inspired by two different research fields: sensory substitution and haptic and touch perception. We define an Audio Virtual Surface as a region of space that triggers sounds when the user touches it or moves into it. First, we describe an example of interactive setup using an AVS to simulate a sonic interaction with a virtual water tank. Then, we present an experiment designed to investigate the ability of blindfolded adults to discriminate between concave and convex AVSs using only the gesture-sound interaction. Two groups received different sound feedback, a static one indicating presence in the AVS, and a static+dynamic one (related to the component of the hand velocity tangential to the surface). In order to demonstrate that curvature direction was correctly perceived, we estimated their discrimination thresholds with a psychophysical staircase procedure. Results show that most of the participants were able to learn the task. The best results were obtained with the additional dynamic feedback. Gestural patterns emerged from the interaction, suggesting the use of auditory representations of the virtual object. This work proposes a contribution to the introduction in Virtual Reality of sonic interactions with auditory virtual objects. The setups we present raise new questions at both experimental (sensory substitution) and application levels (design of gesture-sound interaction for virtual reality).
Conference Paper
Full-text available
This paper presents the creation of two music pieces using motivic through-composition techniques applied to a previously presented NIME: a Network of Intelligent Sound Agents. The compositions take advantage of the characteristics of the system, which is designed to monitor, predict and react to the performer's level of engagement. In the set of visual and written instructions comprising the scores, the network is fed with pre-composed musical motifs , stimulating responses from the system that relate to the inputted musical material. We also discuss some of the key challenges in applying traditional composition techniques to an intelligent interactive system.
Conference Paper
Full-text available
In this paper we present the rationale and design for two systems (developed by the Integra Lab research group at Birmingham Conservatoire) implementing a common approach to interactive visualisation of the spatial position of ‘sound-objects’. The first system forms part of the AHRC-funded project ‘Transforming Transformation: 3D Models for Interactive Sound Design’, which entails the development of a new interaction model for audio processing whereby sound can be manipulated through grasp as if it were an invisible 3D object. The second system concerns the spatial manipulation of ‘beatboxer’ vocal sound using handheld mobile devices through already-learned physical movement. In both cases a means to visualise the spatial position of multiple sound sources within a 3D ‘stereo image’ is central to the system design, so a common model for this task was therefore developed. This paper describes the ways in which sound and spatial information are implemented to meet the practical demands of these systems, whilst relating this to the wider context of extant, and potential future methods for spatial audio visualisation.
Conference Paper
Full-text available
In this paper, we report on Encounters, an interactive public installation that provides a basis for studying the effect of dance performances on the emergence of creative, social experiences. Based on observations and interviews with dancers and participants, we identified a range of tensions that arise from integrating a staged performance with participatory interaction. These tensions occurred among both participants and performers, and influenced the social and performative experience. Based on our analysis, we propose several strategies to smoothen the integration of performative and participatory interaction. These strategies reconsider the role of the interactive installation, the effect of digital cues that draw on existing conventions, and mechanisms to direct gaze. We believe our findings and strategies are valuable to HCI researchers and performative artists seeking to design for public participation in interactive experiences.
Conference Paper
Full-text available
Sound spatialisation is an important component in interactive per- formances as well as in game audio and virtual or mixed reality systems. HCI practices are increasingly focused on creating a natural user experience and embodied interaction through gestu- ral control. Body movements that coincide with sounds consist of both performed ‘sound producing‘ gestures, and ancillary and communicative movements. Thus, different gestural typologies may relate to the same audio source. Furthermore, gestures may depend on the context in which they have been expressed; in other words, they can carry different semantic or semiotic meanings in relationship to the situation and environment or reality in which they have been enacted. In order to explore these research themes, we are developing gSPAT: a software and hardware system able to drive live sound spatialisation for interactive audio performance using gestural control based on human-meaningful gesture-sound relationships. The ultimate aim is to provide a highly natural and musically expressive sound spatialisation experience for the per- former. Here we describe three experiments conducted to explore the possible directions for the future of gSPAT’s development. The tests employ a range of practice-based and ethnographic research methods to establish applicability, naturalness and usability across a range of approaches to the interaction design of the system.
Conference Paper
'Kontraktion' is an embodied musical interface using biosignals to create an immersive sonic performance setup. It explores the energetic coupling between digital synthesis and musical expression by reducing the interface to an embodied instrument and therefore tightening the connection between intention and sound. By using the setup as a biofeedback system the user explores his own subconscious gestures with a heightened sensitivity. Even subtle, usually unaware neural impulses are brought to conscious awareness by sensing muscle contractions with an armband and projecting them outward into space with sound in realtime. The users gestural expressions are embodied in sound and allow for an expressive energetic coupling between the users body and a virtual agent. Utilizing the newly adopted awareness of his body the user can take control of the sound and perform with it using the metagestures of his body as an embodied interface. The body itself is transformed into a musical instrument, controlled by neurological impulses and sonified by a virtual interpreter.
Article
This article presents a set of interactive audio applications that have been realised between 2009 and 2012. The applications have been created to foster the engagement of users in playing sound and music—alone or collectively. After introducing the underlying concepts and technical framework, the article briefly describes each scenario and discusses the involved metaphors.
Conference Paper
Feature extraction is meant to get representation and information that embedded in the signals, this is necessary to minimize complexity of implementation and reduce the cost of information processing. Recently, there are many methods for features extraction. This research is comparing five feature extractions from eight channels electromyography (EMG) signals that obtained from Myo Armband located on forearm muscles in order to get significant differences when hand do some movements. The time series features extraction that evaluated are Mean Absolute Value (MAV), Variance (VAR), Willison Amplitude (WAMP), Waveform Length (WL), and Zero Crossing (ZC). The variety of hand movement are fist, rest, half-fist, gun-point, and mid-finger fold. Moreover, the result shows that the rank of evaluated features extraction always showhs same results in four experiment, MAV is always giving the best performance WL. From this finding, MAV and WL are two recommendation for time series features extraction. This rank of time series features extraction gives worthiness when process information in future development research.