Content uploaded by Jiajun Yang
Author content
All content in this area was uploaded by Jiajun Yang on May 04, 2015
Content may be subject to copyright.
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
SONIC TRAINER: REAL-TIME SONIFICATION OF MUSCULAR ACTIVITY AND LIMB
POSITIONS IN GENERAL PHYSICAL EXERCISE
Jiajun Yang
Audio Lab
University of York, UK
jy682@york.ac.uk
Andy Hunt
Audio Lab
University of York, UK
andy.hunt@york.ac.uk
ABSTRACT
The research outlined in this paper uses real-time sonic feedback
to help improve the effectiveness of a user’s general physical train-
ing. It involves the development of a device to provide sonified
feedback of a user’s kinesiological and muscular state while un-
dertaking a series of exercises. Customised sonification software
is written in Max/MSP to deal with data management and the soni-
fication process with four types of sound feedback available for the
participants.
In the pilot study, 9people used the sonification device in a
‘biceps curl’ exercise routine. Four different sonification methods
were tested on the participants over two sessions. Clear improve-
ment of the movement quality was observed in the second session
as participants tended to slow down their movements in order to
avoid a noise alert. No obvious improvement in the physical range
of movement was found between these two sessions. The partic-
ipants were interviewed about their experience. The results show
that most participants found the produced sounds to be informa-
tive and interesting. Yet there is room for improvement mainly
regarding the sound aesthetic.
This study shows the potential of using real-time interactive
sonification to improve the quality of resistance training by pro-
viding useful cues about movement dynamic and velocity. Suitable
sonification algorithms could help to improve training motivation
and ease the sensation of fatigue.
1. INTRODUCTION
The movement of the human body often produces acoustic en-
ergy. We can gain information about that movement by perceiving
motion-related sounds. For instance, the loudness of a badminton
racket swing can reflect the strength and speed of the swing.
Effenberg describes the relationships between music and sport
as ‘interwoven’ [1]. Music is an essential part of many rhythm-
driven sports, such as figure skating and synchronized swimming,
both for aesthetic and informative reasons. Also, many people like
to listen to music while doing physical exercise. Apart from sim-
ply enjoying some favourite music the sound itself provides useful
cues for maintaining good rhythmic motor coordination and re-
laxation, and it can also lead to a positive mood and a raising of
confidence and motivation [2, 3, 4].
Computer technologies have traditionally used visual displays,
and so data analysis has been carried out with graphical techniques.
The relatively recent development and study of auditory display
techniques, conveying information through the use of sound ob-
jectively [5], provides us with new opportunities for analysing data
and feeding back information to human users.
There are many advantages to using sound to study and inter-
act with data. Firstly, sound allows a screen-free scenario which
enables users to focus more on their main physical task. For in-
stance, an auditory monitoring system can help anaesthetists to
improve their working efficiency during an operation, as it reduces
the mental workload of having to focus on visual monitors while
carrying out many other responsibilities [6].
Secondly, sound shows its superiority in attracting people’s at-
tention. A visual alert may be easily neglected if a person’s visual
attention is focused elsewhere. However, sound is highly suitable
for alarm systems because not only can it attract people’s attention
while they are looking elsewhere, but the sound itself can carry
extra implicit information, e.g., “this is a fire alarm; leave now”
[7].
In the domain of general physical exercise, such as free weight
training, there is a common problem that many people tend to fo-
cus more on quantity rather than quality. People in a gym are likely
to carry out a certain number of repetitions without as much regard
for the smoothness of the movement or the way that sets of mus-
cles are activated. This problem is compounded when exercising
at home, because of the absence of professional trainers. Although
this may not seem much of a problem to general public, it becomes
immensely important for patients who require physiotherapy treat-
ment following an accident or operation.
This paper considers how we can help people to improve the
quality of their physical exercise by introducing auditory feedback
to their exercise routines. The research has potential applications
in daily physical exercise, elite sport or physical rehabilitation. Ar-
tificial auditory signals can be generated based on the user’s real-
time movement, using computer technology to play the role of a
virtual trainer, by guiding the movement and potentially leading to
an improvement of the exercise. Hence, we present a sonification
system that provides real-time auditory feedback of a user’s exer-
cising movement as a tool aiming to help improve the quality of
the training.
In this pilot study, the main aim was to investigate subjects’ ex-
periences in four different sonification modes, and test how these
four modes of sonification influence the exercise quality across two
identical sessions. As such it did not include a control group, but
a control-based comparison experiment will be conducted in the
future research as explained later in this paper.
The structure of this paper is as follows: Section 2 demon-
strates the concept of interactive sonification referring to literature
about sonifying human body movement. Sections 3 and 4 present
an overview of the sonification system we have developed, with
the usage demonstrated in Section 5. Section 6 contains the pro-
cedure, results and implications of a pilot study. Finally Section 7
ISon13-1
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
discusses further work and potential extension of the work done so
far.
2. SONIFICATION OF HUMAN BODY MOVEMENT
Sonification is a subset of the area of auditory display. It is defined
as the interpretation and transformation of data into perceivable
non-speech acoustic signals for the use of conveying information
[8]. Interactive sonification serves an additional purpose which
allows the manipulation of data based on the sonified feedback. In
this research, we hypothesise that sonic feedback can serve as a
real-time training quality monitor and motivator to help maintain
a good quality of exercise.
The research concept is concerned with whether we can ex-
pand the richness of naturally occurring acoustic cues by produc-
ing artificial sonic feedback to give extra information about the
quality of the exercise to the user, in order that they can make ap-
propriate adjustment in response.
Vogt et al. [9] developed PhysioSonic in 2009, using a camera
tracking system with markers placed on a user to study shoulder
movement and provide both metaphorical and musical audio feed-
back. The system motivated patients with arm abduction and ad-
duction problems via the synthesised or sampled feedback. Kleiman-
Weiner and Berger [10] developed an approach to sonify the mo-
tion of the arm to improve the action of a golfer’s swing. Barrass
et. al. [11] studied how different sonification methods performed
in outdoor jogging. Other researches on the sonification of human
body movement can be found in [1, 14, 15, 12].
Two types of bio-information were sonified in this study. Firstly,
the visible kinetic aspects of the movement were captured using a
Microsoft Kinect system. Such visible motion reflects the most
straightforward impression of movement quality, such as displace-
ment, dynamics and speed. There are also hidden attributes such as
strength, which is harder to observe visually. Strength, effort and
tension are generated from within the muscles and therefore this
requires a more direct and dedicated muscle measurement system,
for which we use an electromyography (EMG) sensor.
When a muscle is activated, muscle cells produce electrical
potential. The resultant electrical signal can be detected by EMG
sensors. EMG is widely used in the study of postural tasks, func-
tional movements and training regimes [13]. Pauletto and Hunt
sonified EMG data from leg muscles in 2006 [14]. They developed
an alternative way of portraying the data from EMG sensors using
sonification instead of a visual display. EMG sonification can also
be seen in [15], where muscular activity of a timpani player’s per-
formance was sonified.
The following section explains the construction of the sonifi-
cation device, which is capable of extracting both kinetic and mus-
cular data in real time. A diagrammatic overview of the system is
shown in figure 1.
3. SONIFICATION SYSTEM - HARDWARE
Two types of sensory devices are used to capture arm movement
and muscular activity separately. The first is a Microsoft Kinect
sensor (fig. 2) to capture real-time limb movement in a format of
2D coordinates (left-right, up-down) related to the centre of mass.
The frame rate is 30fps. Extrapolated from the basic coordinates,
we also calculate the vertical component of the velocity, which is
a key indicator for the biceps curl exercise quality.
Figure 1: Physical exercise sonification system
To measure the muscular activity, a wearable EMG belt shown
in fig. 3 was designed to manage the myoelectric signal acquisi-
tion and wireless transmission to the computer. This belt com-
prises an EMG sensor unit1powered by two 9v batteries, an Ar-
duino Duemilanove microprocessor (9600 baud) and a Bluetooth
modem.
4. SONIFICATION SYSTEM - SOFTWARE
The sonification software (fig.4) was developed using Max/MSP2.
It consists of three main functions, described in the following para-
graphs.
4.1. Data management
The data management section handles EMG data and Kinect data
acquisition through serial communication (sampling rate 500Hz)
and the Open Sound Control (OSC)3protocol. The EMG device
introduces a baseline offset of approximately 0.170.03v (signal
ranges between 0 to 5v). Hence, baseline adjustment was used
to remove the offset. In order to give participants a more obvi-
ous alteration in sound between muscle rest state and contraction
state, EMG normalisation was also used to ensure all users ben-
efitted from the full range of data mapping. A data recorder was
1http://www.advancertechnologies.com/
2http://cycling74.com/
3http://opensoundcontrol.org/
ISon13-2
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
Figure 2: Kinect motion cap-
ture camera Figure 3: EMG sensory belt
used to store all the bio-information into a text file. Hence, bio-
information can be sonified or studied either in real time or offline.
In addition, plots of the Kinect and EMG data are shown graph-
ically. Kinect data - in the form of the positions of body joints
- is presented through several knobs (for display only) shown on
the right side of fig.4. The Kinect data acquisition allows a total
display of up to 15 body joints, however only a few were numer-
ically displayed in real time to make the display more compact.
The EMG data can be monitored through the oscilloscope on the
left side
4.2. Sound engine
The sound engine is designed separately and is linked with the
main interface through the data mapping patch, (explained in 4.3).
Hence, it is not graphically displayed on the main interface while
the system is in use. The sound engine consists of a subtractive
synthesizer and an audio sampler. Theoretically, every parameter
in the sound engine can be controlled by the movement data. How-
ever, in practice, only a few parameters have been chosen for the
control (based on some initial tests) in order to produce the most
distinguishable acoustic results. These parameters are: loudness,
pitch, filter cut-off frequency (brightness), sample playback speed
and noise level.
Figure 4: Main interface of the sonification software
For the sound design, we customised four sonification map-
ping schemes for the pilot test with different acoustic textures and
responses. These four schemes are selectable by the user. The as-
sumption was that the majority of intended users would not have
a background in audio synthesis or programming, so a selection
from pre-sets was the best way of presenting a choice.
•Linear Frequency Synthesis Sound Mode
In this mode, the synthesiser is set to produce a sound with
rich spectral content. It consists of a combination of trian-
gular and square waves. In terms of the mapping, the cur-
rent vertical position (low to high) of the hand is mapped to
a linear scale of frequency (valid frequency: 20 to 570Hz).
The velocity of movement is set to trigger a white noise
sound when it exceeds a threshold value, which notifies the
user of movement that is too fast. The use of noise for this
notification helps to distinguish the ‘speeding’ indication
sound from the main sonic feedback. To avoid annoyance,
if the noise sound occurs too frequently due to bad quality
of movement, the white noise is softened by using a band-
pass filter and an amplitude envelope with a slow attack
time.
The EMG signal is mapped to the cut-off frequency of a
band-pass filter. This mapping allows the EMG data to af-
fect the brightness of the sound. Larger EMG values (in-
dicating more muscle power) lead to a brighter and clearer
tonal quality.
•‘MIDI Note’ Synthesis Sound Mode
The same timbre and mapping scheme are used as the pre-
vious mode. Yet instead of playing the sound with a linear
pitch change, the full vertical range of arm movement is
divided into 10 sections. Each section plays a note on the
synthesiser which is quantised in pitch to an equal tempera-
ment scale in the range of C4 to E5 with fixed velocity and
length. To avoid boredom for the listener, the note selection
is not fixed, but based on two customised first order Markov
chain probability tables. This means that the current note is
selected based on the previous note. Considering each note
as a state, each state will generate one of only a few other
states. For example, when the current state is C4, the next
state has a 45% chance to be D4, 25% chance to be E4,
10% to remain the same note and 20% chance to be E4.
Therefore, tonally, this will result in a similar (but differ-
ent) progression of notes in each set of movement. Differ-
ent melodic patterns are played according to the direction
of the arm movement. Contraction of the biceps results in
an ascending melody while extension produces a descend-
ing pattern. The melody is different each time because of
the probability tables.
•Rhythmic Sound Mode
This mode emits a rhythmic arpeggiator loop when the user
starts moving the forearm to a certain height. Then the loop
will keep playing along with the movement until the user’s
forearm is back at the original height level again, indicating
the completion of a repetition. The purpose is to help the
user scale the timing of a full repetition to match the full
length of the musical loop. The white noise sound is again
used as an indication of moving too fast.
•Ambient Sound Mode
Similar to the rhythmic mode, this triggers a sample of sea
waves instead. It aims to create a relaxing sensation for the
user rather than giving precise information on the move-
ment. Because of the richness in the spectrum of the sound,
playing a noise as a warning for moving too fast becomes
hardly audible as it is masked by the ambient sound. There-
fore, the noise was replaced by a sine wave beep.
Audio examples can be downloaded, see section 8.
ISon13-3
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
There are two main reasons for providing multiple types of
sound for the same movement set. 1) People have different per-
sonal preference for sounds. Therefore, consideration needs to be
given about how to provide sonic options for each user. 2) Each
mode type has its own emphasis in terms of providing sonic feed-
back. The linear frequency can represent the most straightforward
vertical displacement of the hand. The MIDI mode focuses more
on reminding users to slow down their movements, in order to gen-
erate a measured progression of a melodic pattern. The rhythmic
mode aims to improve the steadiness of the movement, whilst the
ambient mode aims to help users to relax.
Audio examples can be downloaded in the footnote below 4.
4.3. Data mapping
The final major functionality is the mapping patch, which links the
bio-information from the data management section with various
sound parameters from the sound engine such as pitch, filter cut-
off frequency, volume, etc. Parameter mapping [5, 8] is used as the
main mapping method. The EMG data and Kinect data are scaled
appropriately in the patch in order to result in the correct range of
values to control the sound parameters.
5. HOW TO USE THE SYSTEM
The user wears the EMG belt and has electrodes placed on the skin
surface directly over the dedicated muscle, in this case the biceps.
Technical details of the electrode placement are not included in
this paper; for more information, please refer to [13]. The user
also stands in front of the Kinect sensor, facing towards it. When
the device is activated the user can hear sounds being generated
according to their arm movement.
Figure 5: Demonstration of using the device
4https://sites.google.com/a/york.ac.uk/jiajun/shared-files
This paragraph describes a set of benchmark data recorded
from a regular gym trainer. As shown on fig.6, the position changed
smoothly and slowly (approximately 8 seconds per repetition). Within
each repetition, in muscle contraction, the EMG signal rose slowly
and peaked at roughly the highest vertical position. Then in muscle
extension, there is another small EMG peak indicating the subject
tried to prevent the dumbbell from lowering too fast.
Figure 6: Hand vertical position and EMG signal of a set of good
quality movement (benchmark)
6. PILOT STUDY AND DISCUSSION
6.1. Overview
The purpose of the pilot study was to examine the use of the device
and to gather user experience and suggestions. We also gained an
initial impression on how this sonification system can influence the
user’s body movement during biceps curls by interviewing partic-
ipants after their exercise sessions.
Nine participants (all male, mean age 25.8±3.0)were re-
cruited to participate in a test made up of two sessions. In each
session, participants were asked to do four sets of dumbbell curls
with one of the four sonification modes played in each set. Partic-
ipants were told to listen to the sonic feedback and try to respond
to the sound while exercising. Each participant experienced all
four sonification modes and therefore we could study their relative
experience of each via a post-session interview. Their Kinect and
EMG data were both recorded for offline sonification study and
analysis purposes.
We defined a good quality of exercise as consisting of the fol-
lowing two criteria: 1. The maximum dynamic range of move-
ment possible, which means that the forearm should aim to reach
the lowest and highest positions while the upper part of the arm
remains still. 2. The concentric and eccentric contractions should
be executed at a steady and relatively slow speed, with a total of 4
to 8 seconds per repetition. This has been shown to help improve
blood flow which can lead to a better training results [16].
ISon13-4
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
6.2. First Session
At the beginning of the session, a copy of the consent form was
given to the participant to sign and the purpose and procedures
of the test were clearly explained. An adjustable dumbbell was
prepared and the participant could adjust the weight by adding or
removing plates to the two sides of the dumbbell.
The participant was then fitted with the EMG device and po-
sitioned to stand in front of the Kinect sensor. A set of Sennheiser
HD 201 headphones was provided for the participant to listen to
the sonification. Prior to the session, a trial was conducted to give
the participant some familiarity with the exercise and the resultant
sound.
During the test, participants did several sets of exercises, each
with a different sound mapping. The repetition quantity in each
set was entirely up to the participant to decide upon, based on
their own motivation and physical condition. 1-2 minutes rest was
given between each set. After the session, a copy of the question-
naire was given to the participant and they were asked to rate each
sonification mode in terms of its comprehensibility and preference
from an integral scale between 1 to 5 (very poor, poor, moderate,
good, excellent). Each participant was also asked to comment on
the experience of using the device with each mode. Comments
were recorded either orally at the session or in written form.
6.3. Second Session
In the second session, participants were asked to complete the
same four sets of biceps curls with the same sonification modes.
After the session, participants again rated the four sonification
modes. The reason for conducting an identical second session is
because, at the first session, a participant may have been unfamil-
iar with the whole process and found the sounds strange to listen
to. Therefore, we looked for any difference in both the exercis-
ing quality and subjective opinions of the sonification, after they
became more familiar with the sound and system.
6.4. Quantitative Results
Figure 7 shows the plots of both EMG signal strength and the
hand’s y coordinate (dumbbell height) during a set of repetitions
using the linear frequency mode. The EMG data was normalised
(0 to 1.0) so that it could be viewed more easily together with the
y coordinate. Peaks in the EMG signal can be seen to be occurring
during vertical lifting, which is what would be expected, but also
in the lowest part of the movement, where the dumbbell is being
decelerated.
Figure 8 represents the velocity progression of the same set of
repetitions as the previous graph. We defined a velocity threshold
of vt=±0.78 whereby the white noise would be sounded if the
absolute velocity |v|was greater than vt.
The mean movement dynamic range and mean repetition time
gathered from the participants’ two-sessions of exercise were anal-
ysed. We had hypothesised that an improvement of mean dynamic
range and repetition time would be found in the second session as
participants gained familiarity with the system.
In terms of the mean dynamic range, such improvement could
not be statistically supported (table 1). A paired-samples T test
shows a significance level with p=0.191 and a low correlation
of 0.138. However the table demonstrates that for several partic-
ipants there was indeed an improvement from the first session to
the second.
Figure 7: EMG and dumbbell height plotted together
Figure 8: Changes of hand velocity throughout a whole set of
movements using the linear frequency mode.
The same test was conducted to study for the mean repetition
time. The result shows a significance level with p=0.003, and an
average increase in the repetition time of 1.58 second.
The different extents of improvement can be seen from table
2. Slower movements were executed in the second sessions for all
participants, (remembering that in curls, a slow and steady move-
ment is desired as opposed to a fast and spiky movement). During
the second session, no extra instructions were given to the partici-
pants. Therefore we did not purposely introduce factors that may
have led to a change of curl velocity. Two participants (No.2 and
No.5) made the least improvement on average time per repetition
with only 3% and 5% increment respectively. Yet the mean repe-
tition time of participant 5 already lies in the high standard range.
A greater amount of improvement was achieved by the other par-
ticipants.
6.5. Qualitative Results
The questionnaire collected subjective opinions of participants’
experience. Participants rated each mode in terms of the compre-
hensibility and preference from a scale of 1 to 5, where 1 means
‘highly disliked’ and 5 means ‘highly favoured’. The results show
a moderate overall rating (across all four modes) in comprehen-
sibility and preference with 3.71 and 3.41 out 5 respectively. As
shown in table 3, on average, participants found that the linear
frequency mode delivered a better sonic representation of the curl
ISon13-5
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
Table 1: Mean Dynamic per Repetition
Participant 1st Session 2nd Session Differences
1 1.247 1.653 +33%
2 1.569 1.450 -8%
3 1.235 1.693 +37%
4 1.586 1.531 -3%
5 1.558 1.480 -5%
6 1.512 1.524 +1%
7 1.399 1.539 +10%
8 1.650 1.791 +9%
9 1.265 1.255 -1%
Table 2: Average Time per Repetition (unit: second)
Participant 1st Session 2nd Session Differences
1 3.23 6.85 +121%
2 3.58 3.75 +5%
3 4.26 6.73 +58%
4 5.50 6.99 +27%
5 7.45 7.66 +3%
6 3.11 4.01 +29%
7 4.66 6.24 +34%
8 7.37 9.48 +29%
9 3.78 5.42 +43%
compared to the others. It scored 4.22 on mean comprehensibil-
ity with a standard deviation of 1.31. The majority of participants
found this mode sufficiently informative and only one participant
thought it was confusing. The rhythmic mode seems to be the least
informative among all four. This may be caused by the specifics
of this mode’s mapping; the vertical movement only control the
initial activation of the sound – once activated the sound plays in-
dependently until the position is back to the initial level (where the
arm is in a natural straighten position). The movement does not
alter the sound greatly apart from the brightness changes due the
change of the EMG data. Therefore, participants generally felt less
in control over the sound.
Table 3: Mean rating and standard deviation of four sonifications
Comprehensibility Preference
Mode Mean Std Mean Std
Linear frequency 4.22 1.31 3.56 1.54
MIDI note 3.56 1.15 3.33 1.24
Rhythmic loop 3.29 1.18 3.67 1.28
Ambient sound 3.78 1.11 3.06 1.43
As shown in figure 9, apart from the rhythmic mode, the up-
per quartile of each of the other modes is equal to the maximum
rating of 5. This is also an indication that using sound to provide
movement feedback is effective. Ratings for all four modes range
from ‘moderate’ to ‘excellent’ and users are able to understand the
sonic feedback easily.
The users’ preference in sound aesthetic varies more signifi-
cantly as shown in figure 10. This is also apparent in the subjects’
comments. These pinpoint the fact that there is still room for im-
provement in terms of sound aesthetics.
Based on the interviews, not all participants responded posi-
Figure 9: Comprehensibility rating
Figure 10: Preference rating
tively to all four modes of the sonification, yet at least one mode is
favoured by each participant either from a comprehensibility point
of view or by preference. Listed below are summarised comments
abstracted from the interviews about participants’ experience of
each mode. These comments have been re-worded into categories
based on their meaning.
1. Linear Frequency Synthesis Sound Mode
“It is easy to understand and it functioned clearly;
the dynamic representation is very clear.”
“You can listen to the change of the muscle and
it is the most raw presentation.”
“The noise indication is really useful. In terms
of the movement, specific motions are easy to
repeat.”
“Aesthetically not good enough. The sound is
noisy.”
“I like the sound because it is new to me. I
slowed down more than I would usually do to
prevent hearing the noise.”
Most participants (89%) agreed that this mode gave suffi-
cient information reflecting their exercise. Yet their major
ISon13-6
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
concern is the unfamiliarity with the linear synthesis sound
and aesthetic preference. 33% report that they did not enjoy
listening to this type of sound because they do not regard it
as a musical tone.
2. MIDI Note Synthesis Sound Mode
“I don’t think it provides as good feedback as
the frequency mode.”
“Faster notes seem to indicate worse exercise.
But it also creates a big leap if I moved too fast.
So I didn’t enjoy it that much.”
“The sound was unrepeatable, so the feedback
felt a little random.”
“This mode is the most interactive one. I needed
to slow down my pace a lot to generate a clear
melodic pattern. And the melody is different
each time. But it didn’t seem to inform me much
about the dynamics of my movement.”
“It was difficult to understand.”
This mode was designed to split the movement range into
10 steps. However, generally, people without much training
experience tend to do curls much faster than desired (less
than 4 seconds per repetition). This results in a quicker
MIDI note change, which leads to less clear melodic pro-
gression. One of the participants called it a ‘big leap’. Hence,
the preference for this mode is inversely related to the move-
ment speed; people who moved slower enjoyed the sound
more than people who moved quickly.
3. Rhythmic Sound Mode
“There is a progression I enjoyed listening to.
But the loop starts again every time I finished
one repetition. I would rather be able to hear
the whole melody.”
“I didn’t like it because I kept getting the wrong
sound. It was distracting. It motivated me though
to try to do it right because I hated the wrong
sound though.”
“It is a good idea. But at the moment it doesn’t
help me too much. It would be better if the
sound could be changed to my own mp3 files.”
“This one is very interesting. The exercise is
periodic, just like most music. So I adjusted
my pace to try and fit with the rhythm of the
sound.”
“The sound was pleasant to listen to.”
This mode provides the most musical content compared to
the other three. It is interesting that it became the most pop-
ular mode in the second session with an average preference
rating of 4.0 and a standard deviation of 1.0. It transferred
periodic movement into periodic music. Yet it has the prob-
lem of being too repetitive, and because of this a few par-
ticipants suggested making the music selectable from their
own music playlist.
4. Ambient Sound Mode
“It has the right balance between information
and aesthetic. It was pleasant and natural.”
“Generally it is good but it is too relaxing and
makes it harder for me to concentrate.”
“Comprehensible; the louder and more intense
the sound means more muscle strength.”
“It is quite random.” “I felt less control over
the sound.” “Not enough feedback.”
“It is special and immersive.” “It is relatively
easy to recognise.”
Currently this mode has the lowest ratings in preference
from both sessions, and received the most negative com-
ments (56% of negative opinions). Despite ranking second
in mean comprehensibility for both test sessions, interviews
still showed that people thought they had less control over
the sound. Only one participant showed support for this
mode. The positive response reflects the purpose of this
mode for creating a relaxing sonic atmosphere. Yet having
such a low popularity clearly indicates that this mode ei-
ther requires a major improvement or faces removal in the
planed future tests.
6.6. Discussion
The results from the pilot study indicate that a novel approach of
providing real-time sonic feedback of biceps curl exercises can
produce useful cues to the user and can influence the quality of
the exercise. Comparing the results in dynamic range and repeti-
tion time between two sessions, we did not observe a significant
result in the change of movement dynamic range. However, a sig-
nificant increase in repetition time was achieved. Overall, subjec-
tively, most participants found the device useful for maintaining a
good pace of movement, and good for reducing the sensation of fa-
tigue. Yet there are concerns over the listening experience, which
is mainly due to personal preference of the sounds.
Our initial plan was to provide four types of sonification so
that there were several choices to accommodate the issue of per-
sonal music preference. The rating of the questionnaire supports
this concept as all participants have at least one preferred sound
that they found both informative and enjoyable. However, further
development of the sound design is essential to provide a better
listening experience. It is also suggested that improvement is re-
quired of the sonification mapping for a clearer indication of the
dynamics of the arm movement.
We believe that the sonification device has great potential to
improve the quality of general exercise. However, due to the de-
sign of the pilot study, we focused more on the user experience in
order to help us improve the system for a future test. This study did
not include a control group to provide comparative statistical evi-
dence to support the hypothesis. Therefore, a thorough hypothesis
test will be conducted in the near future including both latitudinal
and longitudinal experiments to compare the exercise results be-
tween a group of participants with the sonification feedback and
a group without. In addition, the subsequent experiment will also
study on the influence of fatigue and whether the sonification feed-
back has a positive or negative effect when user is feeling tired.
ISon13-7
Proceedings of ISon 2013, 4th Interactive Sonification Workshop, Fraunhofer IIS, Erlangen, Germany, December 10, 2013
7. FURTHER WORK AND CONCLUSION
We are developing a game-based difficulty system that introduces
a “hi-score” concept to motivate the user do to better each time
they use the device. We aim to provide more tasks to further pro-
fessionalise the user’s movement through sonic feedback, and to
further optimise the sound design.
In the subsequent hypothesis test, a latitudinal experiment and
longitudinal experiment will be conducted. These two tests aim
to discover and track the differences in exercising quality between
participants who use the real-time sonification feedback and a con-
trol group who do the same exercise but without audio feedback.
We will be looking into factors such as movement dynamic and
velocity, repetition, EMG patterns, and subjective comments. Ap-
propriate statistical methods such as student’s T test and Pearson’s
chi-squared test will be used for comparative analytical purposes.
One of the possible extensions of the project to the area of
physiotherapy is to use the sonification device in rehabilitation
training. In this context sonified bio-feedback could be used to
correct the patient’s prescribed exercise. This has the potential of
accelerating the recovery process from conditions such as strokes,
which often requires a sustained level of rehabilitation exercises.
Such a device could be used at home so that patients can receive
feedback without the constant presence of a physiotherapist.
Another prospect is to migrate the sonification device to a
smartphone external device or watch-based wearable computer
with a suitable software application. This would offer better acces-
sibility to users and allow more possibilities of getting feedback for
outdoor exercise.
8. RESOURCES
The software and audio examples can be downloaded from the fol-
lowing link:
https://sites.google.com/a/york.ac.uk/jiajun/shared-files
9. REFERENCES
[1] A.O. Effenberg, “Movement Sonification: Effects on Percep-
tion and Action” in Multimedia, IEEE, 12.2(2005):53-59.
[2] N. Oliver and F. Flores-Mangas, “MPTrain: A Mobile, Mu-
sic and Physiology-Based Personal Trainer” in MobileHCI
’06 Proceedings of the 8th Conference on Human-Computer
Interaction with Mobile Devices and Services, 21-28, 2006.
[3] A. Beckett “The Effects of Music on Exercise as Determined
by Physiological Recovery Heart Rate and Distance” in Jour-
nal of Music Therapy, 27:126-136, 1990.
[4] N. M. Harmon and L. Kravitz, “The Effects of Music on Ex-
ercise” in IDEA Fitness Journal, 4(8), 72-77, 2007.
[5] T. Hermann, “Taxonomy and Definitions for Sonification and
Auditory Display” in Proceedings of the 14th International
Conference on Auditory Display, France: IRCAM, 2008
[6] R. G. Loeb and W. T. Fitch, “A Laboratory Evaluation of an
Auditory Display Designed to Enhance Intraoperative Mon-
itoring”, in Anesthesia & Analgesia, 94.2 : 362-368, 2002.
[7] D. D. Woods, “The Alarm Problem and Directed Attention in
Dynamic Fault Management”, in Ergonomics, 38:11, 2371-
2393, 1995.
[8] T. Hermann, A. Hunt and J. G. Neuhoff, “The Sonification
Handbook”, Logos Verlag, 2011.
[9] K. Vogt, D. Pirro, I. Kobenz, R. H ¨
oldrich, and G. Eckel,
“Physiosonic - Movement Sonification as Auditory Feed-
back”, in Proceedings of the 15th International Conference
on Auditory Display, 2009.
[10] M. Kleiman-Weiner and J. Berger, “The Sound of One Arm
Swinging: A Model for Multidimensional Auditory Display
of Physical Motion”, in Proceedings of the 12th International
Conference on Auditory Display, 2006.
[11] S. Barrass, N. Schaffert and T. Barrass, “Probing Preferences
between Six Designs of Interactive Sonifications for Recre-
ational Sports, Health and Fitness”, in Human Interaction
with Auditory DisplaysProceedings of the Interactive Sonifi-
cation Workshop, pp. 23-30. 2010.
[12] C. Ghez, T. Rikakis, R. L. DuBois and P. Cook, “An Auditory
Display System for Aiding Interjoint Coordination”, in Pro-
ceedings of International Conference on Auditory Display,
2000.
[13] P. Konrad, “The ABC of EMG: A Practical Introduction to
Kinesiological Electromyography”, Noraxon, USA, 2005.
[14] S. Pauletto and A. Hunt, “The Sonification of EMG Data”,
in Proceedings of the 12th International Conference on Au-
ditory Display, 2006.
[15] F. Grond, A. Bou¨
enard, T. Hermann and M. M. Wanderley,
“Virtual Auditory Myography of Timpani-playing Avatars”,
in Proceedings of the International Conference on Digital
Audio Effects (DAFX), pp. 135-142, 2010.
[16] http://stronger-slimmer.com/weight-lifting-tips.html
ISon13-8