Content uploaded by Mads Haahr
Author content
All content in this area was uploaded by Mads Haahr on Nov 26, 2017
Content may be subject to copyright.
AES 41st International Conference, London, UK, 2011 February 2–4
1
LOCATION AWARE INTERACTIVE GAME AUDIO
NATASA PATERSON2, KATSIARYNA NALIUKA1, TARA CARRIGY1, MADS HAAHR2,
FIONNUALA CONWAY2
1 Trinity College Dublin, College Green, Dublin 2, Ireland
katsiaryna.naliuka@ndrc.ie,tara.carrigy@ndrc.ie
2 Trinity College Dublin, College Green, Dublin 2, Ireland
patersn@tcd.ie,mads.haahr@cs.tcd.ie,conwayfi@tcd.ie
With the affordability and pervasiveness of GPS enabled smart phones, downloadable location aware applications have
become increasingly popular. These applications include the overlay of local information onto physical sites to
multiplayer games and tourism. However, many of these applications rely predominantly on graphical overlays and
simple user interface audio. For narrative led location aware experiences, audio can play an important role in
immersing the player in the story line and physical location. Coupled interactivity and adaptability of audio to player
movements encourage the user to remain engaged with the physical location. This paper describes the theory and
implementation of interactive audio for a location aware tourism application. The aim of the application is to create an
immersive gaming experience where the audio supports the historical setting and given narrative.
INTRODUCTION
Location aware augmented reality applications refer to
the overlaying of a digital world onto a real world space
with physical locations providing contextual cues for
narrative and gameplay elements. Currently many
augmented reality applications include the use of visual
and audio overlays onto physical locations in order to
present local information of the site or for use in gaming
or artistic installations. One installation that focuses on
audio is The Tactical Sound Garden [12] by Mark
Shepard. Participants download the software and travel
to the specified location of the implemented soundscape
within an urban place. As the user moves around the
space a real-time audio mix is heard that responds to
movements within the given location. As well as
listening to a pre-defined soundscape, participants may
also create their own ‘sound garden’ by uploading audio
files to chosen physical positions. In regards to
narrative led historical dramas in location aware
applications, audio can play an important role in
involving participants in the setting and creating
empathy towards characters [11]. Correlations have
been found to exist between increased immersion and
the association of a physical object to audio. Visual cues
are often important for sound localisation therefore
adding to location involvement [5]. Hence the
importance of utilising real world visual cues and an
adaptive and interactive soundscape to increase the
engagement in a given narrative.
Audio as yet has not been thoroughly explored for its
potential in creating an immersive environment for this
medium. There are examples of rich cinematic
soundtracks overlaid onto a real world space such as
SoundWalk [13]. However, most augmented reality
audio walks do not include traditional gaming concepts
such as non-linearity and interactive audio.
In this paper we shall overview non-linear adaptive
audio in composition and gaming together with
examples of historical location aware applications that
use audio as part of their interface. Falkland Ghost
Hunt, a location aware historical game, will then be
presented together with the soundscape implementation.
Results of a commercial evaluation of the overall game
experience will be discussed in regards to the role of
interactive audio for game immersing and involvement.
1 RELATED BACKGROUND
Game audio production and composition techniques hail
back to the experience of film and music composition.
Not only is film audio production knowledge of value
but composers and sound designers also draw on the
compositional aesthetics and also apply this to gaming.
In gaming, music serves as one important component of
the spectrum of sounds that includes a musical score,
ambient sound, dialogue, sound effects and even
silence. Although sound effects and game soundtracks
are based on the same principles of film, a slightly
different approach must be taken when it comes to game
audio. In addition to a cinematic soundscape,
compositional non-linearity, adaptability and
interactivity must be considered when designing the
Paterson et al. Location Aware Interactive Game Audio
AES 41st International Conference, London, UK, 2011 February 2–4
2
sound. Therefore in gaming scenarios, music acts as an
accompaniment while gameplay elements and the
communication of narrative information is of paramount
importance.
Interactive audio in a location aware scenario describes
the process where sound and music is passively
triggered by a user’s movement in a given location.
Audio content activated by interaction describes the
process of ‘generative art’ whereby new content is
created with the use of algorithmic processes [1]. The
first known example of generative interactive
composition goes as far back as the 18th century with
Johann Philip Kirn Berger’s musical dice game Der
allezeit fertige Polonoisen und Menuetten companist
(Berlin, 1757). The “Minuet Composer” allowed
Berger to compose by throwing a dice and consulting a
table to determine which musically predefined card to
use. Through mathematics the player was ensured an
“original” work with most of the variation occurring in
the melody and the harmonic content remaining more
controlled. This random approach to music generation
popularised aleatoric (according to the dice) or dice
music and was popular for those new to musical
composition. Many other examples of ‘dice
composition’ existed with Mozart’s Musikalisches
Wurfelspiel first published in 1792 in Berlin, being the
most famous. More recently the technique has been
used by the avant garde composers such as Henry
Cowell’s String Quartet No. 3, Cage’s Winter Music
and Stockhausen’s Klavierstuck XI. In Klavierstuck X, a
note sheet with nineteen separated segments are given to
the performer to be played at random [2]. Information
in regards to tempo and dynamics for the next segment
is stated at the end of each segment and the whole piece
is finished when one of the segments has been repeated
twice. This is an early example of ‘mobile’ interactive
music.
Due to the presence of mixers for digital audio on the
mobile platform, interactive generative systems can now
be developed. One example is Lasorsal and
Lemordant’s [7] interactive format called AMML,
(Advanced/Audio Multimedia Markup Language). This
sound manager is designed for the iPhone and J2ME
phones. This example uses the process of selecting
short files in creating a continuous immersive
soundtrack. Random file selection is used together with
randomised triggering of sound effects and sleep times
between file playback. DSP (Digital Signal Processing)
effects and virtual 3D (three dimensional) audio is
implemented in real-time. This example provides an
immersive non-linear soundtrack without the use of
dialogue elements and does not utilise the triggering of
audio files by user movement in relation to GPS (Global
Positioning System) predefined locations.
Due to the ubiquitous nature of mobile technology,
tourism is one area that can greatly benefit from
downloadable applications. This is especially true for
historical sites where location aware technology can
allow for history to ‘come alive’ for the participant. The
Battel of Culloden, developed by ZolkC is an interactive
visual and audio tourist guide retelling the historical
events of the battle between the Jacobite and British
Government in 1746 [10]. The core content is triggered
automatically and delivered through a single earpiece
with the visual display providing information such as
maps and a menu – with the user controlling how much
they want to listen to and see. However the mono audio
deployment does not allow for any spatial awareness of
audio and may effect engagement. Additionally, the
audio presentation was linear with no use of generative
audio content.
The use of interactive dramas for reliving historical sites
and events is used in the location aware audio drama
Riot! 1831 [11]. Participant movements trigger a variety
of sound files with audio content presenting dramatised
real world historical events. In general, moving into a
region would trigger one of the sound files to start
playing and moving out of the region would cause it to
stop. However this application was not based on the
mobile phone platform and required the use of PDA
(Personal Digital Assistant) and external GPS receiver.
The aim of Falkland Ghost Hunt is to combine location
aware technology on the mobile phone platform with
that of an interactive and generative soundscape in order
to engage participants in the historical setting of
Falkland palace and gaming elements.
2 GAME PROTOTYPE
The working prototype, Falkland Ghost Hunt is a
location aware gaming application designed to make
local history more accessible to a younger audience.
The application is developed on the Android platform
and based on the Samsung Galaxy S, which contains an
integrated GPS receiver, internal compass and
accelerometer and requires for the participant to wear
stereo headphones. The game narrative is set during the
time Queen Mary spent at Falkland Castle in Scotland.
It is located in the gardens of Falkland Castle with a
contextually based audio and visual virtual world
overlay. This narrative led game requires the player to
act as a paranormal investigator in order to find and
release ghostly manifestations. Different graphical
modes of interaction assist the player in locating and
capturing the image of the spirit in order to hear their
story. Audio plays a significant role in creating a
ghostly, immersive atmosphere that is used as a
backdrop for sound effects and dialogue.
Paterson et al. Location Aware Interactive Game Audio
AES 41st International Conference, London, UK, 2011 February 2–4
3
Figure 1: Capturing the ghost
Once the player has found and released all required
ghosts they are able to review their evidence in a
casebook where they can hear the dialogue and see the
images they have captured. Therefore the aim of the
sound was to reinforce the paranormal investigator role-
play element within the historical setting.
3 INTERACTIVE AND GENERATIVE AUDIO
According to the game composer Koji Kondo [2] game
audio must be adaptive, interactive, change with each
gameplay and provide tension and surprise. Keeping
these principles in mind, the sound design for Falkland
Ghost Hunt utilises the concept of generative stochastic
background sound by the randomisation and layering of
‘wavelets’ (small audio files). Generating music by
using digital audio wavelets can be used to build a
soundtrack and is used for games on consoles and now
is also possible on the mobile platform. The process of
generating music from short audio files can be first
traced back to 2001 when Kenneth Kirschner a New
York based composer, used Adobe Flash (multimedia
platform) to compose ever-changing pieces of digital
music [4]. This process generally consists of a number
of audio files that are randomly chosen and layered
simultaneously and can play for as long as the player
wishes. There is no inherent end and it is suited for
games requiring continuously changing music, hence
avoiding the problem of boredom and habituation often
evident in the looping of files in console gaming. From
the listeners' point of view the soundscape will always
be perceived as linear even though it is non-linear in
implementation.
The use of interdeterminate adaptive audio wavelets is
ideal for the mobile platform due to the limited memory
and CPU (Central Processing Unit) available. In the
prototype, the use of wavelets forms the continuous
background sound that is able to play for as long as the
player remains within a given location, therefore
avoiding the looping of sound files [2]. This is
implemented by using the SoundPool class in the
Android operating system. The maximum audio file size
was 750 KB, and of approximately 2–4 seconds in
length. Playback of the wavelets was randomised within
the application code so that the files overlapped to
create a continuous background sound. It was found that
SoundPool had a quick file loading time and hence
worked well for looping and playing back smaller
numerous files simultaneously. As well as randomising
the file selection, sleep time between file playback was
also randomised in order to avoid any perceptual pattern
recognition. Hence the background was perceived as
continuous and ever changing so that the player always
had a different soundscape experience each time they
played the game. SoundPool was also used to playback
isolated sound effects. The Android MediaPlayer class
presented the dialogue as it was capable of streaming
larger files in a linear format [9].
In addition to the generative background sound,
interactivity of the audio to player movement was
implemented by linking predefined GPS co-ordinates to
the triggering of audio files. Concentric rings were
created with differing radial distances from the central
GPS location so when a player enters the different
regions, relevant audio files are triggered, consisting of
the coded generative audio files and isolated sound
effects. When a ghost is encountered in the central
region, the player is required to capture its image in
order to listen to the historically related game dialogue.
Figure 2: Concentric rings of audio
Synchronicity between the audio, visuals and player
movement is paramount for the application, as tight
coupling between all sensory modalities provides a
sense of realism and immersion [8]. By utilising
location technology such as GPS to trigger content
delivery, it enables a subconscious interaction with the
game space and physical location. Hence the
Paterson et al. Location Aware Interactive Game Audio
AES 41st International Conference, London, UK, 2011 February 2–4
4
technology doesn’t interfere with the experience and
makes the interaction pervasive and ubiquitous.
Figure 3: Falkland Ghost Hunt
Within the sound design, user interface sounds are also
implemented using the SoundPool class. These sounds
were designed to reinforce the concept of the handset
acting as a paranormal investigative device. Hence
sounds reminiscent of EVP (Electronic Voice
Phenomenon) recorders and radar were used to
complete the role-play aspect of the game. This
represents another aspect of audio interactivity, that it
provides feedback to the player of their investigative
role in the game space and physical location.
As well as considering interactive and generative audio,
compositional styles in creating suspense and tension
reminiscent of cinema are also implemented in the
sound design. The combination of these factors allow
for the creation of an engaging, narrative led audio
experience for the player that supports the game
architecture and virtual world overlay in the given
location.
4 EVALUATION
A commercial evaluation of the Falkland Ghost Hunt
was undertaken in the grounds of Falkland Castle,
Scotland. The aim of the trial was to assess the
commercial viability of the application and to evaluate
the players’ experience of the game. The trial ran for 12
days with 319 people playing the game. Participant
ages ranged from 6 to 65+ years with varying
technological and cultural backgrounds. Hence player
demographics were wide and diverse. After completing
the game mission, participants completed a
questionnaire consisting of open-ended questions and
quantitative five point scale (1–5) questions.
Approximately 98% of people responded that they liked
the game and 71% felt they had learned something
about the site. The average enjoyment rate was 4.4 out
of 5 (5 stating they enjoyed it very much). Overall this
is a favourable result as one of the original aims of the
application was to make history enjoyable and
accessible to a wider audience, especially a younger
one:
“It occupied the children with historical information…”
(32 year old mom with children of 10 and 6)
“Makes history come alive.” (29 year old)
Another aim in developing the game was to encourage
the participant to experience interactivity with the
virtual game world and historical site. Players were
asked what they liked most about this experience:
“Going around the garden, interactivity, ghosts,
historical aspect.” (8 and 12 year olds)
“(The game was) easy to use, something
different/modern/interactive.” (28 year old)
“Interactivity, visuals, audio.” (44 year old)
“Innovative, fun, the use of technology in a traditional
setting.” (40 year old man)
Players also responded positively to the overall sound
design:
“Sound effects were tremendous, very atmospheric.”
(29 year old man)
“I liked the sounds.” (12 year old)
“I liked listening to the voices, fun and scary.” (13 year
old)
“I liked the sound, birds, atmospheric sounds.” (60 year
old woman)
Therefore the sound design contributed to participants
being able to engage and interact with the historical site
in a way that was immersive and enjoyable for all ages
[6].
5 DISCUSSION
Interactive and generative audio for location aware
historical applications can facilitate the engagement of
players in the virtual world and historical site. The
responsiveness of the audio to the physical location and
the generative, ever changing atmospheric background
together with visual clues encourage an involvement
with historical events. However, in regards to
generative audio there are limitations and difficulties.
Non-linearity does pose the problem of the inability to
Paterson et al. Location Aware Interactive Game Audio
AES 41st International Conference, London, UK, 2011 February 2–4
5
create a tension build up and resolution popular in linear
forms, which may limit the composer’s control of
musical elements. This may result in the difficulty in
mapping potential player emotion progression during
gameplay which would aid in sound design composition
[3]. Another difficulty for location aware augmented
reality sound designers and composers is the lack of
standards for hardware devices that may not have the
same features in memory and processing power. This
results in variable playback of sound content and
quality. As well as device dependency, processing
constraints result in the inability of real-time DSP
(Digital Signal Processing) mixing of effects on the
Android platform.
Additionally GPS inaccuracies or lack of updates can
interfere with the triggering of audio files in the
specified locations hence interfering with the
experience. However, in spite of the difficulties of
game audio and current mobile platform technology,
adaptive interactive game audio in location aware
augmented reality gaming can result in a creative and
entertaining experience.
6 CONCLUSIONS AND FUTURE WORK
Location aware game audio that includes cinematic
soundscapes, interactivity and non-linear generative
audio together with an engaging virtual world can
successfully be created on the mobile platform. Future
work in this area could focus on the inclusion of
variability within audio file playback. Changing the
speed of audio playback would change the pitch, or the
addition of real-time granulation would change audio
file timbres. This would create sound variability and
reusability of files hence reducing the need for
additional memory in creating complex soundscapes.
REFERENCES
[1] Beilharz, K. “Interactively Determined
Generative Sound Design for Sensate
Environments: ExtendingCyborg Control”, Y.
Pisan (eds), InteractiveEntertainment '04, pp. 11-
18 (2004).
[2] Collins, K., Game Sound: An introduction to the
history, theory and practice of video game music
and sound design. MIT Press, USA, (2008).
[3] Glassner, A.S., Interactive storytelling:
techniques for 21st century fiction. Coyote Wind,
Canada, (2004).
[4] Guerraz, A., Lemordant, J. Indeterminate
Adaptive Digital Audio for Games on Mobiles: In
From Pac-Man to Pop Music, editors: Karen
Collins. Ashgate, (2007).
[5] Griesinger, D.H. “Concert Hall Acoustics and
Audience Perception” IEEE Signal Processing
Magazine March (2007)
[6] www.hauntedplanet.com
[7] Lasorsa1, Y., Lemordant, J. “An Interactive
Audio System for Mobiles” Audio Engineering
Society Convention Paper, Presented at the
127th Convention October 9–12 New York, NY,
USA (2009).
[8] Laurel, B., Computers as theatre, New York,
Addison-Wesley, (1993)
[9] Paterson, N., Naliuka, K., Jensen, S.K., Carrigy,
T., Haahr, M., Conway, F. “Spatial Audio and
Reverberation in an Augmented Reality Game
Sound Design,” 40th AES Conference: Spatial
Audio, Tokyo, Japan, October 8-10, 2010, Audio
Engineering Society, (2010).
[10] Pfeifer, T., Savage, P., Robinson, B. “Managing
the Culloden Battlefield Invisible Mobile
Guidance Experience”, MUCS2009, June 15,
Barcelona, Spain (2009).
[11] Reid, J., Geelhoed, E., Hull, R., Cater, K.,
Clayton, B. “Parallel Worlds: Immersion in
location-based experiences”. CHI '05 extended
abstracts on Human factors in computing
systems, (2005).
[12] Shepard, M. “The Tactical Sound Garden [TSG}
Toolkit”. International Conference on Computer
Graphics and Interactive Techniques, ACM
SIGGRAPH 2007 p 219 (2007).
[13] www.soundwalk.com
Paterson et al. Location Aware Interactive Game Audio
