Designing Immersive Sound for a Pervasive Mobile Game.
- [Show abstract] [Hide abstract]
ABSTRACT: Mobile phones are a widespread platform for ICT applications because they are highly pervasive in contemporary society. Hence, we can think of mobile gaming as a serious candidate to being a prominent form of entertainment in the near future. However, most games (for computers, console and mobile devices) make extensive use of the visual medium, which tends to exclude visually-impaired users from the play. While mobile gaming could potentially reach many visually-impaired users, who are very familiar with this technology, currently there seems to be only very few alternatives for this community. In an attempt to explore new interactive possibilities for such users, this work presents an initial study on non-visual interfaces for mobile phone games. It is based on Semiotic Engineering principles, emphasizing communication through aural, tactile and gestural signs, and deliberately excluding visual information. Results include a number of issues that can be incorporated to a wider research agenda on mobile gaming accessibility, both for the visually-impaired and sighted.Journal of the Brazilian Computer Society 01/2009; 15:45-58.
- [Show abstract] [Hide abstract]
ABSTRACT: Game audio has been suggested as a means of enhancing the play experiences on mobile devices. However, the field has seen little practical research on the subject. This study investigated the role of sound design in a mobile pervasive game in a prototype mobile game called The Songs of North. We examine the challenges faced by designers of mobile sound and demonstrate how some of the challenges can be overcome. Our design demonstrates how using sounds as a primary information channel can facilitate the use of physical movement as a main game mechanics. However, results from user tests highlight that using sound to convey information is an unfamiliar game mechanic to sighted players. Pervasive game sound also challenges the players' personal sound environment as it competes with music and causes social disturbance. We discuss the implications on mobile game sound in general and suggest directions for future work.Proceedings of the 2 nd AudioMostly Conference,(Röntgenbau, Ilmenau, Germany, September 27–28, 2007). AM; 01/2007
Designing Sound for a Pervasive Mobile
Inger Ekman*, Jussi Lahti, Jani Nummela, Petri Lankoski, Frans Mäyrä
University of Tampere
33014 University of Tampere
+358 3 215 7813*
firstname.lastname@example.org*, email@example.com, firstname.lastname@example.org, email@example.com,
We examine the role of sound design in designing pervasive mobile games. As a case study, we
examine the sound design process and solutions of a working prototype game called The Songs
of North. The goal has been a design that allows sounds to be used as a primary information
channel facilitating the use of physical movement as a main game mechanics. We discuss
insights from the sound design process and also generally consider the role of sound in
producing immersive gaming experiences.
Mobile games, pervasive gaming, sound design
Traditionally games are often thought of as a dominantly visual medium, and sound usually has
only a marginal role in the game design and development process. Although there are examples
of ingenious use of sound in mainstream games (for example the Thief series by Eidos), only a
few games give sound a central function in play functionality; many games are even fully
playable without sound. The same cannot be said for the opposite; and games rely heavily on
visual information, often making it necessary to constantly watch the screen while playing.
Recently, the development of mobile phones as gaming platforms has opened new opportunities
for novel game types. Whereas there has long existed handheld gaming devices (e.g. the
GameBoy), combining gaming with the communication possibilities of mobile phones has
opened a door to new applications, such as location-aware multiplayer games. The fact that one
has to consider when developing games for such platforms is the limited display capability of
such game devices. Also, there are several situations in which looking at a mobile device’s
screen may be impossible or unwanted, such as when driving a car or walking in a highly
trafficated area. By limiting the interaction to situations in which the player is stationary, one is
also setting a limit on the modes of gameplay; e.g. excluding games in which a player’s
movements in the physical world are a crucial aspect of playing. However, when the visual
capability of a person is limited (you can only look at a few things at the time and looking at
your gaming device will probably interfere with your looking at the street), the auditory channel
* Corresponding author
Proceedings of DiGRA 2005 Conference: Changing Views – Worlds in Play.
© 2005 Authors & Digital Games Research Association DiGRA. Personal and educational classroom use of this paper is
allowed, commercial use requires specific permission from the author.
can still be used for conveying information about the game; we can disregard something in the
audio landscape by not paying attention to it, which is what we usually do with muffled
conversations in café tables. On the other hand, when necessary, we can also focus our attention
towards a certain sound and really listen to it to extract the information that is being laid out for
our ears to hear.
This article examines the sound design process for a pervasive mobile game (Songs of North)
that is being developed as a research prototype to highlight the possibilities and design issues in
this game development area. The purpose is to identify what elements of sound design can be
used to support the different aspects of playability of a location-aware game that is being played
on a mobile phone. The article builds on ongoing research done at the Hypermedia Laboratory at
the University of Tampere. The structure of the paper is as follows: First we take a brief look at
related research in the areas of game sound and mobile audio. We then describe our prototype
game, The Songs of North, and the design and implementation of the game sounds and how the
purpose of the game affected the design choices made. We conclude with a discussion on how
the role of sound may change in mobile games and propose some directions for future work.
To date, there has been remarkably little research on game audio. Most often audio solutions for
gaming are searched for in earnest only when designing games for the blind and visually
impaired; see e.g. [4, 5, 13]. Thus, the motivation for much audio game development has been
accessibility (see e.g. the IGDA accessibility whitepaper ) rather than a realization of sounds
potential for richer interaction. However, a few approaches have also considered sound as an
alternative form of interaction for sighted players as well [3, 6, 11].
The recent development of mobile phones as gaming platforms may call for a change in this
attitude and highlight the importance of sound as offering novel opportunities from an interface
design perspective. Limitations on display capability of mobile devices restrict the games visual
expression, and using non-speech sound has been suggested to be the solution for making these
user interfaces easier to navigate. Research on mobile sound, on the other hand, has focused on
non-gaming applications. The Sonic City project  explored mobile interaction and wearable
technology as a tool for music creation. Sonic City is a wearable music system, which creates a
personal soundscape based on physical movement, local activity and urban ambiance. In another
project, Nomadic Radio, different information sources received by the mobile device are
combined to construct a dynamic information experience to the user .
THE SONGS OF NORTH: A CASE STUDY
The Songs of North is an enhanced-reality multiplayer game. The spirit world exists in parallel
with the physical world. To transport the spirit to a certain place in the spirit world, players have
to move around in the physical world. The game receives the movements of players through
GSM cell positioning.  A schematic overview of the game world is presented in Figure 1.
Players act as shamans fulfilling quests in the game by gathering and manipulating objects and
interacting with real and virtual characters. Interaction occurs via a virtual shaman drum
(implemented in the mobile interface). The interaction between players and non-player
characters can take the form of fights or collaboration. The game also supports messaging
Figure 1: An overview of the enhanced-reality game world. The shaman
spirit world of the game exists in parallel with the real world. Players’
movements are fed into the game system depending on their location in
the physical world.
Sound Design in The Songs of North
One of the main goals for the sound design of The Songs of North is to support playing without
watching the game device. Because this, sound should provide enough information for some
decision making in the game. With information as audio, players can move safely and monitor
the location of their spirits and events in the game world. We also wanted to provide strong
auditory support for visually perceivable events, to heighten the sense of an immersive game
world. The moments when the player looks at the visual information simultaneously with hearing
audio can also act as learning points introducing audio as a primary information channel.
The basic idea in the audiovisual design of the game is that the spirit world can be heard real-
time, whereas getting up-to-date visual information requires casting spells with a shaman drum
(mobile device). This way, audio is ‘free’ in the sense that it doesn’t require the player to do
anything. Since information is available visually as well, players can gradually shift to using
audio information as they realize the benefits of movement for playing. A listener can also
process audio in the background and notice changes in sounds even if they were not directly
attending to them . This way, since audio is not the only channel of information, sound can be
used as a compromise between active playing and not attending to the game at all. Thus, when
players notice something requiring their attention, they can pick up their shaman drum and
require visual information. This design is intended to encourage players to use the audio
information as an event signal even before they know exactly what a particular sound means, and
thus support physical movement as a main game mechanic from the very beginning of the game.
Game sounds are used to convey information of places, objects and characters nearby as well as
movement or other actions of characters within a certain range. Characters, places and game
objects have specific sounds, which are played depending on their range from the player. As the
players move in the real world, or as NPCs move within the game world, the distances change.
Movements are reflected in the soundscape as addition or deletion of sounds.
There were seven types of sounds to be played for a player:
·Player’s own drum hit sounds
·Drum hit pattern sounds for other players’ actions
·Item sounds for different item classes
· Place sounds either for unique places or place types in the game world
· Attack sounds for different attack types (or elements)
·Ambient sounds that represent the world’s current state (good, neutral or evil)
The Java-implementation for Nokia N-Gage (our test device) did not support playing several
sounds at once. Because of this, sounds had to be queued in order of play priority and played
when possible. Since the drum is the player’s primary tool for manipulating the game world, a
design decision was made to give the topmost priority to the player’s own drum hits. Whenever a
player uses the drum to make a spell, all other game sounds quiet down and the drum sound is
All other sounds are queued and played one by one. Every time the game device observes the
game world (at least once every 10 seconds), the game world is examined to decide, what
objects, characters and places are presently in the proximity of the character. The sounds of
these, as well as the state sound describing the nature of the game world are queued in
chronological order, making the most recent sound last in the queue. The maximum number of
sounds in the queue at once is eight and too old sounds are deleted from the queue as newer ones
arrive. The same sound can appear in the queue only once, thus a sound is not added if the queue
already holds one prior instance of it. Sounds are popped and played from the queue with a
random amount of silence between them. Additionally, to avoid unnecessary repetition, the
ambient sound (representing the game world’s current state), which would otherwise always be
in the queue, was played only once every three minutes.
To make the soundscape more diverse, characters could produce several different sounds.
Character sounds were either specific or generic. The specific character sound was a sonic
symbol for the type and current state of the character (indication of totem animal, modified
according to three possible character states: attack, damage and normal). Alternatively the
character’s presence was conveyed via a generic footstep sound. Sometimes characters did not
produce sounds at all. The probabilities for choosing the character sound are: 0.5 for the generic
movement sound, 0.33 for character class/state and 0.17 for silence.
In order to make the identification of sounds as easy as possible, we chose to use very realistic
sounds to depict events and characters (footsteps for walking, the sounds of a howling wolf for
wolf-shamans etc.). An alternative would have been to use more abstract sound, such as earcons
(see e.g. ). Research on different types of sonic identifiers has shown that auditory icons are
more readily learned and remembered than abstract sound . Moreover, the use of more
realistic sounds for game objects and events was not only based on the recognition of sounds, but
also a stylistic choice to lend the game world a soundscape that could be perceived as a shaman
Emergent Issues in Sound Design for Mobile Games
The first and foremost issue we had to deal with when making the sound design and
implementation was the technological limitations of the mobile device. Technically, such issues
as limited storage and the low-level controllability of sound somewhat tie the hands of designers
if they are accustomed to the solutions available in pc or console gaming. Moreover, the sound
quality of the mobile devices is often optimized for speech, and may not scale very well for all
Another issue worth considering is the difference in listening situation compared to a traditional
stationary gaming setting. The usual speaker settings that are part of home systems will often
have some spatial sound available (at least a left/right distinction), but this is not guaranteed with
mobile games Although headphones are available, it is uncertain whether people will use them or
rather have the phone in their pocket. If sounds are played on the speakers of the phone, they do
not necessarily serve to immerse players in the game world, but may instead highlight the
presence of the game device at the expense of immersion. Also, with enhanced-reality mobile
gaming, spatial sound cues require a knowledge of which way the player is headed, which sets
the demand for special technology such as, for example, a compass integrated into headphones
with feedback to the game device. Furthermore, bringing gaming out of the living room will
mean that the sound design has to take into account all sorts of noises, which may affect the
intelligibility of sounds relevant for the player. In a mobile gaming situation, the sound has to
deal with the various sound contexts of everyday life. Background noise, conversations and other
activities may affect the intelligibility of game audio or mask the sounds so that they cannot be
Also something worth consideration is the social playability of mobile games. Whereas sound
design for console and PC games could concentrate just on making a brilliant soundscape, social
playability is an important issue in mobile game sound design. This is an issue that is rather
unique for mobile games. Sound design for mobile devices has to consider also how other
people, not at all engaged with playing, will relate to the soundscape. For example, the ideal
volume of the sounds may vary depending on the situation a player is in, not only providing
maximum intelligibility, but social acceptability as well. Social playability also means
considering the gaming context in a broader sense. This involves designing the game interaction
as a whole in such a way that it fits in with the players activities. This is not to imply that games
should not try to change players’ attitudes. However, what we argue is that when designing
games to be played in a social context, sound design (as well as the design of any other
information channel) should not be seen as independent design tasks. For example, we have
argued that the use of sound can be used as a means to minimize the need for visual interaction
with the device. In addition to being essential to play functionality (allowing the players to use
movement as a game mechanics), this also means that it is possible for a player to most of the
time play unnoticed by others. This may be a crucial factor for some player groups, if they do not
want to be recognized as players when out in the city.
The notion of social playability can also be thought of as connected with a larger shift in playing
habits. What mobile is bringing to gaming is the ability for people to engage in play wherever
they are – at a bus stop, in the workplace cafeteria, on the train. Together with the
communication possibilities of mobile devices, there is an underlying potential for a shift in how
we look at the whole (computer) gaming experience: from a solitary and foremost private
activity towards social interaction within a gaming context. With time, it is probable also that
attitudes towards gaming in a social context changes and games and gaming (and, also, game
sounds) are seen as a normal part of the everyday activities of people.
As a final note, we view enhanced reality gaming not only as providing dizzying gaming
experiences. Ultimately, we think that the combination of moving around in the physical world
in parallel with the sense of moving around in a fictional world, where the player has the ability
to manipulate and affect the environment, could enhance the players’ experience of their
everyday surroundings. Successful design for immersion may encourage players to view the
physical world with a new attitude. Eventually, the game world might well add color and
meaning to our everyday events, making a trip to the supermarket a challenging endeavor into a
fierce battleground or walking the dog a trip to the treasury.
CONCLUSIONS AND FUTURE WORK
We have examined the role of sound design in a mobile pervasive game in a prototype mobile
game called The Songs of North. The work presented here is part of ongoing research done at the
Hypermedia Laboratory at the University of Tampere. In the next stage of research, we will
continue to develop the game prototype as well as perform user tests on the sound design and
implementation. One task is to make the audiovisual integration stronger, and more profoundly
define the roles the visual and audio components play in the interface.
Another interesting question related to this is what different roles sound can play in the interface.
The research described in this paper examined the use of sound to provide real-time task-oriented
information on events, objects, characters and places within proximity of the player in the game
world. In the future we will examine how (or, indeed if) sound can be used in other roles as well,
such as providing background information on the game world or to lend emotional impact to
events and places in the game world.
We thank Satu Heliö for doing a tremendous job with the quest and story design in The Songs of
North. The project is funded by National Technology Agency TEKES, Codetoys, the Finnish
national lottery Veikkaus, Nokia Research Center and TeliaSonera Finland.
1Blattner, M., Sumikawa, D. and Greenberg, R. Earcons and icons: Their structure and common design
principles. Human Computer Interaction, 4, 1 (1989), 11–44.
2Cohen, J. Monitoring background activities. In Kramer, G. (ed.) Auditory Display. SFI Studies in the
Sciences of Complexity. Addison-Wesley, 1994.
Drewes, T.; Mynatt, E. & Gandy, M. Sleuth: An Audio Experience. Proc. International Conference on
Auditory Display. Atlanta, GA, USA, 2000.
Eriksson, Y & Gärdenfors, D. Computer Games for Children with Visual Impairments. Proc. 5th
International Conference of Disability, Virtual Reality & Associated Technologies, Oxford, UK, 2004,
Friberg, J. & Gärdenfors, D. Audio Games: New Perspectives on Game Audio. Proc. Advances in
Computer Entertainment Technology. Singapore, 2004.
Hadrup, R., Jakobsen, P. S., Juul, M. S., Lings, D., Magnúsdóttir, Å., O. Designing an Auditory W-
LAN%20based%20Game.pdf [Accessed February 15, 2005.]
IGDA. Accessibility in Games: Motivations and Approaches. International Game Developers
Association white paper. 2004. http://www.igda.org/accessibility/IGDA_Accessibility_WhitePaper.pdf
[Accessed February 15, 2005.]
Keller, P. & Stevens, C. (2004). Meaning From Environmental Sounds: Types of Signal-Referent
Relations and Their Effect on Recognizing Auditory Icons. Journal of Experimental Psychology:
Applied, 10, 1 (2004), 3–12.
Lankoski, P.; Heliö, S.; Nummela, J.; Lahti, J.; Mäyrä, F.; Ermi, L. (2004). A Case Study in Pervasive
Game Design: The Songs of North. Proc. NordiCHI, Tampere, Finland, 2004, 413–416.
Mazé, R. & Jacobs, M. Sonic City: Prototyping a Wearable Experience. Proc. 7th IEEE International
Symposium on Wearable Computing. 2003.
McCrindle, R. J. & Symons, D. Audio Space Invaders. Proceedings of the 3rd International
Conference of Disability, Virtual Reality & Associated Technologies, Alghero, Italy, 2000, 59–65.
Sawhney, N. & Schmandt, C. Nomadic Radio: Speech and Audio Interaction for Contextual
Messaging in Nomadic Environments. ACM Transactions on Computer-Human Interaction, 7, 3
Targett, S. & Fernström, M. Audio Games: Fun for All? All for Fun? Proc. International Conference
on Auditory Display, Boston, MA, USA, 2003.