ChapterPDF Available

When the Soundtrack Is the Game: From Audio-Games to Gaming the Music

Authors:

Abstract

There has been an increasing popularity in audio-only games, particularly for mobile devices. These are usually based on spoken-word, sound-effect scene setting, or DJ/musical-instrument simulation. This chapter surveys the field and discusses the concept of games which are entirely abstract musical experiences, presenting a first example based on Minecraft and music memory games: Musicraft. The comparisons with Minecraft can be extended into the idea of a creative mode: a quite natural approach in music-only games which has not generally been considered in audiogame research.
... People with visual impairment (PVI) have difculty accessing digital games due to the inherently visual nature of such games [15,98]. Although some solutions for PVI to engage with digital games have emerged, particularly in the form of audiogames and text-based adventures, providing PVI access to mainstream gaming experiences remains a challenge [4,38,85]. One particular challenge PVI face when engaging with digital games is difculty in acquiring information about the spatial characteristics of the virtual world. ...
... There is a growing interest in digital games targeted to PVI (commonly referred to as audiogames) and in improving accessibility in mainstream games [4,38,85]. However, for PVI, the transition from 2D to 3D graphics was an efective barrier that has prevented them from engaging with modern mainstream games [4,85]. ...
... It has been argued that playing games is a fundamental human desire [34] and specifc tools have been designed to bring access to PVI to specifc games [41,84,97]. However, one recurrent challenge these tools face is the sequential and non-persistent nature of sound [2,38]. Integrating diferent means to use active echolocation in virtual worlds-through sound pulses, footsteps, clapping sounds or others-may help address these challenges by giving agency to the player, who will decide when and how to receive information from their surroundings. ...
Article
Full-text available
In virtual environments, spatial information is communicated visually. This prevents people with visual impairment (PVI) from accessing such spaces. In this article, we investigate whether echolocation could be used as a tool to convey spatial information by answering the following research questions: What features of virtual space can be perceived by PVI through the use of echolocation? How does active echolocation support PVI in acquiring spatial knowledge of a virtual space? And what are PVI’s opinions regarding the use of echolocation to acquire landmark and survey knowledge of virtual space? To answer these questions, we conducted a two-part within-subjects experiment with 12 people who were blind or had a visual impairment and found that size and materials of rooms and 90-degree turns were detectable through echolocation, participants preferred using echoes derived from footsteps rather than from artificial sound pulses, and echolocation supported the acquisition of mental maps of a virtual space. Ultimately, we propose that appropriately designed echolocation in virtual environments improves understanding of spatial information and access to digital games for PVI.
... Some games have focused on assisting PVI to navigate physical locations by reproducing these locations in a virtual environment, obtaining improved orientation and mobility outcomes [32,94,95]. Others have focused on the use of videogames for educational purposes, designing interesting interaction mechanisms such as compass, radar, and musical metaphors [66,103]. Others have argued for the need of equitably accessible games, and designed videogames for PVI for hedonic purposes [100]. ...
... By being text-based, MUDs are accessible to existing accessibility software. Audiogames are a type of digital game that includes an entirely auditory interface and may or may not include a graphical interface [45,66]. The engagement of GVI with mainstream games depends on their remaining sensory capabilities. ...
... For example, some games have focused on assisting PVI to navigate physical locations by reproducing these locations in a virtual environment, obtaining improved orientation and mobility outcomes [34,94,95]. Some videogames have been designed for use in formal educational settings, utilising interesting interaction mechanisms which have been replicated in audiogames, such as compass, radar, and musical metaphors [66,103]. Other researchers have argued for the need for equitably accessible games, and designed videogames for PVI for hedonic purposes [100]. ...
Conference Paper
Full-text available
Previous research on games for people with visual impairment (PVI) has focused on co-designing or evaluating specific games - mostly under controlled conditions. In this research, we follow a game-agnostic, "in-the-wild" approach, investigating the habits, opinions and concerns of PVI regarding digital games. To explore these issues, we conducted an online survey and follow-up interviews with gamers with VI (GVI). Dominant themes from our analysis include the particular appeal of digital games to GVI, the importance of social trajectories and histories of gameplay, the need to balance complexity and accessibility in both games targeted to PVI and mainstream games, opinions about the state of the gaming industry, and accessibility concerns around new and emerging technologies such as VR and AR. Our study gives voice to an underrepresented group in the gaming community. Understanding the practices, experiences and motivations of GVI provides a valuable foundation for informing development of more inclusive games.
... color and contrast options) and depend on audio feedback. Past research has explored diferent strategies to adapt games that require vision to an accessible audio-based gameplay [4,33,39,40,48,58,61]. In most cases a substantial redesign is needed to ensure that the player is able to perceive the information, determine the correct action, and provide input. ...
... In some audio-based games there's an analogy with real-life navigation systems, such as compass and sonar like features [2,58]. Musicraft [33] is an audio game inspired by Minecraft, in which the world is depicted by abstract musical representations. ...
... Compilations such as Audio Game Hub 31 and Sammy Senter 32 explore different mechanics and similar mechanics applied to different narrative contexts. Curious games originate from real-life challenges where seeing is unimportant such as lock-picking 33 and musical performance [19,49,50]. ...
... https://store.steampowered.com/app/247020/Cook_ Serve_Delicious/ (Last visited on October 23rd, 2020) 50 VR The Diner Duo. https://store.steampowered.com/app/530120/VR_The_ ...
Thesis
Full-text available
Noticeably, the majority of mainstream games — digital games and tabletop games — are still designed for players with a standard set of abilities. As such, people with some form of disability, often face insurmountable challenges to play mainstream games or are limited to play games specifically designed for them. By conducting an initial study, we share multiplayer gaming experiences of people with visual impairments collected from interviews with 10 adults and 10 minors, and 140 responses to an online survey. We include the perspectives of 17 sighted people who play with someone who has a visual impairment, collected in a second online survey. We found that people with visual impairments are playing diverse games, but face limitations in playing with others who have different visual abilities. What stood out is the lack of intersection in gaming opportunities, and consequently, in habits and interests of people with different visual abilities. In this study, we highlight barriers associated with these experiences beyond inaccessibility issues and discuss implications and opportunities for the design of mixed-ability gaming. As expected, we found a worrying absence of games that cater to different abilities. In this context, we explored ability-based asymmetric roles as a design approach to create engaging and challenging mixed-ability play. We designed and developed two collaborative testbed games exploring asymmetric interdependent roles. In a remote study with 13 mixed-visual-ability pairs we assessed how roles affected perceptions of engagement, competence, and autonomy, using a mixed-methods approach. The games provided an engaging and challenging experience, in which differences in visual ability were not limiting. Our results underline how experiences unequal by design can give rise to an equitable joint experience.
... color and contrast options) and depend on audio feedback. Past research has explored different strategies to adapt games that require vision to an accessible audio-based gameplay [4,33,39,40,48,58,61]. In most cases a substantial redesign is needed to ensure that the player is able to perceive the information, determine the correct action, and provide input. ...
... In some audio-based games there's an analogy with real-life navigation systems, such as compass and sonar like features [2,58]. Musicraft [33] is an audio game inspired by Minecraft, in which the world is depicted by abstract musical representations. ...
Preprint
Full-text available
The landscape of digital games is segregated by player ability. For example, sighted players have a multitude of highly visual games at their disposal, while blind players may choose from a variety of audio games. Attempts at improving cross-ability access to any of those are often limited in the experience they provide, or disregard multiplayer experiences. We explore ability-based asymmetric roles as a design approach to create engaging and challenging mixed-ability play. Our team designed and developed two collaborative testbed games exploring asymmetric interdependent roles. In a remote study with 13 mixed-visual-ability pairs we assessed how roles affected perceptions of engagement, competence, and autonomy, using a mixed-methods approach. The games provided an engaging and challenging experience, in which differences in visual ability were not limiting. Our results underline how experiences unequal by design can give rise to an equitable joint experience.
... Interactive fction (also known as text adventures), text-based MMORPGs and MUDs are also popular among people with visual impairments [2]. Past research shows rhythm/music games to be efective in engaging both visually impaired and sighted players [18,29,30,53]. Other studies aimed to design multiplayer games for groups with mixed visual abilities. ...
Conference Paper
Full-text available
Games bring people together in immersive and challenging interactions. In this paper, we share multiplayer gaming experiences of people with visual impairments collected from interviews with 10 adults and 10 minors, and 140 responses to an online survey. We include the perspectives of 17 sighted people who play with someone who has a visual impairment, collected in a second online survey. Our focus is on group play, particularly on the problems and opportunities that arise from mixed-visual-ability scenarios. These show that people with visual impairments are playing diverse games, but face limitations in playing with others who have diferent visual abilities. What stands out is the lack of intersection in gaming opportunities, and consequently, in habits and interests of people with diferent visual abilities. We highlight barriers associated with these experiences beyond inaccessibility issues and discuss implications and opportunities for the design of mixed-ability gaming.
... These include two key relationships and a set of in-game metaphors that support these relationships. Whereas previous research has provided an overview of games commonly played by PVI [49], or has explored the opinions, habits, and concerns of GVI [6,85], to the best of our knowledge this is the first attempt at providing a broader exploration of the elements that enable PVI to access digital games. With this research, we aim to provide a tool for game designers and researchers to think about the key relationships that enable PVI to access digital games. ...
... One of the most well researched areas in game accessibility is visual disabilities, and this trend seems to continue. Recent contributions to this field of research include Musicraft [30], a novel audio game concept for music creation where the soundtrack is the game itself. Furthermore, there is SoniFight, a free software under MIT licence to make fighting games accessible with a retrofit approach, including customisation of profiles and sounds for different games as well as screen reader support [31]. ...
Article
Full-text available
INTRODUCTION: The problem addressed in this work is the lack of knowledge of what inclusive game design would mean in practice within existing design processes of game companies. A pilot project was devised to involve both the game industry and disabled people. OBJECTIVES: The goal in this study was to identify activities that constitute the biggest obstacles to realising sustainable design processes for inclusive game design. METHODS: The study is mainly based on two full-day workshops with the game industry and three game studios, three organisations of disabled youth and authorities. RESULTS: Five activities were identified in the analysis of the workshops: 1) Find opportunities for inclusive game design; 2) Raise awareness about inclusive game design; 3) Handle integrity and security; 4) Recruit the right competence; and 5) Adapt workplaces and tools. CONCLUSION: The five main activities should be considered to achieve sustainable inclusive game design processes.
Article
As virtual environments—in the form of videogames and augmented and virtual reality experiences—become more popular, it is important to ensure that they are accessible to all. Previous research has identified echolocation as a useful interaction approach to enable people with visual impairment to access virtual environments. In this paper, we further investigate the usefulness of echolocation to explore virtual environments. We follow a participatory design approach that comprised a focus group session coupled with two fast prototyping and evaluation iterations. During the focus group session, expert echolocators produced a series of seven design recommendations, of which we implemented and trialed four. Our trials revealed that the use of ambient sounds, the ability to place landmarks, directional control, and the ability to use pre-recorded mouth-clicks produced by expert echolocators improved the overall experience of our participants by facilitating the detection of openings and obstacles. The recommendations presented and evaluated in this paper may help to develop virtual environments that support a broader range of users while recognising the value of the lived experience of people with disability as a source of knowledge.
Chapter
Gamemusik unterhält unter anderem eine wichtige Verbindung zu Phänomenen am Rande des Musikalischen. Geräusche und andere amorphe Schallvorgänge können sich mit der Musik in Spielen verbinden und gemeinsam ganz neue Ästhetiken ausformen. In unserem Beitrag analysieren wir eine Passage aus Inside (Playdead, 2016), in der es zu einer solchen Verbindung kommt und beleuchten sie im Zusammenhang mit dem Begriff der musique concrète. Dabei zeigen wir auf, was die Musik von Inside einzigartig, innovativ und wegweisend für die Zukunft macht.
Conference Paper
Full-text available
Training blind children to use audio-based navigation is a demanding and risky task, as children can walk into objects and hurt themselves. Furthermore, training outdoors is dangerous due to traffic, noise and weather conditions. Having a controlled indoor environment is safer but not always available. To tackle this problem, we developed an audio-based computer game, Legend of Iris (LOI), specifically designed to train navigation skills. The game is a 3D exploration game, which uses the headtracking capabilities of the Oculus Rift to create an immersive experience, and the new sound libraries AstoundSound and Phonon3D, to generate an accurate and realistic soundscape. These libraries use a head-related transfer function, allowing the player to localize the audio source in 3D space. The design of LOI involved selecting sounds that are easily recognizable to provide cues to blind people playing the game. A subset of these cues were incorporated into the game. To verify the effectiveness of the game in developing audio orientation and navigation skills, we performed a preliminary qualitative experiment with blind children in a dedicated school. LOI scored high in terms of accuracy and immersion, but a larger test is required to make statistical conclusions.
Conference Paper
Full-text available
Ambisonics is a 3D recording and playback method that is based on the representation of the sound field excitation as a decomposition into spherical harmonics. This representation facilitates spatial sound production that is independent of the playback system. The adaptation to a given playback system (loudspeakers or motion-tracked headphones) is achieved by a suitable decoder. This contribution gives an overview of the current state-of-the-art in Ambisonics including content produc- Tion using Ambisonic main microphone arrays or panning of virtual sources, spatial effects, and reproduction by loudspeakers and headphones. The software for the whole production chain is already available as a VST-plugin suite for digital audio workstations.
Article
Full-text available
This paper summarizes recent developments in audio and tactile feedback based assistive technologies targeting the blind community. Current technology allows applications to be efficiently distributed and run on mobile and handheld devices, even in cases where computational requirements are significant. As a result, electronic travel aids, navigational assistance modules, text-to-speech applications, as well as virtual audio displays which combine audio with haptic channels are becoming integrated into standard mobile devices. This trend, combined with the appearance of increasingly user-friendly interfaces and modes of interaction has opened a variety of new perspectives for the rehabilitation and training of users with visual impairments. The goal of this paper is to provide an overview of these developments based on recent advances in basic research and application development. Using this overview as a foundation, an agenda is outlined for future research in mobile interaction design with respect to users with special needs, as well as ultimately in relation to sensor-bridging applications in general.
Article
Full-text available
Text-based games offer a compelling opportunity for sighted and non-sighted players to interact. Studies on navigation show that spatial mental models are encoded differently across both groups of players. While sighted people tend to view the environment from an overview representation (allocentric), non-sighted people more frequently do so from an egocentric perspective. We investigate whether a text-based game‟s navigation system should be presented using both perspectives to support immersion for sighted and non-sighted players alike. An evaluation of the fit of both systems using a between-subject user study supports the hypothesis that egocentric directions are easier to use and more immersive for non-sighted players, while sighted players perform better with allocentric directions. The discussion offers insights into how these results can be interpreted and operationalised for the design of text-based games or other inclusive software.
Chapter
Over the past four decades video games have become a worldwide phenomenon and one of the preferred leisure options for many. However, game accessibility remains a pending issue, due to the visual and interactive nature of the medium. Mangiron and Zhang analyse the current state of game accessibility, particularly for the blind and visually impaired. They discuss the barriers and gaming options for this target group, such as audio games and ‘video-less’ games. Finally, the chapter explores the feasibility of applying audio description to video games, which would potentially enhance game accessibility for blind and visually impaired players.
Conference Paper
Audio-Games (AGs) are electronic games that mainly (or exclusively) implement auditory means to express the game’s plot, mechanics and content. AG players need to concentrate on perceiving sound, in order to understand and play the game. Recent developments in the field of edutainment suggest that AGs can be implemented not only as entertaining systems, but also as valid tools for research and education on any curriculum related (but no limited) to acoustics and sound studies. This paper will first discuss existing approaches to organizing the sonic content of an AG. Then it will propose a methodology for designing an AG for educational purposes. Finally, a case–study consisting of the development of two educational AGs related to the issue of noise will be demonstrated, on which the proposed methodology is applied.
Article
A lot of video games on the market are inaccessible to players with visual impairments because they rely heavily on use of graphical elements. This paper presents a project aimed at developing a point-and-click adventure game for smart phones and tablets that is equally functional and enjoyable by blind and sighted players. This will be achieved by utilizing audio to give blind players all necessary information and enjoyment without graphics. In addition to creating the game, the aim of the project is to identify design aspects that can be applied to more types of games to include more players. This paper also presents a pilot study that has been conducted on an early version of the game and the preliminary findings are discussed.
Conference Paper
This paper examines the use of sound in designing and structuring the gameplay of Audio-Based games. The classification into different categories and the discussion of existing techniques conclude that there are issues that need to be dealt with: the conflict between a strict system of rules and free improvisation, the sonic description of visual elements and the limited educational role. A brief outline of the work-in-project Audio-Based Role-Playing-Game 'Kronos' suggests how a game can be merged with an instrument on an educational basis.