Atau Tanaka

Goldsmiths, University of London · Department of Computing

This chapter proposes a parametric model that is useful in audio-visual instrument design, composition and performance. We can draw a separation between those activities, but in practice that separation might not be so obvious: ultimately, the iterative creation process must always consider the final, global experience. Derived from a perceptual ap...

INTRODUCTION: We propose AudioVisual User Interfaces (AVUI), a novel type of UI linking interaction, sound and image. It extends the concept of Graphical User Interface (GUI) by adding interconnected sound and image. OBJECTIVES: We aim to situate AVUIs in relation to identified relevant fields: Human-Computer Interaction (HCI), sonic interaction de...

This chapter explores three systems for mapping embodied gesture, acquired with electromyography and motion sensing, to sound synthesis. A pilot study using granular synthesis is presented, followed by studies employing corpus-based concatenative synthesis, where small sound units are organized by derived timbral features. We use interactive machin...

This chapter presents an overview of Interactive Machine Learning (IML) techniques applied to the analysis and design of musical gestures. We go through the main challenges and needs related to capturing, analysing, and applying IML techniques to human bodily gestures with the purpose of performing with sound synthesis systems. We discuss how diffe...

Musical performance can be thought of in multimodal terms – physical interaction with musical instruments produces sound output, often while the performer is visually reading a score. Digital Musical Instrument (DMI) design merges tenets of HCI and musical instrument practice. Audiovisual performance and other forms of multimedia might benefit from...

This work explores the potential of a set comprised of wearable sensors, a performative lighting installation, and a public museum space, to inspire performative and collaborative social behavior among members of the public. Our installation, The Light, was first exhibited as part of the Late at Tate Britain event in 2019. In this paper we discuss...

Digital Musical Instruments as Probes: How Computation Changes the Mode-of-Being of Musical Instruments - Volume 25 Issue 2 - Koray Tahiroğlu, Thor Magnusson, Adam Parkinson, Iris Garrefls, Atau Tanaka

We present a sonic interaction design approach that makes use of deep reinforcement learning to explore many mapping possibilities between input sensor data streams and sound synthesis parameters. The user can give feedback to an artificial agent about the mappings proposed by the latter while playing the synthesiser and trying the new mappings on...

This article explores how computation opens up possibilities for new musical practices to emerge through technology design. Using the notion of the cultural probe as a lens, we consider the digital musical instrument as an experimental device that yields findings across the fields of music, sociology, and acoustics. As part of an artistic-research...

This paper presents a method for mapping embodied gesture , acquired with electromyography and motion sensing, to a corpus of small sound units, organised by derived timbral features using concatenative synthesis. Gestures and sounds can be associated directly using individual units and static poses, or by using a sound tracing method that leverage...

In this paper, we describe the definition of the body being extended by the concepts of other disciplines, such as philosophy, where some define it as decided by the potential of its actions. Inspired by these philosophical ideas of the body, a dance performance, Skin-awareness, was developed to explore and experiment with the body as a self-aware...

The term 'spatial presence' refers to the feeling of presence in a mediated space. This subjective experience has been discussed in media theory, sound art, film and performance. It depends on multiple variables, or parameters. This paper presents a parametric model that can be used to analyze those variables and their relationships. It exposes met...

The term ‘spatial presence’ refers to the feeling of presence in a mediated space. This subjective experience has been discussed in media theory, sound art, film and performance. It depends on multiple variables, or parameters. This article presents a parametric model that can be used to analyze those variables and their relationships. It exposes m...

We present a system that allows users to try different ways to train neural networks and temporal modelling to associate gestures with time-varying sound. We created a software framework for this and evaluated it in a workshop-based study. We build upon research in sound tracing and mapping-by-demonstration to ask participants to design gestures fo...

In this paper, we consider the effect of co-adaptive learning on the training and evaluation of real-time, interactive machine learning systems, referring to specific examples in our work on action-perception loops, feedback for virtual tasks, and training of regression and temporal models. Through these studies we have encountered challenges when...

Music is a natural partner to human-computer interaction, offering tasks and use cases for novel forms of interaction. The richness of the relationship between a performer and their instrument in expressive musical performance can provide valuable insight to human-computer interaction (HCI) researchers interested in applying these forms of deep int...

The presence of the phenomenological body is central to music in all of its varieties and contradictions. With the explosion of scholarly works on the body in virtually every field in the humanities, the social as well as the biomedical sciences, the question of how such a complex understanding of the body is related to music, with its own complexi...

A 'quick and easy' solution for exploring Thalmic Labs Myo's potential for realising new interfaces for musical expression.

Myo Mapper is a free and open source cross-platform application to map data from the gestural device Myo armband into Open Sound Control (OSC) messages. It provides an easy to use tool for musicians to explore the Myo's potential for creating new gesture-based musical interfaces. Together with details of the software, this paper reports on projects...

There is an identified lack of visual feedback in electronic music performances. Live visuals have been used to fill in this gap. However, there is a scarcity of studies that analyze the effectiveness of live visuals in conveying feedback. In this paper, we aim to study the contribution of live visuals to the understanding of electronic music perfo...

In this paper we present two datasets of instrumental gestures performed with expressive variations: five violinists performing standard pedagogical phrases with variation in dynamics and tempo; and two pianists performing a repertoire piece with variations in tempo, dynamics and articulation. We show the utility of these datasets by highlighting t...

Physiological computing has become widespread, as the democratization of biomedical technolo- gies has facilitated take-up in interactive computing systems. Signals from the human body can be detected by a wide array of sensors and digitized, providing computational systems information on individual identification, body states, and gross and fine l...

The new field of mobile music emerges at the intersection of ubiquitous computing, portable audio technology and NIME. We have held a series of international workshop on this topic with leading projects and speakers, in order to establish a community and stimulate the development of the field. In this report, we define mobile music, and map out the...

This paper describes a technique of multimodal, multichannel control of electronic musical devices using two control methodologies, the electromyogram (EMG) and relative position sensing. Requirements for the application of multimodal interaction theory in the musical domain are discussed. We introduce the concept of bidirectional complementarity t...

The combined use of sound and image has a rich history, from audiovisual artworks to research exploring the potential of data visualization and sonification. However, we lack standard tools or guidelines for audiovisual (AV) interaction design, particularly for live performance. We propose the AVUI (AudioVisual User Interface), where sound and imag...

Movement design is typically based on evoking shapes in space. In interactive systems, user movement is often dictated by the system's sensing capabilities. In neither of these cases are the differences across individual users or expressive variations they make accommodated. We present an exploratory study that uses Laban Movement Analysis as a fra...

This note presents a system that learns expressive and idiosyncratic gesture variations for gesture-based interaction. The system is used as an interaction technique in a music conducting scenario where gesture variations drive music articulation. A simple model based on Gaussian Mixture Modeling is used to allow the user to configure the system by...

We present the Haptic Wave, a device that allows cross-modal mapping of digital audio to the haptic domain, intended for use by audio producers/engineers with visual impairments. We describe a series of participatory design activities adapted to non-sighted users where the act of prototyping facilitates dialog. A series of workshops scoping user ne...

Machine learning is one of the most important and successful techniques in contemporary computer science. It involves the statistical inference of models (such as classifiers) from data. It is often conceived in a very impersonal way, with algorithms working autonomously on passively collected data. However, this viewpoint hides considerable human...

We demonstrate the Haptic Wave, a device that allows audio engineers with visual impairments to "feel" the amplitude of sound, gaining salient information that sighted engineers get through visual waveforms. The demo will allow visitors, sighted or visually-impaired, to sweep backwards and forwards through audio recordings (snippets of pop songs an...

We examine how auditory displays, sonification and haptic interaction design can support visually impaired sound engineers, musicians and audio production specialists access to digital audio workstation. We describe a user-centred approach that incorporates various participatory design techniques to help make the design process accessible to this p...

We present a user-centered approach for prototyping tools for performance with procedural sound and graphics, based on a hackathon. We also present the resulting prototypes. These prototypes respond to a challenge originating from earlier stages of the research: to combine ease-of-use with expressiveness and visibility of interaction in tools for a...

Sonic interaction is the continuous relationship between user actions and sound, mediated by some technology. Because interaction with sound may be task oriented or experience-based it is important to understand the nature of action-sound relationships in order to design rich sonic interactions. We propose a participatory approach to sonic interact...

Expressivity is a visceral capacity of the human body. To understand what makes a gesture expressive, we need to consider not only its spatial placement and orientation, but also its dynamics and the mechanisms enacting them. We start by defining gesture and gesture expressivity, and then present fundamental aspects of muscle activity and ways to c...

The author presents the challenges and opportunities in the use of the electromyogram (EMG), a signal representing muscle activity, for digital musical instrument applications. The author presents basic mapping paradigms and the place of the EMG in multimodal interaction and describes initial trials in machine learning. It is proposed that nonlinea...

Whalley, Mavros and Furniss (this issue) explore questions of agency, control and interaction, as well as the embodied nature of musical performance in relation to the use of human-computer interaction through the work Clasp Together (beta) for small ensemble and live electronics. The underlying concept of the piece focuses on direct mapping of a h...

The design of and performance with sensor-based musical instruments poses specific opportunities and challenges in the translation of the performer's physical gestures into sound. The use of muscle biosignals allows directly integrating aspects of a performer's physical gesture into the human-machine interaction and compositional strategies which c...

This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations bet...

This article presents a gesture recognition/adaptation system for human--computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a...

There is a growing interest in audiovisual performance and composition. In this paper, we would like to investigate if the tools for audiovisual performance and composition have caught up with this growing interest and the practices in the field. In particular, we are interested in tools that use computer-­generated graphics. To address these issue...

The text reports a study, which draws upon methods from experimental psychology to inform audio-visual instrument design. The study aims at gleaning how an audio-visual mapping can produce a sense of causation, and simultaneously confound the actual cause and effect relationships. We call this a fungible audio-visual mapping. The participants in th...

This volume investigates the ramifications of mobile music devices and technologies on musical/sonic performance and aesthetics. It thinks together artistic production with the performances of quotidian life and argues that “mobility” is not the same thing as actual “movement,” demonstrating that mobile performance concerns the perception or signif...

This paper presents a process-based approach to considering workshops as a route to participation in collective creative musical practice. We evoke the notion of the Music One Participates In, which focuses on a shift from the listener-as-consumer to participant-actor actively engaged in sound perception and production. We look at the range of diff...

We report on our experiences of two recent sonification pieces we have been involved in; composing for Peter Sinclair’s RoadMusic project, and creating a multi-speaker installation with sound artist Kaffe Matthews which sonifies the movement of sharks around the Galapagos islands. We describe these pieces in terms of translation of data into differ...

We present a study that explores the affordance evoked by sound and sound-gesture mappings. In order to do this, we make use of a sensor system with minimal form factor in a user study that minimizes cultural association. The present study focuses on understanding how participants describe sounds and gestures produced while playing designed sonic i...

Gesture-based interaction is widespread in touch screen interfaces. The goal of this paper is to tap the richness of expressive variation in gesture to facilitate continuous interaction. We achieve this through novel techniques of adaptation and estimation of gesture characteristics. We describe two experiments. The first aims at understanding whet...

This SIG intends to investigate the ongoing dialogue between music technology and the field of human-computer interaction. Our specific aims are to consider major findings of musical interface research over recent years and discuss how these might best be conveyed to CHI researchers interested but not yet active in this area, as well as to consider...

We present a ludic interactive music performance that allows live recorded sounds to be re-rendered through the users' movements. The interaction design made the control similar to a shaker where the motion energy drives the energy of the played music piece. The instrument has been designed for musicians as well as non-musicians and allows for mult...

An overview of emerging topics, theories, methods, and practices in sonic interactive design, with a focus on the multisensory aspects of sonic experience. Sound is an integral part of every user experience but a neglected medium in design disciplines. Design of an artifact's sonic qualities is often limited to the shaping of functional, representa...

We present the first combined use of the electromyogram (EMG) and mechanomyogram (MMG), two biosignals that result from muscular activity, for interactive music applications. We exploit differences between these two signals, as reported in the biomedical literature, to create bi-modal sonification and sound synthesis mappings that allow performers...

This paper presents work in progress on applying a Multimodal interaction (MMI) approach to studying interactive music performance. We report on a study where an existing musical work was used to provide a gesture vocabulary. The biophysical sensing already used in the work was used as input modality, and augmented with several other input sensing...

We present an overview of machine learning (ML) techniques and their application in interactive music and new digital instrument design. We first provide the non-specialist reader an introduction to two ML tasks, classification and regression, that are particularly relevant for gestural inter- action. We then present a review of the literature in c...

Mobile music making is an area where many of the specific design challenges and specific affordances of mobile technologies can be explored. Music applications have often been at the forefront of research in social-interactive aspects of emerging technologies. Music, as a social activity and time-based medium makes demands in terms of intuitive and...

The establishment of a Digital Arts Featured Community at CHI 2012 indicates the general acceptance of mutually beneficial synergies between digital arts and HCI. At this juncture, the Digital Arts Community has an opportunity to build upon this established community platform to begin articulating lines of research. This SIG initiates this essentia...

The public release of datasets on the internet by government agencies, environmental scientists, political groups and many other organizations has fostered a social practice of data visualization. The audiences have expectations of production values ...

During a regular day while on the move, most people interact with multiple portable devices: a personal music player, mobile phone, and digital camera. People driving cars in addition may also use navigation systems. Whereas each of these devices are getting more and more sophisticated, and packed with numerous functionalities, they are each optimi...

This paper is a comparative study of gestural interaction with musical sound, designed to gain insight into the notion of musical affordance on interactive music systems. We conducted an interview base user study trialing three accelerometer based devices, an iPhone, a Wii-mote, and an Axivity Wax prototype, with four kinds of musical sound, includ...

This paper presents The Quiet Walk, an interactive mobile artwork for sonic explorations of urban space. The goal of TQW is to find the quietest place. An interface on the mobile device directs the user to avoid noisy areas of the city, giving directions to find quiet zones. Data collected by the system generates a geo-acoustic map of the city that...

In this paper a comparative study gestural interaction with musical sound, to gain insight into the notion of musical affordance on interactive music systems. We conducted an interview base user study trialing three accelerometer based devices, an iPhone, a Wii-mote, and an Axivity Wax prototype, with four kinds of musical sound, including percussi...

This article pinpoints a specific movement within the broad spectrum of music technology to identify a musical and instrument-building tradition concerned with gesture. This area of research and creative practice, often labeled NIME after the international conference series New Interfaces for Musical Expression, goes beyond enhancing traditional in...

This chapter looks at technologies of biosensing and techniques of sonification that invoke physicalization of sound to create a sense of intimacy in musical performance and gallery exhibition. By comparing the dynamic across stage and gallery settings, I will discuss how the corporeal activation of sound can become a key not just to decode experim...

We describe a collaborative design project with a group of young people in which an interactive educational information pack for teenagers was implemented. Instead of just providing input to a design project, the young people initiated, controlled and partially implemented the project themselves, with the support of an interdisciplinary research te...

Digital music has undergone fundamental shifts -- it has gone real time, it has become interactive, it has become miniaturized, and completely democratized. I'll map out my personal trajectory in this time to look at broader evolutions in the field with sensors, networks, and mobility. These are not just technological changes, but changes that brin...

With the advent of interactive digital media, people are no longer simply 'users'. They actively shift between various roles: author, collaborator, and even performer. We coin the term "user in flux" to problematize static definitions of "the user" and highlight how people's roles and practices switch and evolve when engaged in such interactions. D...

We apply techniques drawn from interactive media art in fieldwork for social inclusion. Advanced mobile media and grassroots DIY techniques are used to bring creative practice with digital media into community based outreach work. We use these techniques in a participatory context that encourages the co-production of cultural output. We triangulate...

SIDE is a UK-based research project investigating the social benefits of digital technologies for marginalized social groups. The Creative Media Group works in particular with creative practices and young people, with a twofold research focus: the fostering of engagement through digital creativity, and the support of youth-led innovation with digit...

Analog audio needs a separate physical circuit for each channel. Each microphone in a studio or on a stage, for example, must have its own circuit back to the mixer. Routing of the signals is inflexible. Digital audio is frequently wired in a similar way to analog. Although several channels can share a single physical circuit (e.g., up to 64 with A...

This paper describes a second-generation mobile music system that adds qualities of physical interaction to previous participative, networked, multi-user systems. We call upon traditions in interactive sensor music instrument building to inform this process. The resulting system underscores its dual personal/community context awareness with a techn...

The paper discusses the digital artwork series Impossible Geographies, works that weave dynamic cartographies of invisible, fragile and hybrid spaces. Impossible Geographies 01: Memory invokes the memory space of a gallery, producing cracks through which past events seep into the physical present. Net_Dérive negotiates the space between a gallery a...

The new field of mobile music emerges at the intersection of ubiquitous computing, portable audio technology and NIME. We have held a series of international workshop on this topic with leading projects and speakers, in order to establish a community ...

The new field of mobile music emerges at the intersection of ubiquitous computing, portable audio technology and NIME. We have held a series of international workshop on this topic with leading projects and speakers, in order to establish a community ...