Marcelo Wanderley

McGill University | McGill · Centre for Interdisciplinary Research in Music Media & Technology (CIRMMT)

This paper examined how rotary force feedback in knobs can enhance control over musical techniques, focusing on both performance and user experience. To support our study, we developed the Bend-aid system, a web-based sequencer with pre-designed haptic modes for pitch modulation, integrated with TorqueTuner, a rotary haptic device that controls pit...

R-FF (Reed Force Feedback) is a library of clarinet reed equations adapted for use on the TorqueTuner (TT), a one degree of freedom, rotary force-feedback (FF), haptic device. We created the R-FF library by analyzing the clarinet's physical model and adapting the reed component that defines the forces between the user and the mouthpiece into force-...

While urban sound management often focuses on sound as a nuisance, soundscape research suggests that proactive design approaches involving sound art installations can enhance public space experience. Nevertheless, there is no consensus on a methodology to inform the composition of sound installations through soundscape evaluation, and little resear...

This chapter focuses on different aspects of sound interaction ranging from how we hear, how we create, and the technologies that we use to interact with sound in different contexts. Three different contexts of uses are used to illustrate a wide range of users and areas of application. This includes interactive soundscape planning of public spaces,...

When designing a sound installation in public spaces, creators consider a wide range of factors related to the site where it will be deployed as part of the artistic statement. However, anticipating the impact of the sound installation on user experience is difficult in the absence of established methods to inform the design and evaluate the outcom...

This paper investigates the integration of force feedback in Digital Musical Instruments (DMI), specifically evaluating the reproduction of intricate vibrato techniques using haptic feedback controllers. We introduce our system for vibrato modulation using force feedback, composed of Bend-aid (a web-based sequencer platform using pre-designed hapti...

This chapter discusses ways to study sonic design from the perspective of musical performances with Digital Musical Instruments (DMIs). We first review the specificities of DMIs in terms of their unique affordances and limitations and comment on instrument availability, longevity, and stability issues, which impact the use of DMIs in musical practi...

Music learning and practice may be enhanced by the use of biofeedback based on both learners' and teachers' muscle activity, an essential component of music performance typically unavailable to listeners. By incorporating haptic vibrations, MappEMG enables the audience to experience the performers' muscle effort. This paper updates the MappEMG syst...

Music educators and researchers have grown increasingly aware of the need for traditional musical practices to promote inclusive music for disabled people. Inclusive music participation has been addressed by Accessible Digital Musical Instruments (ADMIs), which welcome different ways of playing and perceiving music, with considerable impact on musi...

When sight-reading a score, a timpanist needs to decide in real-time which stick to use to play a specific note while interpreting the musical material. Our main point of inquiry seeks to understand which sticking patterns performers employ and how they are affected by rhythmic stability. This paper analyzes the bi-manual sequencing (i.e., sticking...

Recorded on March 23 and 2023 [...]

A growing body of work on musical haptics focuses on vibrotactile feedback, while musical applications of force feedback, though more than four decades old, are sparser. This paper reviews related work combining music and haptics, focusing on force feedback. We then discuss the limitations of these works and elicit the main challenges in current ap...

This Special Issue in Arts aligns with the journal’s established theme of ‘Music and the Machine’ [...]

The Karlax is a gestural controller which arose substantial interest among composers since its inception in 2010 and continues to be commonly used in solo and group performances. One of the reasons for its longevity is the device’s remarkable adaptability, especially in musical contexts involving acoustic instruments. This article analyses six cham...

This paper introduces the T-Patch, a software application that streamlines the use of T-Stick Digital Musical Instruments (DMIs). It offers a user-friendly interface for gesture extraction, mapping, signal conditioning, sound synthesis, and sequencing with cues, enabling composers to create music without programming, therefore lowering the entry fe...

In this paper, the authors describe working with and on the T-Tree, a device that integrates multiple instances of a gestural controller known as the T-Stick. The T-Tree is used in two public performance contexts; the results of those performances are summarized, potential improvements to the design of the hardware and software are introduced, and...

The eTu{d,b}e framework adapts existing improvising musical agents (MA) for performance with an augmented instrument called the eTube. This instrument has been developed with deliberate musical and technological limitations including a simple two-button controller and restricted pitch capacity. We will present case studies which outline our researc...

This article introduces a series of workshop activities carried out with expert musicians to imagine new musical instruments through design fiction. At the workshop, participants crafted nonfunctional prototypes of instruments they would want to use in their own performance practice. Through analysis of the workshop activities, a set of design spec...

Different technologies can acquire data for gait analysis, such as optical systems and inertial measurement units (IMUs). Each technology has its drawbacks and advantages, fitting best to particular applications. The presented multi-sensor human gait dataset comprises synchronized inertial and optical motion data from 25 participants free of lower-...

Urban sound management often amounts to reducing sound levels with the underlying assumption of sound/noise as a nuisance. However, a reduction in sound level does not necessarily lead to a more pleasant auditory experience, especially in urban public spaces where vibrancy can be sought after. A proactive design approach that accounts for the human...

Digital musical instruments (DMIs) built to be used in performance settings need to go beyond the prototypical stage of design to become robust, reliable, and responsive devices for extensive usage. This paper presents the Tapbox and the Slapbox, two generations of a standalone DMI built for percussion practice. After summarizing the requirements f...

ForceHost is an opensource toolchain for generating firmware that hosts authoring and rendering of force-feedback and audio signals and that communicates through I2C with guest motor and sensor boards. With ForceHost, the stability of audio and haptic loops is no longer delegated to and dependent on operating systems and drivers, and devices remain...

Motion capture (mocap)—the recording of three-dimensional movement using high-accuracy systems—has become a standard research tool in the analysis of music performances in the last two decades. A variety of systems is currently available, ranging from optical, multi-camera (passive and/or active) infra-red systems and inertial systems (using orient...

Digital musical instruments are frequently designed in research and experimental performance contexts but few are taken up into sustained use by active and professional musicians. To identify the needs of performers who use novel technologies in their practices, a survey of musicians was conducted that identified desirable qualities for instruments...

Different technologies can acquire data for gait analysis, such as optical systems and inertial measurement units (IMUs). Each technology has its drawbacks and advantages, fitting best to particular applications. The presented multi-sensor human gait dataset comprises synchronized inertial and optical motion data from 25 subjects free of lower-limb...

The aims of this study were to i) evaluate proximal-to-distal sequencing (PDS) in pianists’ attack and release movements during pressed-staccato keystrokes, and ii) investigate if trunk motion facilitates PDS of upper-limb movements. Nine expert pianists performed a series of loud pressed-staccato keystrokes. Kinematic data was recorded with a 3 D...

This paper introduces the musical haptic wearables for audiences (MHWAs), a class of wearable devices for musical applications targeting audiences of live music performances. MHWAs are characterized by embedded intelligence, wireless connectivity to local and remote networks, a system to deliver haptic stimuli, and tracking of gestures and/or physi...

Recent works recognize musicality is based on and constrained by our cognitive and biological system. Taking in account a concept from cognitive science - cognitive offloading - as a principle for technology-supported musical activities, in this paper we discuss some principles (guidelines) to be taken into account when designing, developing and ev...

This paper presents a theoretical framework for describing interactive sound installations, along with an interactive database, on a web application, for visualizing various features of sound installations. A corpus of 195 interactive sound installations was reviewed to derive a taxonomy describing them across three perspectives: Artistic Intention...

This paper describes Le Bâton, a new digital musical instrument based on the nonlinear dynamics of the triple pendulum. The triple pendulum is a simple physical system constructed by attaching three pendulums vertically such that each joint can swing freely. When subjected to large oscillations, its motion is chaotic and is often described as unexp...

High-level improvising musicians master idiosyncratic gesture vocabularies that allow them to express themselves in unique ways. The full use of such vocabularies is nevertheless challenged when improvisers incorporate electronics in their performances. To control electronic sounds and effects, they typically use commercial interfaces whose physica...

We report on a conceptual framework for describing interactive sound installations from three complementary perspectives: artistic intention, interaction and system design. Its elaboration was informed by a systematic review of 181 peer-reviewed publications retrieved from the Scopus database, which describe 195 interactive sound installations. The...

This paper reviews the existing literature on input device evaluation and design in human-computer interaction (HCI) and discusses possible applications of this knowledge to the design and evaluation of new interfaces for musical expression. Specifically, a set of musical tasks is suggested to allow the evaluation of different existing controllers.

The internet has allowed for the flourishing of several shared projects. Developers from different parts of the world can work asynchronously on the same project, formulating ideas, and implementing complex systems. For remote collaboration in software development projects, sharing information and code becomes more straightforward, given the textua...

Haptics, audio and human–computer interaction are three scientific disciplines that share interests, issues and methodologies. Despite these common points, interaction between these communities are sparse, because each of them have their own publication venues, meeting places, etc. A venue to foster interaction between these three communities was c...

The integration of vibrotactile feedback in digital music instruments (DMIs) is thought to improve the instrument’s response and make it more suitable for expert musical interactions. However, given the extreme requirements of musical performances, there is a need for solutions allowing for independent control of frequency and amplitude over a wide...

This paper reports ongoing initiatives to explore how digital and augmented musical instruments gestural vocabulary is developed over time, and how embedding the high-level gesture descriptors in those instruments can fomenting their use and contribute to their maturity toward established music instruments. The instruments included in this research...

The definition of mapping strategies between input devices and sound generating algorithms is an essential process in computer music. Traditionally, mapping has been considered in the context of digital musical instruments where the control of low level sound features (note articulation, timbre) is the goal. This paper describes the motivation, imp...

The CIRMMT Distinguished Lecture (DL) series consists of over seventy 1-h videos of leading scholars and artists discussing their interdisciplinary work, a unique video encyclopedia of Music, Science and Technology. Though this collection in itself is already an invaluable research and pedagogical tool, searching videos in the collection is not a t...

So far, NIME research has been mostly inward-looking, dedicated to divulging and studying our own work and having limited engagement with trends outside our community. Though musical instruments as cultural artefacts are inherently political, we have so far not sufficiently engaged with confronting these themes in our own research. In this paper we...

Music performance involves continuous muscle activity of different body parts. Activity of more distal body segments which directly interact with the instrument usually relates to the actual production of sound. However, muscle activations of more proximal segments, including the trunk, might relate to the expressive content of the musical idea bei...

L’approche classique – désincarnée – de la cognition musicale conçoit l’action et la perception comme des processus séparés, périphériques. En revanche, une vision incarnée de la cognition musicale donne une place centrale au couplage étroit entre action et perception. Il est un fait communément établi que la perception oriente l’action. Nous prése...

In this paper we investigate the role of haptic stimuli in affecting the perception of live music. We designed a study where a smart mandolin performer played live for audience members wearing a gilet-based musical haptic wearable, which provided vibro-tactile sensations in response to the performed music. Six performances were conducted, each of w...

The increasing availability of accessible sensor technologies, single board computers, and prototyping platforms have resulted in a growing number of frameworks explicitly geared towards the design and construction of Digital and Augmented Musical Instruments. Developing such instruments can be facilitated by choosing the most suitable framework fo...

This paper documents the key issues of performance and compatibility working with Musical Instrument Digital Interface (MIDI) over Bluetooth Low Energy (BLE) as a wireless interface for sensor or controller data and inter-module communication in the context of building interactive digital systems. An overview of BLE MIDI is presented along with a c...

We present the body:suit:score, a vibrotactile musical score that displays instructions and signals directly on musicians’ bodies using an array of vibrating motors. Initially inspired by the third author’s need for a new kind of score in his compositional practice, the body:suit:score facilitates the exploration of novel musical experiences and pe...

This chapter discusses possible links between the fields of computer music (CM) and human-computer interaction (HCI), particularly in the context of the MIDWAY project between Inria, France and McGill University, Canada. The goal of MIDWAY is to construct a “musical interaction design workbench” to facilitate the exploration and development of new...

This is the introductory chapter of a book dedicated to new research in, and emerging new understandings of, music and human-computer interaction—known for short as music interaction. Music interaction research plays a key role in innovative approaches to diverse musical activities, including performance, composition, education, analysis, productio...

Wendy Mackay is a Research Director at INRIA Saclay—Île-de-France where she founded and directs the ExSitu research group in Human-Computer Interaction. She is a former chair of CHI. Her research interests include multi-disciplinary, participatory design methods, tangible computing, interactive paper, and situated interaction. In this interview, Ma...

Established pedagogical theories for classical piano usually do not consider the essential relationship between the musical structure, whole body movements, and expression. Research focusing on musicians' expression has shown that body movements reflect the performer's understanding of the musical structure. However, most studies to date focus on t...

This paper describes two augmented nylon-string guitar projects developed in different institutions. GuitarAMI uses sensors to modify the classical guitars constraints while GuiaRT uses digital signal processing to create virtual guitarists that interact with the performer in real-time. After a bibliographic review of Augmented Musical Instruments...

Despite the proliferation of new digital musical instruments (DMIs) coming from a diverse community of designers, researchers and creative practitioners, many of these instruments experience short life cycles and see little actual use in performance. There are a variety of reasons for this, including a lack of established technique and repertoire f...

Contemporary classical music conductors are often required to perform music that requires a click track for accurate tempo following. Auditory click tracks can be obtrusive as they block an ear and interfere with the conductor’s perception of the music being performed. We propose a device and framework that uses the tactile sense to provide a simil...

We present a gestural control system to augment harp performance with real-time control of computer-based audio affects and processing. While the lightweight system was designed for use alongside any instrument, our choice of the concert harp represented a unique case study in gestural control of music. The instrument's large size and physically de...

The swaying action coming from the pianists' hip region constitutes the pivotal point for upper body movements. In this paper, we evaluate the relation between the use of vertical force and structural characteristics of technically different excerpts when pianists play with varied levels of expression. We also analyze the common patterns within the...

The term 'NIME'-New Interfaces for Musical Expression-has come to signify both technical and cultural characteristics. Not all new musical instruments are NIMEs, and not all NIMEs are defined as such for the sole ephemeral condition of being new. So, what are the typical characteristics of NIMEs and what are their roles in performers' practice? Is...

The Digital Orchestra Toolbox for Max is an open-source collection of small modular software tools for aiding the development of Digital Musical Instruments. Each tool takes the form of an “abstraction” for the visual programming environment Max, meaning it can be opened and under- stood by users within the Max environment, as well as copied, modif...

This paper presents the work to maintain several copies of the digital musical instrument (DMI) called the T-Stick in the hopes of extending their useful lifetime. The T-Sticks were originally conceived in 2006 and 20 copies have been built over the last 12 years. While they all preserve the original design concept, their evolution resulted in vari...

Haptics, and specifically vibrotactile-augmented interfaces, have been the object of much research in the music technology domain: In the last few decades, many musical haptic interfaces have been designed and used to teach, perform, and compose music. The investigation of the design of meaningful ways to convey musical information via the sense of...

Motor skill acquisition inherently depends on the way one practices the motor task. The amount of motor task variability during practice has been shown to foster transfer of the learned skill to other similar motor tasks. In addition, variability in a learning schedule, in which a task and its variations are interweaved during practice, has been sh...

We articulate a need for the representation of temporal objects represented by dynamic, short-lived mapping connections instantiated from a template, in tools for designing and using interactive media systems. A list of requirements is compiled from examination of existing tools, practical use cases, and abstract considerations of node connectivity...