Institute for Research and Coordination in Acoustics/Music

User interface (UI) sounds communicate information and guide users as they navigate an interface. Research exploring the direct links between these sounds’ physical characteristics and their communicative power is lacking. In the present study, we asked 100 participants to report the emotional and functional connotations of 30 UI sounds from household devices. For each sound, they provided ratings on eight emotion scales and then categorized the sound by its position in time within a user experience (beginning, middle, or end) and by its function (a confirmation, a notification, or a cancelation). We then explored the relation between acoustic characteristics (attack time, release time, spectral centroid height, and mean fundamental frequency) and the effects of the change in perceived pitch and loudness over time (melodic and loudness contour) on participants’ ratings of emotion and categorization of function and time. Results showed effects of the acoustic characteristics, especially release time, as well as effects of melodic and loudness contour on emotion, function, and time in experience. This study provides a basic set of guidelines for intuitive UI sound design and informs the discussion on how sound can be leveraged to enhance tech experiences.

Social interaction research is lacking an experimental paradigm enabling researchers to make causal inferences in free social interactions. For instance, the expressive signals that causally modulate the emergence of romantic attraction during interactions remain unknown. To disentangle causality in the wealth of covarying factors that govern social interactions, we developed an open-source video-conference platform enabling researchers to covertly manipulate the social signals produced by participants during interactions. Using this platform, we performed a speed-dating experiment where we aligned or misaligned the facial smiles of participants in real time with face transformation algorithms. Even though participants remained totally unaware that their faces were being manipulated, aligning their smiles causally enhanced the romantic attraction they felt toward each other, compared to unaligned scenarios. Manipulations also influenced how participants synchronized and vocally reacted to each other. This paradigm causally manipulates the emergence of romantic attraction in free social interactions. Moreover, our methodology opens the possibility to perform causal inferences during free social interactions.

Somax2 is an AI-based multiagent system for human-machine coimprovisation that generates stylistically coherent streams while continuously listening and adapting to musicians or other agents. The model on which it is based can be used with little configuration to interact with humans in full autonomy, but it also allows fine real-time control of its generative processes and interaction strategies, closer in this case to a “smart” digital instrument. An offspring of the Omax system, conceived at the Institut de Recherche et Coordination Acoustique/Musique, the Somax2 environment is part of the European Research Council Raising Cocreativity in Cyber-Human Musicianship (REACH) project, which studies distributed creativity as a general template for symbiotic interaction between humans and digital systems. It fosters mixed musical reality involving cocreative AI agents. The REACH project puts forward the idea that cocreativity in cyber-human systems results from the emergence of complex joint behavior, produced by interaction and featuring cross-learning mechanisms. Somax2 is a first step toward this ideal, and already shows life-size achievements. This article describes Somax2 extensively, from its theoretical model to its system architecture, through its listening and learning strategies, representation spaces, and interaction policies.

In this paper we report on the inaugural meetings of the Musical Care International Network held online in 2022. The term “musical care” is defined by Spiro and Sanfilippo (2022) as “the role of music—music listening as well as music-making—in supporting any aspect of people's developmental or health needs” (pp. 2–3). Musical care takes varied forms in different cultural contexts and involves people from different disciplines and areas of expertise. Therefore, the Musical Care International Network takes an interdisciplinary and international approach and aims to better reflect the disciplinary, geographic, and cultural diversity relevant to musical care. Forty-two delegates participated in 5 inaugural meetings over 2 days, representing 24 countries and numerous disciplines and areas of practice. Based on the meetings, the aims of this paper are to (1) better understand the diverse practices, applications, contexts, and impacts of musical care around the globe and (2) introduce the Musical Care International Network. Transcriptions of the recordings, alongside notes taken by the hosts, were used to summarise the conversations. The discussions developed ideas in three areas: (a) musical care as context-dependent and social, (b) musical care's position within the broader research and practice context, and (c) debates about the impact of and evidence for musical care. We can conclude that musical care refers to context-dependent and social phenomena. The term musical care was seen as useful in talking across boundaries while not minimizing individual disciplinary and professional expertise. The use of the term was seen to help balance the importance and place of multiple disciplines, with a role to play in the development of a collective identity. This collective identity was seen as important in advocacy and in helping to shape policy. The paper closes with proposed future directions for the network and its emerging mission statement.

As interaction designers are venturing to design for others based on autobiographical experiences, it becomes particularly relevant to critically distinguish the designer’s voice from others’ experiences. However, few reports go into detail about how self and others mutually shape the design process and how to incorporate external evaluation into these designs. We describe a one-year process involving the design and evaluation of a prototype combining haptics and storytelling, aiming to materialise and share somatic memories of earthquakes experienced by a designer and her partner. We contribute with three strategies for bringing others into our autobiographical processes, avoiding the dilution of first-person voices while critically addressing design flaws that might hinder the representation of our stories.

In the context of this special issue on the course-of-experience framework, we were fortunate to receive far-reaching commentaries from a wide variety of disciplines including enaction, phenomenology, ecological psychology, ecological dynamics, cognitive anthropology and archaeology and the philosophy of mind. The result is an extraordinarily rich set of reflections that critically engage in debate and lay the groundwork for future discussions and empirical research. In this reply, the authors try to highlight the originality of the course-of-experience framework, which is not yet widely known but has the potential to offer a breakthrough in the present-day international ecological-enactive mainstream and an alternative to (Husserlian) phenomenologically inspired enactivist approaches. They also try to set things right concerning the critical dimensions of cognitive ecology, such as the role of material and ‘others’ that they didn’t mention in their original article (in view of the objectives pursued).

We present a three dimensional nonlinear string model based on a geometrically exact beam. The beam model is obtained by applying a variational principle using a covariant Lagrangian formulation; in particular, the equations of motion and the boundary conditions are treated in an unified manner. Following an analogous discrete variational principle, a Lie group variational integrator is given. The energy and momentum conservation properties of the integrator are discussed and illustrated. This geometrically exact beam serves as a basis to formulate a prestressed damped string model with coupled non trivial boundary conditions. Simulation results are discussed and validated against analytical solutions obtained in the context of a small displacement hypothesis.

Musical prosody is characterized by the acoustic variations that make music expressive. However, few systematic and scalable studies exist on the function it serves or on effective tools to carry out such studies. To address this gap, we introduce a novel approach to capturing information about prosodic functions through a citizen science paradigm. In typical bottom-up approaches to studying musical prosody, acoustic properties in performed music and basic musical structures such as accents and phrases are mapped to prosodic functions, namely segmentation and prominence. In contrast, our top-down, human-centered method puts listener annotations of musical prosodic functions first, to analyze the connection between these functions, the underlying musical structures, and acoustic properties. The method is applied primarily to the exploring of segmentation and prominence in performed solo piano music. These prosodic functions are marked by means of four annotation types—boundaries, regions, note groups, and comments—in the CosmoNote web-based citizen science platform, which presents the music signal or MIDI data and related acoustic features in information layers that can be toggled on and off. Various annotation strategies are discussed and appraised: intuitive vs. analytical; real-time vs. retrospective; and, audio-based vs. visual. The end-to-end process of the data collection is described, from the providing of prosodic examples to the structuring and formatting of the annotation data for analysis, to techniques for preventing precision errors. The aim is to obtain reliable and coherent annotations that can be applied to theoretical and data-driven models of musical prosody. The outcomes include a growing library of prosodic examples with the goal of achieving an annotation convention for studying musical prosody in performed music.

The article presents the course-of-experience framework and how it contributes to studying cognition in practice. The aim is twofold: (a) to argue for a phenomenologically and semiotically inspired enactivist approach to practice and cognition in practice and (b) to describe research methods that provide rigorous first-person data in relation to practice—in other words, a view “from within” of practice. Practice is considered to be a relevant unit of analysis for studying cognition-in-the-world and is defined as enacted, lived, situated, embodied, and enculturated. Practice is not viewed as a “context for” but as “constitutive” of the cognitive process itself. This article describes (a) the epistemological foundation and general assumptions of the course-of-experience framework, (b) the associated way of looking at pre-reflective self-consciousness and its relation to practice, (c) the analytical hypothesis derived from Peirce’s semeiotic, and (d) some methodological considerations related to data collection, data processing, and analysis. In the concluding section, we outline the added value of the course-of-experience framework for cognitive science, and we indicate possible directions for further research.

In the latter years, we have been observing a growth in wearable technology for personal use. However, an analysis of the state of the art for wearable technology shows that most devices perform data acquisition from individual subjects only, relying on communication technologies with drawbacks that prevent their use in collective real-world scenarios (e.g. a cinema, a theatre, and related use cases). When analysing the emotional response in groups, two types of emotions appear: individual (influenced by the group) and group-based emotions (towards the group as an identity). To fill the existing gap, we propose a biocybernetic engine for real-time data acquisition of multimodal physiological data in real-world scenarios. Our system extends the state of the art with: (1) real-time data acquisition for the signals being acquired (20 devices at 25 Hz; 10 devices at 60 Hz); (2) creation of a standalone local infrastructure with end-user interface for monitoring the data acquisition; (3) local and cloud-based data storage. We foresee that this platform could be the basis for the creation of large databases in diverse real-world scenarios, namely health and wellbeing, marketing, art performances, and others. As a result, this work will greatly contribute to simplify widespread biosignals data collection from unobtrusive wearables. To evaluate the system, we report a comprehensive assessment based on a set of criteria for data quality analysis.

The effects of music on bodily movement and feelings, such as when people are dancing or engaged in physical activity, are well-documented—people may move in response to the sound cues, feel powerful, less tired. How sounds and bodily movements relate to create such effects? Here we deconstruct the problem and investigate how different auditory features affect people’s body-representation and feelings even when paired with the same movement. In three experiments, participants executed a simple arm raise synchronised with changing pitch in simple tones (Experiment 1), rich musical sounds (Experiment 2) and within different frequency ranges (Experiment 3), while we recorded indirect and direct measures on their movement, body-representations and feelings. Changes in pitch influenced people’s general emotional state as well as the various bodily dimensions investigated—movement, proprioceptive awareness and feelings about one’s body and movement. Adding harmonic content amplified the differences between ascending and descending sounds, while shifting the absolute frequency range had a general effect on movement amplitude, bodily feelings and emotional state. These results provide new insights in the role of auditory and musical features in dance and exercise, and have implications for the design of sound-based applications supporting movement expression, physical activity, or rehabilitation.

Performance-score synchronization is an integral task in signal processing, which entails generating an accurate mapping between an audio recording of a performance and the corresponding musical score. Traditional synchronization methods compute alignment using knowledge-driven and stochastic approaches, and are typically unable to generalize well to different domains and modalities. We present a novel data-driven method for structure-aware performance-score synchronization. We propose a convolutional-attentional architecture trained with a custom loss based on time-series divergence. We conduct experiments for the audio-to-MIDI and audio-to-image alignment tasks pertained to different score modalities. We validate the effectiveness of our method via ablation studies and comparisons with state-of-the-art alignment approaches. We demonstrate that our approach outperforms previous synchronization methods for a variety of test settings across score modalities and acoustic conditions. Our method is also robust to structural differences between the performance and score sequences, which is a common limitation of standard alignment approaches.

Combining sketch studies, musical analysis, and acoustic analysis, this article examines the impact of spectral models on Grisey’s creative process in the 1970s. First, it focuses on Grisey’s establishment of initial theoretical models in Dérives and Périodes, composed during his residence at the Villa Medici (1972–74), during which period he conducted his own study of acoustics. The article then looks at how, back in Paris, the composer started studying with the acoustician Émile Leipp and then worked with Michèle Castellengo, with whom in 1977 he undertook for the first time the spectral analysis of instrumental sounds. Finally the article examines in detail how some spectrograms, once transcribed, were used in Modulations (1976–77) to construct a complex spectral polyphony. If Grisey draws his inspiration from sound phenomena and the characteristics of auditory perception, he does not for all that neglect structuralism: spectral models serve both to simulate natural phenomena and to formalise his compositional processes.

No PDF available ABSTRACT Visual and acoustic environment may influence the perception of auditory distance. In the context of Audio-only augmented reality (AAR), the coherence of the perceived virtual sound sources with the apparent room geometry and acoustics cannot always be guaranteed. The perceptual consequences of these incoherences are not well known. We conducted two online perceptual studies with a sound distance rendering model based on measured spatial room impulse responses (SRIR). A first study evaluated the perceptual performances of the model in incongruent visual contexts. The incongruent environment-related visual cues (spatial visual boundary and room volume) demonstrated a significant effect on the auditory distance perception (ADP) of virtual sound sources, through a calibration effect. A second study evaluated the impact of acoustical incongruence. Virtual sound sources distances were judged after the participant listened to distracting sound sources conveying distance cues relative to a different acoustical environment. When this distracting sound sources corresponded to a larger room than the one reproduced by the model, a higher compression effect was observed on the ADP of virtual sound sources. However, when the intensity cue conveyed by the distracting sound sources were coherent with the acoustical environment simulated by the model, their distracting effect were negligible.