Sandra Pauletto

• PhD in Music Technology
• Professor (Associate) at KTH Royal Institute of Technology

We present a design exploration into the use of sound and listening to embody concepts relevant to energy consumption with the aim to affect attitudes towards it.We developed and evaluated two sound-based interactive prototypes aimed at sensitizing participants to the topic of electricity consumption and sustainable energy choices. These prototypes...

Energy from fossil fuel is responsible for about three-quarters of global greenhouse gases. Transitioning to sustainable solutions can greatly contribute to limiting global heating. To do this, people need to gain a global, as well as a national, perspective on energy consumption and production. This paper presents the design of Earth Electric, a c...

Recent research identified sixteen dimensions along which sound designers make decisions when ideating and sketching sounds. These 16 dimensions can be grouped into “Overarching strategies”, “Design Strategies” and “Approaches to sound”. In this paper, the ideation process of two professional sound designers - both working in media production - is...

Addressing climate change and related societal challenges requires the acknowledgement that our resources are limited, that a sustainable future depends on how we act now, and that responsibilities are shared. In this context, the choices we make when designing the world around us are of great significance and consequence. In this conversation, we...

The Sound-Driven Design track invites researchers and practitioners to delve into the multifaceted nature of sound, unravelling its physical, perceptual, emotional, and socio-technological dimensions, and contribute to the discovery and development of design methods and tools. In the sound-driven perspective, listening emerges as an element that ad...

What is sound when it is not heard? How does this unheard sound affect us? What might such sound reveal to us and how would we know? How do we recall sounds when their sources are no longer accessible? Traces of Sound. Reflections of Sounds Unheard invites scholars of music archaeology, German literature, media technology, sound art, human percepti...

Sonification and sonic interaction design aim to create meaningful displays and digital interactions using data and information from the most disparate fields (astronomy, finance, health, and security, for example) as the basis of the design. To date, there are no standards and conventions on how to meaningfully link data to sound; therefore, desig...

This paper describes the development of three digital sound prototypes based on three sounding objects selected from those in use at the Swedish Radio Sound Studio. Knowledge about historical and current creative practice was gathered through an in-depth interview with the Swedish Radio sound engineer and sound maker Michael Johansson and the obser...

In this paper we describe the development of two heat and movement-sensitive sonic textile prototypes. The prototypes interactively sonify in real-time the bodily temperature of the person who wears them, complementing the user's felt experience of warmth. The main aim is making users aware of the heat exchanges between the body, the fabric, and th...

In this paper we present a system for the sonification of the electricity drawn by different household appliances. The system uses SpecSinGAN as the basis for the sound design , which is an unconditional generative architecture that takes a single one-shot sound effect (e.g., a fire crackle) and produces novel variations of it. SpecSinGAN is based...

Even if images of astrophysical objects are used by professional astronomers for research and by the public for outreach, we are all basically blind to the Universe. Challenging the idea that we should always use visualisations, there has been a growing interest in converting astronomical phenomena into sound, motivated by: making astronomy more ac...

The Audible Universe project aims at making dialogue between two scientific domains investigating two distinct research objects, briefly said, Stars and Sound. It has been instantiated within a collaborative workshop that started to mutually acculturate both communities, by sharing and transmitting respective knowledge, skills and practices. One ma...

The Audible Universe project aims to create dialogue between two scientific domains investigating two distinct research objects: stars and sound. It has been instantiated within a collaborative workshop that began to mutually acculturate the two communities, by sharing and transmitting respective knowledge, skills and practices. One main outcome of...

This paper describes initial research conducted in The Radio Sound Studio Project. 1 The aim of the project is to develop novel sound design tools by digitally model historical sound effects, found in the radio drama studio, utilis-ing methods from sound computing, ethnography, and design. These novel sound tools will address both artistic sound pr...

Sonic interaction design and sonification have the potential to provide new ways to display and interpret data and information. Data from a number of domains have been sonified: astronomy, finance, health, security, and many more. However in recent years, research in auditory displays has highlighted the importance of using participatory methods to...

As buildings become increasingly automated and energy efficient, the relative impact of occupants on the overall building carbon footprint is expected to increase. Research shows that by changing occupant behaviour energy savings between 5 and 15 % could be achieved. A commonly used device for energy-related behaviour change is the smart meter, a v...

This paper reports on the procedure and results of a preliminary experiment to evaluate participants’ perceptual experiences of a mechanical theatre sound effect and its digital counterpart. The theatre sound effect chosen - an acoustic wind machine - affords a simple rotational gesture; turning its crank handle at varying speeds produces a convinc...

Knocking sounds are highly meaningful everyday sounds. There exist many ways of knocking, expressing important information about the state of the person knocking and their relationship with the other side of the door. In media production, knocking sounds are important storytelling devices: they allow transitions to new scenes and create expectation...

Knocking sounds are highly expressive. In our previous research we have shown that from the sound of knocking actions alone a person can differentiate between different basic emotional states. In media productions, such as film and games, knocks can be very important storytelling de- vices as they allow the story to transition from one part to anot...

Assisted by the technological advances of the past decades, stereoscopic 3D cues are being integrated increasingly in both interactive and non-interactive media. Arguably, the main focus of this effort is placed on the creation of an increased sense of visual depth. Considering that human perception relies heavily on the audiovisual integration rat...

In this paper we explore the synthesis of sound effects using conditional generative adversarial networks (cGANs). We commissioned Foley artist Ulf Olausson to record a dataset of knocking sound effects with different emotions and trained a cGAN on it. We analysed the resulting syn-thesised sound effects by comparing their temporal acoustic feature...

Knocking sounds are highly meaningful everyday sounds. There exist many ways of knocking, expressing important information about the state of the person knocking and their relationship with the other side of the door. In media production, knocking sounds are important storytelling devices: they allow transitions to new scenes and create expectation...

Assisted by the technological advances of the past decades, stereoscopic 3D media are currently making another attempt to be established as mainstream forms of entertainment. Arguably, the main focus of this effort is placed on the creation of immersive 3D visual worlds. However, with few exceptions, little attention has been given so far to the po...

Assisted by the technological advances of the past decades, stereoscopic 3D media are currently making another attempt to be established as mainstream forms of entertainment. Arguably, the main focus of this effort is placed on the creation of immersive 3D visual worlds. However, with few exceptions, little attention has been given so far to the po...

Technological advances have always had an impact on the development of new audio-visual aesthetics. Recently, exploiting the spatial capabilities of immersive sound technology in the form of Dolby Atmos, Alfonso Cuarón introduced in Gravity (2013) an innovative sound design approach that enhances the illusion of ‘presence’ in the space of the diege...

The use of real-time sound synthesis for sound effects can improve the sound design of interactive experiences such as video games. However, synthesized sound effects can be often perceived as synthetic, which hampers their adoption. This paper aims to determine whether or not sounds synthesized using filter-based modal synthesis are perceptually c...

This paper reports on the procedure and results of an experiment to evaluate a continuous sonic interaction with an everyday wind-like sound created by both acoustic and digital means. The interaction is facilitated by a mechanical theatre sound effect, an acoustic wind machine, which is performed by participants. This work is part of wider researc...

Despite the resurgence of stereoscopic 3D cinema over the past decade, a relatively small number of movies are designed in such a way as to take full advantage of the technology. Additionally, even movies designed with 3D immersion in mind cannot be adventurous with the new possibilities, as they have to be designed with 2D compatibility in mind (M...

In this article, we firstly provide a brief historical background on stereoscopic 3D (S3D) cinema and highlight some of the challenges S3D filmmakers face. Then, we propose a number of ways in which the soundtrack can be used to complement S3D cinematic productions. These propositions are based on the unique spatial characteristics of S3D cinema an...

The cinematic sound design practice of Foley developed from a rich history of performative sound design through materials, objects and mechanical devices created for theatrical performance, magic lantern shows and silent cinema screenings in the late nineteenth and early twentieth century. Now, as virtual and digital methods become available in fil...

This paperpresents the next stage of an investigation into the potential of historical theatre sound effects as a resource for Sonic Interaction Design (SID). An acoustic theatre wind machine was constructed, and a digital physical modelling-based version of this specificmachine was programmed using the Sound Designer’s Toolkit (SDT) in Max/MSP. Th...

Excessive consumption of alcohol has been recognised as a significant risk factor impacting the health of young people. Effective communication of such risk is considered to be one key step to improve behaviour. We evaluated an innovative multimedia intervention that utilised audio (sonification—using sound to display data—and music) and interactiv...

Digital audio took its first, tentative steps towards widespread acceptance over forty years ago with the adoption of the Compact Disc as a de-facto standard for auditioning music, with cinema embracing digital encoding and reproduction a decade later. Somehow, broadcast audio production and post-production processes managed to remain some distance...

As part of an investigation into the potential of historical theatre sound effects as a resource for Sonic Interaction Design (SID), an acoustic theatre wind machine was constructed and analysed as an interactive sounding object. Using the Sound Designer’s Toolkit (SDT), a digital, physical modelling- based version of the wind machine was programme...

The interdisciplinary research project, Using Sonification to COmmunicate public health Risk data (SCORe), aims to experimentally test how sonification, interactivity in combination with music, could increase the communicative potential of a visual presentation directed to young people and focused on health risk data of alcohol consumption. Specifi...

Sonification designs have been increasingly used to complement data presentation and exploration in various disciplines. In medical research they have been applied as an aid for analysis and diagnosis, disease treatment, to assist visually impaired patients, and in physiotherapy. Interactive sonification as a communication channel has the potential...

The idea for this special issue came to me when I was organising the symposium Perspectives on Sound Design that took place at The University of York in July 2013. The purpose of the symposium was to bring together researchers, practitioners and theorists to discuss what it means to design sound in various contexts, and whether it is possible and d...

Assisted by the technological advances of the past decades, stereoscopic 3D (S3D) cinema is currently in the process of being established as a mainstream form of entertainment. The main focus of this collaborative effort is placed on the creation of immersive S3D visuals. However, with few exceptions, little attention has been given so far to the p...

* Accompanying reference video here: https://vimeo.com/channels/neilhillman By looking at the relationship between audience emotions and a film’s soundtrack, this research started by asking the question ‘Are a listening-viewers Emotional Reactions to moving-picture sound design identifiable, predictable and repeatable?’. If so, can we pre-determin...

Emotion in audio-voice signals, as synthesized by text-to-speech (TTS) technologies, was investigated to formulate a theory of expression for user interface design. Emotional parameters were specified with markup tags, and the resulting audio was further modulated with post-processing techniques. Software was then developed to link a selected TTS s...

The voice is the most important sound of a film soundtrack. It represents a character and it carries language. There are different types of cinematic voices: dialogue, internal monologues, and voice-overs. Conventionally, two main characteristics differentiate these voices: lip synchronization and the voice's attributes that make it appropriate for...

Sonic interaction design studies how digital sound can be used in interactive contexts to convey information, meaning, aesthetic and emotional qualities. This area of research is positioned at the intersection of sound and music computing, auditory displays and interaction design. The key issue the designer is asked to tackle is to create meaningfu...

The present paper provides an insight into the acoustic characteristics of one of the spaces used for the performance of the York Mystery Plays: Stonegate, a street in central York (UK). An experiment conducted in a controlled environment with the aim of determining the optimal method for acoustic measurements on this site is first presented. The m...

In films, the voice is the principal audio channel carrying both a character’s expressivity and storytelling information. It is also a unique sound as it carries language and explicit meaning. The voices we hear in the cinema usually emanate flawlessly and sound natural, as if the characters on screen were really speaking to us. In actual fact thes...

Acoustic measurements of outdoor sites require researchers to carefully consider the appropriate method to ensure reliable results. This entails the consideration of the signal-to-noise ratio (SNR), the presence of visitors as well as restrictions that are specific to the site. The present paper presents the results of an experiment conducted in a...

The term audio film refers to a novel format of sonic art that explores the possibilities of telling a story solely through sound, sound processing and spatialisation, without the need for visual elements or a narrator. The term audio film was coined by the authors as part of the research project entitled 'The Design of an Audio Film for the Visual...

This project examines how the emotional content of a synthesised and robotic-sounding speech signal can be modified by manipulating high-level acoustic parameters using commonly available sound design digital tools. Stimuli were created on the basis of trends described by the literature and verified via our own analysis of emotional speech produced...

The focus of this project is the manipulation of a robotic voice signal for the purpose of adding emotional expres-sion. In particular, the main aim was to design the emo-tion expressed by a robotic voice by manipulating spe-cific acoustic parameters such as pitch, amplitude and speech rate. The three basic emotions considered were: anger, happines...

In this paper we present two experiments on implementing interaction in sonification displays: the first focuses on recorded data (interactive navigation) and the second on data gathered in real time (auditory feedback).Complex synthesised data are explored in the first experiment to evaluate how well the known characteristics present in the data a...

Throughout its evolution, cinema sound has changed the way we watch and understand films. Initially sound carried little narrative information. Today's multi-channel soundtrack, on the other hand, is able to enlarge the diegetic space and immerse the audience in it. However, it could be argued that the multi-channel soundtrack's potential as a narr...

Nowadays, audio description is used to enable visually impaired people to access films. It presents an important limitation, however, which consists in the need for visually impaired audiences to rely on a describer, not being able to access the work directly. The aim of this project was to design a format of sonic art called audio film that elimin...

Sonic Interaction Design (SID) is an emerging field that is positioned at the intersection of auditory display, ubiquitous computing, interaction design, and interactive arts. SID can be used to describe practice and inquiry into any of various roles that sound may play in the interaction loop between users and artifacts, services, or environments,...

The aim of the experiment described in this paper is to evaluate and compare three different methods for interacting with an algorithm for the sonification of data streams. The experiment was carried out using an existing Interactive Sonification Toolkit as a high fidelity prototype. The experiment focused on measuring and comparing the efficiency...

This paper describes the sonification of electromyographic (EMG) data and an experiment that was conducted to verify its efficacy as an auditory display of the data. A real-time auditory display for EMG has two main advantages over the graphical representation: it frees the eyes of the analyst, or the physiotherapist, and it can be heard by the pat...

Creativity research is a large and varied field in which the subject is characterized on many different levels. The arrival of digital media and computational tools has opened up new possibilities for creative practice. The cutting edge in the digital arts is a highly fertile ground for the investigation of creativity and the role of new technologi...

This paper describes an experiment to compare user understanding of complex data sets presented in two different modalities, a) in a visual spectrogram, and b) via audification. Many complex time-series data sets taken from helicopter flight recordings were presented to the test subjects in both modalities separately. The aim was to see if a key se...

This paper outlines the evolution, in the history of art and technology, of image-sound interface systems. This can be called the evolution of the "audiovisual discourse" in art and technology. Recent software for audio and visual material manipulation is briefly described and audiovisual digital art works, developed during the COSTART Research Pro...

This paper stresses the importance of the human user being tightly embedded within an interactive control loop for exploring data sets using sound. We consider the quality of interaction, and how this can be improved in computer systems by learning from real-world acoustic interactions. We describe how different sonification methods can utilise the...

An abstract is not available.

This paper describes work-in-progress on an Interactive Sonification Toolkit which has been developed in order to aid the analysis of general data sets. The toolkit allows the designer to process and scale data sets, then rapidly change the sonification method. The human user can then interact with the data in a fluid manner, continually controllin...

The paper reports on the use of a visual programming environment and discusses its impact on thecreative process. It draws upon research into the collaborative interactive art development process.The context of the work is COSTART, a research project on computer systems for creative work that was undertaken at the Creativity and Cognition Research...

The Workshop on 'Sonic Interaction Design and its relation to Film and Theatre Sound Design’ was organised as part ofthe European COST project on Sonic Interaction Design (COST-SID). The aim of the workshop was to explore how existingpractices in theatre can inform the design and evaluation of sonic interfaces. In this paper we describe the process...

1. ABSTRACT Nowadays, Audio Description is used to enable visually impaired people to access films. However, it presents an important limitation, which consists in the need of the visually impaired audiences to rely on a describer, not being able to access the work directly. The aim of this project was to design a format of sonic art called audio f...

Creating interactive artworks involves the construction of complex computer systems. For artists this often requires the formation of collaborative arrangements with technologists. Collaboration involves multi-disciplinary teamwork and this has implications for the creative process. In June and July of 2002, a group of people came together under th...

Sonic Interaction Design (SID) is an emerging field that is positioned at the intersection of auditory display, ubiquitous computing, interaction design, and interactive arts. SID can be used to describe practice and inquiry into any of various roles that sound may play in the interaction loop between users and artifacts, services, or environments,...