About
106
Publications
30,045
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
501
Citations
Introduction
My research in sound perception is anchored in artistic work as a composer and designer of interactive sound installations and computer music. I am interested in cross-modal correspondences and how individual differences such as personality traits influence perceptual, cognitive and psychophysiological responses to sound. My long-term aim is to enrich the science-art bond by applying empirical results in the design of artwork as well as extracting analytic data from artful sensory experiences.
Current institution
Additional affiliations
November 2007 - September 2017
March 2007 - November 2007
March 2007 - November 2007
Education
December 2009 - December 2015
Field of study
- Sound and Music Communication (School of Computer Science and Communication)
August 2001 - August 2003
August 1998 - June 1999
Field of study
- Cursus de composition et d'informatique musicale
Publications
Publications (106)
The well-known Proust Effect suggests the powerful impact of sensory cues to trigger nostalgia and recall memories of our past experiences. In the Proust example, there are detailed descriptions of spatial scenarios leading to the ‘centre place’ of the memory associated with the taste of madeleine made by his aunt. Lots of architectural writings al...
Sleep is an essential part of health. One factor that affects sleep quality is soundscape, the acoustic environment as perceived. In the SleepSound project, we aim to collect field data and harness machine listening for an AI-supported assessment of soundscape quality for healthy sleep. Why is this important? Our initial literature review reveals t...
This Workshop on the Open Ambisonics Toolkit will have practical, pedagogical, and theoretical elements. The increasing number of possible applications for spatial audio technologies has caused renewed interest on the subject from academic institutions and resulted in a more widespread diffusion of techniques and practices. However, the lack of an...
This Workshop on the Open Ambisonics Toolkit will have practical, pedagogical, and theoretical elements. The increasing number of possible ap- plications for spatial audio technologies has caused renewed interest on the subject from academic institutions and resulted in a more wide-spread diffusion of techniques and practices. However, the lack of...
Push The ENVELOPE Sound Art Initiative is a two-year Program curated by Contemporary Musiking Hong Kong and designed by a team of professional sound artists, instrument designers, experimental musicians and composers to introduce sound as a medium for creation and collaboration. This paper describes the background to and impetus of the initiative;...
The sensory cultural heritage creates identity and cohesion in a community. The Multimodal Hong Kong (MMHK) started in January 2023 and aims to build a database of soundscape and smellscape at culturally significant sites. This paper will introduce the project and its objectives, present results from a pilot survey, and describe the development of...
In prior research on Hong Kong's acoustic environment, noise has been the primary focus, highlighting its diverse sources and its detrimental impact on physical and mental health. However, there is still limited exploration of multi-sensory perspectives in Hong Kong. Mong Kok is one of the world's most densely populated areas, with a density of 130...
Many people in Hong Kong regularly visit Chinese temples, and see this not necessarily as a religious activity, but as a traditional practice. Temples represent the local intangible cultural heritage. The dense urbanity and mixed population characterise Hong Kong and make it valuable to study its sounds and smells. Most temples existed earlier than...
This paper reviews an experimental workshop exploring creative ways to use soundscape and smellscape protocols for better spatial understandings. It aims to find out how visual methods such as drawing and making could contribute to the recall process. The reflections of the experimental work address the challenge of gaining integrated spatial infor...
Despite the importance of soundscape and smellscape in intangible cultural heritage, little is known about multisensory interactions. To explore the impact of multiple senses on environmental perception, the present study focused on six stalls at Shek Kip Mei wet market in Hong Kong. In the first part of the study, an online survey collected percep...
Despite the importance of soundscape and smellscape in intangible cultural heritage, little is known about multisensory interactions. To explore the impact of multiple senses on environmental perception, the present study focused on six stalls at Shek Kip Mei wet market in Hong Kong. In the first part of the study, an online survey collected percep...
The sensory cultural heritage creates identity and cohesion in a community. The Multimodal Hong Kong (MMHK) started in January 2023 and aims to build a database of soundscape and smellscape at culturally significant sites. This paper will introduce the project and its objectives, present results from a pilot survey, and describe the development of...
This paper reviews an experimental workshop exploring creative ways to use soundscape and smellscape protocols for better spatial understandings. It aims to find out how visual methods such as drawing and making could contribute to the recall process. The reflections of the experimental work address the challenge of gaining integrated spatial infor...
Many people in Hong Kong regularly visit Chinese temples, and see this not necessarily as a religious activity, but as a traditional practice. Temples represent the local intangible cultural heritage. The dense urbanity and mixed population characterise Hong Kong and make it valuable to study its sounds and smells. Most temples existed earlier than...
The hospital soundscape is known for high noise levels and a perception of chaos, leading to concerns about its impact on patients, families, professionals, and other hospital staff. This study investigates the relationship between sound, Annoyance, and sleep quality in a multi-patient neurology ward. A mixed-methods approach was employed. Intervie...
The district of Mong Kok in central Kowloon, Hong Kong, is one of the most densely populated areas in the world. Our current project focuses on investigating a pedestrian footbridge system and designing a multisensory augmented reality (AR) map. In this paper, we discuss current on-site maps and how an AR system can make significant improvements, b...
This study focuses on a corpus of 445 sonification projects currently available in the Data Sonification Archive (DSA). The DSA develops in a collaborative process that involves researchers and creative communities and has been online since early 2021. Projects are heuris-tically classified according to several aspects, in particular their intended...
The innovation of sound recording and reproduction technologies some 145 years ago spurred composers to imagine and indeed create works of extreme duration: very long-lasting days, years, or more-and conversely, very short-miniatures of a few seconds that nevertheless encapsulate 'large' expressions or denote a corpus of pre-existing music. Compari...
摘要 :在气候危机的大背景下，当代声音艺术家如何才能进行可持续实 践?显然，音乐家、制作人、作曲家和音响设计师不能再盲目地进行艺术创 作。富有情感的旋律、激动人心的节奏、丰富的音色，不能仅仅是为了吸引 听 众 短 暂 地 进 入 “ 极 乐 世 界 ” (S c h a r a f f e n l a n d )。 艺 术 不 仅 仅 是 娱 乐 。 声 音 艺 术家有责任应对气候危机。虽然包括我在内的许多声音艺术家都与学术机构 有着密切的联系，这些机构为我们提供了充分的资源和时间，但我们必须认 识到，这些便利也意味着巨大的责任。事实上，我们不能仅仅满足于提高危 机意识，而必须努力做出行为上的改变。在这次讲座中，我介绍了流行音乐 家，声音艺术家对可持续实践的倡议。在此基础上，我讨论了在学术界...
Since the closing of the first wave of ubiquitous music research, our community has expanded the diversity and depth of its challenges. Some threads have followed a steady path of development and at this point may be considered established frameworks. This state of affairs-featuring composites of musical outcomes, technological deployments and conc...
Generating music with pitch and rhythm has received ample attention from researchers. By contrast, noise music as a special genre is largely ignored so far. This paper aims to generate noise music in 'Merzbow style' using a neural network called variational autoencoder (VAE). The results show that our proof-of-concept system can generate noise musi...
Reverberation is an audio effect used in music production affecting the audio spectrum, as well as the timbre, of music. Technical ear training concerning the ways that reverberation works on music samples is important, since in order to achieve the desired texture sound engineers need to be able to perceive subtle audio changes. However, "dry" rec...
With the recent extension of application of spatial audio in the entertainment industries, an increasing number of academic institutions are renewing their interest in the subject. However, the plethora of techniques and practices, together with the relative novelty of them, has resulted in a lack of a methodology for teaching these technologies. T...
Hospital soundscapes are often associated with unhealthy sound levels and an overall perception of chaos and annoyance. Over the past four decades, concerns about the harmful effects of environmental noise on hospital stakeholders (patients, families, and healthcare professionals) were repeatedly raised by the scientific community. In this paper, t...
With the recent extension of application of spatial audio in the entertainment industries, an increasing number of academic institutions are renewing their interest in the subject. However, the plethora of techniques and practices, together with the relative novelty of them, has resulted in a lack of a methodology for teaching these technologies. T...
5 Sonification and visualisation are tools for communicating a deeper understanding of the climate crisis
6 and for stimulating action. Technically, they are umbrella terms for the translation of data into sound and
7 light, respectively. They are examples of perceptualization, the study and design of ways to recast scientific
8 knowledge into expr...
Reverberation is an audio effect frequently used in music production, which affects spectrum of audio as well as the perception on timbre of music. Whereas, de-reverberation, which refers to the process of removing reverberation from sound as an emergent topic attracts the attentions from researchers, due to the sound tasks such as music informatio...
In the context of the climate crisis, what makes for a sustainable practice for contemporary sound artists? It is clear that musicians, producers, composers, and sound designers can no longer blindly pursue the creation of beauty-emotional melodies, exciting rhythms, lush timbres-simply to lure listeners on a leisurely, alas temporary, escape into...
This is a first draft of a longer report on the development of SoundLab, a 3D spatial audio research and practice unit at the School of Creative Media, City University of Hong Kong, co-directed by the authors. We outline how we have developed the lab's objectives in terms of research, artwork, teaching, and outreach. We give a brief account of the...
As a result of the recent extension of the fields of application of spatial audio in the entertainment industries, an increasing number of academic institutions are renewing their interest in the subject, adding it to their curricula. However, the plethora of techniques and practices, together with the relative novelty of them, has resulted in a la...
The increasing number of possible applications for spatial audio technologies has caused renewed interest on the subject from academic institutions and resulted in a more widespread diffusion of techniques and practices. However, the lack of an integrated methodology for teaching these technologies is clear. This served as a motivation for our team...
Exhibition as part of GRF project and Symposium on Inclusive Design with Multi-senses :: https://mmhk.scm.cityu.edu.hk/index.php/2023/04/11/exhibition-the-dual-city-multi-sensory-experience-of-markets/
Introduction
It has proven a hard challenge to stimulate climate action with climate data. While scientists communicate through words, numbers, and diagrams, artists use movement, images, and sound. Sonification, the translation of data into sound, and visualization, offer techniques for representing climate data with often innovative and exciting...
Artificial intelligence (AI) technologies have been applied in music production to create various sound effects, including reverberation. However, evaluation of such applications has not yet been fully explored in research studies. This paper reports results from a study comparing reverberation processing on six recordings of guitars, with musical...
This article analyses recent developments of sonic art in Hong Kong. Based on a series of in-depth interviews with 23 local sonic art practitioners over the past six years, we discuss the contextual understanding of what constitutes ‘sonic art’ among local practitioners, along neighbouring terms such as ‘electroacoustic music’, ‘experimental music’...
The presentation is a progress report on the design and installation of SoundLab, [1] a physical art/research space with a hemispherical loudspeaker array dedicated to high spatial resolution audio at the School of Creative Media in City University of Hong Kong that was initiated in November 2020. We also introduce a study on the local context of s...
DACA 2022 is a dual-hub conference on interactive sonification and visualisation for climate science communication. It is supported by the Environment and Conservation Fund, the School of Creative Media, and the Cultural and Sports Committee at City University, in partnership with Hong Kong Observatory, Institute of Electronic Music and Acoustics a...
In Norse mythology, Helheim is the lowest part of the afterworld: the world of the dead. It is ruled by the goddess of death, Hel, a daughter of Loki. As visitors tread the steps of the central staircase of the Run Run Shaw Creative Media Centre, they are met with sonic objects falling onto them; cascading down the steps; obscenely bouncing like yo...
The paper introduces SoundLab, a 3D spatial audio research and practice unit at the School of Creative Media, City University of Hong Kong, co-directed by PerMagnus Lindborg and Ryo Ikeshiro, and serves as a progress report. It begins with a description and its objectives involving research, artwork, teaching and outreach. It then gives a brief acc...
The smellscape is the olfactory environment as perceived and understood, consisting of odours and scents from multiple smell sources. To what extent can audiovisual information evoke the smells of a real, complex, and multimodal environment? To investigate smellscape imagination, we compared results from two studies. In the first, onsite participan...
Human-machine communication has evolved from one-to-one to multi-agent systems where the interplay between machines themselves interacts with human perception and behavior, complicated by unconstrained emotion-based variables in social systems. To investigate Human-Robot and Robot-Robot-Human interaction while constraining the interaction variables...
My paper addresses the theme of 'uncertainty' via the subject matter of a creative project in my remote-learning classes in radio play, audio narrative, and soundscape composition. The focus is on remoteness and the toll that extreme environmental conditions may have on individuals and small communities of people living on isolated islands in the g...
Environmental sounds are a key component of the human experience of a place as they carry meanings and contextual information, together with providing situational awareness. They have the potential to either support or disrupt specific activities as well as to trigger, to inhibit, or simply to change human behaviors in context. The experience of ac...
The advent of computer for consumers have been supporting artists in various fields to develop their creativity into new territories. Computer-assisted sonification is one of the modern techniques which is available for the contemporary artist. Music and sound art usually derive from abstract inspiration and the same approach ais true for aesthetic...
As AI technology is applied in sound effects plug-ins, machines start to undertake part of sound engineering in studio music production. In this case, the musical outputs from intelligent tools should be evaluated. In this paper, we focus on reverberation. Previous studies have explored perception of reverberation effects e.g. early decay time, how...
New conference formats are emerging in response to COVID-19 and climate change. Virtual conferences are sustainable and inclusive regardless of participant mobility (financial means, caring commitments, disability), but lack face-to-face contact. Hybrid conferences (physical meetings with additional virtual presentations) tend to discriminate again...
Loki’s Pain is an immersive 3D audio sonification of seismological activity. Visitors take the place of Loki, who was punished by the gods and caused earthquakes. We designed an auditory display structure in the shape of a hemi-dodecahedron and built a prototype with a low-budget DIY approach. Seismological data were retrieved from the Internet. Lo...
Loki’s Pain is an immersive 3D audio installation artwork, a sonification of seismic activity. Visitors take the place of Loki, who was punished by the gods and caused earthquakes. We designed an auditory display in the shape of a hemi-dodecahedron and built a prototype with a low-budget, DIY approach. Seismic data were retrieved from the Internet....
The coronavirus disease 2019 (COVID-19) lockdown meant a greatly reduced social and economic activity. Sound is of major importance to people’s perception of the environment, and some remarked that the soundscape was changing for the better. But are these anecdotal reports based in truth? Has traffic noise from cars and airplanes really gone down,...
The idea for When We Collide sprang from Douglas Hofstader’s metaphor of creativity as the meeting between records and record players, appearing in his 1979 book "Gödel, Escher, Bach: An Eternal Golden Braid". In our case, the records are soundfiles, whilst the record player is a generative system. The player analyses, selects, mixes, transforms, a...
We report an approach to communicating cross-cultural differences in sentiment data through sonification. Sonification is a powerful technique for the translation of patterns into sound that is understandable, accessible, and musically pleasant. A machine-learning classifier was trained on sentiment information of two samples of Tweets with the key...
There are several established methodologies for the analysis of electroacoustic music, but color association may not have been previously employed. Using a non-verbal response interface such as color might allow analysists and composers to gain a broader understanding of how music affects listeners generally. This understanding would stimulate the...
The International Computer Music Association aims to be an inclusive association with the goal of promoting computer music in all its forms of expression. ICMA embraces styles, genres, thinking, and tools, that active- ly, passionately, and profoundly con- nect music and computing. Member- ship is open to individuals of all ethni- cities, countries...
Situations of listening As I close my door and walk down the stairs, the motor-driven lock mechanism heralds a sequence of percussive metallic clicks, in counterpoint with my creaking shoes and, through their door, laughter from the neighbor's child. I'm in a space with stone floor and concrete walls. I stop, hold my breath, and wait for the reverb...
(https://www.mitpressjournals.org/doi/abs/10.1162/leon_a_01339) To work flexibly with the sound design for The Locust Wrath, a multimedia dance performance on the topic of climate change, we developed a software for interactive sonification of climate data. An open-ended approach to parameter mapping allowed tweaking and improvisation during rehear...
In what ways do we listen to the soundscape? How do our concurrent activities, moods, and abilities determine the listening mode? What is it that allows us to experience arbitrary sounds in an everyday environment as elements in a musical composition? Why do different ways of listening sometimes feel categorically different? Are there multiple para...
Noise has become integral to electroacoustic music aesthetics. In this paper, we define noise as sound that is high in auditory roughness, and examine its effect on cross-modal mapping between sound and visual shape in participants. In order to preserve the ecological validity of contemporary music aesthetics, we developed Rama, a novel interface,...
In music cognition, dissonance has been linked to musical features such as functional harmony and external contexts (e.g. working memory). However, researchers have also suggested that low-level psychoacoustic features, such as auditory roughness, are important to dissonance perception. Similarly, cross-modal mapping has been shown to be unconsciou...
This paper studies the use of Gumowski-Mira maps for sonic arts. Gumowski-Mira maps are a set of chaotic systems that produce many organic orbits that resemble cells, flowers and other life forms. This has prompted mathematicians and eventually artists to study them. These maps carry a potential for use in the sonic arts, but until now such use i...
Pacific Belltower tolls for you… to remind of the fragility of the Earth’s crust, and the reality faced by people around the Pacific Ocean exposed to the terrifying power of unpredictable earthquakes and volcanoes. The ’tower’ is installed at the centre of the lobby at Xuhui Art Museum, Shanghai, China. Loudspeakers are turned outwards, so that wal...
Pacific Belltower tolls for you… to remind of the fragility of the Earth's crust, and the reality faced by people around the Pacific Ocean exposed to the terrifying power of unpredictable earthquakes and volcanoes. The tower is installed at the centre of a public space, such as a lobby, using parametric 'beam' speakers and wall reflections to diffu...
There have been few empirical investigations of how individual differences influence the perception of the sonic environment. The present study included the Big Five traits and noise sensitivity as personality factors in two listening experiments (n = 43, n = 45). Recordings of urban and restaurant soundscapes that had been selected based on their...
Restaurants are complex environments engaging all our senses. More or less designable sound sources, such as background music, voices, and kitchen noises, influence the overall perception of the soundscape. Previous research suggested typologies of sounds in some environmental contexts, such as urban parks and offices, but there is no detailed acco...
Welcome to this Special Issue of Array: Proceedings of Si15, the 2nd International Symposium on Sound and Interactivity.
The articles in the present issue originated in the Si15 Soundislands Festival, which was held in Singapore 18–23 August 2015. The festival events included five invited artist performances, two scientific keynotes and two days o...
This dissertation is about sound in context. Since sensory processing is inherently multimodal, research in sound is necessarily multidisciplinary. The present work has been guided by principles of systematicity, ecological validity, complementarity of methods, and integration of science and art. The main tools to investigate the mediating relation...
This dissertation is about sound in context.
Since sensory processing is inherently multimodal, research in sound is necessarily multidisciplinary.
The present work has been guided by principles of systematicity, ecological validity, complementarity of methods, and integration of science and art.
The main tools to investigate the mediating re...
Crossmodal associations may arise at neurological, perceptual, cognitive, or emotional levels of brain processing. Higher-level modal correspondences between musical timbre and visual colour have been previously investigated, though with limited sets of colour. We developed a novel response method that employs a tablet interface to navigate the CIE...
This paper looks into how music composers’ rights to their work are dependent on the nature of a work as well as its origin and underlying philosophy. Some differences and overlaps between French Droit d’auteur, Anglo-Saxon Copyright and the Copyleft movement will be explored. I will briefly touch on examples from my own compositional practice. Acc...
Mobile devices have been used in soundscape installations and performances over the past decade or longer, often to emphasize social interaction. Multichannel sonification has been found to successfully represent data describing kinematic phenomena. However, there are few if any examples where these two approaches are combined. The Locust Wrath pro...
Audio quality is known to cross-modally influence reaction speed, sense of presence, and visual quality. We designed an experiment to test the effect of audio quality on source localization. Stimuli with different MP3 compression rates, as a proxy for audio quality, were generated from drum samples. Participants (n = 18) estimated the position of a...
The SI13 NTU/ADM Symposium on Sound and Interactivity in Singapore provided a meeting point for local researchers, artists, scholars and students working creatively with sound and interactivity, as well as the foundation for an issue exploring sound and interactivity in the Southeast Asian country.
The School of Art Design Media of Singapore’s Nany...
This text is a “constructed multilogue” oriented around a set of questions about sound art in Singapore. I have lived here since 2007 and felt that a “community report” should aim to probe recent history deeper than what I could possibly do on my own, in order to give a rich perspective of what is happening here today. I was very happy when Pete Ke...
Skalldans is an audiovisual improvisation framework for a solo laptop performer. The framework is an instrument for performance, developed in Max[2]. Sound and video syntheses are piloted with a MIDI interface, a camera, and a Wiimote; also, audiovisual streams influence each other. The present text discusses some of the hardware and software point...
The article outlines a psychoacoustically founded method to describe the acoustic performance of earphones in two dimensions, Spectral Shape and Stereo Image Coherence. In a test set of 14 typical earphones, these dimensions explained 66.2% of total variability in 11 acoustic features based on Bark band energy distribution. We designed an interacti...
In this digital music era, sorting and discovery of songs is getting harder and more time consuming than before, due to the large pool of songs out there. Many music recom-mendation system and other similar applications in the market make use of collaborative filtering and social rec-ommendation to suggest music to listeners. However, the problem a...
There is no exact model for the relationship between the autonomic nervous system (ANS) and evoked or per-ceived emotion. Music has long been a privileged field for exploration, while the contribution of soundscape research is more recent. It is known that health is influenced by the sonic environment, and the study here presented aimed to investig...
This pilot study investigated patterns in the way different people experience an interactive installation. The Canopy (Lindborg, Koh & Yong 2011, with support from NTU’s Institute of Media innovation)
The facility under study is “NTU Innovation Centre Block 1/1 Multi-Purpose Hall (T,G)” and will here be referred to as ”ICMG”. It is a free-standing building with brick walls. The inner dimensions are: 17.1 m long, 5.4 m wide, for a floor surface of 92.3 m2. The facade wall has a double glass door and 4 large glass windows. The facing long wall has...
“On the String” is an installation-performance scored for sound sculptures, real-time synthesis, musicians, light display and an immersive sound diffusion system. The composition, inspired by String Theory, called for the design of an immersive sound design that could be flexibly adapted to different diffusion situations. The software design aimed...
This overview of research in 3D sound focusses on brief descriptions of the major 3-dimensional sound technologies that meet challenges of sonic perception in different ways: binaural techniques, ambisonics and wave field synthesis.
This article reports results from a study of perceived emotion portrayal in cartoons by different groups of subjects. A set of audiovisual stimuli was selected through a procedure in two steps. First, 6 ‘judges’ evaluated a large number of random snippets from all Mickey Mouse cartoons released between 1928 and -35. Analysis singled out the five fi...
Singapore Voices is an interactive installation, integrating sound and image in a series of touch-sensitive displays. Each display shows the portrait of an elderly person, standing with the hand turned outwards, as if saying: “I built this nation”. Two displays can be seen in Figure 1 below. When the visitor touches the hand or shoulder, they hear...
Questions
Questions (2)
Dimensional personality models (also mood) were used in music emotion research by Jonna Vuoskoski et al. (http://users.ox.ac.uk/~musf0093/publications.html) - is someone extending this work to soundscapes? Soundscape research has used e.g. the Weinstein Noise Sensitivity index, however this describes a specific trait, and not an individual's personality as a whole. There are several models for assessing soundscape quality.
I suspect that "Orientation" could be a latent factor for some measurable faculties, including directed (split) attention, facility, acuity, sophistication, etc in perceptual and cognitive processing of bimodal sensory input..
