Conference PaperPDF Available

The Impact of Sound Systems on the Perception of Cinematic Content in Immersive Audiovisual Productions

Authors:
A preview of the PDF is not available
... That same study showed that participants observed larger areas of the screen and were therefore less able to focus specifically on the relevant screen area when more enveloping 3D sound was used. A more recent study replicated those findings using different soundtrack immersiveness levels in a movie reproduction [9]. It was found that subjects spent more time looking at non task-related screen areas with surrounding sound conditions, as opposed to when the sound came exclusively from the central part of the screen. ...
... These results clearly show that participants move their eyes more and pay attention to a greater variety of visual elements when the sound is more distributed. This is not a novel finding, as it has been reported before [9,12], but it remains surprising. It would be expected that, when given a specific task, participants would be able to intentionally control their attention and focus on the relevant stimuli. ...
Chapter
Full-text available
The goal of this study was to explore the effects of different spatial sound configurations on visual attention and cognitive effort in an immersive environment. For that purpose, different groups of people were exposed to the same immersive video, but with different soundtrack conditions: mono, stereo, 5.1 and 7.4.1. The different sound conditions consisted of different artistic adaptations of the same soundtrack. During the visualization of the video, participants wore an eye-tracking device and were asked to perform a counting task. Gaze direction and pupil dilation metrics were obtained, as measures of attention and cognitive effort. Results demonstrate that the conditions 5.1 and 7.4.1 were associated with larger distributions of the visual attention, with subjects spending more time gazing at task-irrelevant areas on the screen. The sound condition which led to more concentrated attention on the task-relevant area was mono. The wider the spatial sound configuration, the greater the gaze distribution. Conditions 7.4.1 and 5.1 were also associated with larger pupil dilations than the mono and stereo conditions, showing that these conditions might lead to increased cognitive demand and therefore increased task difficulty. We conclude that sound design should be carefully planned to prevent visual distraction. More surrounding spatialized sounds may lead to more distraction and more difficulty in following audiovisual contents than less distributed sounds. We propose that sound spatialization and soundtrack design should be adapted to the audiovisual content and the task at hand, varying in immersiveness accordingly.
... Another study with continuous realistic stimuli revealed that visual attention and visual event detection in a counting task were affected by different sound event distributions [19], broader distributions leading to worse visual perception performance. Another similar preliminary study found that broader sound distributions led to wider distribution of visual attention and also to worse performance in a visual event counting task [20]. From these studies, it can be argued that sound might be able to affect basic visual perception mechanisms in complex scenarios. ...
... Likewise, the study [22], through an experiment, suggests that binaural sound significantly reduces a visual search or attention task in VR. The study [23] evaluates the impact of sound systems or sound design formats on the perception of cinematic virtual reality by the audience. As part of the study, participants were given a visual attention task in four different sound conditions (mono, stereo, 5.1 and 7.1.4.). ...
Conference Paper
Full-text available
In screen-based cinema, the relationship between audiences and the cinema screens remains fixed. The visual frame on-screen serves as a reference for all filmmaking conventions, including sound design. In virtual reality films, this visual reference no longer exists. The audience as a user can freely choose the viewing direction and interact with the space. This presents new challenges for spatial sound design in terms of on-location sound recording, situating sound elements in 360-degree space, perspective, and guiding viewers’ attention. Still, the research on spatial sound design in cinematic virtual reality remains limited. A scoping review of the literature was undertaken to explore the breadth of literature available on spatial sound design for cinematic virtual reality focusing on live-action filmmaking. Searches were conducted in Web of Science, SCOPUS, IEEE Explore, ACM Digital Library, and Google scholar, followed by backward and forward-searching of relevant references. Searches were restricted to publications in the English language from 2015 onwards. After eliminating the duplicate entries, the search results were screened for the title and abstract first and later for full text. Ten studies were included in the final review to explore different aspects of sound design, challenges, and opportunities. The current literature review aims to contribute to the extant body of knowledge on sound design in cinematic virtual reality and identify gaps for further studies.
... Dwyer and Perkins note that sound design is one of the additional factors, along with composition of shots and camera movement, that need to be related to the length of shots so that visual momentum can be properly understood (2018,120). Recent examples of relevant eye-tracking research include work on the way the direction of viewers' gaze can be affected by film music (Mera and Stumpf 2014), the combination of music, dialogue, and sound effects (Batten and Smith 2018), or even the availability and setup of sound equipment that is used during movie-watching (Korshunova et al. 2019). All this work is an important contribution to the wider literature of film theory, supported by an increased sense of credibility that the use of eye-tracking is capable of promising. ...
Article
Full-text available
The use of eye-tracking in data collection, when accompanied by the proper research questions and methodology, is a powerful tool that may provide invaluable insights into the way viewers perceive and experience movies. Film theory can use eye-tracking to test and verify research hypotheses not only with unprecedented accuracy, but also with the ability to address a significant variety of theoretical questions. Eye-tracking can help build contemporary film theory by supporting its various fields of research, and also even assist the production of films themselves by helping filmmakers make more informed creative decisions. The present article is an overview of eye-tracking and its gradual implementation in cinema research; in the context of discussing some recent examples of academic work based on eye-tracking, it considers the technology of eye-trackers and the way in which human vision handles visual information on screen. By testing the attentional behaviour of viewers, eye-tracking can produce more solid answers to questions regarding the way films are experienced; therefore, it may very well prove to be the spearhead of a more robust body of film theory in the near future.
Article
Building on several decades of research and development, the recent progress in virtual reality (VR) and augmented reality (AR) devices with spatial computing technologies marks a significant leap in human–computer interaction, with applications ranging from entertainment and education to e‐commerce and healthcare. Advances in these technologies promise immersive experiences by simulating and augmenting the real world with computer‐generated digital content. The core objective of the VR and AR systems is to create convincing human sensory perceptions, thereby creating immersive and interactive experiences that bridge the gap between virtual and physical realities. However, achieving true immersion remains a goal, and it necessitates a comprehensive understanding of the neuroscience of human multisensory perception and accurate technical implementations to create a consistency between natural and synthetic sensory cues. This paper reviews the human sensory‐perceptual requirements vital for achieving such immersion, examines the current status and challenges, and discusses potential future advancements.
Chapter
Full-text available
Presence, the “perceptual illusion of non-mediation,” is often a central goal in mediated and mixed environments, and sound is believed to be crucial for inducing high-presence experiences. This chapter provides a review of the state of the art within presence research related to auditory environments. Various sound parameters such as externalization and spaciousness and consistency within and across modalities are discussed in relation to their presence-inducing effects. Moreover, these parameters are related to the use of audio in mixed realities and example applications are discussed. Finally, we give an account of the technological possibilities and challenges within the area of presence-inducing sound rendering and presentation for mixed realities and outline future research aims. KeywordsPresence–Auditory–Auralization–Sound–Acoustics–Virtual environments–Mixed reality–Augmented reality
Article
Full-text available
Selective attention contributes to perceptual efficiency by modulating cortical activity according to task demands. Visual attention is controlled by activity in posterior parietal and superior frontal cortices, but little is known about the neural basis of attentional control within and between other sensory modalities. We examined human brain activity during attention shifts between vision and audition. Attention shifts from vision to audition caused increased activity in auditory cortex and decreased activity in visual cortex and vice versa, reflecting the effects of attention on sensory representations. Posterior parietal and superior prefrontal cortices exhibited transient increases in activity that were time locked to the initiation of voluntary attention shifts between vision and audition. These findings reveal that the attentional control functions of posterior parietal and superior prefrontal cortices are not limited to the visual domain but also include the control of crossmodal shifts of attention.
Article
Full-text available
Virtual Reality (VR), sometimes called Virtual Environments (VE) has drawn much attention in the last few years. Extensive media coverage causes this interest to grow rapidly. Very few people, however, really know what VR is, what its basic principles and its open problems are. In this paper a historical overview of virtual reality is presented, basic terminology and classes of VR systems are listed, followed by applications of this technology in science, work, and entertainment areas. An insightful study of typical VR systems is done. All components of VR application and interrelations between them are thoroughly examined: input devices, output devices and software. Additionally human factors and their implication on the design issues of VE are discussed . Finally, the future of VR is considered in two aspects: technological and social. New research directions, technological frontiers and potential applications are pointed out. The possible positive and negative influence of VR on li...
Book
In The Sonic Persona, Holger Schulze undertakes a critical study of some of the most influential studies in sound since the 19th century in the natural sciences, the engineering sciences, and in media theory, confronting them with contemporary artistic practices, with experimental critique, and with disturbing sonic experiences. From Hermann von Helmholtz to Miley Cyrus, from FLUXUS to the Arab Spring, from Wavefield Synthesis to otoacoustic emissions, from premillennial clubculture to postdemocratic authoritarianism, from signal processing to human echolocation: This book presents a fundamental critique concerning recent sound theories and their anthropological concepts - and proposes an alternate, a more plastic, a visceral framework for research in the field of a cultural anthropology of sounding and listening. This anthropology of sound takes its readers and listeners on a research expedition to the multitude of alien humanoids and their surprising sonic personae: in dynamic and generative tension between predetermined auditory dispositives, miniscule and not seldomly ignored sound practices, and idiosyncratic sensory corpuses: a critique of the senses
Conference Paper
The design of an immersive audiovisual environment for researching the aspects of the perception of spatial sound in presence of surrounding moving visual image is presented. The system consists in a visual screen with a wide field-of-view based on acoustically transparent screens that span 226° in the horizontal plane and 57° in the vertical plane. In addition, a 3D multi-channel sound reproduction system with 29 active loudspeakers is installed. The total system is optimized for audio playback, and measured acoustical responses of the system are presented. The system is equipped with a tracking system based on infrared cameras, which enable headtracking for headphone listening, and also interaction based on gestures. This audiovisual system aims to be a tool for researching of spatial audio, crossmodal interaction and psychoacoustics, auralization and gaming.
Article
Directional audio coding (DirAC) is a method for spatial sound representation, applicable for different sound reproduction systems. In the analysis part the diffuseness and direction of arrival of sound are estimated in a single location depending on time and frequency. In the synthesis part microphone signals are first divided into nondiffuse and diffuse parts, and are then reproduced using different strategies. DirAC is developed from an existing technology for impulse response reproduction, spatial impulse response rendering (SIRR), and implementations of DirAC for different applications are described.
Article
An experiment was conducted to address the relation between theories of selective attention and theories of divided attention by examining the pattern of task interference between visual scanning as a sequential and selective attention process and other concurrent spatial or verbal processing tasks. A distinction is proposed between visual scanning with or without spatial uncertainty regarding their possible differential effects on interference with other concurrent processes. The experiment required the subjects to perform a primary tracking task, which was concurrently performed with a secondary spatial or verbal decision task. The relevant information that was needed to perform the decision tasks was displayed with or without spatial uncertainty. The results provide evidence that visual scanning as a spatial exploratory activity produces greater task interference with concurrent spatial tasks than with verbal tasks. Furthermore, spatial uncertainty in scanning is identified to be the crucial factor in producing this differential effect.
Article
The pinna, or external ear, has been shown experimentally to perform an acoustical transformation essential to localization in human hearing. The mathematical form of the transformation is given, the inverse to the transformation is shown to exist, and a theory of localization constructed from the evidence. The theory is extended to reverberation and other facets of human hearing. The basic theorem applied to the study is 'in order to know, the inverse to the transformation of perception must be constructed'.
Perceptual optimization of audio-visual media: Moved by sound
  • väljamäe
A. Väljamäe and A. Tajadura-Jimenéz, "Perceptual optimization of audio-visual media: Moved by sound," Narration and Spectatorship in Moving Images, 2007.