Conference PaperPDF Available

The Effect Of 3D Audio And Other Audio Techniques On Virtual Reality Experience

DOI:10.3233/978-1-61499-595-1-44
  • Conference: Studies in health technology and informatics
  • Volume: 219
Authors:
Willem-Paul Brinkman at Delft University of Technology
Willem-Paul Brinkman
Allart R D Hoekstra
Allart R D Hoekstra
Allart R D Hoekstra
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Rene Vanegmond at Delft University of Technology
Rene Vanegmond
Download full-text PDF
Read full-text
Download citation

Abstract

Three studies were conducted to examine the effect of audio on people's experience in a virtual world. The first study showed that people could distinguish between mono, stereo, Dolby surround and 3D audio of a wasp. The second study found significant effects for audio techniques on people's self-reported anxiety, presence, and spatial perception. The third study found that adding sound to a visual virtual world had a significant effect on people's experience (including heart rate), while it found no difference in experience between stereo and 3D audio.
Content uploaded by Willem-Paul Brinkman
Author content
All content in this area was uploaded by Willem-Paul Brinkman on Jan 23, 2016
Content may be subject to copyright.
Pre-print of: Brinkman, W.P., Hoekstra, A.R.D., van Egmond, R. (2015). The
effect of 3D audio and other audio techniques on virtual reality experience. Studies in
health technology and informatics, 219, 44-48
The effect of 3D audio and other audio
techniques on virtual reality experience
Willem-Paul BRINKMANa,1, Allart R.D. HOEKSTRAa, René van EGMONDa
a
Delft University of Technology, The Netherlands
Abstract. Three studies were conducted to examine the effect of audio on people’s
experience in a virtual world. The first study showed that people could distinguish
between mono, stereo, Dolby surround and 3D audio of a wasp. The second study
found significant effects for audio techniques on people’s self-reported anxiety,
presence, and spatial perception. The third study found that adding sound to a
visual virtual world had a significant effect on people’s experience (including heart
rate), while it found no difference in experience between stereo and 3D audio.
Keywords. 3D audio, audio, presence, anxiety, spatial perception.
Introduction
A recent meta-analysis showed a positive association between self-reported level of
presence and anxiety [6]. The ability to elicit anxiety is considered a key ingredient in
the success of virtual reality exposure therapy in the treatment of anxiety disorders.
This has motivated research into factors that influence presence such as individual
characteristics [5], or technology factors such as stereoscopic viewing [3] or the field of
view [4]. Relatively little is known about the impact different audio techniques have on
people’s feeling of presence in a virtual world. Several audio techniques exist, such as
mono (1-channel), stereo (2-channels), Dolby surround (multiple-channels), and 3D
audio (realistic audio representation). Unlike the other audio techniques, 3D audio
provides information about the sound source location outside the observer’s head on a
horizontal and vertical plane and also information about the distance toward the sound
source. For this it can use several elements, such as binaural cues, head-related transfer
function (HRTF), head movement, and reverberation. 3D audio can be offered using
speakers or a headphone. This paper examines the effect of different audio techniques
on how people experience a virtual world that used sound, specifically a flying wasp.
1. ABX perceptual difference listening study
The first study tested whether people are able to hear the difference between 3D audio,
Dolby surround, stereo and mono with headphones. This study was setup as an ABX
discrimination test, which is a double blind method to compare two stimuli.
Participants were presented with three audio fragments: A, B and X, whereby X could
either be A or B which was randomly chosen. Participants were asked to determine
1 Corresponding Author, w.p.brinkman@tudelft.nl
Pre-print of: Brinkman, W.P., Hoekstra, A.R.D., van Egmond, R. (2015). The
effect of 3D audio and other audio techniques on virtual reality experience. Studies in
health technology and informatics, 219, 44-48
whether audio fragment X was similar to fragment A or B. While listing they could
directly switch between the 3 sound fragments. They were asked to do this four times
for the six combinations of four audio techniques resulting in 24 trails for each
participant. To control for potential order and learning effect the order of the trails was
balanced following Balanced Latin square. The experiment was performed in an
acoustically isolated room. Participants wear a Beyerdynamic DT 770 headphone
(frequency response 5 – 35.000 Hz, 250 Ohms impedance, ambience noise reduction
approximately 18dB(A)). A mono-recorded sound fragment of flying wasp2 was placed
in 3D world using the 3D audio tool SoundLocus. The 3D audio was created using
HRTF, human hearing modeling, and a small Doppler effect. A 57 seconds sound
fragment of a flying wasp was created with a constant movement path. Details of the 3
studies can be found in [2]. Twenty-two individuals (15 males, 7 females) with a mean
age of 27.7 years (SD = 8.4) participated. None of the participants suffered from total
deafness in one of their ears. Only one participant indicated to have hearing capacity of
5% in the left ear. All other participants indicated to have no hearing impairments. The
university human research ethics committee approved all 3 studies.
1.1. Results
The comparison of two sound techniques were regarded as a Bernoulli trail, where a
participant either matches a stimuli correct or incorrect with a 50% gamble chance. For
each combination this resulted in 88 tests. For mono – stereo comparison 84 correct
matches were made, for mono – Dolby surround comparison 86 correct matches were
made, for all other comparisons all 88 matches were correct. All comparisons were
significant (p. < .001) above mean gamble chance of 44 correct matches.
1.2. Conclusion
The nearly perfect matching found shows that participants were well able to hear a
distinction between the four different audio techniques.
2. Sound experience study without visuals
As participants were able to distinguish between sounds produced by different
techniques, the next question was whether the four audio techniques had a different
impact on people’s experience, i.e. level of anxiety, presence, and spatial perception.
The same participants, equipment and stimuli material were used as in the previous
study. In additions participants’ heart rate was measured with a Mobi8 device from
TMSi with a Xpod Oximeter. Participants wear a black eye-mask to blindfold them,
and placed their head on a chin-rest to keep their head on a fixed position and
orientation. Participants were exposed to the wasp sound fragment four times, each
time using a different sound technique. Again the order in which conditions were
presented was balanced following a Balanced Latin square. After each sound fragment
participants were asked to rate their level of discomfort on the Subjective Units of
Discomfort (SUD) [8] scale, their level of presence on the Igroup Presence
Questionnaire (IPQ) [7], their fear of wasps on, for this study created, the Fear of
2 Fragments from www.audiosparx.com/sa/summary/play.cfm/crumb.31/crumc.0/sound_iid.407163
Pre-print of: Brinkman, W.P., Hoekstra, A.R.D., van Egmond, R. (2015). The
effect of 3D audio and other audio techniques on virtual reality experience. Studies in
health technology and informatics, 219, 44-48
Wasps Scale (FWS), and their spatial perception on the Spatial Perception
Questionnaire (SPQ) [2]. SPQ was created for this study to measure perceptual strength
of the spatial attribute in the perceived stimuli. SPQ includes 10 items related to
localization, distance/depth, externalization, movement, sense of space, and quality.
FWS is a single 10-point scale with the question: Do you have a fear of wasps? ranging
from 0 (no fear at all) to 10 (very much). To establish a baseline heart rate
measurement, participants had to sit in total silence for 5 minutes at the start of
experiment, after which they were asked for a SUD score. Data of one participant was
discarded because of an administrative error.
2.1. Results
A Friedman test on the mean IPQ score found a significant (
χ
2(3) = 12.26, n = 22, p
= .007) effect for the four audio techniques. Wilcoxon Signed-Rank Tests showed a
significant higher level of presence for 3D audio (Mdn = 1.29) compared to (Z = 2.90, p.
= .004) Dolby surround (Mdn = 0.71), and (Z = 3.51, p. < .001) mono (Mdn = -0.86)
sound. Furthermore, significant higher level of presence was found for stereo (Mdn =
0.29) compared to (Z = 2.71, p. = .007) mono, and for Dolby surround compared to (Z
= 2.67, p. = .008) mono.
A Friedman test found a significant (
χ
2(3) = 19.75, n = 22, p < .001) effect for the
audio techniques on SPQ score. Wilcoxon Signed-Rank Tests showed a significant
higher spatial perception score for 3D audio (Mdn = 1.6) compared to (Z = 3.74, p.
< .001) mono (Mdn = -0.9), and (Z = 2.11, p. = .035) Dolby surround (Mdn = 1). On
the other hand, significant lower special perception score was given for mono
compared to (Z = 3.27, p. = .001) stereo (Mdn = 1.3), and (Z = 2.67, p. = .007) Dolby
surround.
A Friedman test found a significant effect (
χ
2(4) = 31.44, n = 22, p < .001) for the
four audio techniques and the baseline conditions in SUD scores. Wilcoxon Signed-
Rank Tests showed a significant lower SUD score for baseline (Mdn = 1) compared to
(Z = 2.91, p. = .004) mono (Mdn = 2), (Z = 3.18, p. = .001) stereo (Mdn = 3), (Z = 3.06,
p. = .002) Dolby surround (Mdn = 3), and (Z = 3.75, p. < .001) 3D audio (Mdn = 4).
Significant higher SUD score was also found for 3D audio compared to (Z = 3.09, p.
= .002) Dolby surround, and (Z = 2.29, p. = .022) mono sound.
After visually inspecting the histogram of the FWS score, two groups were
identified: a lower fear group with scores between 0 and 2 (n = 16) and a higher fear
group with scores between 4 and 8 (n = 5). Mann-Whitney tests found significant
difference between two groups on SUD score for (Z = 2.25, p. = .025) mono (Mdnlower
= 2, Mdnhigher = 5), (Z = 2.06, p. = .039) stereo (Mdnlower = 2.5, Mdnhigher = 4), (Z = 2.22,
p. = .027) Dolby surround (Mdnlower = 2, Mdnhigher = 5), and (Z = 2.00, p. = .046) 3D
audio (Mdnlower = 3, Mdnhigher = 6) conditions.
A repeated measure ANOVA on heart rate found for the four audio techniques and
the baseline conditions (taking only last 2 minutes) an effect (F(3, 60) = 2.41, p = .076)
with a p-value that only approached the threshold level of .05.
2.2. Conclusions
Anxiety reported for the stimuli material seems related to people’s fear for wasps as
anxiety differences were found between the lower and higher wasp fear groups.
Pre-print of: Brinkman, W.P., Hoekstra, A.R.D., van Egmond, R. (2015). The
effect of 3D audio and other audio techniques on virtual reality experience. Studies in
health technology and informatics, 219, 44-48
Furthermore, significant variations found in the level of the presence, anxiety, and
spatial perception, showed that the four audio techniques had different impact on how
the participants experienced the sound fragment. Surprisingly, a significant lower level
for presence was found for Dolby headphones compared to stereo. This might be a
consequence of the 5.1 channel Dolby Headphone algorithm used to simulate a sense
of Dolby surround with headphones, instead of actually reconstructing it by using
multiple loudspeakers.
3. Sound experience study with visuals
The last study tested whether the different sound techniques have a different impact on
people’s experience when sound is integrated into a visual virtual environment. The
study included three conditions: no sound (only visual environment), stereo, and 3D
audio. The visual environment consisted of a 3D wasp flying in an in-door town hall
environment, which was taken from the Vizard tutorial on stereoscopic panoramas. The
wasp flew and crawled for 51 seconds, following the same path in all three conditions.
The pathway consisted out of the following four elements: 1) far away in front of the
observer, 2) close in front of the observer landing near the left ear, 3) close in front of
the observer landing near the right ear, and 4) wasp sitting and walking on the table.
SoundLocus was used to create the sound for the wasp to match its visual fly path.
One member of the new group of 25 participants (9 female, 16 male), consisting of
mainly students and university staff with average age of 28 years (SD = 8.2), reported
to suffer from 30dB loss on both ears. Three other mentioned to have small hearing
impairment. Participants wore the Beyerdaynamic DT 770 headphone, a Sony HMZ-
T2 head mounted display, and Mobi8. Participants again placed their head on a chin-
rest to keep their head on a fixed position and orientation. Also, head tracking was not
supported. The order of the three conditions was again balanced using a Latin square.
Before exposure to the town hall world, baseline SUD and heart rate measurement was
collected in 3 minutes exposure in a neutral virtual reality environment of a waiting
room [1]. After each exposure conditions participants completed IPQ, SPQ and SUD
score. SUD scores were collected at the start and end of the exposure.
3.1. Results
A Friedman test on the mean IPQ score found a significant (
χ
2(2) = 24.15, n = 25, p
< .001) effect for the 3 audio conditions. Wilcoxon Signed-Rank Tests found a
significant lower presence level for the no sound condition (Mdn = -0.29) compared to
(Z = 3.54, p. < .001) stereo (Mdn = 0.64), and the (Z = 3.79, p. < .001) 3D audio
condition (Mdn = 0.43).
A Friedman test found no significant effect (
χ
2(1) = 0.73, n = 25, p = .394)
between stereo and the 3D audio condition on the SPQ scores.
A Friedman test found a significant (
χ
2(2) = 12.22, n = 25, p = .002) effect for 3
conditions on the increment SUD scores i.e. post – pre exposure SUD score. Wilcoxon
Signed-Rank Tests found a lower increment SUD score for the no audio condition
(Mdn = 0) compared to (Z = 2.68, p. = .007) stereo (Mdn = 1) and (Z = 3.04, p. = .002)
3D (Mdn = 1) condition. Splitting the participants group based on the median FWS
score of 3, resulted in lower and higher fear for wasp group. Mann-Whitney tests found
Pre-print of: Brinkman, W.P., Hoekstra, A.R.D., van Egmond, R. (2015). The
effect of 3D audio and other audio techniques on virtual reality experience. Studies in
health technology and informatics, 219, 44-48
significant (Z = 1.99, p. = .047) difference for two groups on increment SUD score only
in the 3D audio (Mdnlower = 0, Mdnhigher = 2) condition.
Heart rate of 5 participants were not recorded successfully. Furthermore, probably
because of anticipation anxiety, the heart rate of one participant was considered an
extreme outlier (> 90 BMP) in the baseline measurement and first wasp exposure
condition. This participant was therefore removed for heart rate analysis. A Friedman
test found a significant (
χ
2(2) = 9.79, n = 19, p = .007) effect for the 3 conditions on the
heart rate. Wilcoxon Signed-Rank Test found a significant lower heart rate for the no
audio (Mdn = 70.41) condition compared to (Z = 2.58, p. = .010) the stereo (Mdn =
73.73) and the (Z = 2.01, p. = .044) 3D audio (Mdn = 71.15) condition.
3.2. Conclusions
The significant variants found in the level of the self-reported presence, anxiety and
heart rate between no audio and the audio conditions suggest that adding audio to a
visual stimuli environment has added value. No significant difference was however
found between the stereo and 3D audio condition.
4. Discussion and Conclusion
A number of conclusions can be drawn in the case of this wasp virtual world. First,
sound on its own can elicit anxiety. Second, if only audio stimulus is provided,
people’s experience is affected by the type of audio technique. Third, adding sound to a
visual environment can enhance the experience. Four, it seems unlikely that compared
to stereo sound, 3D audio will add much to individuals’ experience when exposed to
either an audio only stimuli world, or an audio combined with visual stimuli world.
References
[1] B. Busscher, D. de Vliegher, Y. Ling, and W.P. Brinkman, Physiological measures and self-report to
evaluate neutral virtual reality worlds, Journal of Cyber Therapy and Rehabilitation 4 (2011), 15-25.
[2] A.R.D. Hoekstra, 3D audio for virtual reality exposure therapy, MSc, Delft university of technology,
2013.
[3] Y. Ling, W.P. Brinkman, H.T. Nefs, C. Qu, and I. Heynderickx, Effects of Stereoscopic Viewing on
Presence, Anxiety, and Cybersickness in a Virtual Reality Environment for Public Speaking, Presence-
Teleoperators and Virtual Environments 21 (2012), 254-267.
[4] Y. Ling, H.T. Nefs, W.P. Brinkman, C. Qu, and I. Heynderickx, The Effect of Perspective on Presence
and Space Perception, PLOS ONE 8 (2013).
[5] Y. Ling, H.T. Nefs, W.P. Brinkman, C. Qu, and I. Heynderickx, The relationship between individual
characteristics and experienced presence, Computers in Human Behavior 29 (2013), 1519-1530.
[6] Y. Ling, H.T. Nefs, N. Morina, I. Heynderickx, and W.P. Brinkman, A Meta-Analysis on the
Relationship between Self-Reported Presence and Anxiety in Virtual Reality Exposure Therapy for
Anxiety Disorders, PLOS ONE 9 (2014).
[7] T. Schubert, F. Friedmann, and H. Regenbrecht, The experience of presence: Factor analytic insights,
Presence-Teleoperators and Virtual Environments 10 (2001), 266-281.
[8] J. Wolpe, Psychotherapy by reciprocal inhibition, Stanford University Press, Stanford, Calif.,, 1958.
... This includes studies that focus on accessible or assistive technologies, including audio-only VR for people with visual impairments (Bălan et al. 2018;Picinali et al. 2014;Siu et al. 2020;Wedoff et al. 2019) and the investigation of age (Souza Silva et al. 2019) and visual impairments (Dong, Wang, and Guo 2017) with regards to audio feedback in VR. Many of these medium-specific studies have a focus on technical aspects of audio, investigating the experiential effects of technological factors such as 3D/binaural audio (Brinkman, Hoekstra, and van Egmond 2015;Davies et al. 2017;Hong et al. 2018;Huang et al. 2019;Olko et al. 2017; (2017) (Mehra et al. 2015). The 'no domain' remainder of these studies exploring the medium of VR refers to the large group of papers studying some other aspect of the VR medium. ...
... Related to this (Lindquist et al. 2020), studied the experiential differences between generic ambiences, hich had no effect on perceived recreational value, and detailed ambience created for specific visuals, leading to increased perceived recreational value. The large variability in study outcomes for emotional/affective measures can be partially explained by the differing use of audio in this area, for example, using it for short-term increases in discomfort or fear/anxiety (Brinkman, Hoekstra, and van Egmond 2015;Josman et al. 2008) as well as for decreasing them with repeated exposure (Johnston, Egermann, and Kearney 2020;North et al. 2008). While one study (Lee, Bruder, and Welch 2017) found no effect of audio on fear/anxiety, it should be noted that the base rates for negative affect was perceived as relatively high, which might explain the lack of a significant change. ...
... The two therapeutic areas where VR was most common were exposure therapy (7 studies) and mindfulness practice (6 studies). The use of audio was found to be effective both in terms of creating the experience of anxiety in the short term (Brinkman, Hoekstra, and van Egmond 2015;Josman et al. 2008) and in reducing it through repeated exposure (Johnston, Egermann, and Kearney 2020;North et al. 2008), which provides supporting evidence for the effectiveness of audio in a virtual reality exposure therapy (VRET) context. There was also some support for the use of audio in a mindfulness context to increase relaxation and decrease activation/arousal (Gomez et al. 2017;Nararro-Haro et al. 2016;Pizzoli et al. 2019;Prpa et al. 2017Prpa et al. , 2018Seabrook et al. 2020), although the bulk of these results focused on qualitative data which means that the reporting and analysis of these results was out of the scope of the current study; these might be explored further in other types of review papers, such as narrative reviews. ...
The effect of audio on the experience in virtual reality: a scoping review
Article
Full-text available
The use of virtual reality (VR) has seen significant recent growth and presents opportunities for use in many domain areas. The use of head-mounted displays (HMDs) also presents unique opportunities for the implementation of audio feedback congruent with head and body movements, thus matching intuitive expectations. However, the use of audio in VR is still undervalued and there is a lack of consistency within audio-centedd research in VR. To address this shortcoming and present an overview of this area of research, we conducted a scoping review (n = 121) focusing on the use of audio in HMD-based VR and its effects on user/player experience. Results show a lack of standardisation for common measures such as pleasantness and emphasize the context-specific ability of audio to influence a variety of affective, cognitive, and motivational measures, but are mixed for presence and generally lacking for social experiences and descriptive research.
View
... Existing studies suggest that the perceiving of auditory and visual information of an environment is an interconnected process. Studies like the ones reported by Rohrmann and Bishop research found that providing sound in an environmental simulation "is important in enhancing perceived realism (and also fosters attention and recognition)" (2002, p. 328), A similar study, by Brinkman et al. (2015) observed that adding sound to a visual virtual world had a significant effect on people's experience. Morinaga et al. (2003) refers that visual information has greater effects on the impression related to pleasant factor than auditory information. ...
... These results show that participants perceive a specific virtual environment differently if it has environmental sound or not, following Brinkman et al. (2015) and Serafin et al. (2018). The novelty of our study is the use of the gaze data acquired from an eye-tracking device to prove this relation. ...
The impact of sound in people's behaviour in outdoor settings: A study using virtual reality and eye-tracking
Article
This paper presents an analysis of space perception and how visual cues, such as landmarks and sound, are perceived and impact people's behaviour while exploring a given outdoor space. The primary goal of the research is to investigate how auditory sensations and visual stimuli influence people's behaviour in outdoor built environments. Our technique compares people's perception of the built environment in different conditions: the real world and a replicated virtual world. As a case study, a university campus was used, and four experimental conditions were designed. The study followed a between-subjects design, and the data collection included gaze data acquired from an eye-tracking device as well as self-reports. The study concludes that sound influences human behaviour in such settings. More specifically conclusions are that: i) human behaviour in virtual replications of the real space, including both visual and sound stimuli, is tendentially more similar to human behaviour in the real world than in simulations omitting sound; and ii) there is a difference in human behaviour when people explore the same virtually replicated outdoor space, by varying the presence of sound. This study is particularly useful for researchers working on the comparison between human behaviour in virtual and real environments, related to visual and sound stimuli.
View
... In a VR environment, sound is of great benefit to enhance users' immersion and presence (see Figure 1) [1]. Combined with stronger visual stimuli in VR, sound can play a greater role in enhancing the VR experience [2]. Sound can direct users' attention and behaviour in VR and elicit psychological anxiety or tension. ...
Research on the Application of Sound in Virtual Reality
Article
Full-text available
  • Apr 2023
Immersive sound is an essential element of any Virtual Reality (VR) experience. In recent years, increasing popularity of VR devices and numerous advancements in hardware and software technologies, the development of immersive interactive sound has become more potent. This article explores VR through the sonic perception, presents a review of the Method of combining and applying sound and VR, with the goal of providing more references and suggestions for the sound of VR. This includes the creative and technical work of music and spatial sound in practical projects in recent years. Some examples of musical VR are provided in this article to compare their characteristics. A review of the state of the art of sound simulations, focusing on the various factors that ensure a complete and immersive audio experience, while trying to strike an appropriate balance between realism and accuracy. This article finally discusses the future and challenges of sound in VR.
View
... Regarding auditory displays, it is well known that high quality audio significantly contributes to an immersive experience [46]. The headphones embedded in commercially available HMDs are still rather low quality, and usually an external additional pair of headphones is used for higher fidelity auditory display. ...
A Review of Virtual Reality for Individuals with Hearing Impairments
Article
Full-text available
  • Mar 2023
Virtual Reality (VR) technologies have the potential to be applied in a clinical context to improve training and rehabilitation for individuals with hearing impairment. The introduction of such technologies in clinical audiology is in its infancy and requires devices that can be taken out of laboratory settings as well as a solid collaboration between researchers and clinicians. In this paper, we discuss the state of the art of VR in audiology with applications to measurement and monitoring of hearing loss, rehabilitation, and training, as well as the development of assistive technologies. We review papers that utilize VR delivered through a head-mounted display (HMD) and used individuals with hearing impairment as test subjects, or presented solutions targeted at individuals with hearing impairments, discussing their goals and results, and analyzing how VR can be a useful tool in hearing research. The review shows the potential of VR in testing and training individuals with hearing impairment, as well as the need for more research and applications in this domain.
View
... However, there is limited research with mixed results investigating its capability as a primary tool for inducing the required amount of anxiety for successful exposure therapy. Brinkman et al. [36] compared the effects of different audio rendering techniques over headphones on participants' self-reported anxiety, indicating that compared to mono, stereo and Dolby 5.1 surround, binaural 3D audio generated significantly higher levels of anxiety. Finally during the study conducted by the authors [25], participants diagnosed with ASD reported negative emotional responses whilst listening to adverse auditory stimuli rendered using binaural based spatial audio. ...
The Use of Binaural Based Spatial Audio in the Reduction of Auditory Hypersensitivity in Autistic Young People
Article
Full-text available
Individuals diagnosed with autism spectrum disorder (ASD) are characterised as experiencing impairments in social-emotional interaction and communication, alongside frequently displaying repetitive behaviours and interests. Further to this, they are often described as experiencing difficulties in processing sensory information, with particular prevalence within the auditory modality. Provoked by common environmental sounds, auditory hypersensitivity can result in self-regulatory fear responses. Rather than a physiological pain reaction, literature suggests that these hypersensitivities are resulting through irrational fear of the sounds. This investigation evaluates the use of binaural based spatial audio as a rendering technique for delivering realistic simulations of averse stimuli within a virtual reality (VR) exposure based computer game intervention for auditory hypersensitivity in autism. Over multiple experimental sessions, 20 autistic participants experiencing auditory hypersensitivity were exposed to either spatial audio or stereo renders of target stimuli during the intervention. Measurements of self-reported emotions displayed significant reductions in associated negative emotional reactions to target stimuli for all participants. However, significant improvements were experienced by those listening to spatial audio simulations. Moreover, tracked voluntary interactions with exposure based game-mechanics increased as the study progressed. Providing further evidence of increased tolerance towards averse auditory stimuli.
View
... Neuroimaging research has shown that human brains are more attuned to complex, naturalistic stimuli over those that are simple and artificial [15]. VR technology can be customized to present three-dimensional images [16][17][18], create the illusion of distant sounds [19,20], and provide haptic feedback to create engaging, multimodal stimuli that represent the lived experiences of research participants [21][22][23]. VR can also incorporate a high degree of control in a realistic and multisensory environment, ideal for high quality basic research. ...
Rapid assessment of hand reaching using virtual reality and application in cerebellar stroke
Article
Full-text available
The acquisition of sensory information about the world is a dynamic and interactive experience, yet the majority of sensory research focuses on perception without action and is conducted with participants who are passive observers with very limited control over their environment. This approach allows for highly controlled, repeatable experiments and has led to major advances in our understanding of basic sensory processing. Typical human perceptual experiences, however, are far more complex than conventional action-perception experiments and often involve bi-directional interactions between perception and action. Innovations in virtual reality (VR) technology offer an approach to close this notable disconnect between perceptual experiences and experiments. VR experiments can be conducted with a high level of empirical control while also allowing for movement and agency as well as controlled naturalistic environments. New VR technology also permits tracking of fine hand movements, allowing for seamless empirical integration of perception and action. Here, we used VR to assess how multisensory information and cognitive demands affect hand movements while reaching for virtual targets. First, we manipulated the visibility of the reaching hand to uncouple vision and proprioception in a task measuring accuracy while reaching toward a virtual target (n = 20, healthy young adults). The results, which as expected revealed multisensory facilitation, provided a rapid and a highly sensitive measure of isolated proprioceptive accuracy. In the second experiment, we presented the virtual target only briefly and showed that VR can be used as an efficient and robust measurement of spatial memory (n = 18, healthy young adults). Finally, to assess the feasibility of using VR to study perception and action in populations with physical disabilities, we showed that the results from the visual-proprioceptive task generalize to two patients with recent cerebellar stroke. Overall, we show that VR coupled with hand-tracking offers an efficient and adaptable way to study human perception and action.
View
... This is where 3D audio comes in. 3D audio allows for the simulation of sounds from specific directions, allowing a more realistic feeling [3]. At the present moment, there lacks sufficient tools for users to design immersive audio experiences that fully exploit the abilities of 3D audio. ...
An Context-Aware Intelligent System to Automate the Conversion of 2D Audio to 3D Audio using Signal Processing and Machine Learning
Conference Paper
  • Sep 2022
As virtual reality technologies emerge, the ability to create immersive experiences visually drastically improved [1]. However, in order to accompany the visual immersion, audio must also become more immersive [2]. This is where 3D audio comes in. 3D audio allows for the simulation of sounds from specific directions, allowing a more realistic feeling [3]. At the present moment, there lacks sufficient tools for users to design immersive audio experiences that fully exploit the abilities of 3D audio. This paper proposes and implements the following systems [4]: 1. Automatic separation of stems from the incoming audio file, or letting the user upload the stems themselves 2. A simulated environment in which the separated stems will be automatically placed in 3. A user interface in order to manipulate the simulated positions of the separated stems. We applied our application to a few selected audio files in order to conduct a qualitative evaluation of our approach. The results show that our approach was able to successfully separate the stems and simulate a dimensional sound effect.
View
... The findings of the study by Brinkman et al. suggest that adding sound to the virtual world has a significant effect on people's experience though no difference in experience between stereo and 3D audio was reported [42]. In the same line, another study by Kobayashi et al. suggest that the sympathetic nervous system activated to a greater extent by the spatialized sound [43]. ...
Sound Design for Cinematic Virtual Reality: A State-of-the-Art Review
Chapter
Full-text available
  • Jun 2022
In the last decade, the rapid development of low-cost 360° cameras, Head Mounted Displays, smartphones, and open-source softwares have created new possibilities of immersive storytelling in virtual reality, popularly known as cinematic virtual reality or virtual reality films. Even in 360° videos, the visual experience remains limited to field of view of head mounted displays. Hence, the spatial audio is considered essential to create ‘Presence’ and ‘Immersion’. The viewer is part of the virtual environment with the freedom to interact, navigate, and choose the viewing direction. This presents challenge of sound recording, designing and playback. The recent development in spatial audio recording, sound synthesis and rendering has addressed some of these challenges. The current study presents the state-of-the-art of sound design in cinematic virtual reality through literature review and textual analysis of the relevant publication in the field since 2015. The objective is to identify further scope of studies. This review could be useful for filmmakers, sound designers, and scholars working in the field of cinematic virtual reality.KeywordsVirtual realityImmersive storytelling360°Spatial audio Film Sound design
View
Исследования эффекта присутствия в виртуальной реальности: современное состояние и перспективы
Article
Full-text available
  • Jun 2020
В статье рассматриваются современные исследования эффекта присутствия – психологического феномена, в котором человек испытывает иллюзию нахождения в виртуальной среде, создаваемой специальными техническими устройствами. Рассматриваются технические, стимульные и субъективные факторы, оказывающие влияние на эффект присутствия. Критически анализируются субъективные и объективные методы измерения эффекта присутствия. Рассмотрены факторы, которые нарушают переживание погружения в виртуальную среду. Обозначены проблемы и задачи, стоящие перед исследователями и разработчиками технологий виртуальной реальности.
View
View
A Meta-Analysis on the Relationship between Self-Reported Presence and Anxiety in Virtual Reality Exposure Therapy for Anxiety Disorders
Article
Full-text available
In virtual reality exposure therapy (VRET) for anxiety disorders, sense of presence in the virtual environment is considered the principal mechanism that enables anxiety to be felt. Existing studies on the relation between sense of presence and level of anxiety, however, have yielded mixed results on the correlation between the two. In this meta-analysis, we reviewed publications on VRET for anxiety that included self-reported presence and anxiety. The comprehensive search of the literature identified 33 publications with a total of 1196 participants. The correlation between self-reported sense of presence and anxiety was extracted and meta-analyzed. Potential moderators such as technology characteristics, sample characteristics including age, gender and clinical status, disorder characteristics and study design characteristics such as measurements were also examined. The random effects analysis showed a medium effect size for the correlation between sense of presence and anxiety (r = .28; 95% CI: 0.18-0.38). Moderation analyses revealed that the effect size of the correlation differed across different anxiety disorders, with a large effect size for fear of animals (r = .50; 95% CI: 0.30-0.66) and a no to small effect size for social anxiety disorder (r = .001; 95% CI: -0.19-0.19). Further, the correlation between anxiety and presence was stronger in studies with participants who met criteria for an anxiety disorder than in studies with a non-clinical population. Trackers with six degrees of freedom and displays with a larger field of view resulted in higher effect sizes, compared to trackers with three degrees of freedom and displays with a smaller field of view. In addition, no difference in effect size was found for the type of presence measurement and the type of anxiety measurement. This meta-analysis confirms the positive relation between sense of presence and anxiety and demonstrates that this relation can be affected by various moderating factors.
View
The Effect of Perspective on Presence and Space Perception
Article
Full-text available
In this paper we report two experiments in which the effect of perspective projection on presence and space perception was investigated. In Experiment 1, participants were asked to score a presence questionnaire when looking at a virtual classroom. We manipulated the vantage point, the viewing mode (binocular versus monocular viewing), the display device/screen size (projector versus TV) and the center of projection. At the end of each session of Experiment 1, participants were asked to set their preferred center of projection such that the image seemed most natural to them. In Experiment 2, participants were asked to draw a floor plan of the virtual classroom. The results show that field of view, viewing mode, the center of projection and display all significantly affect presence and the perceived layout of the virtual environment. We found a significant linear relationship between presence and perceived layout of the virtual classroom, and between the preferred center of projection and perceived layout. The results indicate that the way in which virtual worlds are presented is critical for the level of experienced presence. The results also suggest that people ignore veridicality and they experience a higher level of presence while viewing elongated virtual environments compared to viewing the original intended shape.
View
The relationship between individual characteristics and experienced presence
Article
Full-text available
People experience different levels of presence (e.g. and ), and different levels of cybersickness (Sharples, Cobb, Moody, & Wilson, 2008) even though they are immersed in the same virtual environment setting. In the current study, we raised the question how differences in individual characteristics might relate to differences in sensed presence for a virtual environment related to public speaking. The individual characteristics included in the experiment were related to visual abilities, personality traits, cognitive styles, and demographic factors. We recruited 88 participants, who were first immersed in a non-stereoscopic neutral environment, and then in a public speaking world: once with stereoscopic rendering and once without stereoscopic rendering, in a counter-balanced order. The results showed that immersive tendency and monocular visual ability were significantly correlated with presence and these correlations were consistent among the three virtual environments. Immersive tendencies and its subscale “involvement” were also found to be significantly correlated with cybersickness in all three worlds. Screening people on these variables may help to recognize the users who are more likely to benefit from virtual reality applications and may help to reduce the number of dropouts during virtual reality exposure therapy.
View
Physiological measures and self-report to evaluate neutral virtual reality worlds
Article
Full-text available
  • Jan 2011
Using virtual reality technology for exposure therapy to treat patients with anxiety disorders is attracting considerable research attention. The ability to monitor patient anxiety level helps therapists to set appropriate anxiety arousing situations. Physiological measures have been put forward as objective indicators of anxiety levels. Because of individual variation, they need a baseline recording which is often conducted in neutral virtual worlds which do not include phobic stressors. Still because of the novelty of the virtual worlds, reports in the literature suggest that individuals already show some level of arousal when placed in these worlds. This paper presents two studies which look at the effect two different neutral virtual worlds can have on individuals. Findings suggest that a neutral world does not have to result in an increased level of arousal.
View
Effects of Stereoscopic Viewing on Presence, Anxiety, and Cybersickness in a Virtual Reality Environment for Public Speaking
Article
Full-text available
In this study, we addressed the effect of stereoscopy on presence, anxiety, and cybersickness in a virtual public speaking world, and investigated the relationships between these three variables. Our results question the practical relevance of applying stereoscopy in head-mounted displays (HMDs) for virtual reality exposure therapy (VRET) in a virtual public speaking world. In VRET, feelings of presence improve the efficacy (B. K. Wiederhold & M. D. Wiederhold, 2005). There are reports of a relatively large group of dropouts during VRET at low levels of presence in the virtual environment (Krijn, Emmelkamp, Olafsson, & Biemond, 2004). Therefore, generating an adequate level of presence is essential for the success of VRET. In this study, 86 participants were recruited and they were immersed in the virtual public speaking world twice: once with stereoscopic rendering and once without stereoscopic rendering. The results showed that spatial presence was significantly improved by adding stereoscopy, but no difference for reported involvement or realism was found. The heart rate measurements also showed no difference between stereoscopic and nonstereoscopic viewing. Participants reported similar anxiety feelings about their talk and similar level of cybersickness in both viewing modes. Even though spatial presence was significantly improved, the size of statistical effect was relatively small. Our results therefore suggest that stereoscopic rendering may not be of practical importance for VRET in public speaking settings.
View
The Experience of Presence: Factor Analytic Insights
Article
Full-text available
  • Jun 2001
Within an embodied cognition framework, it is argued that presence in a virtual environment (VE) develops from the construction of a spatial-functional mental model of the VE. Two cognitive processes lead to this model: the representation of bodily actions as possible actions in the VE, and the suppression of incompatible sensory input. It is hypothesized that the conscious sense of presence reflects these two components as spatial presence and involvement. This prediction was confirmed in two studies (N = 246 and N = 296) assessing self-reports of presence and immersion experiences. Additionally, judgments of “realness” were observed as a third presence component. A second-order factor analysis showed a distinction between presence, immersion, and interaction factors. Building on these results, a thirteen-item presence scale consisting of three independent components was developed and verified using confirmatory factor analyses across the two studies.
View
Psychotherapy By Reciprocal Inhibition
Article
  • Oct 1968
Reciprocal inhibition is a process of relearning whereby in the presence of a stimulus a non-anxiety-producing response is continually repeated until it extinguishes the old, undesirable response. A variety of the techniques based on reciprocal inhibition, such as systematic desensitization, avoidance conditioning, and the use of assertion, are described in detail. Behavior therapy techniques evaluated on the basis of their clinical efficacy are found to have striking success over traditional psychoanalytic methods. Currently, more comparative studies are required which will validate the merit of behavior therapy in the psychotherapeutic field while experimental research should continue to refine the techniques.
View
3D audio for virtual reality exposure therapy, MSc, Delft university of technology
  • Jan 2013
  • A R D Hoekstra
A.R.D. Hoekstra, 3D audio for virtual reality exposure therapy, MSc, Delft university of technology, 2013.
  • Jan 2012
  • 254-267
  • Y Ling
  • W P Brinkman
  • H T Nefs
  • C Qu
  • I Heynderickx
Y. Ling, W.P. Brinkman, H.T. Nefs, C. Qu, and I. Heynderickx, Effects of Stereoscopic Viewing on Presence, Anxiety, and Cybersickness in a Virtual Reality Environment for Public Speaking, Presence- Teleoperators and Virtual Environments 21 (2012), 254-267.