Conference PaperPDF Available

Effects of Manipulating Physiological Feedback in Immersive Virtual Environments

Authors:

Abstract and Figures

Virtual environments have been proven to be effective in evoking emotions. Earlier research has found that physiological data is a valid measurement of the emotional state of the user. Being able to see one's physiological feedback in a virtual environment has proven to make the application more enjoyable. In this paper, we have investigated the effects of manipulating heart rate feedback provided to the participants in a single user immersive virtual environment. Our results show that providing slightly faster or slower real-time heart rate feedback can alter participants' emotions more than providing unmodified feedback. However, altering the feedback does not alter real physiological signals.
Content may be subject to copyright.
Effects of Manipulating Physiological Feedback in
Immersive Virtual Environments
Arindam Dey
University of South Australia
Adelaide, Australia
arindam.dey@unisa.edu.au
Hao Chen
University of Canterbury
Christchurch, New Zealand
hao.chen@pg.canterbury.ac.nz
Mark Billinghurst
University of South Australia
Adelaide, Australia
mark.billinghurst@unisa.edu.au
Robert W. Lindeman
University of Canterbury
Christchurch, New Zealand
gogo@hitlabnz.org
ABSTRACT
Virtual environments have been proven to be effective in evok-
ing emotions. Earlier research has found that physiological
data is a valid measurement of the emotional state of the user.
Being able to see one’s physiological feedback in a virtual en-
vironment has proven to make the application more enjoyable.
In this paper, we have investigated the effects of manipulating
heart rate feedback provided to the participants in a single user
immersive virtual environment. Our results show that provid-
ing slightly faster or slower real-time heart rate feedback can
alter participants’ emotions more than providing unmodified
feedback. However, altering the feedback does not alter real
physiological signals.
Author Keywords
Immersive Virtual Reality, Physiological Signals,
Multi-Sensory Feedback, Emotion
INTRODUCTION
In this paper, we explore how manipulating physiological feed-
back in an immersive virtual environment (VE) can influence
user’s emotions in VE. Virtual reality (VR) can create immer-
sive VEs that are capable of evoking emotional responses. VR
is now widely used in different application domains such as
gaming, movies, education, and therapy. VR has the ability to
immerse users in environments that are within or beyond the
constraints of their current physical world. Creating empathy
in VR, where one person can understand the feelings of an-
other person, has been argued to be one of the most promising
applications by Thomas [30]. Earlier work found that VEs
are capable of creating targeted emotions in users, such as
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia.
Copyright © 2018 Association of Computing Machinery.
ACM ISBN 978-1-4503-5624-4/18/10 ...$15.00.
http://dx.doi.org/10.1145/3242671.3242676
happiness [10] and scariness [22, 24]. VR was also found to
be able to create empathy with virtual characters [17].
Emotion is defined by Cabanac [8] as “any mental expe-
rience with high intensity and high hedonic content (plea-
sure/displeasure).” However, there is no consensus on the
definition of emotions. While earlier research has focused
on creating and measuring emotions in VR, little work has
been done to investigate the effects of providing physiological
feedback to users, and measuring its effects on emotions and
interaction. Dey et al. [12], in a collaborative VR setup, re-
ported that showing real time heart rate to collaborators could
create higher subjective connections. In a single user setup,
Chen et al. [9] investigated the most effective multi-sensory
way of providing heart rate feedback to the users. In neu-
roscience research, it was reported that manipulating heart
rate variability can create different response to anger-inducing
stimuli [16]. Another study reported that providing accurate
heart rate biofeedback is an efficient way to control autonomic
physiological reactions when people are exposed to negative
stimuli [25]. In VR, Ueoka et al. [31] created a system to
provide pseudo heart rate feedback to increase the scariness
in horror VR experiences. However, they did not thoroughly
measure the effect of such feedback.
In this paper, we present one of the first works that system-
atically investigates the emotional and physiological effects
of providing manipulated heart rate feedback to users in VEs.
Our study focuses on providing users with their own heart
rate feedback—in decreased, increased, and non-manipulated
ways—in real time through audio-haptic channels. We mea-
sure which emotions can be altered, and how real physiological
signals get affected by this modulation. Subjective measure-
ments have been the primary mode of investigating emotions
in VR. In our work, we use both subjective and physiolog-
ical measurements. The main motivation of this work is to
examine how additional physiological feedback, on top of
traditional audio-visual effects in VR, can affect the overall
experience, and whether or not some specific emotions can be
enhanced or reduced by manipulating the feedback. This is
important because nowadays VR applications are commonly
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
101
used to treat various phobias [34, 23] and disorders [28]. If
we can establish the effects of heart rate manipulation found
by [25] and [16] in VR applications as well, then treating
conditions related to negative emotions using VR can become
more effective than now. Similarly, in the case of entertaining
and gaming VR applications we could induce higher level of
emotions by manipulating the heart rate feedback.
To investigate these effects, we designed five VEs of similar
quality and experience. We collected users’ heart rate data
and provided the feedback of their own heart rate to them in
real time using audio and haptic sensations. However, we
manipulated the feedback in five ways. We decreased the
frequency of heart beats by 30% and 15%, increased by 30%
and 15%, and the control condition we did not manipulate
and provided the feedback as is. In a within-subjects user
study, participants relayed their experiences through two state-
based emotion measuring subjective instruments. We also
collected heart rate and galvanic skin response data during the
experience.
Novelty and Contribution:
The key novelty of the work is
that, for the first time in VR research, the effect of manipu-
lated physiological feedback on emotion is measured using a
combination of physiological and subjective measurements.
The primary contribution of the work is that we have found,
for the first time, that manipulated heart rate feedback can
significantly affect five emotions in virtual experiences. The
results can be used in future VR applications where amplifying
a particular emotion is important, such as cognitive training
applications, VR movies, or games.
The rest of the paper is organized as follows. In the Related
Work section, we discuss some of the key earlier work that
has been done in this domain. In the following section (User
Evaluation), we provide details of the VEs used in the study,
the overall system, and the study design, including the experi-
mental procedure. Next we report the results, followed by a
discussion of them in detail. Finally, we conclude by directing
toward future research.
RELATED WORK
In our work, we are investigating the effects of heart rate
feedback manipulation on emotions in VR. Emotions in VR
have been researched over the past few years, and many re-
searchers have proposed that emotions similar to real world
can be elicited in VEs as well [3, 14, 15, 26]. Researchers have
investigated the possibility of enhancing emotional feelings
and/or empathy by adding some physiological signal cues in
both single-user [9, 32, 31] and collaborative tasks [11, 12,
29].
Felnhofer et al. reported that VEs accurately triggered emo-
tions in five different kinds of virtual park scenarios, as mea-
sured by emotional arousal and presence using GSR [15]. Riva
et al. argued that the degree of presence experienced in a VE
has a strong influence on the experienced emotional states
[26]. Emotional arousal has been measured and used in dif-
ferent military applications. For example, Rizzo et al. created
emotional states for military training in a post-war scenario
[27] and Roy et al. used emotion elicitation in VR to treat
post traumatic stress disorder (PTSD) [28]. In other work,
Wiederhold et al. [34] reported treating people with flying
phobia using VR and measured physiological measures such
as heart rate and skin conductance.
There are other medical applications where VR has been used
to treat a condition where emotion was used as a key mea-
surement. Gomez et al. [18] reported using VR to facilitate
dialectical behavioral therapy and found over time positive
emotions increased and negative emotions decreased. A sim-
ilar effect was noticed in work by Banos et al. [4], where
elderly participants were found to have an increase in posi-
tive emotions and a decrease in negative emotions after being
exposed to specially designed VEs. Researchers using VR-
based interventions for fibromyalgia [19] and autism spectrum
disorder (ASD) [21, 20] measured emotions using subjec-
tive instruments. Overall, this group of work establishes that
measuring emotion in VR shows good promise as a research
method for evoking appropriate emotions.
There is another, yet small, group of work where besides just
measuring emotions and/or physiological data, researchers
have investigated ways to visualize and communicate that data.
This is the topic in which our current research fits. Visualizing
physiological data in a collaborative video conferencing was
invested by Tan et al. [29], but such reported work in VR is
rare.
In a recent work, Dey et al. [12] researched the effect of
sharing the real-time audio-visual heart rate cues of one player
to another in two different (scary and calm) collaborative
VR gaming environments. They reported a trend that being
able to see the collaborator’s heart rate in real time created
higher positive affect than not seeing it, and that this made
collaborators connect better with each other. Bernal and Meas
[5] recently reported work where they visualized emotions
using avatars in VEs. After measuring GSR and heart rate
data, they represented emotions visually in two different ways:
(1) growing the fur on the skin of an avatar when arousal is
high, and (2) intensifying the brightness or colour change to
highlight the avatar when arousal is high. In other work, Chen
et al. [9] investigated how to provide physiological feedback
to users in a VE using multi-sensory channels, and whether
showing the real-time physiological cues in VEs can make
the user more aware of his/her own emotional state. They
reported that participants preferred having sensory feedback,
and among combinations of audio, visual, and haptic feedback,
participants preferred audio-haptic feedback the most, while
visual feedback was found to be distracting. Our current work
is influenced by the findings of Chen et al., however, instead
of just providing feedback, we are investigating the effects of
manipulated heart rate feedback.
We are not aware of any other work that has investigated the ef-
fects of manipulated physiological feedback in immersive VR.
However, there is some work that manipulated physiological
feedback. For example, Ueoka et al. [32] attempted to amplify
the horror experience of watching 3D movies by providing
pseudo heart rate, and did another study [31] that did similar
work using a horror VR experience, trying to amplifying the
scariness of the experience further. They provided the heart
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
102
rate feedback through a vibrating floor. However, in their
work, they only used a horror VR experience, whereas we
use VEs with different emotional segments—happy, anxious,
disgusting, fearful, and sad.
Summary:
Earlier research has shown that VR can create
emotions akin to the real world. Providing physiological feed-
back can enhance the experience and increase positive affect
in collaborative setups. There are different ways of visualiz-
ing physiological and emotional data in VR. Multi-sensory
audio-haptic feedback has been shown to be an effective way
of communicating physiological states in real time. Some
researchers have investigated ways to manipulate heart rate
feedback, but the effect of such a manipulation has not been
investigated in detail, particularly in immersive VR with mul-
tiple emotional VEs. We address these issues and investigate
such effects using both subjective instruments and physiologi-
cal measurements.
USER EVALUATION
The main goal of the user evaluation was to investigate the
emotional and physiological effects of providing manipulated
(increased, decreased, and accurate) real time heart rate feed-
back to users in an immersive VE.
Experimental Virtual Environment
The VR experiences were based on a jungle safari with various
animals (including dinosaurs) moving through in the environ-
ment and supplemented with appropriate sound effects (Figure
1). Each participant was a tourist on a virtual safari, placed in
a standing position on the back of a virtual car moving along
without any interaction from the player. We used this scenario
for two reasons: (a) to provide participants with the great-
est opportunity to explore the environment without worrying
about controlling their movement, similar to most real-world
jungle safaris, and (b) we wanted to control the path of the car
for consistency in our experiment, as we placed different emo-
tional triggers in the VE at particular locations. The emotional
triggers were carefully placed in a manner that the participants
could not avoid receiving them. Most of the visual effects of
interest were presented in front of the participant’s eyes within
a 200
°
horizontal field of view. However, there were sound
effects that originated behind the participant in the VE. We did
not provide any avatar for the participants for self-awareness.
Using the Unity 3D game engine [2], we created five similar
VEs for this experiment, each lasting for four minutes. In each
of the environments, there was a mixture of five different kinds
of experiences—happy, anxious, scary, disgusting, and sad.
We ran expert reviews and pilot studies to make sure that the
emotion experience triggers are appropriate for their intended
purposes. For example, to trigger happiness, we showed a
waterfall and many butterflies flying around, while for fear, we
displayed roaring panthers, dinosaurs, and snakes attacking the
car and to trigger sadness, we designed a deer hit the pickup
accidentally and died. However, in our measurements we
took a holistic approach and did not distinguish between the
experiences as it is difficult to ensure that participants would
feel only one emotion at any point in time.
(a) Happy
(b) Anxious
(c) Scary
(d) Disgusting
(e) Sad
Figure 1. Representative images from the experimental virtual environ-
ments.
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
103
Figure 2. The system overview.
In the physical world, the subject was standing with a hand-
rest in front to maintain balance, if needed. The subject was
allowed to look around and rotate his/her head to experience
the VE at will. However, s/he was not allowed to walk as
we wanted to avoid any elevated physiological signals due to
locomotion.
Experimental System and Setup
The scenes were experienced by the participants through an
HTC Vive head-mounted display (HMD) and the audio ef-
fects were provided through noise-cancelling Logitech head-
phones. We also provided heart rate feedback to the partici-
pants while in the virtual experience. Chen et al. found that
audio-haptic multi-sensory feedback of heart rate was most
preferred by users [9], so we provided the heart rate feedback
in the same way. While the sounds of the heart beats were
provided through the headphones, the haptic feedback was pro-
vided through both of the Vive controllers, which participants
were holding in their hands.
To capture the physiological signals including heart rate (HR)
and skin conductance (GSR), we used the ProComp Infiniti 8
Channel Encoder by Thought Technology [1]. The heart-rate
sensor we used was a Polar H7. Through the EKG receiver, the
signal from Polar H7 could be processed through a Procomp
Infiniti Encoder. The GSR sensors were attached to two fingers
of participants, and the Polar H7 was strapped around the
chest. In order to stream the HR in real time, we used the
Generic Attribute (GATT) profile to get the HR data from
Polar H7 via BLE. Figure 2 shows the system we designed
for our user study. In the figure, the laptop labeled as No.1,
running Ubuntu, received the HR data directly from Polar H7
via Bluetooth. Then, the laptop streamed the data to the No.2
computer in real time, where the HR data was visualized in
Unity using audio and haptic cues. At the same time, the GSR
and the HR data were recorded in the No.2 computer using
BioGraph Infiniti software of Thought Technology.The user
was asked to stand and hold the HTC Vive controllers in both
of her hands.
Independent Variable
We ran a very focused within-subjects study. The only in-
dependent variable in this experiment was heart rate manip-
ulation. We experimented with five different levels of the
manipulation—
-30%, -15%, Real (0%), +15%, and +30%
.
In the -30% and -15% conditions, we provided the real time
heart rate feedback after reducing the rate by 30% and 15%,
respectively. Similarly, for +30% and +15%, the heart rate
feedback was 30% and 15% higher than real heart rates. In
the Real condition, the heart rate feedback was given without
any manipulation. These levels of manipulations were chosen
after a short pilot study with five participants where varied
the range between
±50%
with
10%
intervals. We noticed
beyond
±30%
the manipulation becomes too obvious to the
participants.
In the study, each participant experienced the above five levels
using five different VE scenes to avoid any learning effects.
The order of the presentation of manipulated feedback was
counterbalanced using a balanced Latin square.
Dependent Variables
Our main aim was to identify the effect of heart rate manipula-
tion on emotions. As such, we used two validated subjective
emotion-measuring surveys, the Positive And Negative Affect
Schedule (PANAS) [33] and the Self-Assessment Manikin
(SAM) questionnaire [7]. Additionally, as the end of the ex-
periment we semi-formally interviewed participants.
The PANAS scale was used as it provides an overall state-
based positive and negative affect scores, measured through
the scores for 20 individual feelings and emotions. SAM
provides scores for overall arousal, valence, and dominance.
Besides the subjective measurements, we also collected HR
and GSR data during the virtual experience sessions.
Experimental Procedure
After welcoming the participant, we explained the task and
asked them to fill out the consent form and demographic ques-
tionnaires. We clearly informed the participant that the heart
rate feedback they would be getting would be their own real
time heart rate. However, we did not disclose that their heart
rate feedback would be manipulated between the sessions. Af-
ter attaching the sensors to the body, we asked them to wait
for three minutes to bring their heart rate to a normal level,
following which we collected baseline HR and GSR data for
two minutes. During this time, we asked the participant to
stand as the experimental task also required them to stand.
After collecting the baseline data, the participant waited, sit-
ting for a at least two minutes, before starting their first ex-
perimental session, after which they answered the PANAS
and SAM questionnaires, and waited another minute before
starting the next session. This process was repeated five times.
After all experimental sessions were done, we interviewed the
participants. After that, we debriefed them about the study
and mentioned that their heart rate feedback was manipulated
in four of the five sessions withing the range of
±30%
. We
also explained then that this manipulation was not disclosed to
them at the beginning to avoid this knowledge affecting their
performance and consequently the results. We provided them
with a chocolate gift before sending them off. On average the
experiment took about an hour per participant.
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
104
(a) Interested (b) Excited (c) Scared
(d) Nervous (e) Afraid
Figure 3. In the PANAS questionnaire, we noticed significant effects of heart rate manipulation in (a) interested, (b) excited, (c) scared, (d) nervous, and
(e) afraid. Whiskers represent ±95% confidence interval.
Participants
For this experiment, we recruited 20 participants from uni-
versity students and staff or from personal contacts. One
participant had to drop out after two sessions due to personal
emergency, which resulted in the study being finalized with 19
participants (two female) with ages ranging between 21 and
45 years (m=30.6, sd=7.1).
Almost all participants had prior experience playing video
games, and except for two they had moderate to high experi-
ence with VR. Only two participants reported that they paid
attention to their heart rate in daily life. Eighteen partici-
pants thought their heart rate increased when they felt stressed,
afraid, or anxious, and 16 participants thought their heart rate
decreased when they felt relaxed, bored, or sleepy.
We used G*Power [13] to calculate the sample size for re-
peated measure ANOVAs with a large effect size of (
f=0.4
)
and
α=0.05
, and found we required 13 participants for our
study, meaning we had more than the required number to get
enough power for our statistical tests.
Hypotheses
At the outset, we had the following hypotheses:
H1:
-30%
and
-15%
feedback will cause less interest in the
virtual experience than real, +15%, and +30% feedback.
H2:
+30%
and
+15%
feedback will cause more excite-
ment in the virtual experience than
real, -15%, and -30%
feedback.
H3:
+30%
and
+15%
heart rate feedback will cause more
anxiety, fear, and nervousness than
real, -15%, and -30%
feedback.
H4: There will be a significant difference in HR and GSR
between different conditions.
RESULTS
To analyze the non-parametric subjective data, we used Fried-
man’s ANOVA. For the objective data (HR and GSR), we used
repeated measure ANOVAs. Table 1 gives an overview of
responses from the PANAS and SAM questionnaires. Over-
all, we found that slightly higher HR feedback (
+15%
) than
real
caused more excitement, scariness, nervousness, and
fear, while a slightly lower heart rate (
-15%
) made partici-
pants more interested in the VE. However, the effects were
mainly noticed in subjective responses, and the physiological
responses remained largely unaffected by the different HR
feedback.
Analysis of PANAS and Individual Emotions
We noticed that in all conditions, positive affect was stronger
than negative affect. However, there was no significant differ-
ence between the conditions on either positive affect (p=0.17)
or negative affect (p=0.18). We noticed among the 20 different
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
105
Table 1. Mean and standard deviation values of PANAS and SAM questionnaires. PANAS scores ranged between Very Slightly or Not at All (1) and
Extremely (5).
Table 2. Mean and standard deviation of additional subjective questions.
emotions and feelings measured using PANAS, five individ-
ual emotions were significantly influenced by the heart rate
manipulation—interest, excitement, scariness, nervousness,
and fear. Interestingly, these five emotions are clearly most
relevant for the VEs we designed.
Interest:
With a Friedman test, we found a significant effect
of heart rate manipulation on participant interest while being
in the VE—
χ2(4) = 10.5
, p=0.03. Using post-hoc Wilcoxon
signed rank test we found that
real
heart rate feedback was
significantly less interesting than -30% (Z=-2.1, p=0.03) and
-15%
(Z=-2.5, p=0.01) feedback. It was also almost signif-
icantly less than
+15%
(Z=-1.7, p=0.08). We also found a
strong trend for
+30%
being less interesting than
-15%
feed-
back (p=0.06), which participants reported to be the condition
that made them most interested while experiencing the VE
(Figure 3(a)).
Excitement:
We found a significant effect of heart rate ma-
nipulation on the feeling of excitement while in the VE—
χ2(4) = 15
, p=0.005. With a post-hoc test we found that,
similar to interest,
real
feedback made participants feel signifi-
cantly less excited than
-30%
(Z=-2.1, p=0.03),
-15%
(Z=-2.6,
p=0.008), and
+15%
(Z=-2.5, p=0.01) feedback. We found
that
+30%
feedback made participant almost significantly less
excitement than
+15%
feedback, which was the feedback that
made participants most excited while in VE (Figure 3(b)).
Scariness:
There was a significant effect of heart rate manip-
ulation on evoking scariness in VE—
χ2(4) = 12.9
, p=0.012
(Figure 3(c)). A post-hoc test revealed that
+15%
feedback
was able to create significantly more scariness in participants
than
+30%
(Z=-2.48, p=0.013) and
-30%
(Z=-2.53, p=0.01)
feedback. There was a trend for
Real
feedback making partici-
pants feel less scariness than
+15%
feedback (Z=-1.8, p=0.07).
Real
created significantly less scariness than
-15%
feedback
(Z=-2.3, p=0.02).
Nervousness:
We noticed that heart rate manipulation had a
significant effect on the feeling of nervousness—
χ2(4) = 10.6
,
p=0.03. A post-hoc test showed that
+15%
feedback, which
caused most nervousness (Figure 3(d)), made participants sig-
nificantly more nervous than
real
(Z=-2.38, p=0.018) and
-30%
(Z=-2.3, p=0.02) feedback.
Real
feedback also made
participants less nervous than
-15%
(Z=-2.8, p=0.005) feed-
back and there was a trend for it making them less nervous
than +30% (Z=1.7, p=0.09) feedback.
Fear:
We found a significant effect for heart rate manipula-
tion on feeling afraid—
χ2(4) = 10
, p=0.04 (Figure 3(e)). A
Wilcoxon signed rank post-hoc test found that
+15%
caused
significantly more fear than
real
(Z=-1.9, p
<
.05) and
-30%
(Z=-2.5, p=0.01). Interestingly, there was a trend for
+30%
causing less fear than +15% feedback (Z=-1.8, p=0.07).
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
106
Table 3. Mean and standard deviation of physiological data. Heart Rate variability indicates Root Mean Square of the Successive Differences (RMSSD).
Differences indicate differences from baseline.
Self Assessment Manikin (SAM)
The SAM questionnaire pictorially measures valance, arousal,
and dominance. We did not find any significant difference in
any of the measures of the SAM questionnaire as shown in
Table 1.
Subjective Questions
We asked participants to respond to five additional questions
on five-point Likert scales after each session: (1) How much
did you pay attention to your heart rate when in the VE? (2)
How much did you feel your heart rate when in the VE? (3)
How much do you agree that your heart rate was represented
accurately in the VE? (4) How much did the heart rate visu-
alization add to the experience when in the VE? and (5) How
much do you agree that the heart rate visualization distracted
your experience when in the VE?
Overall, we did not notice any significant effect of the heart
rate manipulation on any of the questions (Table 2). Generally,
participants somewhat agreed that they did pay moderate at-
tention to their heart rate. They felt their heart rate moderately
and thought that their heart rate was accurately represented.
They agreed that heart rate feedback slightly enhanced the
experienced and did not distract too much.
Interviews
After all the sessions we asked a few questions of the partic-
ipants. They reported that the VEs were indeed interesting
and made them feel differently in different segments. P5 men-
tioned that ". . . the night scene where dinosaurs attacked the
car was very scary." P12 reported that he/she would have more
feeling while in the VEs if the graphical fidelity was higher.
This participant reported a lot of experience playing VR games
using the HTC Vive.
Only one participant (P15) reported feeling a bit dizzy, how-
ever he/she was recovering from a flu condition. Most interest-
ing insights came when we asked participants about whether
or not they felt their heart rate was accurately represented in
the sessions. P15 also made an interesting comment that “. ..if
I was in control of the car then I might have felt higher level of
emotions.” All participants, except for two, thought their heart
rate was a bit faster than what he/she was expecting in one
condition (
+30%
), otherwise the heart rate was accurate. The
other two participants thought their heart rate was shown ac-
curately in all conditions. Two participants explicitly reported
that they thought their real heart rate went up in the
+30%
condition. In other words, from the interview it was clear that
elevating heart rate feedback to
+30%
was noticeable, but
other conditions (including -30%) went unnoticed.
Participants reported that they paid attention to the heart rate
feedback more when at the beginning of the experiences and
when nothing much was happening in the VE (e.g., in happy
segments). As the experiences progressed and more things
stared happening, the feedback became part of the experience
and they did not notice it anymore.
Analysis of Physiological Data
Besides qualitative responses, we were also interested to in-
vestigate whether heart rate manipulation could affect the real
physical HR and GSR of participants.
Heart Rate:
In the case of heart rate, we analyzed raw heart
rate, heart rate change from the baseline, and heart rate vari-
ability. For heart rate variability, we measured root mean
square of the successive differences (RMSSD). However, we
did not notice any significant difference in any of the heart rate
measures. This indicates that increasing or decreasing heart
rate feedback does not alter real heart rate (Table 3).
Galvanic Skin Response:
We measured raw GSR and differ-
ence in GSR between baseline and other heart rate manipu-
lation conditions. In the case of GSR, we noticed a signifi-
cant difference between baseline and all other VR exposure
conditions—F(2.12, 37.9) = 8.82, p = .001,
η2
p
=.33, observed
power=.97 (Table 3). As the data did not meet the assump-
tion of sphericity we used Greenhouse-Geisser adjustments.
GSR in the baseline was significantly lower than all other
conditions. It is clear that being exposed to VR increased the
arousal in participants [6]. However, we did not notice any
difference between the heart rate manipulation conditions. Al-
though, a Wilcoxon Signed-rank test indicated that there was
trend of
real
heart rate feedback being almost significantly
less arousing than
+30%
condition—Z=-1.69, p=.09 (Figure
4).
DISCUSSION
In this experiment, we found that manipulating the heart rate
feedback had a significant effect on emotions, but not on
physiological signals. As expected, we noticed that positive
affect was significantly stronger than negative affect in our
VEs. However, there was no difference between the conditions,
though
real
(not manipulated) heart rate feedback had the
lowest positive affect and lowest negative affect.
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
107
Figure 4. Mean GSR data shows a significant increase in arousal in VR
conditions than in Baseline. Whiskers represent ±95% confidence inter-
val.
Besides overall positive and negative affect, we were also
interested in five key emotions that were relevant for our VEs—
interest, excitement, scariness, nervousness, and fear. Inter-
estingly, in all five emotions, there were significant effects of
heart rate manipulation.
In our first hypothesis (H1), we expected that providing feed-
back that represented a slower heart rate than what it was in
reality (
-30%
and
-15%
feedback) would decrease interest in
the VE in comparison with all other feedback. Contrary to our
expectations, we found that
-15%
heart rate feedback made
participants significantly more interested in the VE, and
real
heart rate feedback made participants the least interested. A
possible reason of this effect is that finding one’s rate to be
slower than what it actually was made him/her feel more in
control and provided courage to explore more, hence increas-
ing interest. Although our hypothesis was not accepted, this is
an interesting finding as future VR applications can provide
real time heart rate feedback that indicates lower heart rate to
participants in order to make them more interested while in
the experience.
Our second hypothesis (H2) predicted that faster heart rate
feedback (
+30%
and
+15%
) would increase excitement while
being in the VE than all other conditions. This hypothesis was
partially accepted, as we found that
+15%
feedback made par-
ticipants most excited of all feedback. However, the difference
was only significant compared to the
real
feedback. Interest-
ingly,
real
heart rate feedback made participants significantly
the least excited of all conditions, except for
+30%
. When
the feedback was increased to the
+30%
level, participants
became less excited. This means that faster-than-real heart
rate feedback can increase excitement, however, when it goes
beyond a threshold (in our case +15%) the effect reverses. We
believe that when heart rate gets too fast, it can become appar-
ent to the participant that they are being manipulated, and the
excitement can fade away. In our interviews with participants,
we found that almost all participants noticed when their heart
rate feedback was provided at the +30% level, but they did not
notice anything unusual when it was at the -30% level. We
think that people expect their heart rate to be lower, and do not
pay attention when they find it that way. Another explanation
of this effect could be, that with a faster heart rate, the sound
frequency of the beats and vibration of the controllers also
increases, which may have made participants disengage with
the experience.
Our third hypothesis (H3) predicted that scariness, nervous-
ness, and fear would increase when faster heart rate feedback
is provided in comparison to real and slower heart rate feed-
back. For all three emotions, we found that
+15%
heart rate
feedback caused the highest effects, although the effect was
not always significant, which indicates that our hypothesis
was partially accepted.
Real
and
-30%
feedback had the
smallest effects in making participants feel scared, nervous, or
afraid. This effect is expected, as often in movies and other
audio-visual media, faster heart rate is associated with negative
emotions. This result is consistent with findings of Perira et al.
[25], where they found that accurate (
real
) heart rate feedback
can help control negative emotions. It also establishes that
manipulating and providing physiological feedback in VR is
consistent with the effects found in physical world, which is
very encouraging for future VR applications particularly those
deal with negative emotions. However, the fact that
+30%
had similar effects as
real
feedback shows that audio-haptic
feedback should not cross a certain upper threshold to keep
the users engaged in the experience.
Our final hypothesis (H4) predicted that there would be a sig-
nificant difference in physiological signals between the heart
rate manipulation levels. However, we did not find any sup-
port for this hypothesis, as there was no noticeable difference
in either heart rate or GSR measures. However, we noticed
higher arousal while being in VR than in the baseline condi-
tion. Although, this hypothesis was not accepted, this is an
encouraging finding, as significantly altering physiological
signals may cause detrimental effects for various medical rea-
sons. The fact that by manipulating heart rate feedback we
can alter emotions, but not physiological signals, makes this
type of feedback safe for future VR applications.
Overall, our results indicate that
-15%, -30%
and
+15%
heart
rate feedback can increase interest and excitement. For increas-
ing negative emotions such as scariness, nervousness, and fear,
+15%
feedback should be used.
Real
feedback creates the
least effects on any of these emotions, whereas
+30%
feed-
back causes detrimental effects and should be avoided. In VR,
emotions can be altered without altering physiological signals,
which are controlled by the autonomic nervous system.
CONCLUSIONS AND FUTURE WORK
In this paper, we have presented the first study in VR where
the effects of manipulated multi-sensory heart rate feedback
on user emotion are investigated. We found that interest, ex-
citement, scariness, nervousness, and fear can be enhanced
by providing manipulated heart rate feedback, although phys-
iological signals remain unaffected. This is an interesting
finding for VR researchers and virtual experience designers,
as using these results they can modulate a user’s emotion in a
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
108
more controlled way than now by simply providing and ma-
nipulating heart rate feedback. We believe our work will have
an impact in creating more empathetic and emotionally aware
VR applications in the future.
VR Experience Design Guidelines
We have received several valuable insights from this study,
which could be useful for future VR application design. We
would like to follow these guidelines in our future research.
Provide ±15% heart rate feedback:
We noticed that ma-
nipulating heart rate can increase interest, excitement, scari-
ness, nervousness, and fear. The effects were most when
either +15% or -15% heart rate feedback was provided.
We would recommend provided slightly increased or de-
creased heart rate feedback when the application requires an
increase in emotions, for example VR games and movies.
Provide real heart rate feedback:
We also noticed that
real
heart rate feedback caused the least emotion among the
manipulation levels with which we experimented, which
is supported by other research in neuroscience that real
heart rate feedback can help control negative emotions [25].
Therefore, we would recommend providing real heart rate
feedback in VR applications where controlling emotions
are required, such as in therapeutic applications.
Let the user drive and/or interact:
Participants in our
study expressed interest to be in control of the car or having
opportunities to interact with the virtual animals such as pat-
ting or pushing, but we did not add these interactions for the
sake of controlling the experimental conditions. However,
in non-experimental experiences, we would recommend
adding more interaction than just looking around.
Limitations
Although we are reporting on the first study investigating these
effects and found interesting results, our experiment had some
limitations. First, our experimental environments were mostly
exploratory, with little interaction. We designed them that
way to reduce any confounding effects of excessive physical
movement on increased heart rate. Now that we have estab-
lished the effects of manipulated heart rate feedback in this
controlled setup, in future we would like to explore other en-
vironments where more interactions are required, such as VR
shooter games. Second, we did not measure the effects of
manipulated heart rate feedback on Presence in the VE as
we wanted to keep the overall task load reasonable for the
participants. However, we understand the need for higher pres-
ence and would like to investigate this effect in a future study.
Third, we used five discrete manipulation factors being
±30%
,
±15%
, and 0%. While we could identify the effects based on
these manipulation factors, they do not confirm whether or not
these are the thresholds where the emotional manipulation is
the highest; there may be other manipulation factors where
we can get better effects. It would be interesting to identify
exactly what levels of manipulation are most effective for each
of the key emotions. Fourth, some of our results are relied
upon the participants’ perception of their own heart rate, for
example subjective question 3. However, heart rate being a
physiological measurement participant may have inaccurate
perception about their own heart rate. Although, this limitation
is not easy to overcome.
Future Work
In the future, we would like to explore how heart rate manip-
ulation affects emotion in other environments and tasks that
require collaboration between two or more users in VR. This
will be interesting, since besides making an individual aware
of his/her own physiological state, we would be providing
cues for the state of collaborators, hopefully creating more
empathy in the collaboration. There are other physiological
measures that can be used to measure emotions such as respi-
ration rate. Besides heart rate and GSR, we would like explore
the physiological effects of VR stimuli on respiration rate. In
this paper, we have provided and manipulated the feedback
of heart rate. It will be interesting to investigate the ways
to provide feedback of additional physiological signals in an
intuitive, yet non-distracting, way to users in VR; similar to
what Tan et al. [29] has done for video-conferencing.
REFERENCES
1. 2016. Thought Technology.
http//thoughttechnology.com/index.php/. (2016).
2. 2017. Unity. https://unity3d.com/. (2017).
3. Rosa María Baños, Cristina Botella, Mariano Alcañiz,
Víctor Liaño, Belén Guerrero, and Beatriz Rey. 2004.
Immersion and emotion: their impact on the sense of
presence. CyberPsychology & Behavior 7, 6 (2004),
734–741.
4. Rosa M Baños, Cristina Botella, Isabel Rubió, Soledad
Quero, Azucena García-Palacios, and Mariano Alcañiz.
2008. Presence and emotions in virtual environments:
The influence of stereoscopy. CyberPsychology &
Behavior 11, 1 (2008), 1–8.
5. Guillermo Bernal and Pattie Maes. 2017. Emotional
Beasts: Visually Expressing Emotions through Avatars in
VR. In Proceedings of the 2017 CHI Conference
Extended Abstracts on Human Factors in Computing
Systems. ACM, 2395–2402.
6. Wolfram Boucsein. 2012. Electrodermal activity.
Springer Science & Business Media.
7. Margaret M Bradley and Peter J Lang. 1994. Measuring
emotion: the self-assessment manikin and the semantic
differential. Journal of behavior therapy and
experimental psychiatry 25, 1 (1994), 49–59.
8. Michel Cabanac. 2002. What is emotion? Behavioural
processes 60, 2 (2002), 69–83.
9. Hao Chen, Arindam Dey, Mark Billinghurst, and
Robert W Lindeman. 2017. Exploring the design space
for multi-sensory heart rate feedback in immersive virtual
reality. In Proceedings of the 29th Australian Conference
on Computer-Human Interaction. ACM, 108–116.
10. Lennard Chua, Jeremy Goh, Zin Tun Nay, Lihui Huang,
Yiyu Cai, and Ruby Seah. 2017. ICT-Enabled Emotional
Learning for Special Needs Education. In Simulation and
Serious Games for Education. Springer, 29–45.
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
109
11. Arindam Dey, Hao Chen, Chang Zhuang, Mark
Billinghurst, and Robert W Lindeman. 2018. Effects of
Sharing Real-Time Multi-Sensory Heart Rate Feedback
in Different Immersive Collaborative Virtual
Environments. In Proceedings of the IEEE International
Symposium on Mixed and Augmented Reality. IEEE.
12.
Arindam Dey, Thammathip Piumsomboon, Youngho Lee,
and Mark Billinghurst. 2017. Effects of Sharing
Physiological States of Players in a Collaborative Virtual
Reality Gameplay. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems.
ACM, 4045–4056.
13. F Faul, E Erdfelder, A Buchner, and AG Lang. 2013. G*
Power Version 3.1. 7 [computer software]. Uiversität
Kiel, Germany. (2013).
14. Anna Felnhofer, Oswald D Kothgassner, Thomas
Hetterle, Leon Beutl, Helmut Hlavacs, and Ilse
Kryspin-Exner. 2014. Afraid to be there? Evaluating the
relation between presence, self-reported anxiety, and
heart rate in a virtual public speaking task.
Cyberpsychology, Behavior, and Social Networking 17, 5
(2014), 310–316.
15. Anna Felnhofer, Oswald D Kothgassner, Mareike
Schmidt, Anna-Katharina Heinzle, Leon Beutl, Helmut
Hlavacs, and Ilse Kryspin-Exner. 2015. Is virtual reality
emotionally arousing? Investigating five emotion
inducing virtual park scenarios. International journal of
human-computer studies 82 (2015), 48–56.
16. Heather M Francis, Kathryn M Penglis, and Skye
McDonald. 2016. Manipulation of heart rate variability
can modify response to anger-inducing stimuli. Social
neuroscience 11, 5 (2016), 545–552.
17.
O Gillath, CA McCall, P Shaver, and J Blascovich. 2008.
Reactions to a needy virtual person: Using an immersive
virtual environment to measure prosocial tendencies.
Media Psychology 11 (2008), 259–282.
18. Jocelyn Gomez, Hunter G Hoffman, Steven L Bistricky,
Miriam Gonzalez, Laura Rosenberg, Mariana Sampaio,
Azucena Garcia-Palacios, Maria V Navarro Haro, Wadee
Alhalabi, Marta Rosenberg, and others. 2017. The use of
Virtual Reality facilitates Dialectical Behavior Therapy
®
“observing sounds and visuals” mindfulness skills training
exercises for a Latino patient with severe burns: A case
study. Frontiers in Psychology 8 (2017), 1611.
19. Rocio Herrero, Azucena Garcia-Palacios, Diana Castilla,
Guadalupe Molinari, and Cristina Botella. 2014. Virtual
reality for the induction of positive emotions in the
treatment of fibromyalgia: a pilot study over acceptability,
satisfaction, and the effect of virtual reality on mood.
Cyberpsychology, Behavior, and Social Networking 17, 6
(2014), 379–384.
20. Horace HS Ip, Simpson WL Wong, Dorothy FY Chan,
Julia Byrne, Chen Li, Vanessa SN Yuan, Kate SY Lau,
and Joe YW Wong. 2016. Virtual reality enabled training
for social adaptation in inclusive education settings for
school-aged children with autism spectrum disorder
(ASD). In International Conference on Blending
Learning. Springer, 94–102.
21. Michelle R Kandalaft, Nyaz Didehbani, Daniel C
Krawczyk, Tandra T Allen, and Sandra B Chapman.
2013. Virtual reality social cognition training for young
adults with high-functioning autism. Journal of autism
and developmental disorders 43, 1 (2013), 34–44.
22.
Jih-Hsuan Tammy Lin. 2017. Fear in virtual reality (VR):
Fear elements, coping reactions, immediate and next-day
fright responses toward a survival horror zombie virtual
reality game. Computers in Human Behavior 72 (2017),
350–361.
23. Alexander Miloff, Philip Lindner, William Hamilton,
Lena Reuterskiöld, Gerhard Andersson, and Per
Carlbring. 2016. Single-session gamified virtual reality
exposure therapy for spider phobia vs. traditional
exposure therapy: study protocol for a randomized
controlled non-inferiority trial. Trials 17, 1 (2016), 60.
24.
Pedro A Nogueira, Vasco Torres, Rui Rodrigues, Eugénio
Oliveira, and Lennart E Nacke. 2016. Vanishing scares:
biofeedback modulation of affective player experiences in
a procedural horror game. Journal on Multimodal User
Interfaces 10, 1 (2016), 31–62.
25. Nathalie Peira, Mats Fredrikson, and Gilles Pourtois.
2014. Controlling the emotional heart: Heart rate
biofeedback improves cardiac control during emotional
reactions. International Journal of Psychophysiology 91,
3 (2014), 225–231.
26. Giuseppe Riva, Fabrizia Mantovani, Claret Samantha
Capideville, Alessandra Preziosa, Francesca Morganti,
Daniela Villani, Andrea Gaggioli, Cristina Botella, and
Mariano Alcañiz. 2007. Affective interactions using
virtual reality: the link between presence and emotions.
CyberPsychology & Behavior 10, 1 (2007), 45–56.
27. Albert Rizzo, Jacquelyn F Morie, Josh Williams, Jarrell
Pair, and J Galen Buckwalter. 2005. Human emotional
state and its relevance for military VR training. Technical
Report. University Of Southern California Marina Del
Rey Ca Inst For Creative Technologies.
28. Michael J Roy, Michelle E Costanzo, Tanja Jovanovic,
Suzanne Leaman, Patricia Taylor, Seth D Norrholm, and
Albert A Rizzo. 2013. Heart rate response to fear
conditioning and virtual reality in subthreshold PTSD.
Annual Review of Cybertherapy and Telemedicine 191
(2013), 115–119.
29. Chiew Seng Sean Tan, Johannes Schöning, Kris Luyten,
and Karin Coninx. 2014. Investigating the effects of using
biofeedback as visual stress indicator during
video-mediated collaboration. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems. ACM, 71–80.
30.
John C Thomas. 1996. The long-term social implications
of new information technology. Hillsdale, NJ: Erlbaum.
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
110
31. Ryoko Ueoka, Ali AlMutawa, and Hikaru Katsuki. 2016.
Emotion hacking VR (EH-VR): amplifying scary VR
experience by accelerating real heart rate using false
vibrotactile biofeedback. In SIGGRAPH ASIA 2016
Emerging Technologies. ACM, 7.
32.
Ryoko Ueoka and Kouya Ishigaki. 2015. Development of
the horror emotion amplification system by means of
biofeedback method. In International Conference on
Human Interface and the Management of Information.
Springer, 657–665.
33. David Watson, Lee A Clark, and Auke Tellegen. 1988.
Development and validation of brief measures of positive
and negative affect: the PANAS scales. Journal of
personality and social psychology 54, 6 (1988), 1063.
34. Brenda K Wiederhold, Dong P Jang, Sun I Kim, and
Mark D Wiederhold. 2002. Physiological monitoring as
an objective tool in virtual reality therapy.
CyberPsychology & Behavior 5, 1 (2002), 77–82.
Session: Paper Presentation
CHI PLAY 2018, October 28–31, 2018, Melbourne, VIC, Australia
111
... Nevertheless, within the dynamic and fast-growing field of VR, the intricate interplay among multisensory stimulation and emotional experiences represents a complex area that remains relatively uncharted, as evidenced by several authors [8][9][10][11][12]. Although some attempts to establish a link between specific stimuli and emotional responses (e.g., [10], [11], [13][14][15][16]), the majority of the studies consider VR experiments based on audiovisual stimuli only [8]. ...
... Nevertheless, within the dynamic and fast-growing field of VR, the intricate interplay among multisensory stimulation and emotional experiences represents a complex area that remains relatively uncharted, as evidenced by several authors [8][9][10][11][12]. Although some attempts to establish a link between specific stimuli and emotional responses (e.g., [10], [11], [13][14][15][16]), the majority of the studies consider VR experiments based on audiovisual stimuli only [8]. However, this focus on primarily audiovisual inputs may limit the overall user experience quality, as human perception of the real world naturally involves a broader range of senses [17]. ...
... Triggering specific emotions in VR can be a complex task, presenting even greater challenges when aiming to evoke particular emotional responses in the users, as highlighted by several authors [8][9][10][11][12]. Emotional elicitation in VR can be categorized as passive, where users are mere observers of an emotional event, or active, involving direct engagement and interaction within the virtual environment. ...
Article
Full-text available
In this study we explore the impact of multisensory stimuli in virtual reality on users’ emotional responses, addressing a knowledge gap in this rapidly evolving field. Utilizing a range of sensory inputs, including taste, haptics, and smell, in addition to audiovisual cues, this study aims to understand how different combinations of these stimuli affect the users’ emotional experience. Two immersive virtual experiences have been developed for this purpose. One included a scenario to evoke positive emotions through selectively chosen pleasant multisensory stimuli, validated in a focus group. The other sought the contrary: to trigger negative emotions by integrating selected combinations of unpleasant multisensory stimuli, also validated in the same focus group. Through a comparative analysis, our findings revealed significant differences in emotional responses between the groups exposed to positive and negative stimuli combinations. Results indicated that combinations involving haptics and taste were particularly effective in eliciting intense emotions using positive stimuli, but their impact was less significant with negative stimuli. This investigation suggests that a fully multisensory virtual environment integrating positive stimuli might lead to cognitive overload, reducing overall emotional responses. In contrast, environments with negative stimuli could enhance emotional engagement and be more likely to avoid cognitive overload. These findings have important implications for designing emotionally resonant and compelling virtual reality experiences. This research enhances the understanding of sensory integration in virtual reality and its effects on emotional engagement, offering valuable insights for developing more impactful virtual experiences.
... The study of emotions can be done consciously, with subjective methods and instruments [29], or unconsciously, resorting to objective methods and instruments. Also, humans' emotional state can be recognized by combining these two methods -a mixed approach [39]. ...
... Bias caused by the sensitivity GSR can be influenced by the room temperature conditions [77]; Physical movements might affect data collection of HR [29] and EDA [55]; "Empatica E4" wearable, used to collect EDA, does not work well consistently throughout the entire time of use (some occasional disconnections and server failures were reported) [55]. ...
... Dey, Chen, Billinghurst, et al. [29] emphasize the importance of understanding what levels of manipulation are most effective for each key emotion; Chin, Thompson and Ziat [76] consider it beneficial to test each stimulus individually to compare modal vs. multimodal effectiveness. Arousal ratings might be biased by the lack of images that induce intense emotional reactions (N=1) ...
Article
Full-text available
Virtual reality and emotions have become inseparable concepts over the past few years, supported by the increasing number of studies relating them. However, these studies' methodologies are often poorly justified or dependent on the authors' subjective definition of emotion and its classification. Moreover, frequently, these studies only consider two stimuli, specifically audiovisual, despite being known the relevance of including a greater variety of sensory channels to improve the relationship between the individual and the virtual environment. So, to address these gaps, and considering the importance of multisensory stimulation, this paper aims to review the methods and instruments found in the literature regarding the analysis of the users' emotions in virtual reality. Also, we provide an overview of the main limitations of such studies. Little information can be found in the literature regarding the connection between the input stimulus and the users’ emotional responses. This corroborates the difficulty in creating and evaluating immersive virtual experiences when stimulating more than two human senses, typically audiovisual. Nevertheless, we address some clues on the impact of visual, auditory, haptic, smell, and taste elements to trigger specific emotions. Also, we address the association between the research area and the method used. Finally, the main gaps and challenges are discussed. We expect that the combination of these results acts as guidelines for designing richer multisensory virtual experiences. Moreover, we intend to contribute to future research on emotions-based immersive virtual reality by providing a review of the most suitable methodologies and instruments for specific contexts.
... Relatedly, Chen et al. [8] explored multi-sensory heart rate (HR) representations, and found that audio-haptic feedback was most preferred while visual feedback was reported as being distracting. Yet other works found that the audio modality either did not facilitate physiological relaxation [94], or did not influence participants' physiological signals [14]. Given the importance of exteroceptive cues, and the unknown role they play in cardiac interoception and "interoceptive illusions" [69], we sought to compare exteroceptive modalities (audio, visual, or audio-visual) for measuring cardiac interoception across screen-based and VR environments. ...
... Whereas these tasks focused on recognition of one's heartbeat with respect to another person's, our task draws on work within immersive virtual environments where participants' own cardiac signals are manipulated through increase/decrease of displayed heartbeats [8,14]. Importantly however, all these previous tasks either rely on expensive measurement apparatus, focus on a single modality, or are designed specifically for psychophysiological experiments using custom software to assess cardiodynamics. ...
... For heartbeats specifically, visualizations typically span visual representations, including skeuomorphic [40], text/numerical [27], screen overlays [29], or holographic displays [82]. However, they can also be represented as auditory heartbeats [15] or as haptic sensations [8,14]. Chen et al. [8] explored multi-sensory heart rate representations, and found that audiohaptic feedback was most preferred while visual feedback was reported as being distracting. ...
Article
Measuring interoception (‘perceiving internal bodily states’) has diagnostic and wellbeing implications. Since heartbeats are distinct and frequent, various methods aim at measuring cardiac interoceptive accuracy (CIAcc). However, the role of exteroceptive modalities for representing heart rate (HR) across screen-based and Virtual Reality (VR) environments remains unclear. Using a PolarH10 HR monitor, we develop a modality-dependent cardiac recognition task that modifies displayed HR. In a mixed-factorial design (N=50), we investigate how task environment (Screen, VR), modality (Audio, Visual, Audio-Visual), and real-time HR modifications ( ± 15%, ± 30%, None) influence CIAcc, interoceptive awareness, mind-body measures, VR presence, and post-experience responses. Findings showed that participants confused their HR with underestimates up to 30%; environment did not affect CIAcc but influenced mind-related measures; modality did not influence CIAcc, however including audio increased interoceptive awareness; and VR presence inversely correlated with CIAcc. We contribute a lightweight and extensible cardiac interoception measurement method, and implications for biofeedback displays.
... Following the guidelines from Chen et al. [8], Dey et al. [16] delivered heart rate feedback through sounds and haptic feedback through hand-held controllers in a VR task. They found that faster heart rate feedback increased self-reported excitement, but excitement decreased beyond +15% higher than real heart rate, potentially due to participants' realization of feedback manipulation. ...
... Firstly, and in answer to RQ2, our results show that high frequencies were able to increase heart rate. This is in contrast to previous projects with similar systems, which failed to achieve this result [8,16]. We believe this difference could be attributed to haptic placement. ...
Conference Paper
Full-text available
This paper explores the impact of vibrotactile haptic feedback on heart activity when the feedback is provided at four different body locations (chest, wrist, neck, and ankle) and with two feedback rates (50 bpm and 110 bpm). A user study found that the neck placement resulted in higher heart rates and lower heart rate variability , and higher frequencies correlated with increased heart rates and decreased heart rate variability. The chest was preferred in self-reported metrics, and neck placement was perceived as less satisfying, harmonious, and immersive. This research contributes to understanding the interplay between psychological experiences and physiological responses when using haptic biofeedback resembling real body signals.
... Regarding the understanding of emotions, Dey et al. [12] investigate the effect of manipulating physiological feedback in VR on emotions. They found that altering such feedback can change the emotions within immersive environments. ...
... Therefore, it is important for any user study that the participants are physically and mentally well enough to participate and not bias the data. We have seen in prior work that at times researchers had to drop participant data due to the participant not doing tasks properly or because the participant left the study mid-way due to frustration [12,47]. We believe that emotional assessment may prove to be helpful to avoid such instances. ...
... The choice of the interactive moments and interfaces will deeply influence the narrative experience and vice versa. In this respect, the incorporation of interfaces that integrate interactors' biological feedback (heart rate, breathing, temperature, sweat, brain waves, among others) with the immersive narrative experience (Reyes 2019, 144;Dey et al. 2018;Chen et al. 2017) would give direct control to the authors over their interactors' emotions bypassing interactors' conscious choices. 4. The Space-Scene writing of events and perceptual stimuli. ...
Chapter
Full-text available
The concept of immersion has come to dominate the extended reality (XR) industries, leading to a renewed interest in the concept's relevance in society. The chapter investigates the evolution of the concept and proposes a model for the analysis of the ideation stage of immersive narrative experiences, as this stage precedes screenwriting in the production workflow of an immersive experience. The concept of immersography is proposed as a way to bridge the gap between colliding epistemologies, and between theory and practice, in the field of immersive narrative experiences. This term emphasizes the narrative connotation of immersion in relation to all-encompassing virtual or in-real-life environments that share three main components: spatial experience of the narrative universe, full-body interaction with the storyworld and its elements, and the capability to generate multiple narrative journeys through interactive narrative structures.
Conference Paper
Full-text available
Virtual Reality (VR) interfaces have been shown to be able to trigger different emotions in users. However, earlier VR interfaces did not provide users with their own heart rate feedback and the effect of such a feedback on overall experience has not been investigated. In this research, we investigated whether providing heart rate feedback enhanced the participant's experience of using VR and we explored the design space of different types of multi-sensory heart rate representations. Through a within-subjects study, and using subjective measurements, we found that participants enjoyed seeing their heart rate feedback when experiencing VR environments. From the different types of heart-rate representations, audio-haptic feedback was the most preferred while visual feedback was reported as being distracting. We report on the implications of this for designing VR experiences and directions for future research¹.
Article
Full-text available
Sustaining a burn injury increases an individual's risk of developing psychological problems such as generalized anxiety, negative emotions, depression, acute stress disorder, or post-traumatic stress disorder. Despite the growing use of Dialectical Behavioral Therapy® (DBT®) by clinical psychologists, to date, there are no published studies using standard DBT® or DBT® skills learning for severe burn patients. The current study explored the feasibility and clinical potential of using Immersive Virtual Reality (VR) enhanced DBT® mindfulness skills training to reduce negative emotions and increase positive emotions of a patient with severe burn injuries. The participant was a hospitalized (in house) 21-year-old Spanish speaking Latino male patient being treated for a large (>35% TBSA) severe flame burn injury. Methods: The patient looked into a pair of Oculus Rift DK2 virtual reality goggles to perceive the computer-generated virtual reality illusion of floating down a river, with rocks, boulders, trees, mountains, and clouds, while listening to DBT® mindfulness training audios during 4 VR sessions over a 1 month period. Study measures were administered before and after each VR session. Results: As predicted, the patient reported increased positive emotions and decreased negative emotions. The patient also accepted the VR mindfulness treatment technique. He reported the sessions helped him become more comfortable with his emotions and he wanted to keep using mindfulness after returning home. Conclusions: Dialectical Behavioral Therapy is an empirically validated treatment approach that has proved effective with non-burn patient populations for treating many of the psychological problems experienced by severe burn patients. The current case study explored for the first time, the use of immersive virtual reality enhanced DBT® mindfulness skills training with a burn patient. The patient reported reductions in negative emotions and increases in positive emotions, after VR DBT® mindfulness skills training. Immersive Virtual Reality is becoming widely available to mainstream consumers, and thus has the potential to make this treatment available to a much wider number of patient populations, including severe burn patients. Additional development, and controlled studies are needed.
Conference Paper
Interfaces for collaborative tasks, such as multiplayer games can enable more effective and enjoyable collaboration. However, in these systems, the emotional states of the users are often not communicated properly due to their remoteness from one another. In this paper, we investigate the effects of showing emotional states of one collaborator to the other during an immersive Virtual Reality (VR) gameplay experience. We created two collaborative immersive VR games that display the real-time heart-rate of one player to the other. The two different games elicited different emotions, one joyous and the other scary. We tested the effects of visualizing heart-rate feedback in comparison with conditions where such a feedback was absent. The games had significant main effects on the overall emotional experience.
Conference Paper
With the advances in Virtual Reality (VR) and physiological sensing technology, even more immersive computer-mediated communication through life-like characteristics is now possible. In response to the current lack of culture, expression and emotions in VR avatars, we propose a two-fold solution. First, integration of bio-signal sensors into the HMD and techniques to detect aspects of the emotional state of the user. Second, the use of this data to generate expressive avatars which we refer to as Emotional Beasts. The creation of Emotional Beasts will allow us to experiment with the manipulation of a user's self-expression in VR space and as well as the perception of others in it, providing some valuable tools to evoke a desired emotional reaction. As this medium moves forward, this and other tools are what will help the field of virtual reality expand from a medium of surface-level experience to one of deep, emotionally compelling human-to-human connection.
Article
This study explores players’ fright reactions and coping strategies in an immersive virtual reality (VR) horror game. Based on Slater’s theory of virtual reality, two dimensions of fear elements in the VR game−the fear of place illusion (PI) and the plausibility illusion (PSI) −were identified by playing a virtual reality survival horror game with a sample of 145 students. Participants reported greater fear toward PSI elements than toward PI elements. Fear of PSI elements positively and strongly predicted disengagement coping strategies and overall fear. Among coping strategies, players mainly adopted approach strategies, followed by avoidance (disengagement and denial), and self-help strategies. A “self-talk” strategy, newly identified in this study, has been reported as an effective means to cope with mediated threat in VR games. Regarding individual differences, sensation seeking and neuroticism influenced participants’ coping strategies and fear. Additionally, males and females employed different coping strategies. Very few students experienced next-day fright, which consists mostly of cognitive reactions and VR-related reactions, such as the Tetris effect and the fear of being attacked from the back. Theoretical frameworks regarding fear elements and coping reactions are proposed to aid future research. Implications for academia, fear conditioning for training, and marketing campaigns are discussed.
Conference Paper
Emotion hacking virtual reality (EH-VR) system is an interactive system that hacks one's heartbeat and controls it to accelerate scary VR experience. The EH-VR system provides vibrotactile biofeedback, which resembles a heartbeat, from the floor. The system determines false heartbeat frequency by detecting user's heart rate in real time and calculates false heart rate, which is faster than the one observed according to the quadric equation model. With the system, we demonstrate "Pressure of unknown" which is a CG VR space originally created to express the metaphor of scare. A user experiences this space by using a wheel chair as a controller to walk through a VR world displayed via HMD while receiving vibrotac-tile feedback of false heartbeat calculated from its own heart rate from the floor.
Chapter
Autism Spectrum Disorders (ASD) is a neuro-developmental disability. Children with ASD often find it difficult to express and recognize emotions which make it hard for them to interact socially. Conventional emotional learning for children with ASD often uses medicinal means, and behavioural analysis that could be costly and less effective. There is a significant need to develop Infocomm Technology (ICT) based methods for effective emotional learning. We are interested to develop an iPAD-based game approach for children with ASD to learn emotion through interactive gaming. This chapter reports our project on Social Emotional Learning (SEL) for children with ASDs using an iPAD app.
Conference Paper
The transition from kindergarten to primary school tends to be challenging for children with special needs. These children may benefit from relevant training in advance or in addition to school, yet it is challenging to support such training in an authentic, safe and controllable environment. In this paper, we present a Virtual Reality (VR) enabled system to facilitate social adaptation training for school-aged children with clinical or suspected diagnosis of Autism Spectrum Disorders (ASD) in the inclusive education setting. Six unique VR training scenarios with corresponding training protocols are designed, implemented and being delivered to over 100 school-aged children with normal-ranged IQ (IQ > 70) via a 4-side fully immersive CAVE™ VR installation in 28 sessions (14 weeks). Preliminary results indicate that after training completion, children show significant improvements in three major designated aspects, including emotion recognition, affective expression and social reciprocity.