ArticlePDF Available

Abstract and Figures

Affective Computing has emerged as an important field of study that aims to develop systems that can automatically recognize emotions. Up to the present, elicitation has been carried out with non-immersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for affective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were designed to elicit four possible arousal-valence combinations, as described in each quadrant of the Circumplex Model of Affects. An experiment involving the recording of the electroencephalography (EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features was extracted from these signals using various state-of-the-art metrics that quantify brain and cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classifier to predict the subject’s arousal and valence perception. The model’s accuracy was 75.00% along the arousal dimension and 71.21% along the valence dimension. Our findings validate the use of Immersive Virtual Environments to elicit and automatically recognize different emotional states from neural and cardiac dynamics; this development could have novel applications in fields as diverse as Architecture, Health, Education and Videogames.
Content may be subject to copyright.
1
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
www.nature.com/scientificreports
Aective computing in virtual
reality: emotion recognition from
brain and heartbeat dynamics using
wearable sensors
Javier Marín-Morales1, Juan Luis Higuera-Trujillo1, Alberto Greco
2, Jaime Guixeres1,
Carmen Llinares1, Enzo Pasquale Scilingo
2, Mariano Alcañiz
1 & Gaetano Valenza2
Aective Computing has emerged as an important eld of study that aims to develop systems that
can automatically recognize emotions. Up to the present, elicitation has been carried out with non-
immersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for
aective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were
designed to elicit four possible arousal-valence combinations, as described in each quadrant of the
Circumplex Model of Aects. An experiment involving the recording of the electroencephalography
(EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features
was extracted from these signals using various state-of-the-art metrics that quantify brain and
cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classier
to predict the subject’s arousal and valence perception. The model’s accuracy was 75.00% along the
arousal dimension and 71.21% along the valence dimension. Our ndings validate the use of Immersive
Virtual Environments to elicit and automatically recognize dierent emotional states from neural and
cardiac dynamics; this development could have novel applications in elds as diverse as Architecture,
Health, Education and Videogames.
Aective Computing (AfC) has emerged as an important eld of study in the development of systems that can
automatically recognize, model and express emotions. Proposed by Rosalind Picard in 1997, it is an interdiscipli-
nary eld based on psychology, computer science and biomedical engineering1. Stimulated by the fact that emo-
tions are involved in many background processes2 (such as perception, decision-making, creativity, memory, and
social interaction), several studies have focused on searching for a reliable methodology to identify the emotional
state of a subject by using machine learning algorithms.
us, AfC has emerged as an important research topic. It has been applied oen in education, healthcare,
marketing and entertainment36, but its potential is still under development. Architecture is a eld where AfC
has been infrequently applied, despite its obvious potential; the physical-environment has on a great impact, on
a daily basis, on human emotional states in general7, and on well-being in particular8. AfC could contribute to
improve building design to better satisfy human emotional demands9.
Irrespective of its application, Aective Computing involves both emotional classication and emotional elic-
itation. Regarding emotional classication, two approaches have commonly been proposed: discrete and dimen-
sional models. On the one hand, the former posits the existence of a small set of basic emotions, on the basis that
complex emotions result from a combination of these basic emotions. For example, Ekman proposed six basic
emotions: anger, disgust, fear, joy, sadness and surprise10. Dimensional models, on the other hand, consider a
multidimensional space where each dimension represents a fundamental property common to all emotions. For
example, the “Circumplex Model of Aects” (CMA)11 uses a Cartesian system of axes, with two dimensions,
proposed by Russell and Mehrabian12: valence, i.e., the degree to which an emotion is perceived as positive or
negative; and arousal, i.e., how strongly the emotion is felt.
1Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, València, Spain.
2Bioengineering and Robotics Research Centre E Piaggio & Department of Information Engineering, University of Pisa,
Pisa, Italy. Correspondence and requests for materials should be addressed to J.M.-M. (email: jamarmo@i3b.upv.es)
Received: 6 December 2017
Accepted: 10 August 2018
Published: xx xx xxxx
OPEN
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
2
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
In order to classify emotions automatically, correlates from, e.g., voice, face, posture, text, neuroimaging, and
physiological signals are widely used13. In particular, several computational methods are based on variables asso-
ciated with central nervous system (CNS) and autonomic nervous system (ANS) dynamics13. On the one hand,
the use of CNS is justied by the fact that human emotions originate in the cerebral cortex, involving several
areas in their regulation and feeling. In this sense, the electroencephalogram (EEG) is one of the techniques
most used to measure CNS responses14, also through the use of wearable devices. On the other hand, a wider
class of aective computing studies consider ANS changes elicited by specic emotional states. In this sense,
experimental results over the last three decades show that Heart Rate Variability (HRV) analyses can provide
unique and non-invasive assessments of autonomic functions on cardiovascular dynamics15,16. To this extent,
there has been a great increase over the last decade in research and commercial interest in wearable systems for
physiological monitoring. e key benets of these systems are their small size, lightness, low-power consump-
tion and, of course, their wearability17. e state of the art1820 on wearable systems for physiological monitoring
highlight that: i) surveys predict that the demand for wearable devices will increase in the near future; ii) there
will be a need for more multimodal fusion of physiological signals in the near future; and iii) machine learning
algorithms can be merged with traditional approaches. Moreover, recent studies present promising results on
the development of emotion recognition systems through using wearable sensors instead of classic lab sensors,
through HRV21 and EEG22.
Regarding emotional elicitation, the ability to reliably and ethically elicit aective states in the laboratory is
a critical challenge in the process of the development of systems that can detect, interpret and adapt to human
aect23. Many methods of eliciting emotions have been developed to evoke emotional responses. Based on the
nature of the stimuli, two types of method are distinguished, the active and the passive. Active methods can
involve behavioural manipulation24, social psychological methods with social interaction25 and dyadic interac-
tion26. On the other hand, passive methods usually present images, sounds or lms. With respect to images, one
of the most prominent databases is the International Aective Picture System (IAPS), which includes over a thou-
sand depictions of people, objects and events, standardized on the basis of valence and arousal23. e IAPS has
been used in many studies as an elicitation tool in emotion recognition methodologies15. With respect to sound,
the most used database is the International Aective Digitalised Sound System (IADS)27. Some researchers also
use music or narrative to elicit emotions28. Finally, audio-visual stimuli, such as lms, are also used to induce
dierent levels of valence and arousal29.
Even when, as far we know, elicitation has been carried out with a non-immersive stimulus, it has been shown
that these passive methods have signicant limitations due to the importance of immersion for eliciting emotions
through the simulation of real experiences30. In the present, Virtual Reality (VR) represents a novel and power-
ful tool for behavioural research in psychological assessment. It provides simulated experiences that create the
sensation of being in the real world31,32. us, VR makes it possible to simulate and evaluate spatial environments
under controlled laboratory conditions32,33, allowing the isolation and modication of variables in a cost and
time eective manner, something which is unfeasible in real space34. During the last two decades VR has usually
been displayed using desktop PCs or semi-immersive systems such as CAVEs or Powerwalls35. Today, the use of
head-mounted displays (HMD) is increasing: these provide fully-immersive systems that isolate the user from
external world stimuli. ese provide a high degree of immersion, evoking a greater sense of presence, under-
stood as the perceptual illusion of non-mediation and a sense of “being-there”36. Moreover, the ability of VR to
induce emotions has been analysed in studies which demonstrate that virtual environments do evoke emotions in
the user34. Other works conrm that Immersive Virtual Environments (IVE) can be used as emotional induction
tools to create states of relaxation or anxiety37, basic emotions38,39, and to study the inuence of the users cultural
and technological background on emotional responses in VR40. In addition, some works show that emotional
content increases sense of presence in an IVE41 and that, faced with the same content, self-reported intensity of
emotion is signicantly greater in immersive than in non-immersive environments42. us, IVEs, showing 360°
panoramas or 3D scenarios through a HMD43, are powerful tools for psychological research43,.
Taking advantage of the IVE’s potentialities, in recent years some studies have used IVE and physiological
responses, such as EEG, HRV and EDA, in dierent elds. Phobias4447, disorders48, driving and orientation49,50,
videogames51, quality of experience52, presence53 and visualization technologies54, are some examples of these
applications. Particularly in emotion research, arousal and relaxation have been analysed in outdoor55,56 and
indoor57 IVEs using EDA. erefore, the state of the art presents the following limitations: (1) few studies ana-
lyse physiological responses in IVEs and, in particular, using an aective approach; (2) there are few validated
emotional IVE sets which include stimuli with dierent levels of arousal and valence: and, (3) there is no aective
computing research that tries to automatically recognize the user’s mood in an IVE through physiological signals
and machine learning algorithms.
In this study, we propose a new AfC methodology capable of recognizing the emotional state of a subject in an
IVE in terms of valence and arousal. Regarding stimuli, IVEs were designed to evoke dierent emotional states
from an architectural point of view, by changing physical features such as illumination, colour and geometry. ey
were presented through a portable HMD. Regarding emotion recognition, a binary classier will be presented,
which uses eective features extracted from EEG and HRV data gathered from wearable sensors, and combined
through nonlinear Support Vector Machine (SVM)15 algorithms.
Material and Methods
Experimental context. is work is part of a larger research project that attempts to characterize the use of
VR as an aective elicitation method and, consequently, develop emotion recognition systems that can be applied
to 3D or real environments.
An experimental protocol was designed to acquire the physiological responses of subjects in 4 dierent stimuli
presentation cases: 2D desktop pictures, a 360° panorama IVE, a 3D scenario IVE and a physical environment.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
3
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
e experiment was conducted in two distinct phases that presented some dierences. Both phases were divided
into 3 stages; the results of the experiment are at Fig.1. Between each stage, signal acquisition was temporarily
halted and the subjects rested for 3 minutes on a chair. Stage 1 consisted of emotion elicitation through a desktop
PC displaying 110 IAPS pictures, using a methodology detailed in previous research15. Stage 2 consisted of emo-
tion elicitation using an HMD based on a new IVE set with four 360° panoramas. Finally, stage 3 consisted of the
free exploration of a museum exhibition.
In the present paper we focus on an analysis of stage 2. e experimental protocol was approved by the ethics
committee of the Polytechnic University of Valencia and informed consent was obtained from all participants.
All methods and experimental protocols were performed in accordance with the guidelines and regulations of the
local ethics committee of the Polytechnic University of Valencia.
Participants. A group of 60 healthy volunteers, suering neither from cardiovascular nor evident mental
pathologies, was recruited to participate in the experiment. ey were balanced in terms of age (28.9 ± 5.44) and
gender (40% male, 60% female). Inclusion criteria were as follows: age between 20 and 40 years; Spanish national-
ity; having no formal education in art or ne art; having no previous experience of virtual reality; and not having
previously visited the particular art exhibition. ey were divided into 30 subjects for the rst phase and 30 for
the second.
To ensure that the subjects constituted a homogeneous group, and that they were in a healthy mental state,
they were screened by i) the Patient Health Questionnaire (PHQ-9)58 and ii) the Self-Assessment Manikin
(SAM)59.
PHQ-9 is a standard psychometric test used to quantify levels of depression58. Signicant levels of depression
would have aected the emotional responses. Only participants with a score lower than 5 were included in the
study. e test was presented in the Spanish language as the subjects were native Spanish speakers. SAM tests
were used to detect if any subject had an emotional response that could be considered as an outlier, with respect
to a standard elicitation, in terms of valence and arousal. A set of 8 IAPS pictures60 (see Table1), representative
of dierent degrees of arousal and valence perception, was scored by each subject aer stage 1 of the experiment.
e z-score of each subject’s arousal and valence score was calculated using the mean and deviation of the IAPS’s
published scores60. Subjects that had one or more z-scores outside of the range 2.58 and 2.58 (α = 0.005) were
excluded from further analyses. erefore, we retained subjects whose emotional responses, caused by positive
and negative pictures, in dierent degrees of arousal, belonged to 99% of the IAPS population. In addition, we
rejected subjects if their signals presented issues, e.g., disconnection of the sensors during the elicitation or if
artefacts aected the signals. Taking these exclusions into account, the number of valid subjects was 38 (age:
28.42 ± 4.99; gender: 39% male, 61% female).
Set of Physiological Signals and Instrumentation. e physiological signals were acquired using the
B-Alert x10 (Advanced Brain Monitoring, Inc., USA) (Fig.2). It provides an integrated approach for wireless
wearable acquisition and recording of electroencephalographic (EEG) and electrocardiographic (ECG) signals,
sampled at 256 Hz. EEG sensors were located in the frontal (Fz, F3 and F4), central (Cz, C3 and C4) and parietal
(POz, P3, and P4) regions with electrode placements on the subjects’ scalps based on the international 10–20 elec-
trode placement. A pair of electrodes placed below the mastoid was used as reference, and a test was performed to
Figure 1. Experimental phases of the research.
IAPS picture Arousal Valenc e
7234 3.41 ± 2.29 4.01 ± 1.32
5201 3.20 ± 2.50 7.76 ± 1.44
9290 4.75 ± 2.20 2.71 ± 1.35
1463 4.61 ± 2.56 8.17 ± 1.48
9181 6.20 ± 2.23 1.84 ± 1.25
8380 5.84 ± 2.34 7.88 ± 1.37
3102 6.92 ± 2.50 1.29 ± 0.79
4652 7.24 ± 2.09 7.68 ± 1.64
Table 1. Arousal and valence score of selected IAPS pictures from56.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
4
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
check the conductivity of the electrodes, aiming to keep the electrode impedance below 20k. e le ECG lead
was located on the lowest rib and the right lead on the right collarbone.
Stimulus elicitation. We developed an aective elicitation system by using architectural environments dis-
played by 360° panoramas implemented in a portable HMD (Samsung Gear VR). is combination of environ-
ments and display-format was selected due to its capacity for evoking aective states. e bidirectional inuence
between the architectural environment and the user’s aective-behavioural response is widely accepted: even
subtle variations in the space may generate dierent neurophysiological responses61. Furthermore, the 360°
panorama-format provided by HMD devices is a valid set-up to evoke psychological and physiological responses
similar to those evoked by physical environments54. us, following the combination of the arousal and valence
dimensions, which gives the four possibilities described in the CMA62, four architectural environments were
proposed as representative of four emotional states.
e four architectural environments were designed based on Kazuyo Sejimas “Villa in the forest” scenario63.
is architectural work was considered by the research team as an appropriate base from which to make the mod-
ications designed to generate the dierent aective states.
e four base-scenario congurations were based on dierent modications of the parameters of three design
variables: illumination, colour, and geometry. Regarding illumination, the parameters “colour temperature”,
“intensity”, and “position” were modied. e modication of the “colour temperature” was based on the fact
that higher temperature may increase arousal, being registrable at the neurophysiological level64,65. “Intensity”
was also modied in the same way to try to increase or reduce arousal. e “position” of the light was direct,
in order to try to increase arousal, and indirect to reduce it. e modications of these last two parameters
were based on the design experience of the research team. Regarding colour, the parameters “tone”, “value”, and
“saturation” were modied. e modication of these parameters was performed jointly on the basis that warm
colours increase arousal and cold ones reduce it, being registrable at the psychological66 and neurophysiological
levels6771. Regarding geometry, the parameters “curvature, “complexity”, and “order” were modied. “Curvature
was modied on the basis that curved spaces generate a more positive valence than angular, being registrable at
psychological and neurophysiological levels72. e modication of the parameters “complexity” and “order” was
performed jointly. is was based on three conditions registrable at the neurophysiological level: (1) high levels of
geometric “complexity” may increase arousal and low levels may reduce arousal73; (2) high levels of “complexity”
may generate a positive valence if they are submitted to “order”, and negative valence if presented disorderly74; and
(3) content levels of arousal generated by geometry may generate a more positive valence75. e four architectural
environments were designed on this basis. Table2 shows the conguration guidelines chosen to elicit the four
aective states.
In a technical sense, the four architectural environments were developed in similar ways. e process con-
sisted of modelling and rendering. Modelling was performed by using Rhinoceros v5.0 (www.rhino3d.com).
Figure 2. Exemplary experimental set-up.
High-Arousal &
Negative-Valence
(Room 1)
High-Arousal &
Positive-Valence
(Room 2)
Low-Arousal &
Negative-Valence
(Room 3)
Low-Arousal &
Positive-Valence
(Room 4)
Illumination
Colour temperature 7500 K 7500 K 3500 K 3500 K
Intensity High High Low Low
Position Mainly Direct Mainly Direct Mainly Indirect Mainly Indirect
Colour
Ton e
Warm colours Warm colours Cold colours Cold coloursVal u e
Saturation
Geometry
Curvature Rectilinear Curved Rectilinear Curved
Complexity High Low-Medium Medium-High Low
Order Low High Low-Medium High
Table 2. Conguration guidelines chosen in each architectural environment conguration.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
5
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
e 3D-models used for the four architectural environments were 3446946, 3490491, 3487660, and 3487687
polygons. On completion of this process, they were exported in.dwg format for later rendering. e rendering
was performed using the VRay engine v3.00.08 (www.vray.com), operating with Autodesk 3ds Max v2015 (www.
autodesk.es). 15 textures were used for each of the four architectural environments. Congured as 360° pano-
ramas, renders were exported in.jpg format with resolutions of 6000 × 3000 pixels at 300 dots per inch. ese
were implemented in the Samsung Gear VR HMD device. is device has a stereoscopic screen of 1280 × 1440
pixels per eye and a 96° eld of view, supported by a Samsung Note 4 mobile telephone with a 2.7 GHz quad-core
processor and 3GB of RAM. e reproduction of the architectural 360° panoramas was uid and uninterrupted.
Prior to the execution of the experimental protocol, a pre-test was performed in order to ensure that the archi-
tectural 360° panoramas would elicit the aective states for which they had been designed. It was a three-phased
test: individual questionnaires, a focus-group session conducted with some respondents to the questionnaire
and individual validation-questionnaires. e questionnaires asked the participants to evaluate the architectural
360° panoramas. A SAM questionnaire, embedded in the 360° panorama, was used, with evaluations ranging
from 4 (totally disagree) to 4 (totally agree) for all the emotion dimensions. 15 participants (8 men and 7
women) completed the questionnaires. First, the participants freely viewed each architectural environment, then
the SAM questionnaires were presented and the answers given orally. Figure3 shows an example of one of these
questionnaires. Aer the questionnaire sessions had been completed, a focus group session, which was a carefully
managed group discussion, was conducted76. Five of the participants (3 men and 2 women) with the most unfa-
vourable evaluations in phase 1 were selected as participants and one of the members of the research team, with
previous focus-group experience, moderated. e majority of the changes were performed to Room 3, due to the
discordances between the self-assessment and their theoretical quadrant. Once the changes were implemented, a
similar evaluation to phase 1 was performed. Table3 shows the arousal and valence ratings of the four architec-
tural 360° panoramas of this pre-test phase. Aer these phases, no new variations were considered necessary. is
procedure allowed us to assume some initial reliability in the design of the architectural environments. Figure4
shows these nal congurations. High quality images of the stimuli are included in the supplementary material.
None of the pre-test participants was included in the main study. Regarding the experimental protocol, each
room was presented for 1.5 minutes and the sequence of presentation was counter-balanced using the Latin
Square method. Aer viewing the rooms, the users were asking to orally evaluate the emotional impact of each
room using a SAM questionnaire embedded in the 360° photo.
Signal processing. Heart rate variability. e ECG signals were processed to derive HRV series77. e
artefacts were cleaned by the threshold base artefacts correction algorithm included in the Kubios soware78.
In order to extract the RR series, the well-known algorithm developed by Pan-Tompkins was used to detect the
R-peaks79. e individual trends components were removed using the smoothness prior detrending method80.
Figure 3. Example of SAM questionnaire embedded in the room 1. Simulation developed using Rhinoceros
v5.0, VRay engine v3.00.08 and Autodesk 3ds Max v2015.
Arousal Valenc e
High-Arousal & Negative-Valence
(Room 1) 2.23 ± 1.59 2.08 ± 1.71
High-Arousal & Positive-Valence
(Room 2) 1.25 ± 1.33 1.31 ± 1.38
Low-Arousal & Negative-Valence
(Room 3) 0.69 ± 1.65 1.46 ± 1.33
Low-Arousal & Positive-Valence
(Room 4) 2.31 ± 1.30 1.92 ± 1.50
Table 3. Arousal and Valence resulted in the pre-test with 15 participants. e scores are averaged using mean
and standard deviation for a Likert scale between 4 to +4.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
6
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
We carried out the analysis of the standard HRV parameters, which are dened in the time and frequency
domains, as well as HRV measures quantifying heartbeat nonlinear and complex dynamics77. All features are
listed in Table4.
Time domain features include average (Mean RR) and standard deviation (Std RR) of the RR intervals, the
root mean square of successive dierences of intervals (RMSSD), and the ratio between the number of successive
RR pairs having a dierence of less than 50 ms and the total number of heartbeat analyses (pNN50). e triangu-
lar index was calculated as a triangular interpolation of the HRV histogram. Finally, TINN is the baseline width
of the RR histogram, evaluated through triangular interpolation.
In order to obtain the frequency domain features, a power spectrum density (PSD) estimate was calculated for
the RR interval series by a Fast Fourier Transform based on Welch’s periodogram method. e analysis was car-
ried out in three bands: very low frequency (VLF, <0.04 Hz), low frequency (LF, 0.04–0.15 Hz) and high frequency
(HF, 0.12–0.4 Hz). For each frequency band, the peak value was calculated, corresponding to the frequency with
the maximum magnitude. e power of each frequency band was calculated in absolute and percentage terms.
Moreover, for the LF and HF bands, the normalized power (n.u.) was calculated as the percentage of the signals
subtracting the VLF to the total power. e LF/HF ratio was calculated in order to quantify sympatho-vagal bal-
ance and to reect sympathetic modulations77. In addition, the total power was calculated.
Regarding the HRV nonlinear analysis, many measures were extracted, as they are important quantiers of
cardiovascular control dynamics mediated by the ANS in aective computing15,16,77,81. Pointcaré plot analysis is a
quantitative-visual technique, whereby the shape of a plot is categorized into functional classes. e plot provides
summary information as well as detailed beat-to-beat information on heart behaviour. SD1 is related to the fast
beat-to-beat variability in the data, whereas SD2 describes the longer-term variability of R–R77. Approximate
Entropy (ApEn) and Sample Entropy (SampEn) are two entropy measures of HRV. ApEn detects the changes
in underlying episodic behaviour not reected in peak occurrences or amplitudes82, whereas SampEn statistics
provide an improved evaluation of time-series regularity and provide a useful tool in studies of the dynamics of
human cardiovascular physiology83. DFA correlations are divided into short-term and long-term uctuations
through the α1 and α2 features. Whereas α1 represents the uctuation in the range of 4–16 samples, α2 refers to
the range of 16–64 samples84. Finally, the correlation dimension is another method for measuring the complexity
Figure 4. 360° panoramas of the four IVEs. Simulations developed using Rhinoceros v5.0, VRay engine
v3.00.08 and Autodesk 3ds Max v2015.
Time domain Frequency domain Other
Mean RR VLF peak Pointcaré SD1
Std RR LF peak Pointcaré SD2
RMSSD HF peak Approximate Entropy (ApEn)
pNN50 VLF power Sample Entropy (SampEn)
RR triangular index VLF power % DFA α1
TINN LF power D FA α2
LF power % Correlation dimension (D2)
LF power n.u.
HF power
HF power %
HF power n.u.
LF/HF power
Total power
Table 4. List of used HRV features.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
7
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
or strangeness of the time series; it is explained by the D2 feature. It is expected to give information on the mini-
mum number of dynamic variables needed to model the underlying system85.
Electroencephalographic signals. In order to process the EEG signals, the open source toolbox EEGLAB86 was
used. e complete processing scheme is shown at Fig.5.
Firstly, data from each electrode were analysed in order to identify corrupted channels. ese were identied by
computing the fourth standardized moment (kurtosis) along the signal of each electrode87. In addition, if the signal
was atter than 10% of the total duration of the experiment, the channel was classied as corrupted. If one of the
nine channels was considered as corrupted, it could be interpolated from neighbouring electrodes. If more than one
channel was corrupted, the subject would be rejected. Only one channel among all of the subjects was interpolated.
e baseline of EEG traces was removed by mean subtraction and a band pass lter between 0.5 and 40 Hz
was applied. e signal was divided into epochs of one second and the intra-channel kurtosis level of each epoch
was computed in order to reject the epochs highly damaged by noise87. In addition, automatic artefact detection
was applied, which rejects the epoch when more than 2 channels have samples exceeding an absolute threshold of
>100.00 µV and a gradient of 70.00 µV between samples88.
e Independent Component Analysis (ICA)89 was then carried out using infomax algorithm to detect and
remove components due to eye movements, blinks and muscular artefacts. Nine source signals were obtained
(one per electrode). A trained expert manually analysed all the components, rejecting those related to artefacts.
e subjects who had more than 33% of their signals aected by artefacts were rejected.
Aer the pre-processing, spectral and functional connectivity analyses were performed.
EEG spectral analysis, using Welch’s method90, was performed to estimate the power spectra in each epoch,
with 50% overlapping, within the classical frequency bandwidth θ (4–8 Hz), α (8–12 Hz), β (13–25 Hz), γ (25–
40 Hz). Frequency band δ (less than 4 Hz) was not taken into account in this study because it relates to deeper
stages of sleep. In total, 36 features were obtained from the nine channels and 4 bands.
A functional connectivity analysis was performed using Mean Phase Coherence91, for each pair of channels:
φφRE Esin[cos()][()](1)
222
where
R
is the MPC, Δφ represents the relative phase diference between two channels derived from the instanta-
neous dierence of the analytics signals from the Hilbert transform, and
E
is the expectation operator. By deni-
tion, MPC values ranged between 0 and 1. In the case of strong phase synchronization between two channels, the
MPC is close to 1. If the two channels are not synchronized, the MPC remains low. 36 features were derived from
each possible combination of a pair of 9 channels in one specic band. In total, 144 features were created using the
4 bands analysed.
Feature reduction and machine learning. Each room was presented for 1.5 minutes and was considered as
an independent stimulus. In order to characterize each room, all HRV features were calculated using this time window.
In the case of EEG, in both the frequency band power and mean phase connectivity analyses, we considered the mean
of all the epochs of each stimulus as the representative value of the stimulus time window. Altogether, 209 features
described each stimulus for each subject. Due to the high-dimensional feature space obtained, a feature reduction
strategy was adopted for decreasing this dimension. We implemented the well-known Principal Component Analysis
method (PCA)92. is mathematical method is based on the linear transformation of the dierent variables in the prin-
cipal components, which can be assembled in clusters. We select the features that explain 95% of the variability of the
dataset. e PCA was applied three times: (1) in the HRV set, reducing the features from 29 to 3; (2) in the frequency
band power analysis of the EEG, reducing the features from 36 to 4; and (3) in the mean phase coherency analysis of the
EEG, reducing the features from 144 to 12. Hence, the feature reduction strategy reduces our features to a total of 19.
e machine learning strategy could be summarized as follows:
(1) To divide the dataset into training and test sets.
(2) e development of the model (parameter tuning and feature selection) using cross-validation in the
training set.
(3) To validate the model using the test set.
Figure 5. Block scheme of the EEG signal processing steps.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
8
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
Firstly, the dataset was sliced randomly into 15% for the test set (5 subjects) and 85% for the training set (33
subjects). In order to calibrate the model, the Leave-One-Subject-Out (LOSO) cross-validation procedure was
applied to the training set using Support Vector Machine (SVM)-based pattern recognition93. Within the LOSO
scheme, the training set was normalized by subtracting the median value and dividing this by the median absolute
deviation over each dimension. In each of the 36 iterations, the validation set consisted of one specic subject and
he/she was normalized using the median and deviation of the training set.
Regarding the algorithm, we used a C-SVM optimized using a sigmoid kernel function, changing the
parameters of cost and gamma using a vector with 15 parameters logarithmically spaced between 0.1 and 1000.
Additionally, in order to explore the relative importance of all the features in the classication problem we used
a support vector machine recursive feature elimination (SVM-RFE) procedure in a wrapper approach (RFE was
performed on the training set of each fold and we computed the median rank for each feature over all folds). We
specically chose a recently developed, nonlinear SVM-RFE, which includes a correlation bias reduction strategy
in the feature elimination procedure94. Aer the cross-validation, using the parameters and feature set obtained,
the model was applied to the test set that had not previously been used. e self-assessment of each subject was
used as the output of the arousal and valence model. e evaluation was bipolarized in positive/high (>0) and
negative/low (<=0). All the algorithms were implemented by using Matlab© R2016a, endowed with an addi-
tional toolbox for pattern recognition, i.e., LIBSVM95. A general overview of the analysis is shown in Fig.6.
Results
Subjects’ self-assessment. Figure7 shows the self-assessment of the subjects for each IVE averaged using
mean and standard deviation in terms of arousal (Room 1: 1.17 ± 1.81, Room 2: 2.10 ± 1.59, Room 3: 0.05 ± 2.01,
Room 4: 0.60 ± 2.11) and valence (Room 1: 1.12 ± 1.95, Room 2: 1.45 ± 1.93, Room 3: 0.40 ± 2.14, Room 4:
2.57 ± 1.42). e representation follows the CMA space. All rooms are located in the theoretical emotion quad-
rant for which they were designed, except for Room 3 that evokes more arousal than hypothesized. Due to the
non-Gaussianity of data (p < 0.05 from the Shapiro-Wilk test with null hypothesis of having a Gaussian sam-
ple), Wilcoxon signed-rank tests were applied. Table5 presents the result of multiple comparisons using Tukey’s
Honestly Signicant Dierence Procedure. Signicant dierences were found in the valence dimension between
the negative-valence rooms (1 and 3) and the positive-valence rooms (2 and 4). Signicant dierences were found
in the arousal dimension between the high-arousal rooms (1 and 2) and the low-arousal rooms (3 and 4), but not
for pairs 1 and 3. erefore, the IVEs statistically achieve all the desired self-assessments except for arousal per-
ception in Room 3, which is higher than we hypothesized. Aer the bipolarization of scores (positive/high >0),
they are balanced (61.36% high arousal and 56.06% positive valence).
Arousal classication. Table6 shows the confusion matrix of cross validation and the total average accu-
racy (75.00%), distinguishing two levels of arousal using the rst 15 features selected by the nonlinear SVM-RFE
algorithm. e F-Score of arousal classication is 0.75. e changes in accuracy depending on number of features
are shown in Fig.8, and Table7 presents the list of features used. Table8 shows the confusion matrix of the test set
and the total average accuracy (70.00%) using the parameters and the feature set dened in the cross-validation
phase. e F-score of arousal classication is 0.72 in the test set.
Valence classication. Table9 shows the confusion matrix of the cross validation and total average accu-
racy (71.21%), distinguishing two levels of valence using the rst 10 features selected by the nonlinear SVM-RFE
algorithm. e F-Score of the valence classication is 0.71. e changes in accuracy depending on the number
of features are shown in Fig.9, and Table10 presents the list of features used. Table11 shows the confusion
matrix of the test set and total average accuracy (70.00%), using the parameters and the feature set dened in the
cross-validation phase. e F-score of the valence classication was 0.70 in the test set.
Discussion
e purpose of this study is to develop an emotion recognition system able to automatically discern aective
states evoked through an IVE. is is part of a larger research project that seeks to analyse the use of VR as an
aective elicitation method, in order to develop emotion recognition systems that can be applied to 3D or real
environments. e results can be discussed on four levels: (1) the ability of IVEs to evoke emotions; (2) the ability
of IVEs to evoke the same emotions as real environments; (3) the developed emotion recognition model; and (4),
the ndings and applications of the methodology.
Figure 6. Overview of the feature reduction and classication chain.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
9
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
Regarding the ability of the IVEs to evoke emotions, four versions of the same basic room design were used to
elicit the four main arousal-valence combinations related to the CMA. is was achieved by changing dierent
architectural parameters, such as illumination, colour and geometry. As shown in Fig.7 and Table5, proper elic-
itation was achieved for Room 1 (high arousal and negative valence), Room 2 (high arousal and positive valence)
and Room 4 (low arousal and positive valence), but it overlapped somewhat with the arousal-valence representa-
tion in Room 3: despite the satisfactory pre-test, in the event it evoked higher arousal and valence than expected.
is is due to the diculties we experienced in designing a room to evoke negative emotion with low arousal. It
should be noted that IAPS developers may also have experienced this problem because only 18.75% of the pics
are situated in this quadrant60. Other works based on processing valence and arousal using words show that a
U-model exists in which arousal increases in agreement with valence intensity regardless of whether it is posi-
tive or negative96. Hence, for future works, Room 3 will be redesigned to decrease its arousal and valence and a
self-assessment with a larger sample will be performed, by questionnaire, to robustly assess the IVE. Nonetheless,
aer thresholding the individual self-assessment scores to discern 2 classes (high/low), the IVE set was balanced
in arousal and valence. erefore, we could conclude that the proposed room set can satisfactorily evoke the four
emotions represented by each quadrant of the CMA.
To this extent, although previous studies have presented IVEs capable of evoking emotional states in a con-
trolled way97, to the best of our knowledge we have presented the rst IVE suite capable of evoking a variety of
levels of arousal and valence based on CMA. Moreover, the suite was tested through a low-cost portable HMD,
Figure 7. Self-assessment score in the IVEs using SAM and a Likert scale between 4 and +4. Blue dots
represent the mean whereas horizontal and vertical lines represent standard deviation.
IVE
p-value
Arousal Valenc e
1 2 0.052 10–6 (***)
1 3 0.195 0.152
140.007 (**)10–9 (***)
2310–5 (***)0.015 (*)
2410–8 (***) 0.068
3 4 0.606 10–7 (***)
Table 5. Signication test of the self-assessment of the emotional rooms.
Arousal High Low
High 82.72 17.28
Low 37.25 62.75
Table 6. Confusion matrix of cross-validation using SVM classier for arousal level. Values are expressed as
percentages. Total Accuracy: 75.00%.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
10
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
the Samsung Gear, therefore increasing the possible applications of the methodology. High quality images of the
stimuli are included in the supplementary material. is represents a new tool that can contribute in the eld
of psychology, in general, and in the aective computing eld, in particular, fostering the development of novel
immersive aective elicitation using IVEs.
Figure 8. Recognition accuracy of arousal in cross-validation as a function of the feature rank estimated
through the SVM-RFE procedure.
Rank Feature
1EEG MPC PCA 8
2EEG MPC PCA 9
3EEG MPC PCA 11
4EEG MPC PCA 10
5EEG MPC PCA 7
6EEG MPC PCA 12
7EEG Band Power PCA 3
8EEG Band Power PCA 1
9HRV PCA 1
10 EEG Band Power PCA 4
11 EEG Band Power PCA 2
12 HRV PCA 3
13 EEG MPC PCA 4
14 HRV PCA 2
15 EEG MPC PCA 5
Table 7. Selected features ordered by their median rank over every fold computed during the LOSO procedure
for arousal classication.
Arousal High Low
High 75.00 25.00
Low 33.33 66.67
Table 8. Confusion matrix of test set using SVM classier for arousal level. Values are expressed as percentages.
Total Accuracy: 70.00%.
Val e n ce Positive Negative
Positive 71.62 28.38
Negative 29.31 70.69
Table 9. Confusion matrix of cross-validation using SVM classier for valence level. Values are expressed as
percentages. Total Accuracy: 71.21%.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
11
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
ere are still some topics that need to be researched, relating to the capacity of the IVE display formats, to
ensure that they evoke the same emotions as real environments. Studies comparing display formats show that the
360° IVEs oer results closer to reality, according to the participants’ psychological responses, and 3D IVEs do so
according to their physiological responses54. Moreover, it is quite possible that IVEs will oer the best solutions
at both psychological and physiological levels as they become even more realistic, providing a real improvement
not only at the visual and auditory levels but also at the haptic98. In addition, 3D IVEs allow users to navigate
and interact with the environment. Hence, there are reasons to think that they could be powerful tools for devel-
oping applications for aective computing, but studies comparing human responses in real and simulated IVE
are scarce99101, especially regarding emotional responses; these studies are required. Moreover, every year the
resolution of Head Mounted Displays is upgraded, which brings them closer to eye resolution. us, it is possible
that in some years the advances in Virtual Reality hardware will make the present methodology more powerful.
In addition, works comparing VR devices with dierent levels of immersion are needed in order to give research-
ers the best set-ups to achieve their aims. In future works, we need to consider all these topics to improve the
methodology.
Regarding the emotion recognition system, we present the rst study that develops an emotion recognition
system using a set of IVEs as a stimulus elicitation and proper analyses of physiological dynamics. e accuracy
of the model was 75.00% along the arousal dimension and 71.21% along the valence dimension in the phase
of cross-validation, with average of 70.00% along both dimensions in the test set. ey all present a balanced
confusion matrix. e accuracies are considerably higher than the chance level, which is 58% in brain signal
Figure 9. Recognition accuracy of valence in cross-validation as a function of the feature rank estimated
through the SVM-RFE procedure.
Rank Feature
1EEG MPC PCA 8
2EEG MPC PCA 6
3EEG MPC PCA 11
4EEG MPC PCA 7
5EEG MPC PCA 10
6EEG MPC PCA 12
7EEG MPC PCA 9
8EEG Band Power PCA 3
9EEG Band Power PCA 4
10 EEG MPC PCA 2
Table 10. Selected features ordered by their median rank over every fold computed during the LOSO procedure
for valence classication.
Val e n ce Positive Negative
Positive 75.00 25.00
Negative 37.50 62.50
Table 11. Confusion matrix of test set using SVM classier for valence level. Values are expressed as
percentages. Total Accuracy: 70.00%.
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
12
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
classication and statistical assessment (n = 152, 2-classes, p = 0.05)102. Although the accuracy is lower than other
studies of emotion recognition in images15 and sounds27, our results present a rst proof of concept that suggests
that it is possible to recognize the emotion of a subject elicited through an IVE. e research was developed with
a sample of 60 subjects, who were carefully screened to demonstrate agreement with a “standard” population
reported in the literature46. It should be noted that the possible overtting of the model was controlled using: (1)
a feature reduction strategy with a PCA; (2) a feature selection strategy using a SVM-RFE; (3) a rst validation of
the model using LOSO cross-validation; and (4) a test validation using 5 randomly chosen subjects (15%), who
had not been used before to train or perform the cross-validation of the model. In the arousal model, features
derived from three-signal analyses were selected: 3/3 of HRV, 4/4 of EEG BandPower and 8/12 of EEG MPC.
However, in the valence model only the EEG analysis was used: 0/3 of HRV, 2/4 of EEG BandPower and 8/12
EEG MPC. Moreover, in both models, the rst six features selected by RFE-SVM were derived from an EEG MPC
analysis. is suggests that cortical functional connectivity provides eective correlates of emotions in an IVE.
Furthermore, according to recent evidence22,103, the reliability of emotion recognition outside of the laboratory
environment is improved by wearables. In future experiments, these results could be optimized using further,
maybe multivariate signal analyses and alternative machine learning algorithms87. In addition, the design of new,
controlled IVEs that can increase the number of stimuli per subject, using more combinations of architectural
parameters (colour, illumination and geometry), should also improve the accuracy and robustness of the model.
In future studies, we will improve the set of stimuli presented including new IVEs in order to develop a large set
of validate IVE stimuli to be used in emotion research.
e ndings presented here mark a new step in the eld of aective computing and its applications. Firstly, the
methodology involved in itself a novel trial to overcome the limitations of passive methods of aective elicitation,
in order to recreate more realistic stimuli using 360° IVEs. Nevertheless, the long-term objective is to develop a
robust pre-calibrate model that could be applied in two ways: (1) in 3D environments that would allow the study
of emotional responses to “real” situations in a laboratory environment through VR simulation using HMD
devices and (2) in physical spaces. We hypothesize in both cases that the emotion recognition models developed
through controlled 360° IVEs will work better than the models calibrated by non-immersive stimuli, such as
IAPS. is approach will be discussed in future studies using stage 3 of the experimental protocol.
Regarding the implications for architecture, the methodology could be applied in two main contexts, research
and commercial. On the one hand, researchers could analyse and measure the impact of dierent design param-
eters on the emotional responses of potential users. is is especially important due to the impossibility of devel-
oping researches in real or laboratory environments (e.g. analysing arousal changes caused by the pavement
width on a street). e synergy of aective computing and virtual reality allows us to isolate a parameter design
and measure the emotional changes provoked by making changes to it, while keeping the rest of the environment
identical. is could improve the knowledge of the emotional impact that might be made by dierent design
parameters and, consequently, facilitate the development of better practices and relevant regulations. On the other
hand, this methodology could help architects and engineers in their decision-making processes for the design
of built environments before construction, aiding their evaluations and the selection of the options that might
maximize the mood that they want to evoke: for example, positive valence in a hotel room or a park, low arousal
in a schoolroom or in a hospital waiting room and high arousal in a shop or shopping centre. Nevertheless, these
ndings could be applied to any other eld that needs to quantify the emotional eects of spatial stimuli displayed
by Immersive Virtual Environments. Health, psychology, driving, videogames and education might all benet
from this methodology.
Data Availability
e datasets generated during and/or analysed during the current study are available from the corresponding
authors on reasonable request.
References
1. Picard, . W. Aective computing. (MIT press, 1997).
2. Picard, . W. Aective Computing: Challenges. Int. J. Hum. Comput. Stud. 59, 55–64 (2003).
3. Jerritta, S., Murugappan, M., Nagarajan, . & Wan, . Physiological signals based human emotion ecognition: a review. Signal
Process. its Appl. (CSPA), 2011 IEEE 7th Int. Colloq. 410–415, https://doi.org/10.1109/CSPA.2011.5759912 (2011).
4. Harms, M. B., Martin, A. & Wallace, G. L. Facial emotion recognition in autism spectrum disorders: A review of behavioral and
neuroimaging studies. Neuropsychol. ev. 20, 290–322 (2010).
5. oolagudi, S. G. & ao, . S. Emotion recognition from speech: A review. Int. J. Speech Technol. 15, 99–117 (2012).
6. Gross, J. J. & Levenson, . W. Emotion elicitation using lms. Cogn. Emot. 9, 87–108 (1995).
7. Lindal, P. J. & Hartig, T. Architectural variation, building height, and the restorative quality of urban residential streetscapes. J.
Environ. Psychol. 33, 26–36 (2013).
8. Ulrich, . View through a window may inuence recovery from surgery. Science (80-.). 224, 420–421 (1984).
9. Fernández-Caballero, A. et al. Smart environment architecture for emotion detection and regulation. J. Biomed. Inform. 64, 55–73
(2016).
10. Eman, P. Basic Emotions. Handboo of cognition and emotion 45–60, https://doi.org/10.1017/S0140525X0800349X (1999).
11. Posner, J., ussell, J. A. & Peterson, B. S. e circumplex model of aect: an integrative approach to aective neuroscience, cognitive
development, and psychopathology. Dev. Psychopathol. 17, 715–34 (2005).
12. ussell, J. A. & Mehrabian, A. Evidence for a three-factor theory of emotions. J. es. Pers. 11, 273–294 (1977).
13. Calvo, . A. & D’Mello, S. Aect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans.
Aect. Comput. 1, 18–37 (2010).
14. Valenza, G. et al. Combining electroencephalographic activity and instantaneous heart rate for assessing brain–heart dynamics
during visual emotional elicitation in healthy subjects. Philos. Trans. . Soc. A Math. Phys. Eng. Sci. 374, 20150176 (2016).
15. Valenza, G., Lanata, A. & Scilingo, E. P. e role of nonlinear dynamics in aective valence and arousal recognition. IEEE Trans.
Aect. Comput. 3, 237–249 (2012).
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
13
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
16. Valenza, G., Citi, L., Lanatá, A., Scilingo, E. P. & Barbieri, . evealing real-time emotional responses: a personalized assessment
based on heartbeat dynamics. Sci. ep. 4, 4998 (2014).
17. Valenza, G. et al. Wearable monitoring for mood recognition in bipolar disorder based on history-dependent long-term heart rate
variability analysis. IEEE J. Biomed. Heal. Informatics 18, 1625–1635 (2014).
18. Piwe, L., Ellis, D. A., Andrews, S. & Joinson, A. e ise of Consumer Health Wearables: Promises and Barriers. PLoS Med. 13,
1–9 (2016).
19. Xu, J., Mitra, S., Van Hoof, C., Yazicioglu, . & Mainwa, . A. A. Active Electrodes for Wearable EEG Acquisition: eview and
Electronics Design Methodology. IEEE ev. Biomed. Eng. 3333, 1–1 (2017).
20. umari, P., Mathew, L. & Syal, P. Increasing trend of wearables and multimodal interface for human activity monitoring: A review.
Biosens. Bioelectron. 90, 298–307 (2017).
21. He, C., Yao, Y. & Ye, X. An Emotion ecognition System Based on Physiological Signals Obtained by Wearable Sensors. In Wear abl e
Sensors and obots: Proceedings of International Conference on We ar able Sensors and obots 2015 (eds Yang, C., Vir, G. S. & Yang,
H.) 15–25. https://doi.org/10.1007/978-981-10-2404-7_2 (Springer Singapore, 2017).
22. Naisa, B., astgoo, M. N., Tjondronegoro, D. & Chandran, V. Evolutionary computation algorithms for feature selection of EEG-
based emotion recognition using mobile sensors. Expert Syst. Appl. 93, 143–155 (2018).
23. ory Jacqueline, D. & Sidney, . Aect Elicitation for Aective Computing. In e Oxford Handboo of Aective Computing
371–383 (2014).
24. Eman, P. e directed facial action tas. In Handboo of emotion elicitation and assessment 47–53 (2007).
25. Harmon-Jones, E., Amodio, D. M. & Zinner, L. . Social psychological methods of emotion elicitation. Handb. Emot. elicitation
Assess. 91–105, https://doi.org/10.2224/sbp.2007.35.7.863 (2007)
26. oberts, N. A., Tsai, J. L. & Coan, J. A. Emotion elicitation using dyadic interaction tas. Handboo of Emotion Elicitation and
Assessment 106–123 (2007).
27. Nardelli, M., Valenza, G., Greco, A., Lanata, A. & Scilingo, E. P. ecognizing emotions induced by aective sounds through heart
rate variability. IEEE Trans. Aect. Comput. 6, 385–394 (2015).
28. im, J. Emotion ecognition Using Speech and Physiological Changes. obust Speech ecognit. Underst. 265–280 (2007).
29. Soleymani, M., Pantic, M. & Pun, T. Multimodal emotion recognition in response to videos (Extended abstract). 2015 Int. Conf.
Aect. Comput. Intell. Interact. ACII 2015 3, 491–497 (2015).
30. Baños, . M. et al. Immersion and Emotion: eir Impact on the Sense of Presence. CyberPsychology Behav. 7, 734–741 (2004).
31. Giglioli, I. A. C., Pravettoni, G., Martín, D. L. S., Parra, E. & aya, M. A. A novel integrating virtual reality approach for the
assessment of the attachment behavioral system. Front. Psychol. 8, 1–7 (2017).
32. Marín-Morales, J., Torrecilla, C., Guixeres, J. & Llinares, C. Methodological bases for a new platform for the measurement of
human behaviour in virtual environments. DYNA 92, 34–38 (2017).
33. Vince, J. Introduction to virtual reality. (Media, Springer Science & Business, 2004).
34. Alcañiz, M., Baños, ., Botella, C. & ey, B. e EMMA Project: Emotions as a Determinant of Presence. PsychNology J. 1, 141–150
(2003).
35. Vecchiato, G. et al. Neurophysiological correlates of embodiment and motivational factors during the perception of virtual
architectural environments. Cogn. Process. 16, 425–429 (2015).
36. Slater, M. & Wilbur, S. A Framewor for Immersive Virtual Environments (FIVE): Speculations on the ole of Presence in Virtual
Environments. Presence Teleoperators Virtual Environ. 6, 603–616 (1997).
37. iva, G. et al. Aective Interactions Using Virtual eality: e Lin between Presence and Emotions. CyberPsychology Behav. 10,
45–56 (2007).
38. Baños, . M. et al Changing induced moods via virtual reality. In International Conference on Persuasive Technology (ed. Springer,
Berlin, H.) 7–15, https://doi.org/10.1007/11755494_3 (2006).
39. Baños, . M. et al. Positive mood induction procedures for virtual environments designed for elderly people. Interact. Comput. 24,
131–138 (2012).
40. Gorini, A. et al. Emotional esponse to Virtual eality Exposure across Dierent Cultures: e ole of the AttributionProcess.
CyberPsychology Behav. 12, 699–705 (2009).
41. Gorini, A., Capideville, C. S., De Leo, G., Mantovani, F. & iva, G. e ole of Immersion and Narrative in Mediated Presence: e
Virtual Hospital Experience. Cyberpsychology, Behav. Soc. Netw. 14, 99–105 (2011).
42. Chirico, A. et al. Eectiveness of Immersive Videos in Inducing Awe: An Experimental Study. Sci. ep. 7, 1–11 (2017).
43. Blascovich, J. et al. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychol. Inq. 7965,
103–124 (2012).
44. Peperorn, H. M., Alpers, G. W. & Mühlberger, A. Triggers of fear: Perceptual cues versus conceptual information in spider phobia.
J. Clin. Psychol. 70, 704–714 (2014).
45. McCall, C., Hildebrandt, L. ., Bornemann, B. & Singer, T. Physiophenomenology in retrospect: Memory reliably reflects
physiological arousal during a prior threatening experience. Conscious. Cogn. 38, 60–70 (2015).
46. Hildebrandt, L. ., Mccall, C., Engen, H. G. & Singer, T. Cognitive exibility, heart rate variability, and resilience predict ne-
grained regulation of arousal during prolonged threat. Psychophysiology 53, 880–890 (2016).
47. Notzon, S. et al. Psychophysiological eects of an iTBS modulated virtual reality challenge including participants with spider
phobia. Biol. Psychol. 112, 66–76 (2015).
48. Amaral, C. P., Simões, M. A., Mouga, S., Andrade, J. & Castelo-Branco, M. A novel Brain Computer Interface for classication of
social joint attention in autism and comparison of 3 experimental setups: A feasibility study. J. Neurosci. Methods 290, 105–115
(2017).
49. Eudave, L. & Valencia, M. Physiological response while driving in an immersive virtual environment. 2017 IEEE 14th Int. Conf.
Wearable Implant. Body Sens. Networs 145–148, https://doi.org/10.1109/BSN.2017.7936028 (2017).
50. Sharma, G. et al. Inuence of landmars on waynding and brain connectivity in immersive virtual reality environment. Front.
Psychol. 8, 1–12 (2017).
51. Bian, Y. et al. A framewor for physiological indicators of ow in V games: construction and preliminary evaluation. Pers.
Ubiquitous Comput. 20, 821–832 (2016).
52. Egan, D. et al. An evaluation of Heart ate and Electrodermal Activity as an Objective QoE Evaluation method for Immersive
Virtual eality Environments. 3–8, https://doi.org/10.1109/QoMEX.2016.7498964 (2016).
53. Meehan, M., azzaque, S., Inso, B., Whitton, M. & Broos, F. P. eview of four studies on the use of physiological reaction as a
measure of presence in stressful virtual environments. Appl. Psychophysiol. Biofeedbac 30, 239–258 (2005).
54. Higuera-Trujillo, J. L., López-Tarruella Maldonado, J. & Llinares Millán, C. Psychological and physiological human responses to
simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual eality. Appl. Ergon. 65,
398–409 (2016).
55. Felnhofer, A. et al. Is virtual reality emotionally arousing? Investigating ve emotion inducing virtual par scenarios. Int. J. Hum.
Comput. Stud. 82, 48–56 (2015).
56. Anderson, A. P. et al. elaxation with Immersive Natural Scenes Presented Using Virtual eality. Aerosp. Med. Hum. Perform. 88,
520–526 (2017).
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
14
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
57. Higuera, J. L. et al. Emotional cartography in design: A novel technique to represent emotional states altered by spaces. In D and E
2016: 10th International Conference on Design and Emotion 561–566 (2016).
58. roene, ., Spitzer, . L. & Williams, J. B. W. e PHQ-9: Validity of a brief depression severity measure. J. Gen. Intern. Med. 16,
606–613 (2001).
59. Bradley, M. M. & Lang, P. J. Measuring emotion: e self-assessment maniin and the semantic dierential. J. Behav. er. Exp.
Psychiatry 25, 49–59 (1994).
60. Lang, P. J., Bradley, M. M. & Cuthbert, B. N. International Aective Picture System (IAPS): Technical Manual and Aective atings.
NIMH Cent. Study Emot. Atten. 39–58, https://doi.org/10.1027/0269-8803/a000147 (1997).
61. Nanda, U., Pati, D., Ghamari, H. & Bajema, . Lessons from neuroscience: form follows function, emotions follow form. Intell.
Build. Int. 5, 61–78 (2013).
62. ussell, J. A. A circumplex model of aect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980).
63. Sejima, . azuyo Sejima. 1988–1996. El Croquis 15 (1996).
64. Ochiai, H. et al. Physiological and Psychological Eects of Forest erapy on Middle-Aged Males with High-NormalBlood
Pressure. Int. J. Environ. es. Public Health 12, 2532–2542 (2015).
65. Noguchi, H. & Saaguchi, T. Eect of illuminance and color temperature on lowering of physiological activity. Appl. Hum. Sci. 18,
117–123 (1999).
66. üller, ., Miellides, B. & Janssens, J. Color, arousal, and performance—A comparison of three experiments. Color es. Appl. 34,
141–152 (2009).
67. Yildirim, ., Hidayetoglu, M. L. & Capanoglu, A. Eects of interior colors on mood and preference: comparisons of two living
rooms. Percept. Mot. Sills 112, 509–524 (2011).
68. Hogg, J., Goodman, S., Porter, T., Miellides, B. & Preddy, D. E. Dimensions and determinants of judgements of colour samples and
a simulated interior space by architects and nonarchitects. Br. J. Psychol. 70, 231–242 (1979).
69. Jalil, N. A., Yunus, . M. & Said, N. S. Environmental Colour Impact upon Human Behaviour: A eview. Procedia - Soc. Behav. Sci.
35, 54–62 (2012).
70. Jacobs, . W. & Hustmyer, F. E. Eects of four psychological primary colors on GS, heart rate and respiration rate. Percept. Mot.
Sills 38, 763–766 (1974).
71. Jin, H. ., Yu, M., im, D. W., im, N. G. & Chung, A. S. W. Study on Physiological esponses to Color Stimulation. In International
Association of Societies of Design esearch (ed. Poggenpohl, S.) 1969–1979 (orean Society of Design Science, 2009).
72. Vartanian, O. et al. Impact of contour on aesthetic judgments and approach-avoidance decisions in architecture. Proc. Natl. Acad.
Sci. 110, 1–8 (2013).
73. Tsunetsugu, Y., Miyazai, Y. & Sato, H. Visual eects of interior design in actual-size living rooms on physiological responses.
Build. Environ. 40, 1341–1346 (2005).
74. Stamps, A. E. Physical Determinants of Preferences for esidential Facades. Environ. Behav. 31, 723–751 (1999).
75. Berlyne, D. E. Novelty, Complexity, and Hedonic Value. Percept. Psychophys. 8, 279–286 (1970).
76. rueger, . A. & Casey, M. Focus groups: a practical guide for applied research. (Sage Publications, 2000).
77. Acharya, U. ., Joseph, . P., annathal, N., Lim, C. M. & Suri, J. S. Heart rate variability: A review. Med. Biol. Eng. Comput. 44,
1031–1051 (2006).
78. Tarvainen, M. P., Nisanen, J. P., Lipponen, J. A., anta-aho, P. O. & arjalainen, P. A. ubios HV - Heart rate variability analysis
soware. Comput. Methods Programs Biomed. 113, 210–220 (2014).
79. Pan, J. & Tompins, W. J. A real-time QS detection algorithm. Biomed. Eng. IEEE Trans. 1, 230–236 (1985).
80. Tarvainen, M. P., anta-aho, P. O. & arjalainen, P. A. An advanced detrending method with application to HV analysis. IEEE
Trans. Biomed. Eng. 49, 172–175 (2002).
81. Valenza, G. et al. Predicting Mood Changes in Bipolar Disorder rough HeartbeatNonlinear Dynamics. IEEE J. Biomed. Heal.
Informatics 20, 1034–1043 (2016).
82. Pincus, S. & Viscarello, . Approximate Entropy A regularity measure for fetal heart rate analysis. Obstet. Gynecol. 79, 249–255
(1992).
83. ichman, J. & Moorman, J. Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Hear.
Circ Physiol 278, H2039–H2049 (2000).
84. Peng, C.-., Havlin, S., Stanley, H. E. & Goldberger, A. L. Quantication of scaling exponents and crossover phenomena in
nonstationary heartbeat time series. Chaos 5, 82–87 (1995).
85. Grassberger, P. & Procaccia, I. Characterization of strange attractors. Phys. ev. Lett. 50, 346–349 (1983).
86. Delorme, A. & Maeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent
component analysis. J. Neurosci. Methods 134, 9–21 (2004).
87. Colomer Granero, A. et al. A Comparison of Physiological Signal Analysis Techniques and Classiers for Automatic Emotional
Evaluation of Audiovisual Contents. Front. Comput. Neurosci. 10, 1–14 (2016).
88. ober, S. E., urzmann, J. & Neuper, C. Cortical correlate of spatial presence in 2D and 3D interactive virtual reality: An EEG
study. Int. J. Psychophysiol. 83, 365–374 (2012).
89. Hyvärinen, A. & Oja, E. Independent component analysis: Algorithms and applications. Neural Networs 13, 411–430 (2000).
90. Welch, P. D. e Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Aver. aging Over
Short, Modied Periodograms. IEEE Trans. AUDIO Electroacoust. 15, 70–73 (1967).
91. Mormann, F., Lehnertz, ., David, P. & Elger, E. C. Mean phase coherence as a measure for phase synchronization and its
application to the EEG of epilepsy patients. Phys. D Nonlinear Phenom. 144, 358–369 (2000).
92. Jollie, I. T. Principal Component Analysis, Second Edition. Encycl. Stat. Behav. Sci. 30, 487 (2002).
93. Schöllopf, B., Smola, A. J., Williamson, . C. & Bartlett, P. L. New support vector algorithms. Neural Comput 12, 1207–1245
(2000).
94. Yan, . & Zhang, D. Feature selection and analysis on correlated gas sensor data with recursive feature elimination. Sensors
Actuators, B Chem. 212, 353–363 (2015).
95. Chang, C.-C. & Lin, C.-J. Libsvm: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 1–27 (2011).
96. Lewis, P. A., Critchley, H. D., otshtein, P. & Dolan, . J. Neural correlates of processing valence and arousal in aective words.
Cereb. Cortex 17, 742–748 (2007).
97. McCall, C., Hildebrandt, L. ., Hartmann, ., Baczowsi, B. M. & Singer, T. Introducing the Wunderammer as a tool for emotion
research: Unconstrained gaze and movement patterns in three emotionally evocative virtual worlds. Comput. Human Behav. 59,
93–107 (2016).
98. Blae, J. & Guroca, H. B. Haptic glove with M braes for virtual reality. IEEE/ASME Trans. Mechatronics 14, 606–615 (2009).
99. Heydarian, A. et al. Immersive virtual environments versus physical built environments: A benchmaring study for building design
and user-built environment explorations. Autom. Constr. 54, 116–126 (2015).
100. uliga, S. F., rash, T., Dalton, . C. & Hölscher, C. Virtual reality as an empirical research tool - Exploring user experience in a
real building and a corresponding virtual model. Comput. Environ. Urban Syst. 54, 363–375 (2015).
101. Yeom, D., Choi, J.-H. & Zhu, Y. Investigation of the Physiological Dierences between Immersive Virtual Environment and Indoor
Enviorment in a Building. Indoor adn Built Enviornment 0, Accept (2017).
Content courtesy of Springer Nature, terms of use apply. Rights reserved
www.nature.com/scientificreports/
15
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
102. Combrisson, E. & Jerbi, . Exceeding chance level by chance: e caveat of theoretical chance levels in brain signal classication
and statistical assessment of decoding accuracy. J. Neurosci. Methods 250, 126–136 (2015).
103. He, C., Yao, Y. & Ye, X. An Emotion ecognition System Based on Physiological Signals Obtained by Wearable Sensors. In Wea rab le
Sensors and obots: Proceedings of International Conference on We ar able Sensors and obots 2015 (eds. Yang, C., Vir, G. S. & Yang,
H.) 15–25, https://doi.org/10.1007/978-981-10-2404-7_2 (Springer Singapore, 2017).
Acknowledgements
is work was supported by the Ministerio de Economía y Competitividad. Spain (Project TIN2013-45736-R).
Author Contributions
J.M., J.H., J.G., C.L., and M.A. devised the methodology. J.M., J.H., J.G. and C.L. dened the experimental setup
and acquired the experimental data. J.M., A.G., P.S. and G.V. processed and analysed data. All the authors co-
wrote the manuscript and approved the nal text.
Additional Information
Supplementary information accompanies this paper at https://doi.org/10.1038/s41598-018-32063-4.
Competing Interests: e authors declare no competing interests.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and
institutional aliations.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International
License, which permits use, sharing, adaptation, distribution and reproduction in any medium or
format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Cre-
ative Commons license, and indicate if changes were made. e images or other third party material in this
article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the
material. If material is not included in the article’s Creative Commons license and your intended use is not per-
mitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the
copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
© e Author(s) 2018
Content courtesy of Springer Nature, terms of use apply. Rights reserved
1.
2.
3.
4.
5.
6.
Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:
use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
use bots or other automated methods to access the content or redirect messages
override any security feature or exclusionary protocol; or
share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at
onlineservice@springernature.com
... Seo et al.'s study used only one sensor device; however, a key aspect of context-aware VR systems is to support data collection from diverse sensor sources because this way the system can be more aware of the user's context [6]. For example, researchers have connected wearable physiological sensors related to heart and brain activity to a virtual environment to collect physiological data to be used in a VR experience [7,8]. In previous context-aware VR systems, the method of collecting data from sensors has typically been tightly integrated into a VR project so that it is not easy to adapt the same data collection method to other projects [2,7]. ...
... For example, researchers have connected wearable physiological sensors related to heart and brain activity to a virtual environment to collect physiological data to be used in a VR experience [7,8]. In previous context-aware VR systems, the method of collecting data from sensors has typically been tightly integrated into a VR project so that it is not easy to adapt the same data collection method to other projects [2,7]. Consequently, a context data collection framework that can be easily harnessed in various types of virtual environments is needed to speed up the development of context-aware VR applications. ...
... To the best of our knowledge, this study marks the first occasion of proposing a reusable context data collection framework for the purpose of context-aware VR application development. The previous studies on context-aware VR applications that we reviewed [2,5,7,8,16,17] utilized systems that were built for a purpose specific to the research objective. Additionally, these studies did not report any reusable and extensible method or technique for acquiring rich context data that could be then used for context-aware application development. ...
Article
Full-text available
Virtual Reality (VR) has been adopted as a leading technology for the metaverse, yet most previous VR systems provide one-size-fits-all experiences to users. Context-awareness in VR enables personalized experiences in the metaverse, such as improved embodiment and deeper integration of the real world and virtual worlds. Personalization requires context data from diverse sources. We proposed a reusable and extensible context data collection framework, ManySense VR, which unifies data collection from diverse sources for VR applications. ManySense VR was implemented in Unity based on extensible context data managers collecting data from data sources such as an eye tracker, electroencephalogram, pulse, respiration, galvanic skin response, facial tracker, and Open Weather Map. We used ManySense VR to build a context-aware embodiment VR scene where the user’s avatar is synchronized with their bodily actions. The performance evaluation of ManySense VR showed good performance in processor usage, frame rate, and memory footprint. Additionally, we conducted a qualitative formative evaluation by interviewing five developers (two males and three females; mean age: 22) after they used and extended ManySense VR. The participants expressed advantages (e.g., ease-of-use, learnability, familiarity, quickness, and extensibility), disadvantages (e.g., inconvenient/error-prone data query method and lack of diversity in callback methods), future application ideas, and improvement suggestions that indicate potential and can guide future development. In conclusion, ManySense VR is an efficient tool for researchers and developers to easily integrate context data into their Unity-based VR applications for the metaverse.
... Compared with 2D stimuli, immersive virtual reality simulates the environment in a realistic and possibly interactive modality 5 , ameliorates the spatial perception 19 , and can generate neurophysiological responses as a real scenario 20,21 . Nowadays, virtual reality is considered a powerful device for evoking emotions in laboratory settings [22][23][24][25] compared to the presentation of passive stimuli 26 . Indeed, virtual reality potentiates emotion and attention processes compared to conventional 2D stimulation, ensuring a deeper emotional experience [27][28][29] . ...
... Indeed, virtual reality potentiates emotion and attention processes compared to conventional 2D stimulation, ensuring a deeper emotional experience [27][28][29] . Virtual reality has also been exploited in architecture to investigate the occupant emotions exposed to different architectural features 22,30,31 . Also, immersive virtual environments generated similar behavioral outcomes compared to physically built environments [32][33][34] . ...
Article
Full-text available
The built environment represents the stage surrounding our everyday life activities. To investigate how architectural design impacts individuals' affective states, we measured subjective judgments of perceived valence (pleasant and unpleasant) and arousal after the dynamic experience of a progressive change of macro visuospatial dimensions of virtual spaces. To this aim, we developed a parametric model that allowed us to create 54 virtual architectural designs characterized by a progressive change of sidewalls distance, ceiling and windows height, and color of the environment. Decreasing sidewalls distance, ceiling height variation, and increasing windows height significantly affected the participants' emotional state within virtual environments. Indeed, such architectural designs generated high arousing and unpleasant states according to subjective judgment. Overall, we observed that valence and arousal scores are affected by all the dynamic form factors which modulated the spaciousness of the surrounding. Showing that the dynamic experience of virtual environments enables the possibility of measuring the emotional impact of macro spatial architectural features, the present findings may lay the groundwork for future experiments investigating the effects that the architectural design has on individuals' mental state as a fundamental factor for the creation of future spaces.
... For example, Bored is shown in the lower left quadrant of Figure 1, as it is an unpleasant, mild intensity affective state, while Astonished is shown in the upper right quadrant of Figure 1, as it is a pleasant, high intensity affective state. While VR research investigating valence and arousal dimensions of affect has increased in recent years (Marín-Morales et al., 2018), arousal is investigated more frequently than valence in head-mounted display (HMD) based VR experiences (Marín-Morales et al., 2020). ...
Article
Full-text available
Background: Numerous studies have investigated emotion in virtual reality (VR) experiences using self-reported data in order to understand valence and arousal dimensions of emotion. Objective physiological data concerning valence and arousal has been less explored. Electroencephalography (EEG) can be used to examine correlates of emotional responses such as valence and arousal in virtual reality environments. Used across varying fields of research, images are able to elicit a range of affective responses from viewers. In this study, we display image sequences with annotated valence and arousal values on a screen within a virtual reality theater environment. Understanding how brain activity responses are related to affective stimuli with known valence and arousal ratings may contribute to a better understanding of affective processing in virtual reality. Methods: We investigated frontal alpha asymmetry (FAA) responses to image sequences previously annotated with valence and arousal ratings. Twenty-four participants viewed image sequences in VR with known valence and arousal values while their brain activity was recorded. Participants wore the Oculus Quest VR headset and viewed image sequences while immersed in a virtual reality theater environment. Results: Image sequences with higher valence ratings elicited greater FAA scores than image sequences with lower valence ratings ( F [1, 23] = 4.631, p = 0.042), while image sequences with higher arousal scores elicited lower FAA scores than image sequences with low arousal ( F [1, 23] = 7.143, p = 0.014). The effect of valence on alpha power did not reach statistical significance ( F [1, 23] = 4.170, p = 0.053). We determined that only the high valence, low arousal image sequence elicited FAA which was significantly higher than FAA recorded during baseline ( t [23] = −3.166, p = 0.002), suggesting that this image sequence was the most salient for participants. Conclusion: Image sequences with higher valence, and lower arousal may lead to greater FAA responses in VR experiences. While findings suggest that FAA data may be useful in understanding associations between valence and arousal self-reported data and brain activity responses elicited from affective experiences in VR environments, additional research concerning individual differences in affective processing may be informative for the development of affective VR scenarios.
... Second, it is possible to apply the measure outside of the context of cybersecurity notifications and into other stimuli of interest, such as social media or other notification contexts. A past study by Marín-Morales et al. (2018) applied machine learning to detect EEG signals elicited by affective responses in a virtual reality environment in a paradigm that leveraged IAPS pictures. This suggests that the LPP signal could be applicable in applied, more ecologically valid contexts. ...
Article
Full-text available
Cybersecurity notifications play an important role in encouraging users to use computers safely. Emotional reactions to such notifications are known to positively influence users’ adherence to these notifications, though it is challenging for researchers to identify and quantify users’ emotional reactions. In this study, we explored electroencephalography (EEG) signals that were elicited by the presentation of various emotionally charged image stimuli provided by the International Affective Picture System (IAPS) and compared signals to those elicited by images of cybersecurity notifications and other computer-related stimuli. Participants provided behavioral assessments of valence and arousal elicited by the images which were used to cross-reference the results. We found that EEG amplitudes corresponding to the late positive potential (LPP) were elevated in reaction to images of cybersecurity notifications as well as IAPS images known to elicit strong positive and negative valence, when compared to neutral valence or other computer-related stimuli. These findings suggest that the LPP may account for emotional deliberation about cybersecurity notifications, which could be a useful measure when conducting future studies into the role such emotional reactions play in encouraging safe computer behavior.
... The EmojiGrid, for example, has been proposed as a self-report tool for the assessment of VRevoked emotions [116]. The interested reader is referred to [117][118][119] for more details on user emotional/affective state assessment in VR. ...
Article
Full-text available
Virtual reality (VR) applications, especially those where the user is untethered to a computer, are becoming more prevalent as new hardware is developed, computational power and artificial intelligence algorithms are available, and wireless communication networks are becoming more reliable, fast, and providing higher reliability. In fact, recent projections show that by 2022 the number of VR users will double, suggesting the sector was not negatively affected by the worldwide COVID-19 pandemic. The success of any immersive communication system is heavily dependent on the user experience it delivers, thus now more than ever has it become crucial to develop reliable models of immersive media experience (IMEx). In this paper, we survey the literature for existing methods and tools to assess human influential factors (HIFs) related to IMEx. In particular, subjective, behavioural, and psycho-physiological methods are covered. We describe tools available to monitor these HIFs, including the user’s sense of presence and immersion, cybersickness, and mental/affective states, as well as their role in overall experience. Special focus is placed on psycho-physiological methods, as it was found that such in-depth evaluation was lacking from the existing literature. We conclude by touching on emerging applications involving multiple-sensorial immersive media and provide suggestions for future research directions to fill existing gaps. It is hoped that this survey will be useful for researchers interested in building new immersive (adaptive) applications that maximize user experience.
Article
Although recent developments in VR tracking technology have enabled VR to become more portable and convenient, VR must overcome limited traversable space to be used for more general purposes. Since future VR HMDs aim to embed bio-sensors, such as eye-tracking and electroencephalography (EEG), the present study investigates how bio-sensors could be integrated into VR locomotion and how usability could be improved. In this paper, teleportation methods were developed by combining eye-tracking, EEG, and hand-tracking for location targeting and teleport triggering processes. In a static teleportation task, we compared the efficiency, accuracy, and usability of each method as well as the interactions between different combinations of methods. The experimental results verified that these locomotion methods were overall suitable for hands-free VR contents, and revealed relative superiority of eye-tracking for location targeting and hand-tracking for teleport triggering. Importantly, we identified specific combinations of locomotion methods appropriate for context by showing the interactions between experimental factors, and components that require improvement for future application. The results of this study contribute to development and understanding of novel VR locomotion methods utilizing bio-signal data, for more efficient and natural VR locomotion experience.
Chapter
Marketing scholars and practitioners see extended reality (XR0 as a new tool for both producing optimal consumer experiences and studying consumer behavior aspects. This chapter will explore the use of XR technologies for consumer behavior research. It analyzes XR as a very promising tool to examine various customer behavioral patterns in dynamic, complex, and realistic situations that will enhance our knowledge of new models of buyer–product and buyer–seller relationships. First, it will provide an updated definition of t he consumer shopping experience mediated by highly immersive virtual technology. Moreover, it will discuss how the experience of virtual reality affects human emotions. Finally, it will propose a framework that will allow the creation of controlled laboratory situations to study the factors that affect the acceptability of new products and retail spaces and the influence that the different elements that surround consumers have on their decisions.
Article
Virtual reality (VR) technology offers a great opportunity to explore stress disorder therapies. We created a VR stress training system, which incorporates three highly interactive stressful scenes to elicit stress, and demonstrate the concurrent variations between physiological data (heart rate, electrodermal activity and eye-blink rate) and self-reported stress ratings through a self-designed customized perceived stress questionnaire (SSAI) and wearable devices. Several supervised learning models were rigorously applied to automate stress recognition. Our findings include the evaluations of the VR system by computing Cronbach's alpha (α=0.72) and Kaiser-Meyer-Olkin (KMO) coefficient (η=0.78) through a retrospective survey, which were subsequently confirmed as reliable on four aspects (sense of presence, sense of space, sense of immersion and sense of reality) via factor analysis. Additionally, we demonstrate the effectiveness of physiology-based stress level classification (no stress, low stress and high stress) and continuous SSAI score prediction, with accuracy reaching 0.742 by bagging ensemble learning model and goodness-of-fit reaching 0.44 via multivariate stepwise regression. This study provides detailed insight regarding the effect of objective physiological measures on the validation of subjective self-ratings under a novel complex VR stress training system, which stimulates the further investigations of stress disorder recognition and treatment.
Chapter
Learners with Autism Spectrum Condition (ASC) are often characterized as having deficits in core social competencies affecting ability to communicate and interact with other people across multiple contexts. However, such deficit views of ASC lead to narrow interventions that cannot be generalized to wider learning contexts nor the real world. Taking a non-deficit view can offer a means of broadening understanding of ASC by taking a more holistic view of an individual. Thoughtfully designed VR solutions that take a non-deficit view can provide learning tools allow learners to develop their self-determination awareness and skills such as decision-making, self-advocacy, reflective problem-solving. Furthermore, through the use of machine learning approaches such solutions have the potential to provide real-time monitoring of learners during interaction offering a holistic understanding of individuals with ASC. This position paper sets out the support case for utilizing VR through non-deficit models and examines the role machine learning can play in further understanding of learners with ASC. It is intended that this sets out a case for future research directions.
Conference Paper
Full-text available
Bio-signals as evaluation methods have been increasingly used for design and architectural research over the last few years. Understanding how environmental factors influence people's emotions by using psychophysiological tools represents an opportunity to improve design processes. As a part of an extensive exploration toward emotional maps, EDA signals were used. This preliminary work presents a study in which 12 participants explored an Immersive Virtual Environment (IVE) using a Head-Mounted Display while their sympathetic reaction was being registered. The response to the IVE (3 rooms designed to evoke neutrality, stress and calm states) was measured using both phasic electrodermal activity data gathered during IVE's exploration and psychometric response collected by post-questionnaire. Results show that the IVE evoked the expected emotional states to be measured, EDA-phasic was an appropriate tool to measure emotional states and the "emotional map" created by superimposing an emotional heatmap on the architectural layout confirms that it is possible to represent emotional states altered by spaces graphically.
Article
Full-text available
Spatial navigation is influenced by landmarks, which are prominent visual features in the environment. Although previous research has focused on finding advantages of landmarks on wayfinding via experimentation; however, less attention has been given to identifying the key attributes of landmarks that facilitate wayfinding, including the study of neural correlates (involving electroencephalogram, EEG analyses). In this paper, we combine behavioural measures, virtual environment, and EEG signal-processing to provide a holistic investigation about the influence of landmarks on performance during navigation in a maze-like environment. In an experiment, participants were randomly divided into two conditions, Landmark-enriched (LM+; N = 17) and Landmark-devoid (LM-; N = 18), and asked to navigate from an initial location to a goal location in a maze. In the LM+ condition, there were landmarks placed at certain locations in the maze, which participants could use for wayfinding. However, in the LM-condition, such landmarks were not present. Beyond behavioural analyses of data, analyses were carried out of the EEG data collected using a 64-channel device. Results revealed that participants took less time and committed less errors in navigating the maze in the LM+ condition compared to the LM-condition. EEG analyses of the data revealed that the left-hemispheric activation was more prominent in the LM+ condition compared to the LM-condition. The event-related desynchronization/synchronization (ERD/ERS) of theta-frequency band, revealed activation in the left-posterior inferior and superior regions in the LM+ condition compared to the LM-condition, suggesting an occurrence of an object-location binding in the LM+ condition along with spatial transformation between representations. Moreover, directed transfer function (DTF) method, which measures information flow between two regions, showed more number of active channels in the LM-condition compared to the LM+ condition, exhibiting additional wiring cost associated with the cognitive demands when no landmark was available. These findings reveal pivotal role of the left-hemispheric region (especially, parietal cortex), which indicates the integration of available sensory cues and current memory requirements to encode contextual information of landmarks. Overall, this research helps to understand the role of brain regions and processes that are utilized when people use landmarks in navigating maze-like environments.
Article
Full-text available
Virtual reality (VR) technology represents a novel and powerful tool for behavioral research in psychological assessment. VR provides simulated experiences able to create the sensation of undergoing real situations. Users become active participants in the virtual environment seeing, hearing, feeling, and actuating as if they were in the real world. Currently, the most psychological VR applications concern the treatment of various mental disorders but not the assessment, that it is mainly based on paper and pencil tests. The observation of behaviors is costly, labor-intensive, and it is hard to create social situations in laboratory settings, even if the observation of actual behaviors could be particularly informative. In this framework, social stressful experiences can activate various behaviors of attachment for a significant person that can help to control and soothe them to promote individual’s well-being. Social support seeking, physical proximity, and positive and negative behaviors represent the main attachment behaviors that people can carry out during experiences of distress. We proposed VR as a novel integrating approach to measure real attachment behaviors. The first studies on attachment behavioral system by VR showed the potentiality of this approach. To improve the assessment during the VR experience, we proposed virtual stealth assessment (VSA) as a new method. VSA could represent a valid and novel technique to measure various psychological attributes in real-time during the virtual experience. The possible use of this method in psychology could be to generate a more complete, exhaustive, and accurate individual’s psychological evaluation.
Article
Full-text available
INTRODUCTION: Virtual reality (VR) can provide exposure to nature for those living in isolated confined environments. We evaluated VR-presented natural settings for reducing stress and improving mood. METHODS: There were 18 participants (9 men, 9 women), ages 32 ± 12 yr, who viewed three 15-min 360° scenes (an indoor control, rural Ireland, and remote beaches). Subjects were mentally stressed with arithmetic before scenes. Electrodermal activity (EDA) and heart rate variability measured psycho-physiological arousal. The Positive and Negative Affect Schedule and the 15-question Modified Reality Judgment and Presence Questionnaire (MRJPQ) measured mood and scene quality. RESULTS: Reductions in EDA from baseline were greater at the end of the natural scenes compared to the control scene (−0.59, −0.52, and 0.32 μS, respectively). The natural scenes reduced negative affect from baseline ( 1.2 and 1.1 points), but the control scene did not ( 0.4 points). MRJPQ scores for the control scene were lower than both natural scenes (4.9, 6.7, and 6.5 points, respectively). Within the two natural scenes, the preferred scene reduced negative affect ( 2.4 points) more than the second choice scene ( 1.8 points) and scored higher on the MRJPQ (6.8 vs. 6.4 points). DISCUSSION: Natural scene VR provided relaxation both objectively and subjectively, and scene preference had a significant effect on mood and perception of scene quality. VR may enable relaxation for people living in isolated confined environments, particularly when matched to personal preferences. Anderson AP, Mayer MD, Fellows AM, Cowan DR, Hegel MT, Buckey JC. Relaxation with immersive natural scenes presented using virtual reality. Aerosp Med Hum Perform. 2017; 88(6):520526.
Article
Full-text available
Awe, a complex emotion composed by the appraisal components of vastness and need for accommodation, is a profound and often meaningful experience. Despite its importance, psychologists have only recently begun empirical study of awe. At the experimental level, a main issue concerns how to elicit high intensity awe experiences in the lab. To address this issue, Virtual Reality (VR) has been proposed as a potential solution. Here, we considered the highest realistic form of VR: immersive videos. 42 participants watched at immersive and normal 2D videos displaying an awe or a neutral content. After the experience, they rated their level of awe and sense of presence. Participants’ psychophysiological responses (BVP, SC, sEMG) were recorded during the whole video exposure. We hypothesized that the immersive video condition would increase the intensity of awe experienced compared to 2D screen videos. Results indicated that immersive videos significantly enhanced the self-reported intensity of awe as well as the sense of presence. Immersive videos displaying an awe content also led to higher parasympathetic activation. These findings indicate the advantages of using VR in the experimental study of awe, with methodological implications for the study of other emotions.
Article
There is currently no standard or widely accepted subset of features to effectively classify different emotions based on electroencephalogram (EEG) signals. While combining all possible EEG features may improve the classification performance, it can lead to high dimensionality and worse performance due to redundancy and inefficiency. To solve the high-dimensionality problem, this paper proposes a new framework to automatically search for the optimal subset of EEG features using evolutionary computation (EC) algorithms. The proposed framework has been extensively evaluated using two public datasets (MAHNOB, DEAP) and a new dataset acquired with a mobile EEG sensor. The results confirm that EC algorithms can effectively support feature selection to identify the best EEG features and the best channels to maximize performance over a four-quadrant emotion classification problem. These findings are significant for informing future development of EEG-based emotion classification because low-cost mobile EEG sensors with fewer electrodes are becoming popular for many new applications.
Article
In the domain of building science and architectural design, the immersive virtual environment is being commonly adopted due to its convenience and cost-effectiveness, especially for research relevant to occupant behaviour in a building indoor environmental control. The goal of this study is to investigate whether such an immersive virtual environment condition could affect an occupant's thermal sensation and physiological response to ambient conditions differently, as compared to a real indoor environment, even though those two thermal conditions are the same or very similar. A series of human subject experiments using 18 participants was conducted in an environmental chamber. While thermal conditions were controlled at 20℃ to 30℃ in each environment, respectively, participants were asked to periodically report their thermal sensations on their body. Their heart rates were also continuously measured. The result of our experiments revealed that overall thermal sensations on the whole and local body areas showed some significant differences between the indoor environment and immersive virtual environment conditions during the same thermal conditions. Also, the heart rate difference between two environmental conditions was statistically significant at every thermal sensation level. These findings support the idea that significant physiological response differences could be affected by the immersive virtual environment condition.
Article
Background: We present a novel virtual-reality P300-based Brain Computer Interface (BCI) paradigm using social cues to direct the focus of attention. We combined interactive immersive virtual-reality (VR) technology with the properties of P300 signals in a training tool which can be used in social attention disorders such as autism spectrum disorder (ASD). New method: We tested the novel social attention training paradigm (P300-based BCI paradigm for rehabilitation of joint-attention skills) in 13 healthy participants, in 3 EEG systems. The more suitable setup was tested online with 4 ASD subjects. Statistical accuracy was assessed based on the detection of P300, using spatial filtering and a Naïve-Bayes classifier. Results: We compared: 1 - g.Mobilab+ (active dry-electrodes, wireless transmission); 2 - g.Nautilus (active electrodes, wireless transmission); 3 - V-Amp with actiCAP Xpress dry-electrodes. Significant statistical classification was achieved in all systems. g.Nautilus proved to be the best performing system in terms of accuracy in the detection of P300, preparation time, speed and reported comfort. Proof of concept tests in ASD participants proved that this setup is feasible for training joint attention skills in ASD. Comparison with existing methods: This work provides a unique combination of 'easy-to-use' BCI systems with new technologies such as VR to train joint-attention skills in autism. Conclusions: Our P300 BCI paradigm is feasible for future Phase I/II clinical trials to train joint-attention skills, with successful classification within few trials, online in ASD participants. The g.Nautilus system is the best performing one to use with the developed BCI setup.
Article
Psychological research into human factors frequently uses simulations to study the relationship between human behaviour and the environment. Their validity depends on their similarity with the physical environments. This paper aims to validate three environmental-simulation display formats: photographs, 360° panoramas, and virtual reality. To do this we compared the psychological and physiological responses evoked by simulated environments set-ups to those from a physical environment setup; we also assessed the users' sense of presence. Analysis show that 360° panoramas offer the closest to reality results according to the participants' psychological responses, and virtual reality according to the physiological responses. Correlations between the feeling of presence and physiological and other psychological responses were also observed. These results may be of interest to researchers using environmental-simulation technologies currently available in order to replicate the experience of physical environments.