Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors

Abstract

Affective Computing has emerged as an important field of study that aims to develop systems that can automatically recognize emotions. Up to the present, elicitation has been carried out with non-immersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for affective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were designed to elicit four possible arousal-valence combinations, as described in each quadrant of the Circumplex Model of Affects. An experiment involving the recording of the electroencephalography (EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features was extracted from these signals using various state-of-the-art metrics that quantify brain and cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classifier to predict the subject’s arousal and valence perception. The model’s accuracy was 75.00% along the arousal dimension and 71.21% along the valence dimension. Our findings validate the use of Immersive Virtual Environments to elicit and automatically recognize different emotional states from neural and cardiac dynamics; this development could have novel applications in fields as diverse as Architecture, Health, Education and Videogames.

Full text

1
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
www.nature.com/scientificreports
Aective computing in virtual
reality: emotion recognition from
brain and heartbeat dynamics using
wearable sensors
Javier Marín-Morales1, Juan Luis Higuera-Trujillo1, Alberto Greco
2, Jaime Guixeres1,
Carmen Llinares1, Enzo Pasquale Scilingo
2, Mariano Alcañiz
1 & Gaetano Valenza2
Aective Computing has emerged as an important eld of study that aims to develop systems that
can automatically recognize emotions. Up to the present, elicitation has been carried out with non-
immersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for
aective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were
designed to elicit four possible arousal-valence combinations, as described in each quadrant of the
Circumplex Model of Aects. An experiment involving the recording of the electroencephalography
(EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features
was extracted from these signals using various state-of-the-art metrics that quantify brain and
cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classier
to predict the subject’s arousal and valence perception. The model’s accuracy was 75.00% along the
arousal dimension and 71.21% along the valence dimension. Our ndings validate the use of Immersive
Virtual Environments to elicit and automatically recognize dierent emotional states from neural and
cardiac dynamics; this development could have novel applications in elds as diverse as Architecture,
Health, Education and Videogames.
Aective Computing (AfC) has emerged as an important eld of study in the development of systems that can
automatically recognize, model and express emotions. Proposed by Rosalind Picard in 1997, it is an interdiscipli-
nary eld based on psychology, computer science and biomedical engineering1. Stimulated by the fact that emo-
tions are involved in many background processes2 (such as perception, decision-making, creativity, memory, and
social interaction), several studies have focused on searching for a reliable methodology to identify the emotional
state of a subject by using machine learning algorithms.
us, AfC has emerged as an important research topic. It has been applied oen in education, healthcare,
marketing and entertainment3–6, but its potential is still under development. Architecture is a eld where AfC
has been infrequently applied, despite its obvious potential; the physical-environment has on a great impact, on
a daily basis, on human emotional states in general7, and on well-being in particular8. AfC could contribute to
improve building design to better satisfy human emotional demands9.
Irrespective of its application, Aective Computing involves both emotional classication and emotional elic-
itation. Regarding emotional classication, two approaches have commonly been proposed: discrete and dimen-
sional models. On the one hand, the former posits the existence of a small set of basic emotions, on the basis that
complex emotions result from a combination of these basic emotions. For example, Ekman proposed six basic
emotions: anger, disgust, fear, joy, sadness and surprise10. Dimensional models, on the other hand, consider a
multidimensional space where each dimension represents a fundamental property common to all emotions. For
example, the “Circumplex Model of Aects” (CMA)11 uses a Cartesian system of axes, with two dimensions,
proposed by Russell and Mehrabian12: valence, i.e., the degree to which an emotion is perceived as positive or
negative; and arousal, i.e., how strongly the emotion is felt.
1Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, València, Spain.
2Bioengineering and Robotics Research Centre E Piaggio & Department of Information Engineering, University of Pisa,
Pisa, Italy. Correspondence and requests for materials should be addressed to J.M.-M. (email: jamarmo@i3b.upv.es)
Received: 6 December 2017
Accepted: 10 August 2018
Published: xx xx xxxx
OPEN
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
2
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
In order to classify emotions automatically, correlates from, e.g., voice, face, posture, text, neuroimaging, and
physiological signals are widely used13. In particular, several computational methods are based on variables asso-
ciated with central nervous system (CNS) and autonomic nervous system (ANS) dynamics13. On the one hand,
the use of CNS is justied by the fact that human emotions originate in the cerebral cortex, involving several
areas in their regulation and feeling. In this sense, the electroencephalogram (EEG) is one of the techniques
most used to measure CNS responses14, also through the use of wearable devices. On the other hand, a wider
class of aective computing studies consider ANS changes elicited by specic emotional states. In this sense,
experimental results over the last three decades show that Heart Rate Variability (HRV) analyses can provide
unique and non-invasive assessments of autonomic functions on cardiovascular dynamics15,16. To this extent,
there has been a great increase over the last decade in research and commercial interest in wearable systems for
physiological monitoring. e key benets of these systems are their small size, lightness, low-power consump-
tion and, of course, their wearability17. e state of the art18–20 on wearable systems for physiological monitoring
highlight that: i) surveys predict that the demand for wearable devices will increase in the near future; ii) there
will be a need for more multimodal fusion of physiological signals in the near future; and iii) machine learning
algorithms can be merged with traditional approaches. Moreover, recent studies present promising results on
the development of emotion recognition systems through using wearable sensors instead of classic lab sensors,
through HRV21 and EEG22.
Regarding emotional elicitation, the ability to reliably and ethically elicit aective states in the laboratory is
a critical challenge in the process of the development of systems that can detect, interpret and adapt to human
aect23. Many methods of eliciting emotions have been developed to evoke emotional responses. Based on the
nature of the stimuli, two types of method are distinguished, the active and the passive. Active methods can
involve behavioural manipulation24, social psychological methods with social interaction25 and dyadic interac-
tion26. On the other hand, passive methods usually present images, sounds or lms. With respect to images, one
of the most prominent databases is the International Aective Picture System (IAPS), which includes over a thou-
sand depictions of people, objects and events, standardized on the basis of valence and arousal23. e IAPS has
been used in many studies as an elicitation tool in emotion recognition methodologies15. With respect to sound,
the most used database is the International Aective Digitalised Sound System (IADS)27. Some researchers also
use music or narrative to elicit emotions28. Finally, audio-visual stimuli, such as lms, are also used to induce
dierent levels of valence and arousal29.
Even when, as far we know, elicitation has been carried out with a non-immersive stimulus, it has been shown
that these passive methods have signicant limitations due to the importance of immersion for eliciting emotions
through the simulation of real experiences30. In the present, Virtual Reality (VR) represents a novel and power-
ful tool for behavioural research in psychological assessment. It provides simulated experiences that create the
sensation of being in the real world31,32. us, VR makes it possible to simulate and evaluate spatial environments
under controlled laboratory conditions32,33, allowing the isolation and modication of variables in a cost and
time eective manner, something which is unfeasible in real space34. During the last two decades VR has usually
been displayed using desktop PCs or semi-immersive systems such as CAVEs or Powerwalls35. Today, the use of
head-mounted displays (HMD) is increasing: these provide fully-immersive systems that isolate the user from
external world stimuli. ese provide a high degree of immersion, evoking a greater sense of presence, under-
stood as the perceptual illusion of non-mediation and a sense of “being-there”36. Moreover, the ability of VR to
induce emotions has been analysed in studies which demonstrate that virtual environments do evoke emotions in
the user34. Other works conrm that Immersive Virtual Environments (IVE) can be used as emotional induction
tools to create states of relaxation or anxiety37, basic emotions38,39, and to study the inuence of the users cultural
and technological background on emotional responses in VR40. In addition, some works show that emotional
content increases sense of presence in an IVE41 and that, faced with the same content, self-reported intensity of
emotion is signicantly greater in immersive than in non-immersive environments42. us, IVEs, showing 360°
panoramas or 3D scenarios through a HMD43, are powerful tools for psychological research43,.
Taking advantage of the IVE’s potentialities, in recent years some studies have used IVE and physiological
responses, such as EEG, HRV and EDA, in dierent elds. Phobias44–47, disorders48, driving and orientation49,50,
videogames51, quality of experience52, presence53 and visualization technologies54, are some examples of these
applications. Particularly in emotion research, arousal and relaxation have been analysed in outdoor55,56 and
indoor57 IVEs using EDA. erefore, the state of the art presents the following limitations: (1) few studies ana-
lyse physiological responses in IVEs and, in particular, using an aective approach; (2) there are few validated
emotional IVE sets which include stimuli with dierent levels of arousal and valence: and, (3) there is no aective
computing research that tries to automatically recognize the user’s mood in an IVE through physiological signals
and machine learning algorithms.
In this study, we propose a new AfC methodology capable of recognizing the emotional state of a subject in an
IVE in terms of valence and arousal. Regarding stimuli, IVEs were designed to evoke dierent emotional states
from an architectural point of view, by changing physical features such as illumination, colour and geometry. ey
were presented through a portable HMD. Regarding emotion recognition, a binary classier will be presented,
which uses eective features extracted from EEG and HRV data gathered from wearable sensors, and combined
through nonlinear Support Vector Machine (SVM)15 algorithms.
Material and Methods
Experimental context. is work is part of a larger research project that attempts to characterize the use of
VR as an aective elicitation method and, consequently, develop emotion recognition systems that can be applied
to 3D or real environments.
An experimental protocol was designed to acquire the physiological responses of subjects in 4 dierent stimuli
presentation cases: 2D desktop pictures, a 360° panorama IVE, a 3D scenario IVE and a physical environment.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
3
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
e experiment was conducted in two distinct phases that presented some dierences. Both phases were divided
into 3 stages; the results of the experiment are at Fig.1. Between each stage, signal acquisition was temporarily
halted and the subjects rested for 3 minutes on a chair. Stage 1 consisted of emotion elicitation through a desktop
PC displaying 110 IAPS pictures, using a methodology detailed in previous research15. Stage 2 consisted of emo-
tion elicitation using an HMD based on a new IVE set with four 360° panoramas. Finally, stage 3 consisted of the
free exploration of a museum exhibition.
In the present paper we focus on an analysis of stage 2. e experimental protocol was approved by the ethics
committee of the Polytechnic University of Valencia and informed consent was obtained from all participants.
All methods and experimental protocols were performed in accordance with the guidelines and regulations of the
local ethics committee of the Polytechnic University of Valencia.
Participants. A group of 60 healthy volunteers, suering neither from cardiovascular nor evident mental
pathologies, was recruited to participate in the experiment. ey were balanced in terms of age (28.9 ± 5.44) and
gender (40% male, 60% female). Inclusion criteria were as follows: age between 20 and 40 years; Spanish national-
ity; having no formal education in art or ne art; having no previous experience of virtual reality; and not having
previously visited the particular art exhibition. ey were divided into 30 subjects for the rst phase and 30 for
the second.
To ensure that the subjects constituted a homogeneous group, and that they were in a healthy mental state,
they were screened by i) the Patient Health Questionnaire (PHQ-9)58 and ii) the Self-Assessment Manikin
(SAM)59.
PHQ-9 is a standard psychometric test used to quantify levels of depression58. Signicant levels of depression
would have aected the emotional responses. Only participants with a score lower than 5 were included in the
study. e test was presented in the Spanish language as the subjects were native Spanish speakers. SAM tests
were used to detect if any subject had an emotional response that could be considered as an outlier, with respect
to a standard elicitation, in terms of valence and arousal. A set of 8 IAPS pictures60 (see Table1), representative
of dierent degrees of arousal and valence perception, was scored by each subject aer stage 1 of the experiment.
e z-score of each subject’s arousal and valence score was calculated using the mean and deviation of the IAPS’s
published scores60. Subjects that had one or more z-scores outside of the range −2.58 and 2.58 (α = 0.005) were
excluded from further analyses. erefore, we retained subjects whose emotional responses, caused by positive
and negative pictures, in dierent degrees of arousal, belonged to 99% of the IAPS population. In addition, we
rejected subjects if their signals presented issues, e.g., disconnection of the sensors during the elicitation or if
artefacts aected the signals. Taking these exclusions into account, the number of valid subjects was 38 (age:
28.42 ± 4.99; gender: 39% male, 61% female).
Set of Physiological Signals and Instrumentation. e physiological signals were acquired using the
B-Alert x10 (Advanced Brain Monitoring, Inc., USA) (Fig.2). It provides an integrated approach for wireless
wearable acquisition and recording of electroencephalographic (EEG) and electrocardiographic (ECG) signals,
sampled at 256 Hz. EEG sensors were located in the frontal (Fz, F3 and F4), central (Cz, C3 and C4) and parietal
(POz, P3, and P4) regions with electrode placements on the subjects’ scalps based on the international 10–20 elec-
trode placement. A pair of electrodes placed below the mastoid was used as reference, and a test was performed to
Figure 1. Experimental phases of the research.
IAPS picture Arousal Valenc e
7234 3.41 ± 2.29 4.01 ± 1.32
5201 3.20 ± 2.50 7.76 ± 1.44
9290 4.75 ± 2.20 2.71 ± 1.35
1463 4.61 ± 2.56 8.17 ± 1.48
9181 6.20 ± 2.23 1.84 ± 1.25
8380 5.84 ± 2.34 7.88 ± 1.37
3102 6.92 ± 2.50 1.29 ± 0.79
4652 7.24 ± 2.09 7.68 ± 1.64
Table 1. Arousal and valence score of selected IAPS pictures from56.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
4
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
check the conductivity of the electrodes, aiming to keep the electrode impedance below 20kΩ. e le ECG lead
was located on the lowest rib and the right lead on the right collarbone.
Stimulus elicitation. We developed an aective elicitation system by using architectural environments dis-
played by 360° panoramas implemented in a portable HMD (Samsung Gear VR). is combination of environ-
ments and display-format was selected due to its capacity for evoking aective states. e bidirectional inuence
between the architectural environment and the user’s aective-behavioural response is widely accepted: even
subtle variations in the space may generate dierent neurophysiological responses61. Furthermore, the 360°
panorama-format provided by HMD devices is a valid set-up to evoke psychological and physiological responses
similar to those evoked by physical environments54. us, following the combination of the arousal and valence
dimensions, which gives the four possibilities described in the CMA62, four architectural environments were
proposed as representative of four emotional states.
e four architectural environments were designed based on Kazuyo Sejima’s “Villa in the forest” scenario63.
is architectural work was considered by the research team as an appropriate base from which to make the mod-
ications designed to generate the dierent aective states.
e four base-scenario congurations were based on dierent modications of the parameters of three design
variables: illumination, colour, and geometry. Regarding illumination, the parameters “colour temperature”,
“intensity”, and “position” were modied. e modication of the “colour temperature” was based on the fact
that higher temperature may increase arousal, being registrable at the neurophysiological level64,65. “Intensity”
was also modied in the same way to try to increase or reduce arousal. e “position” of the light was direct,
in order to try to increase arousal, and indirect to reduce it. e modications of these last two parameters
were based on the design experience of the research team. Regarding colour, the parameters “tone”, “value”, and
“saturation” were modied. e modication of these parameters was performed jointly on the basis that warm
colours increase arousal and cold ones reduce it, being registrable at the psychological66 and neurophysiological
levels67–71. Regarding geometry, the parameters “curvature”, “complexity”, and “order” were modied. “Curvature”
was modied on the basis that curved spaces generate a more positive valence than angular, being registrable at
psychological and neurophysiological levels72. e modication of the parameters “complexity” and “order” was
performed jointly. is was based on three conditions registrable at the neurophysiological level: (1) high levels of
geometric “complexity” may increase arousal and low levels may reduce arousal73; (2) high levels of “complexity”
may generate a positive valence if they are submitted to “order”, and negative valence if presented disorderly74; and
(3) content levels of arousal generated by geometry may generate a more positive valence75. e four architectural
environments were designed on this basis. Table2 shows the conguration guidelines chosen to elicit the four
aective states.
In a technical sense, the four architectural environments were developed in similar ways. e process con-
sisted of modelling and rendering. Modelling was performed by using Rhinoceros v5.0 (www.rhino3d.com).
Figure 2. Exemplary experimental set-up.
High-Arousal &
Negative-Valence
(Room 1)
High-Arousal &
Positive-Valence
(Room 2)
Low-Arousal &
Negative-Valence
(Room 3)
Low-Arousal &
Positive-Valence
(Room 4)
Illumination
Colour temperature 7500 K 7500 K 3500 K 3500 K
Intensity High High Low Low
Position Mainly Direct Mainly Direct Mainly Indirect Mainly Indirect
Colour
Ton e
Warm colours Warm colours Cold colours Cold coloursVal u e
Saturation
Geometry
Curvature Rectilinear Curved Rectilinear Curved
Complexity High Low-Medium Medium-High Low
Order Low High Low-Medium High
Table 2. Conguration guidelines chosen in each architectural environment conguration.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
5
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
e 3D-models used for the four architectural environments were 3446946, 3490491, 3487660, and 3487687
polygons. On completion of this process, they were exported in.dwg format for later rendering. e rendering
was performed using the VRay engine v3.00.08 (www.vray.com), operating with Autodesk 3ds Max v2015 (www.
autodesk.es). 15 textures were used for each of the four architectural environments. Congured as 360° pano-
ramas, renders were exported in.jpg format with resolutions of 6000 × 3000 pixels at 300 dots per inch. ese
were implemented in the Samsung Gear VR HMD device. is device has a stereoscopic screen of 1280 × 1440
pixels per eye and a 96° eld of view, supported by a Samsung Note 4 mobile telephone with a 2.7 GHz quad-core
processor and 3GB of RAM. e reproduction of the architectural 360° panoramas was uid and uninterrupted.
Prior to the execution of the experimental protocol, a pre-test was performed in order to ensure that the archi-
tectural 360° panoramas would elicit the aective states for which they had been designed. It was a three-phased
test: individual questionnaires, a focus-group session conducted with some respondents to the questionnaire
and individual validation-questionnaires. e questionnaires asked the participants to evaluate the architectural
360° panoramas. A SAM questionnaire, embedded in the 360° panorama, was used, with evaluations ranging
from −4 (totally disagree) to 4 (totally agree) for all the emotion dimensions. 15 participants (8 men and 7
women) completed the questionnaires. First, the participants freely viewed each architectural environment, then
the SAM questionnaires were presented and the answers given orally. Figure3 shows an example of one of these
questionnaires. Aer the questionnaire sessions had been completed, a focus group session, which was a carefully
managed group discussion, was conducted76. Five of the participants (3 men and 2 women) with the most unfa-
vourable evaluations in phase 1 were selected as participants and one of the members of the research team, with
previous focus-group experience, moderated. e majority of the changes were performed to Room 3, due to the
discordances between the self-assessment and their theoretical quadrant. Once the changes were implemented, a
similar evaluation to phase 1 was performed. Table3 shows the arousal and valence ratings of the four architec-
tural 360° panoramas of this pre-test phase. Aer these phases, no new variations were considered necessary. is
procedure allowed us to assume some initial reliability in the design of the architectural environments. Figure4
shows these nal congurations. High quality images of the stimuli are included in the supplementary material.
None of the pre-test participants was included in the main study. Regarding the experimental protocol, each
room was presented for 1.5 minutes and the sequence of presentation was counter-balanced using the Latin
Square method. Aer viewing the rooms, the users were asking to orally evaluate the emotional impact of each
room using a SAM questionnaire embedded in the 360° photo.
Signal processing. Heart rate variability. e ECG signals were processed to derive HRV series77. e
artefacts were cleaned by the threshold base artefacts correction algorithm included in the Kubios soware78.
In order to extract the RR series, the well-known algorithm developed by Pan-Tompkins was used to detect the
R-peaks79. e individual trends components were removed using the smoothness prior detrending method80.
Figure 3. Example of SAM questionnaire embedded in the room 1. Simulation developed using Rhinoceros
v5.0, VRay engine v3.00.08 and Autodesk 3ds Max v2015.
Arousal Valenc e
High-Arousal & Negative-Valence
(Room 1) 2.23 ± 1.59 −2.08 ± 1.71
High-Arousal & Positive-Valence
(Room 2) 1.25 ± 1.33 1.31 ± 1.38
Low-Arousal & Negative-Valence
(Room 3) −0.69 ± 1.65 −1.46 ± 1.33
Low-Arousal & Positive-Valence
(Room 4) −2.31 ± 1.30 1.92 ± 1.50
Table 3. Arousal and Valence resulted in the pre-test with 15 participants. e scores are averaged using mean
and standard deviation for a Likert scale between −4 to +4.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
6
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
We carried out the analysis of the standard HRV parameters, which are dened in the time and frequency
domains, as well as HRV measures quantifying heartbeat nonlinear and complex dynamics77. All features are
listed in Table4.
Time domain features include average (Mean RR) and standard deviation (Std RR) of the RR intervals, the
root mean square of successive dierences of intervals (RMSSD), and the ratio between the number of successive
RR pairs having a dierence of less than 50 ms and the total number of heartbeat analyses (pNN50). e triangu-
lar index was calculated as a triangular interpolation of the HRV histogram. Finally, TINN is the baseline width
of the RR histogram, evaluated through triangular interpolation.
In order to obtain the frequency domain features, a power spectrum density (PSD) estimate was calculated for
the RR interval series by a Fast Fourier Transform based on Welch’s periodogram method. e analysis was car-
ried out in three bands: very low frequency (VLF, <0.04 Hz), low frequency (LF, 0.04–0.15 Hz) and high frequency
(HF, 0.12–0.4 Hz). For each frequency band, the peak value was calculated, corresponding to the frequency with
the maximum magnitude. e power of each frequency band was calculated in absolute and percentage terms.
Moreover, for the LF and HF bands, the normalized power (n.u.) was calculated as the percentage of the signals
subtracting the VLF to the total power. e LF/HF ratio was calculated in order to quantify sympatho-vagal bal-
ance and to reect sympathetic modulations77. In addition, the total power was calculated.
Regarding the HRV nonlinear analysis, many measures were extracted, as they are important quantiers of
cardiovascular control dynamics mediated by the ANS in aective computing15,16,77,81. Pointcaré plot analysis is a
quantitative-visual technique, whereby the shape of a plot is categorized into functional classes. e plot provides
summary information as well as detailed beat-to-beat information on heart behaviour. SD1 is related to the fast
beat-to-beat variability in the data, whereas SD2 describes the longer-term variability of R–R77. Approximate
Entropy (ApEn) and Sample Entropy (SampEn) are two entropy measures of HRV. ApEn detects the changes
in underlying episodic behaviour not reected in peak occurrences or amplitudes82, whereas SampEn statistics
provide an improved evaluation of time-series regularity and provide a useful tool in studies of the dynamics of
human cardiovascular physiology83. DFA correlations are divided into short-term and long-term uctuations
through the α1 and α2 features. Whereas α1 represents the uctuation in the range of 4–16 samples, α2 refers to
the range of 16–64 samples84. Finally, the correlation dimension is another method for measuring the complexity
Figure 4. 360° panoramas of the four IVEs. Simulations developed using Rhinoceros v5.0, VRay engine
v3.00.08 and Autodesk 3ds Max v2015.
Time domain Frequency domain Other
Mean RR VLF peak Pointcaré SD1
Std RR LF peak Pointcaré SD2
RMSSD HF peak Approximate Entropy (ApEn)
pNN50 VLF power Sample Entropy (SampEn)
RR triangular index VLF power % DFA α1
TINN LF power D FA α2
LF power % Correlation dimension (D2)
LF power n.u.
HF power
HF power %
HF power n.u.
LF/HF power
Total power
Table 4. List of used HRV features.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
7
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
or strangeness of the time series; it is explained by the D2 feature. It is expected to give information on the mini-
mum number of dynamic variables needed to model the underlying system85.
Electroencephalographic signals. In order to process the EEG signals, the open source toolbox EEGLAB86 was
used. e complete processing scheme is shown at Fig.5.
Firstly, data from each electrode were analysed in order to identify corrupted channels. ese were identied by
computing the fourth standardized moment (kurtosis) along the signal of each electrode87. In addition, if the signal
was atter than 10% of the total duration of the experiment, the channel was classied as corrupted. If one of the
nine channels was considered as corrupted, it could be interpolated from neighbouring electrodes. If more than one
channel was corrupted, the subject would be rejected. Only one channel among all of the subjects was interpolated.
e baseline of EEG traces was removed by mean subtraction and a band pass lter between 0.5 and 40 Hz
was applied. e signal was divided into epochs of one second and the intra-channel kurtosis level of each epoch
was computed in order to reject the epochs highly damaged by noise87. In addition, automatic artefact detection
was applied, which rejects the epoch when more than 2 channels have samples exceeding an absolute threshold of
>100.00 µV and a gradient of 70.00 µV between samples88.
e Independent Component Analysis (ICA)89 was then carried out using infomax algorithm to detect and
remove components due to eye movements, blinks and muscular artefacts. Nine source signals were obtained
(one per electrode). A trained expert manually analysed all the components, rejecting those related to artefacts.
e subjects who had more than 33% of their signals aected by artefacts were rejected.
Aer the pre-processing, spectral and functional connectivity analyses were performed.
EEG spectral analysis, using Welch’s method90, was performed to estimate the power spectra in each epoch,
with 50% overlapping, within the classical frequency bandwidth θ (4–8 Hz), α (8–12 Hz), β (13–25 Hz), γ (25–
40 Hz). Frequency band δ (less than 4 Hz) was not taken into account in this study because it relates to deeper
stages of sleep. In total, 36 features were obtained from the nine channels and 4 bands.
A functional connectivity analysis was performed using Mean Phase Coherence91, for each pair of channels:
φφ=Δ+ΔRE Esin[cos()][()](1)
222
where
R
is the MPC, Δφ represents the relative phase diference between two channels derived from the instanta-
neous dierence of the analytics signals from the Hilbert transform, and
E
is the expectation operator. By deni-
tion, MPC values ranged between 0 and 1. In the case of strong phase synchronization between two channels, the
MPC is close to 1. If the two channels are not synchronized, the MPC remains low. 36 features were derived from
each possible combination of a pair of 9 channels in one specic band. In total, 144 features were created using the
4 bands analysed.
Feature reduction and machine learning. Each room was presented for 1.5 minutes and was considered as
an independent stimulus. In order to characterize each room, all HRV features were calculated using this time window.
In the case of EEG, in both the frequency band power and mean phase connectivity analyses, we considered the mean
of all the epochs of each stimulus as the representative value of the stimulus time window. Altogether, 209 features
described each stimulus for each subject. Due to the high-dimensional feature space obtained, a feature reduction
strategy was adopted for decreasing this dimension. We implemented the well-known Principal Component Analysis
method (PCA)92. is mathematical method is based on the linear transformation of the dierent variables in the prin-
cipal components, which can be assembled in clusters. We select the features that explain 95% of the variability of the
dataset. e PCA was applied three times: (1) in the HRV set, reducing the features from 29 to 3; (2) in the frequency
band power analysis of the EEG, reducing the features from 36 to 4; and (3) in the mean phase coherency analysis of the
EEG, reducing the features from 144 to 12. Hence, the feature reduction strategy reduces our features to a total of 19.
e machine learning strategy could be summarized as follows:
(1) To divide the dataset into training and test sets.
(2) e development of the model (parameter tuning and feature selection) using cross-validation in the
training set.
(3) To validate the model using the test set.
Figure 5. Block scheme of the EEG signal processing steps.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
8
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
Firstly, the dataset was sliced randomly into 15% for the test set (5 subjects) and 85% for the training set (33
subjects). In order to calibrate the model, the Leave-One-Subject-Out (LOSO) cross-validation procedure was
applied to the training set using Support Vector Machine (SVM)-based pattern recognition93. Within the LOSO
scheme, the training set was normalized by subtracting the median value and dividing this by the median absolute
deviation over each dimension. In each of the 36 iterations, the validation set consisted of one specic subject and
he/she was normalized using the median and deviation of the training set.
Regarding the algorithm, we used a C-SVM optimized using a sigmoid kernel function, changing the
parameters of cost and gamma using a vector with 15 parameters logarithmically spaced between 0.1 and 1000.
Additionally, in order to explore the relative importance of all the features in the classication problem we used
a support vector machine recursive feature elimination (SVM-RFE) procedure in a wrapper approach (RFE was
performed on the training set of each fold and we computed the median rank for each feature over all folds). We
specically chose a recently developed, nonlinear SVM-RFE, which includes a correlation bias reduction strategy
in the feature elimination procedure94. Aer the cross-validation, using the parameters and feature set obtained,
the model was applied to the test set that had not previously been used. e self-assessment of each subject was
used as the output of the arousal and valence model. e evaluation was bipolarized in positive/high (>0) and
negative/low (<=0). All the algorithms were implemented by using Matlab© R2016a, endowed with an addi-
tional toolbox for pattern recognition, i.e., LIBSVM95. A general overview of the analysis is shown in Fig.6.
Results
Subjects’ self-assessment. Figure7 shows the self-assessment of the subjects for each IVE averaged using
mean and standard deviation in terms of arousal (Room 1: 1.17 ± 1.81, Room 2: 2.10 ± 1.59, Room 3: 0.05 ± 2.01,
Room 4: −0.60 ± 2.11) and valence (Room 1: −1.12 ± 1.95, Room 2: 1.45 ± 1.93, Room 3: −0.40 ± 2.14, Room 4:
2.57 ± 1.42). e representation follows the CMA space. All rooms are located in the theoretical emotion quad-
rant for which they were designed, except for Room 3 that evokes more arousal than hypothesized. Due to the
non-Gaussianity of data (p < 0.05 from the Shapiro-Wilk test with null hypothesis of having a Gaussian sam-
ple), Wilcoxon signed-rank tests were applied. Table5 presents the result of multiple comparisons using Tukey’s
Honestly Signicant Dierence Procedure. Signicant dierences were found in the valence dimension between
the negative-valence rooms (1 and 3) and the positive-valence rooms (2 and 4). Signicant dierences were found
in the arousal dimension between the high-arousal rooms (1 and 2) and the low-arousal rooms (3 and 4), but not
for pairs 1 and 3. erefore, the IVEs statistically achieve all the desired self-assessments except for arousal per-
ception in Room 3, which is higher than we hypothesized. Aer the bipolarization of scores (positive/high >0),
they are balanced (61.36% high arousal and 56.06% positive valence).
Arousal classication. Table6 shows the confusion matrix of cross validation and the total average accu-
racy (75.00%), distinguishing two levels of arousal using the rst 15 features selected by the nonlinear SVM-RFE
algorithm. e F-Score of arousal classication is 0.75. e changes in accuracy depending on number of features
are shown in Fig.8, and Table7 presents the list of features used. Table8 shows the confusion matrix of the test set
and the total average accuracy (70.00%) using the parameters and the feature set dened in the cross-validation
phase. e F-score of arousal classication is 0.72 in the test set.
Valence classication. Table9 shows the confusion matrix of the cross validation and total average accu-
racy (71.21%), distinguishing two levels of valence using the rst 10 features selected by the nonlinear SVM-RFE
algorithm. e F-Score of the valence classication is 0.71. e changes in accuracy depending on the number
of features are shown in Fig.9, and Table10 presents the list of features used. Table11 shows the confusion
matrix of the test set and total average accuracy (70.00%), using the parameters and the feature set dened in the
cross-validation phase. e F-score of the valence classication was 0.70 in the test set.
Discussion
e purpose of this study is to develop an emotion recognition system able to automatically discern aective
states evoked through an IVE. is is part of a larger research project that seeks to analyse the use of VR as an
aective elicitation method, in order to develop emotion recognition systems that can be applied to 3D or real
environments. e results can be discussed on four levels: (1) the ability of IVEs to evoke emotions; (2) the ability
of IVEs to evoke the same emotions as real environments; (3) the developed emotion recognition model; and (4),
the ndings and applications of the methodology.
Figure 6. Overview of the feature reduction and classication chain.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
9
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
Regarding the ability of the IVEs to evoke emotions, four versions of the same basic room design were used to
elicit the four main arousal-valence combinations related to the CMA. is was achieved by changing dierent
architectural parameters, such as illumination, colour and geometry. As shown in Fig.7 and Table5, proper elic-
itation was achieved for Room 1 (high arousal and negative valence), Room 2 (high arousal and positive valence)
and Room 4 (low arousal and positive valence), but it overlapped somewhat with the arousal-valence representa-
tion in Room 3: despite the satisfactory pre-test, in the event it evoked higher arousal and valence than expected.
is is due to the diculties we experienced in designing a room to evoke negative emotion with low arousal. It
should be noted that IAPS developers may also have experienced this problem because only 18.75% of the pics
are situated in this quadrant60. Other works based on processing valence and arousal using words show that a
U-model exists in which arousal increases in agreement with valence intensity regardless of whether it is posi-
tive or negative96. Hence, for future works, Room 3 will be redesigned to decrease its arousal and valence and a
self-assessment with a larger sample will be performed, by questionnaire, to robustly assess the IVE. Nonetheless,
aer thresholding the individual self-assessment scores to discern 2 classes (high/low), the IVE set was balanced
in arousal and valence. erefore, we could conclude that the proposed room set can satisfactorily evoke the four
emotions represented by each quadrant of the CMA.
To this extent, although previous studies have presented IVEs capable of evoking emotional states in a con-
trolled way97, to the best of our knowledge we have presented the rst IVE suite capable of evoking a variety of
levels of arousal and valence based on CMA. Moreover, the suite was tested through a low-cost portable HMD,
Figure 7. Self-assessment score in the IVEs using SAM and a Likert scale between −4 and +4. Blue dots
represent the mean whereas horizontal and vertical lines represent standard deviation.
IVE
p-value
Arousal Valenc e
1 2 0.052 10–6 (***)
1 3 0.195 0.152
140.007 (**)10–9 (***)
2310–5 (***)0.015 (*)
2410–8 (***) 0.068
3 4 0.606 10–7 (***)
Table 5. Signication test of the self-assessment of the emotional rooms.
Arousal High Low
High 82.72 17.28
Low 37.25 62.75
Table 6. Confusion matrix of cross-validation using SVM classier for arousal level. Values are expressed as
percentages. Total Accuracy: 75.00%.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
10
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
the Samsung Gear, therefore increasing the possible applications of the methodology. High quality images of the
stimuli are included in the supplementary material. is represents a new tool that can contribute in the eld
of psychology, in general, and in the aective computing eld, in particular, fostering the development of novel
immersive aective elicitation using IVEs.
Figure 8. Recognition accuracy of arousal in cross-validation as a function of the feature rank estimated
through the SVM-RFE procedure.
Rank Feature
1EEG MPC PCA 8
2EEG MPC PCA 9
3EEG MPC PCA 11
4EEG MPC PCA 10
5EEG MPC PCA 7
6EEG MPC PCA 12
7EEG Band Power PCA 3
8EEG Band Power PCA 1
9HRV PCA 1
10 EEG Band Power PCA 4
11 EEG Band Power PCA 2
12 HRV PCA 3
13 EEG MPC PCA 4
14 HRV PCA 2
15 EEG MPC PCA 5
Table 7. Selected features ordered by their median rank over every fold computed during the LOSO procedure
for arousal classication.
Arousal High Low
High 75.00 25.00
Low 33.33 66.67
Table 8. Confusion matrix of test set using SVM classier for arousal level. Values are expressed as percentages.
Total Accuracy: 70.00%.
Val e n ce Positive Negative
Positive 71.62 28.38
Negative 29.31 70.69
Table 9. Confusion matrix of cross-validation using SVM classier for valence level. Values are expressed as
percentages. Total Accuracy: 71.21%.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
11
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
ere are still some topics that need to be researched, relating to the capacity of the IVE display formats, to
ensure that they evoke the same emotions as real environments. Studies comparing display formats show that the
360° IVEs oer results closer to reality, according to the participants’ psychological responses, and 3D IVEs do so
according to their physiological responses54. Moreover, it is quite possible that IVEs will oer the best solutions
at both psychological and physiological levels as they become even more realistic, providing a real improvement
not only at the visual and auditory levels but also at the haptic98. In addition, 3D IVEs allow users to navigate
and interact with the environment. Hence, there are reasons to think that they could be powerful tools for devel-
oping applications for aective computing, but studies comparing human responses in real and simulated IVE
are scarce99–101, especially regarding emotional responses; these studies are required. Moreover, every year the
resolution of Head Mounted Displays is upgraded, which brings them closer to eye resolution. us, it is possible
that in some years the advances in Virtual Reality hardware will make the present methodology more powerful.
In addition, works comparing VR devices with dierent levels of immersion are needed in order to give research-
ers the best set-ups to achieve their aims. In future works, we need to consider all these topics to improve the
methodology.
Regarding the emotion recognition system, we present the rst study that develops an emotion recognition
system using a set of IVEs as a stimulus elicitation and proper analyses of physiological dynamics. e accuracy
of the model was 75.00% along the arousal dimension and 71.21% along the valence dimension in the phase
of cross-validation, with average of 70.00% along both dimensions in the test set. ey all present a balanced
confusion matrix. e accuracies are considerably higher than the chance level, which is 58% in brain signal
Figure 9. Recognition accuracy of valence in cross-validation as a function of the feature rank estimated
through the SVM-RFE procedure.
Rank Feature
1EEG MPC PCA 8
2EEG MPC PCA 6
3EEG MPC PCA 11
4EEG MPC PCA 7
5EEG MPC PCA 10
6EEG MPC PCA 12
7EEG MPC PCA 9
8EEG Band Power PCA 3
9EEG Band Power PCA 4
10 EEG MPC PCA 2
Table 10. Selected features ordered by their median rank over every fold computed during the LOSO procedure
for valence classication.
Val e n ce Positive Negative
Positive 75.00 25.00
Negative 37.50 62.50
Table 11. Confusion matrix of test set using SVM classier for valence level. Values are expressed as
percentages. Total Accuracy: 70.00%.
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
12
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
classication and statistical assessment (n = 152, 2-classes, p = 0.05)102. Although the accuracy is lower than other
studies of emotion recognition in images15 and sounds27, our results present a rst proof of concept that suggests
that it is possible to recognize the emotion of a subject elicited through an IVE. e research was developed with
a sample of 60 subjects, who were carefully screened to demonstrate agreement with a “standard” population
reported in the literature46. It should be noted that the possible overtting of the model was controlled using: (1)
a feature reduction strategy with a PCA; (2) a feature selection strategy using a SVM-RFE; (3) a rst validation of
the model using LOSO cross-validation; and (4) a test validation using 5 randomly chosen subjects (15%), who
had not been used before to train or perform the cross-validation of the model. In the arousal model, features
derived from three-signal analyses were selected: 3/3 of HRV, 4/4 of EEG BandPower and 8/12 of EEG MPC.
However, in the valence model only the EEG analysis was used: 0/3 of HRV, 2/4 of EEG BandPower and 8/12
EEG MPC. Moreover, in both models, the rst six features selected by RFE-SVM were derived from an EEG MPC
analysis. is suggests that cortical functional connectivity provides eective correlates of emotions in an IVE.
Furthermore, according to recent evidence22,103, the reliability of emotion recognition outside of the laboratory
environment is improved by wearables. In future experiments, these results could be optimized using further,
maybe multivariate signal analyses and alternative machine learning algorithms87. In addition, the design of new,
controlled IVEs that can increase the number of stimuli per subject, using more combinations of architectural
parameters (colour, illumination and geometry), should also improve the accuracy and robustness of the model.
In future studies, we will improve the set of stimuli presented including new IVEs in order to develop a large set
of validate IVE stimuli to be used in emotion research.
e ndings presented here mark a new step in the eld of aective computing and its applications. Firstly, the
methodology involved in itself a novel trial to overcome the limitations of passive methods of aective elicitation,
in order to recreate more realistic stimuli using 360° IVEs. Nevertheless, the long-term objective is to develop a
robust pre-calibrate model that could be applied in two ways: (1) in 3D environments that would allow the study
of emotional responses to “real” situations in a laboratory environment through VR simulation using HMD
devices and (2) in physical spaces. We hypothesize in both cases that the emotion recognition models developed
through controlled 360° IVEs will work better than the models calibrated by non-immersive stimuli, such as
IAPS. is approach will be discussed in future studies using stage 3 of the experimental protocol.
Regarding the implications for architecture, the methodology could be applied in two main contexts, research
and commercial. On the one hand, researchers could analyse and measure the impact of dierent design param-
eters on the emotional responses of potential users. is is especially important due to the impossibility of devel-
oping researches in real or laboratory environments (e.g. analysing arousal changes caused by the pavement
width on a street). e synergy of aective computing and virtual reality allows us to isolate a parameter design
and measure the emotional changes provoked by making changes to it, while keeping the rest of the environment
identical. is could improve the knowledge of the emotional impact that might be made by dierent design
parameters and, consequently, facilitate the development of better practices and relevant regulations. On the other
hand, this methodology could help architects and engineers in their decision-making processes for the design
of built environments before construction, aiding their evaluations and the selection of the options that might
maximize the mood that they want to evoke: for example, positive valence in a hotel room or a park, low arousal
in a schoolroom or in a hospital waiting room and high arousal in a shop or shopping centre. Nevertheless, these
ndings could be applied to any other eld that needs to quantify the emotional eects of spatial stimuli displayed
by Immersive Virtual Environments. Health, psychology, driving, videogames and education might all benet
from this methodology.
Data Availability
e datasets generated during and/or analysed during the current study are available from the corresponding
authors on reasonable request.
References
1. Picard, . W. Aective computing. (MIT press, 1997).
2. Picard, . W. Aective Computing: Challenges. Int. J. Hum. Comput. Stud. 59, 55–64 (2003).
3. Jerritta, S., Murugappan, M., Nagarajan, . & Wan, . Physiological signals based human emotion ecognition: a review. Signal
Process. its Appl. (CSPA), 2011 IEEE 7th Int. Colloq. 410–415, https://doi.org/10.1109/CSPA.2011.5759912 (2011).
4. Harms, M. B., Martin, A. & Wallace, G. L. Facial emotion recognition in autism spectrum disorders: A review of behavioral and
neuroimaging studies. Neuropsychol. ev. 20, 290–322 (2010).
5. oolagudi, S. G. & ao, . S. Emotion recognition from speech: A review. Int. J. Speech Technol. 15, 99–117 (2012).
6. Gross, J. J. & Levenson, . W. Emotion elicitation using lms. Cogn. Emot. 9, 87–108 (1995).
7. Lindal, P. J. & Hartig, T. Architectural variation, building height, and the restorative quality of urban residential streetscapes. J.
Environ. Psychol. 33, 26–36 (2013).
8. Ulrich, . View through a window may inuence recovery from surgery. Science (80-.). 224, 420–421 (1984).
9. Fernández-Caballero, A. et al. Smart environment architecture for emotion detection and regulation. J. Biomed. Inform. 64, 55–73
(2016).
10. Eman, P. Basic Emotions. Handboo of cognition and emotion 45–60, https://doi.org/10.1017/S0140525X0800349X (1999).
11. Posner, J., ussell, J. A. & Peterson, B. S. e circumplex model of aect: an integrative approach to aective neuroscience, cognitive
development, and psychopathology. Dev. Psychopathol. 17, 715–34 (2005).
12. ussell, J. A. & Mehrabian, A. Evidence for a three-factor theory of emotions. J. es. Pers. 11, 273–294 (1977).
13. Calvo, . A. & D’Mello, S. Aect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans.
Aect. Comput. 1, 18–37 (2010).
14. Valenza, G. et al. Combining electroencephalographic activity and instantaneous heart rate for assessing brain–heart dynamics
during visual emotional elicitation in healthy subjects. Philos. Trans. . Soc. A Math. Phys. Eng. Sci. 374, 20150176 (2016).
15. Valenza, G., Lanata, A. & Scilingo, E. P. e role of nonlinear dynamics in aective valence and arousal recognition. IEEE Trans.
Aect. Comput. 3, 237–249 (2012).
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
13
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
16. Valenza, G., Citi, L., Lanatá, A., Scilingo, E. P. & Barbieri, . evealing real-time emotional responses: a personalized assessment
based on heartbeat dynamics. Sci. ep. 4, 4998 (2014).
17. Valenza, G. et al. Wearable monitoring for mood recognition in bipolar disorder based on history-dependent long-term heart rate
variability analysis. IEEE J. Biomed. Heal. Informatics 18, 1625–1635 (2014).
18. Piwe, L., Ellis, D. A., Andrews, S. & Joinson, A. e ise of Consumer Health Wearables: Promises and Barriers. PLoS Med. 13,
1–9 (2016).
19. Xu, J., Mitra, S., Van Hoof, C., Yazicioglu, . & Mainwa, . A. A. Active Electrodes for Wearable EEG Acquisition: eview and
Electronics Design Methodology. IEEE ev. Biomed. Eng. 3333, 1–1 (2017).
20. umari, P., Mathew, L. & Syal, P. Increasing trend of wearables and multimodal interface for human activity monitoring: A review.
Biosens. Bioelectron. 90, 298–307 (2017).
21. He, C., Yao, Y. & Ye, X. An Emotion ecognition System Based on Physiological Signals Obtained by Wearable Sensors. In Wear abl e
Sensors and obots: Proceedings of International Conference on We ar able Sensors and obots 2015 (eds Yang, C., Vir, G. S. & Yang,
H.) 15–25. https://doi.org/10.1007/978-981-10-2404-7_2 (Springer Singapore, 2017).
22. Naisa, B., astgoo, M. N., Tjondronegoro, D. & Chandran, V. Evolutionary computation algorithms for feature selection of EEG-
based emotion recognition using mobile sensors. Expert Syst. Appl. 93, 143–155 (2018).
23. ory Jacqueline, D. & Sidney, . Aect Elicitation for Aective Computing. In e Oxford Handboo of Aective Computing
371–383 (2014).
24. Eman, P. e directed facial action tas. In Handboo of emotion elicitation and assessment 47–53 (2007).
25. Harmon-Jones, E., Amodio, D. M. & Zinner, L. . Social psychological methods of emotion elicitation. Handb. Emot. elicitation
Assess. 91–105, https://doi.org/10.2224/sbp.2007.35.7.863 (2007)
26. oberts, N. A., Tsai, J. L. & Coan, J. A. Emotion elicitation using dyadic interaction tas. Handboo of Emotion Elicitation and
Assessment 106–123 (2007).
27. Nardelli, M., Valenza, G., Greco, A., Lanata, A. & Scilingo, E. P. ecognizing emotions induced by aective sounds through heart
rate variability. IEEE Trans. Aect. Comput. 6, 385–394 (2015).
28. im, J. Emotion ecognition Using Speech and Physiological Changes. obust Speech ecognit. Underst. 265–280 (2007).
29. Soleymani, M., Pantic, M. & Pun, T. Multimodal emotion recognition in response to videos (Extended abstract). 2015 Int. Conf.
Aect. Comput. Intell. Interact. ACII 2015 3, 491–497 (2015).
30. Baños, . M. et al. Immersion and Emotion: eir Impact on the Sense of Presence. CyberPsychology Behav. 7, 734–741 (2004).
31. Giglioli, I. A. C., Pravettoni, G., Martín, D. L. S., Parra, E. & aya, M. A. A novel integrating virtual reality approach for the
assessment of the attachment behavioral system. Front. Psychol. 8, 1–7 (2017).
32. Marín-Morales, J., Torrecilla, C., Guixeres, J. & Llinares, C. Methodological bases for a new platform for the measurement of
human behaviour in virtual environments. DYNA 92, 34–38 (2017).
33. Vince, J. Introduction to virtual reality. (Media, Springer Science & Business, 2004).
34. Alcañiz, M., Baños, ., Botella, C. & ey, B. e EMMA Project: Emotions as a Determinant of Presence. PsychNology J. 1, 141–150
(2003).
35. Vecchiato, G. et al. Neurophysiological correlates of embodiment and motivational factors during the perception of virtual
architectural environments. Cogn. Process. 16, 425–429 (2015).
36. Slater, M. & Wilbur, S. A Framewor for Immersive Virtual Environments (FIVE): Speculations on the ole of Presence in Virtual
Environments. Presence Teleoperators Virtual Environ. 6, 603–616 (1997).
37. iva, G. et al. Aective Interactions Using Virtual eality: e Lin between Presence and Emotions. CyberPsychology Behav. 10,
45–56 (2007).
38. Baños, . M. et al Changing induced moods via virtual reality. In International Conference on Persuasive Technology (ed. Springer,
Berlin, H.) 7–15, https://doi.org/10.1007/11755494_3 (2006).
39. Baños, . M. et al. Positive mood induction procedures for virtual environments designed for elderly people. Interact. Comput. 24,
131–138 (2012).
40. Gorini, A. et al. Emotional esponse to Virtual eality Exposure across Dierent Cultures: e ole of the AttributionProcess.
CyberPsychology Behav. 12, 699–705 (2009).
41. Gorini, A., Capideville, C. S., De Leo, G., Mantovani, F. & iva, G. e ole of Immersion and Narrative in Mediated Presence: e
Virtual Hospital Experience. Cyberpsychology, Behav. Soc. Netw. 14, 99–105 (2011).
42. Chirico, A. et al. Eectiveness of Immersive Videos in Inducing Awe: An Experimental Study. Sci. ep. 7, 1–11 (2017).
43. Blascovich, J. et al. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychol. Inq. 7965,
103–124 (2012).
44. Peperorn, H. M., Alpers, G. W. & Mühlberger, A. Triggers of fear: Perceptual cues versus conceptual information in spider phobia.
J. Clin. Psychol. 70, 704–714 (2014).
45. McCall, C., Hildebrandt, L. ., Bornemann, B. & Singer, T. Physiophenomenology in retrospect: Memory reliably reflects
physiological arousal during a prior threatening experience. Conscious. Cogn. 38, 60–70 (2015).
46. Hildebrandt, L. ., Mccall, C., Engen, H. G. & Singer, T. Cognitive exibility, heart rate variability, and resilience predict ne-
grained regulation of arousal during prolonged threat. Psychophysiology 53, 880–890 (2016).
47. Notzon, S. et al. Psychophysiological eects of an iTBS modulated virtual reality challenge including participants with spider
phobia. Biol. Psychol. 112, 66–76 (2015).
48. Amaral, C. P., Simões, M. A., Mouga, S., Andrade, J. & Castelo-Branco, M. A novel Brain Computer Interface for classication of
social joint attention in autism and comparison of 3 experimental setups: A feasibility study. J. Neurosci. Methods 290, 105–115
(2017).
49. Eudave, L. & Valencia, M. Physiological response while driving in an immersive virtual environment. 2017 IEEE 14th Int. Conf.
Wearable Implant. Body Sens. Networs 145–148, https://doi.org/10.1109/BSN.2017.7936028 (2017).
50. Sharma, G. et al. Inuence of landmars on waynding and brain connectivity in immersive virtual reality environment. Front.
Psychol. 8, 1–12 (2017).
51. Bian, Y. et al. A framewor for physiological indicators of ow in V games: construction and preliminary evaluation. Pers.
Ubiquitous Comput. 20, 821–832 (2016).
52. Egan, D. et al. An evaluation of Heart ate and Electrodermal Activity as an Objective QoE Evaluation method for Immersive
Virtual eality Environments. 3–8, https://doi.org/10.1109/QoMEX.2016.7498964 (2016).
53. Meehan, M., azzaque, S., Inso, B., Whitton, M. & Broos, F. P. eview of four studies on the use of physiological reaction as a
measure of presence in stressful virtual environments. Appl. Psychophysiol. Biofeedbac 30, 239–258 (2005).
54. Higuera-Trujillo, J. L., López-Tarruella Maldonado, J. & Llinares Millán, C. Psychological and physiological human responses to
simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual eality. Appl. Ergon. 65,
398–409 (2016).
55. Felnhofer, A. et al. Is virtual reality emotionally arousing? Investigating ve emotion inducing virtual par scenarios. Int. J. Hum.
Comput. Stud. 82, 48–56 (2015).
56. Anderson, A. P. et al. elaxation with Immersive Natural Scenes Presented Using Virtual eality. Aerosp. Med. Hum. Perform. 88,
520–526 (2017).
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
14
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
57. Higuera, J. L. et al. Emotional cartography in design: A novel technique to represent emotional states altered by spaces. In D and E
2016: 10th International Conference on Design and Emotion 561–566 (2016).
58. roene, ., Spitzer, . L. & Williams, J. B. W. e PHQ-9: Validity of a brief depression severity measure. J. Gen. Intern. Med. 16,
606–613 (2001).
59. Bradley, M. M. & Lang, P. J. Measuring emotion: e self-assessment maniin and the semantic dierential. J. Behav. er. Exp.
Psychiatry 25, 49–59 (1994).
60. Lang, P. J., Bradley, M. M. & Cuthbert, B. N. International Aective Picture System (IAPS): Technical Manual and Aective atings.
NIMH Cent. Study Emot. Atten. 39–58, https://doi.org/10.1027/0269-8803/a000147 (1997).
61. Nanda, U., Pati, D., Ghamari, H. & Bajema, . Lessons from neuroscience: form follows function, emotions follow form. Intell.
Build. Int. 5, 61–78 (2013).
62. ussell, J. A. A circumplex model of aect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980).
63. Sejima, . azuyo Sejima. 1988–1996. El Croquis 15 (1996).
64. Ochiai, H. et al. Physiological and Psychological Eects of Forest erapy on Middle-Aged Males with High-NormalBlood
Pressure. Int. J. Environ. es. Public Health 12, 2532–2542 (2015).
65. Noguchi, H. & Saaguchi, T. Eect of illuminance and color temperature on lowering of physiological activity. Appl. Hum. Sci. 18,
117–123 (1999).
66. üller, ., Miellides, B. & Janssens, J. Color, arousal, and performance—A comparison of three experiments. Color es. Appl. 34,
141–152 (2009).
67. Yildirim, ., Hidayetoglu, M. L. & Capanoglu, A. Eects of interior colors on mood and preference: comparisons of two living
rooms. Percept. Mot. Sills 112, 509–524 (2011).
68. Hogg, J., Goodman, S., Porter, T., Miellides, B. & Preddy, D. E. Dimensions and determinants of judgements of colour samples and
a simulated interior space by architects and nonarchitects. Br. J. Psychol. 70, 231–242 (1979).
69. Jalil, N. A., Yunus, . M. & Said, N. S. Environmental Colour Impact upon Human Behaviour: A eview. Procedia - Soc. Behav. Sci.
35, 54–62 (2012).
70. Jacobs, . W. & Hustmyer, F. E. Eects of four psychological primary colors on GS, heart rate and respiration rate. Percept. Mot.
Sills 38, 763–766 (1974).
71. Jin, H. ., Yu, M., im, D. W., im, N. G. & Chung, A. S. W. Study on Physiological esponses to Color Stimulation. In International
Association of Societies of Design esearch (ed. Poggenpohl, S.) 1969–1979 (orean Society of Design Science, 2009).
72. Vartanian, O. et al. Impact of contour on aesthetic judgments and approach-avoidance decisions in architecture. Proc. Natl. Acad.
Sci. 110, 1–8 (2013).
73. Tsunetsugu, Y., Miyazai, Y. & Sato, H. Visual eects of interior design in actual-size living rooms on physiological responses.
Build. Environ. 40, 1341–1346 (2005).
74. Stamps, A. E. Physical Determinants of Preferences for esidential Facades. Environ. Behav. 31, 723–751 (1999).
75. Berlyne, D. E. Novelty, Complexity, and Hedonic Value. Percept. Psychophys. 8, 279–286 (1970).
76. rueger, . A. & Casey, M. Focus groups: a practical guide for applied research. (Sage Publications, 2000).
77. Acharya, U. ., Joseph, . P., annathal, N., Lim, C. M. & Suri, J. S. Heart rate variability: A review. Med. Biol. Eng. Comput. 44,
1031–1051 (2006).
78. Tarvainen, M. P., Nisanen, J. P., Lipponen, J. A., anta-aho, P. O. & arjalainen, P. A. ubios HV - Heart rate variability analysis
soware. Comput. Methods Programs Biomed. 113, 210–220 (2014).
79. Pan, J. & Tompins, W. J. A real-time QS detection algorithm. Biomed. Eng. IEEE Trans. 1, 230–236 (1985).
80. Tarvainen, M. P., anta-aho, P. O. & arjalainen, P. A. An advanced detrending method with application to HV analysis. IEEE
Trans. Biomed. Eng. 49, 172–175 (2002).
81. Valenza, G. et al. Predicting Mood Changes in Bipolar Disorder rough HeartbeatNonlinear Dynamics. IEEE J. Biomed. Heal.
Informatics 20, 1034–1043 (2016).
82. Pincus, S. & Viscarello, . Approximate Entropy A regularity measure for fetal heart rate analysis. Obstet. Gynecol. 79, 249–255
(1992).
83. ichman, J. & Moorman, J. Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Hear.
Circ Physiol 278, H2039–H2049 (2000).
84. Peng, C.-., Havlin, S., Stanley, H. E. & Goldberger, A. L. Quantication of scaling exponents and crossover phenomena in
nonstationary heartbeat time series. Chaos 5, 82–87 (1995).
85. Grassberger, P. & Procaccia, I. Characterization of strange attractors. Phys. ev. Lett. 50, 346–349 (1983).
86. Delorme, A. & Maeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent
component analysis. J. Neurosci. Methods 134, 9–21 (2004).
87. Colomer Granero, A. et al. A Comparison of Physiological Signal Analysis Techniques and Classiers for Automatic Emotional
Evaluation of Audiovisual Contents. Front. Comput. Neurosci. 10, 1–14 (2016).
88. ober, S. E., urzmann, J. & Neuper, C. Cortical correlate of spatial presence in 2D and 3D interactive virtual reality: An EEG
study. Int. J. Psychophysiol. 83, 365–374 (2012).
89. Hyvärinen, A. & Oja, E. Independent component analysis: Algorithms and applications. Neural Networs 13, 411–430 (2000).
90. Welch, P. D. e Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Aver. aging Over
Short, Modied Periodograms. IEEE Trans. AUDIO Electroacoust. 15, 70–73 (1967).
91. Mormann, F., Lehnertz, ., David, P. & Elger, E. C. Mean phase coherence as a measure for phase synchronization and its
application to the EEG of epilepsy patients. Phys. D Nonlinear Phenom. 144, 358–369 (2000).
92. Jollie, I. T. Principal Component Analysis, Second Edition. Encycl. Stat. Behav. Sci. 30, 487 (2002).
93. Schöllopf, B., Smola, A. J., Williamson, . C. & Bartlett, P. L. New support vector algorithms. Neural Comput 12, 1207–1245
(2000).
94. Yan, . & Zhang, D. Feature selection and analysis on correlated gas sensor data with recursive feature elimination. Sensors
Actuators, B Chem. 212, 353–363 (2015).
95. Chang, C.-C. & Lin, C.-J. Libsvm: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 1–27 (2011).
96. Lewis, P. A., Critchley, H. D., otshtein, P. & Dolan, . J. Neural correlates of processing valence and arousal in aective words.
Cereb. Cortex 17, 742–748 (2007).
97. McCall, C., Hildebrandt, L. ., Hartmann, ., Baczowsi, B. M. & Singer, T. Introducing the Wunderammer as a tool for emotion
research: Unconstrained gaze and movement patterns in three emotionally evocative virtual worlds. Comput. Human Behav. 59,
93–107 (2016).
98. Blae, J. & Guroca, H. B. Haptic glove with M braes for virtual reality. IEEE/ASME Trans. Mechatronics 14, 606–615 (2009).
99. Heydarian, A. et al. Immersive virtual environments versus physical built environments: A benchmaring study for building design
and user-built environment explorations. Autom. Constr. 54, 116–126 (2015).
100. uliga, S. F., rash, T., Dalton, . C. & Hölscher, C. Virtual reality as an empirical research tool - Exploring user experience in a
real building and a corresponding virtual model. Comput. Environ. Urban Syst. 54, 363–375 (2015).
101. Yeom, D., Choi, J.-H. & Zhu, Y. Investigation of the Physiological Dierences between Immersive Virtual Environment and Indoor
Enviorment in a Building. Indoor adn Built Enviornment 0, Accept (2017).
Content courtesy of Springer Nature, terms of use apply. Rights reserved

www.nature.com/scientificreports/
15
SCIEntIfIC RepoRts | (2018) 8:13657 | DOI:10.1038/s41598-018-32063-4
102. Combrisson, E. & Jerbi, . Exceeding chance level by chance: e caveat of theoretical chance levels in brain signal classication
and statistical assessment of decoding accuracy. J. Neurosci. Methods 250, 126–136 (2015).
103. He, C., Yao, Y. & Ye, X. An Emotion ecognition System Based on Physiological Signals Obtained by Wearable Sensors. In Wea rab le
Sensors and obots: Proceedings of International Conference on We ar able Sensors and obots 2015 (eds. Yang, C., Vir, G. S. & Yang,
H.) 15–25, https://doi.org/10.1007/978-981-10-2404-7_2 (Springer Singapore, 2017).
Acknowledgements
is work was supported by the Ministerio de Economía y Competitividad. Spain (Project TIN2013-45736-R).
Author Contributions
J.M., J.H., J.G., C.L., and M.A. devised the methodology. J.M., J.H., J.G. and C.L. dened the experimental setup
and acquired the experimental data. J.M., A.G., P.S. and G.V. processed and analysed data. All the authors co-
wrote the manuscript and approved the nal text.
Additional Information
Supplementary information accompanies this paper at https://doi.org/10.1038/s41598-018-32063-4.
Competing Interests: e authors declare no competing interests.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and
institutional aliations.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International
License, which permits use, sharing, adaptation, distribution and reproduction in any medium or
format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Cre-
ative Commons license, and indicate if changes were made. e images or other third party material in this
article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the
material. If material is not included in the article’s Creative Commons license and your intended use is not per-
mitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the
copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
© e Author(s) 2018
Content courtesy of Springer Nature, terms of use apply. Rights reserved

1.
2.
3.
4.
5.
6.
Terms and Conditions
Springer Nature journal content, brought to you courtesy of Springer Nature Customer Service Center GmbH (“Springer Nature”).
Springer Nature supports a reasonable amount of sharing of research papers by authors, subscribers and authorised users (“Users”), for small-
scale personal, non-commercial use provided that all copyright, trade and service marks and other proprietary notices are maintained. By
accessing, sharing, receiving or otherwise using the Springer Nature journal content you agree to these terms of use (“Terms”). For these
purposes, Springer Nature considers academic use (by researchers and students) to be non-commercial.
These Terms are supplementary and will apply in addition to any applicable website terms and conditions, a relevant site licence or a personal
subscription. These Terms will prevail over any conflict or ambiguity with regards to the relevant terms, a site licence or a personal subscription
(to the extent of the conflict or ambiguity only). For Creative Commons-licensed articles, the terms of the Creative Commons license used will
apply.
We collect and use personal data to provide access to the Springer Nature journal content. We may also use these personal data internally within
ResearchGate and Springer Nature and as agreed share it, in an anonymised way, for purposes of tracking, analysis and reporting. We will not
otherwise disclose your personal data outside the ResearchGate or the Springer Nature group of companies unless we have your permission as
detailed in the Privacy Policy.
While Users may use the Springer Nature journal content for small scale, personal non-commercial use, it is important to note that Users may
not:
use such content for the purpose of providing other users with access on a regular or large scale basis or as a means to circumvent access
control;
use such content where to do so would be considered a criminal or statutory offence in any jurisdiction, or gives rise to civil liability, or is
otherwise unlawful;
falsely or misleadingly imply or suggest endorsement, approval , sponsorship, or association unless explicitly agreed to by Springer Nature in
writing;
use bots or other automated methods to access the content or redirect messages
override any security feature or exclusionary protocol; or
share the content in order to create substitute for Springer Nature products or services or a systematic database of Springer Nature journal
content.
In line with the restriction against commercial use, Springer Nature does not permit the creation of a product or service that creates revenue,
royalties, rent or income from our content or its inclusion as part of a paid for service or for other commercial gain. Springer Nature journal
content cannot be used for inter-library loans and librarians may not upload Springer Nature journal content on a large scale into their, or any
other, institutional repository.
These terms of use are reviewed regularly and may be amended at any time. Springer Nature is not obligated to publish any information or
content on this website and may remove it or features or functionality at our sole discretion, at any time with or without notice. Springer Nature
may revoke this licence to you at any time and remove access to any copies of the Springer Nature journal content which have been saved.
To the fullest extent permitted by law, Springer Nature makes no warranties, representations or guarantees to Users, either express or implied
with respect to the Springer nature journal content and all parties disclaim and waive any implied warranties or warranties imposed by law,
including merchantability or fitness for any particular purpose.
Please note that these rights do not automatically extend to content, data or other material published by Springer Nature that may be licensed
from third parties.
If you would like to use or distribute our Springer Nature journal content to a wider audience or on a regular basis or in any other manner not
expressly permitted by these Terms, please contact Springer Nature at
onlineservice@springernature.com