PreprintPDF Available

Variance in Classifying Affective State via Electrocardiogram and Photoplethysmography

  • Munster Technological University
  • Munster Technological University
Preprints and early-stage research may not have been peer reviewed yet.

Abstract and Figures

Advances in wearable technology have significantly increased the sensitivity and accuracy of devices for recording physiological signals. Commercial off-the-shelf wearable devices can gather large quantities of physiological data un-obtrusively. This enables momentary assessments of human physiology, which provide valuable insights into an individual’s health and psychological state. Leveraging these insights provides significant benefits for human-to-computer interaction and personalised health care. This work contributes an analysis of variance occurring in features representative of affective states extracted from electrocardiograms and photoplethysmography; subsequently identifies the cardiac measures most descriptive of affective states from both signals and provides insights into signal and emotion-specific cardiac measures; finally baseline performance for automated affective state detection from physiological signals is established.
Content may be subject to copyright.
Variance in Classifying Affective State via
Electrocardiogram and Photoplethysmography
1st Zachary Dair
Munster Technological University
2nd Dr Samantha Dockray
University College Cork
3rd Dr Ruairi O’Reilly
Munster Technological University
Abstract—Advances in wearable technology have significantly
increased the sensitivity and accuracy of devices for recording
physiological signals. Commercial off-the-shelf wearable devices
can gather large quantities of physiological data un-obtrusively.
This enables momentary assessments of human physiology, which
provide valuable insights into an individual’s health and psycho-
logical state. Leveraging these insights provides significant bene-
fits for human-to-computer interaction and personalised health-
care. This work contributes an analysis of variance occurring in
features representative of affective states extracted from electro-
cardiograms and photoplethysmography; subsequently identifies
the cardiac measures most descriptive of affective states from
both signals and provides insights into signal and emotion-specific
cardiac measures; finally baseline performance for automated
affective state detection from physiological signals is established.
Index Terms—Affective Computing, Psychophysiology, Elec-
trocardiogram, Photoplethysmography, Affective States
A significant goal of Affective Computing is to improve hu-
man to computer interaction by providing a system with a level
of emotional intelligence that aids natural communications and
is capable of including emotional components [1]. This has
commonly been approached by deriving emotional states from
speech, facial expressions, gesture and body posture analysis.
Utilising physiological signals to communicate psychological
information is a recent exploration in the domain, likely
stemming from the increased and growing accessibility of
signals from wearable devices.
A physiological signal represents an individual’s biologi-
cal processes, derived from core aspects of human biology.
Analysis of these signals can enable diagnostics, for instance,
analysing heart rate to detect arrhythmia [2]. Psychological
analysis can also be enabled as mental states originating from
unconscious effort typically present a noticeable physiological
change in the relevant human system [3]. The combined
analysis enables a richer understanding of individuals’ in terms
of their mental and physical health [4].
Psychological states are complex processes comprised of
several components, including feelings, cognitive reactions,
behaviour and thoughts [5]. Mapping psychological states to a
corresponding experience of the individual provides valuable
information in the context of well-being, health (physical and
mental), social contexts, experiences and emotional responses
Electrocardiograms (ECG) are physiological signals that
measure the electrical activity of the heart. Typically recorded
in a clinical setting using multiple leads and electrodes at-
tached to the chest or whole body of the individual. Recent
improvements have seen the development of wearable ECG
monitors predominately limited to research-grade (RG) and
medical-grade (MG) devices, with a small number of com-
mercial off-the-shelf (COTS) devices. Photoplethysmography
(PPG) was developed to measure heart activity through vari-
ations in blood volume of the skin, using a light-emitting-
diode and photodetector. Until recently, PPG was the sole
method provided by (COTS) devices such as smart-rings or
smart-watches to enable individuals to monitor heart activity
A concern with the analysis of ECG and PPG is the variance
caused by differing sensor placement and signal granularity
[4], [7]. As the sampling frequency of PPG is lower to
reduce battery consumption in COTS devices. These variances
motivate the investigation of the suitability and performance
of both signals to detect a range of affective states.
This work investigates the variance in classifying affective
states from physiological signals representative of heart activ-
ity by addressing the following research aims (i) To evaluate
signal specific variances in standard cardiac measures utilised
for emotive classification; (ii) To highlight the precedence of
cardiac features in classifying affective state via feature im-
portance; (iii) To evaluate the variance in automated affective
state classification between ECG and PPG.
A. Psychological Constucts
Multiple psychological constructs exist to describe human
psychology. These constructs range from discrete models,
where each psychological state is an individual component,
to dimensional models where the emotions span two or more
dimensions. For example, Basic Emotion Theory is a discrete
model containing the emotions: Anger, Joy, Sadness, Surprise,
Fear and Disgust [8], and the Circumplex Model [9] is a two-
dimensional model consisting of, Arousal, the activation level
of the individual commonly seen as excitement, and Valence,
which is the unpleasantness or pleasantness of the experience.
Existing affective state detection research has focused on
discrete and dimensional models [10]. Additionally, specific
arXiv:2207.02916v1 [cs.HC] 6 Jul 2022
mental states such as Stress and Anxiety have received sub-
stantial interest due to their health impact [11], [12]. This work
utilises Arousal and Valence as represented in the Circumplex
model and discrete psychological states relating to stress,
providing analysis from both perspectives. Stress is a complex
mental state, included in some emotion models discretely or
as a combination of high arousal and negative valence [13].
B. Heart-Related Physiological Signals
The prevalence of ECG and heart-related data in wearable
health monitors stems from a desire to monitor a critical organ.
This data has clear ties to health through arrhythmia detection
and heart rate as a measure of fitness [2]. Furthermore, as the
heart is controlled involuntarily through the autonomic nervous
system (ANS), it facilitates the identification of a relationship
between involuntary physiological changes and psychological
states. Multiple psychophysiological theories aim to explain
this relationship, such as Polyvagal Theory [14].
Heart activity is complex to capture. In medicine, the gold
standard utilises a 12-lead ECG, resulting in comprehensive
data recorded from multiple electrodes on the human body.
However, in ambulatory research and daily life, this method is
not feasible. Typically RG equipment uses several electrodes,
commonly 3-lead ECG; and occasionally including PPG as an
additional measure. COTS devices tend to rely solely on PPG
to monitor heart activity, however with recent advances, top
of the range smart-watches (Apple Watch 4-9, Galaxy Active
2, Fitbit) now include a 1-lead ECG, which is promising for
portable ECG analysis [15].
The Apple Watch records the time between R waves of
the QRS signal, these RR intervals (RR) enable Heart Rate
Variability (HRV) analysis for the detection of relaxation
and stress. The statistical analysis approach presented in
[16] demonstrates the capability of COTS devices to suit-
ably detect the RR intervals to a high standard. Notably,
approximately 10% of heartbeats were missing from the Apple
Watch recording. This missing data impacts the computation
of HRV features and subsequently the classification accuracy
of automated affective state detection.
Additional physiological signals such as electrodermal ac-
tivity (EDA), respiration, skin temperature, electromyogram
(EMG), and electrooculogram (EOG) have demonstrated po-
tential for affective state detection [4], [10], however, due to
additional sensor requirements are excluded from this work.
C. Affective ECG Analysis
Numerous studies of affective states conduct custom data
collection, providing precise control over the psychological
domain explored. Varied stimuli have aided the elicitation
of psychological states, for example, images, movie clips,
music, and dedicated tasks to elucidate stress, such as the Trier
Social Stress Test [17], [18]. Additionally a large number of
open-access or on-request datasets have been created, a subset
are utilised in [19], namely AMIGOS”, “DREAMER” [20],
“WESAD” [17], “SWELL-KW” [21]. There is a distinct lack
of emotionally labelled ECG signals from COTS devices. This
is likely due to only recent COTS devices providing ECG
monitoring capabilties [15].
ECG signals contain noise introduced by motion arte-
facts, biological differences and sensor de-attachment. Sig-
nal processing techniques such as Butterworth Bandpass and
Notch filters are utilised to reduce these noise levels [5].
Subsequently, features suitable for emotive classification are
extracted from the pre-processed signals. Often statistical and
frequency measures of ECG are utilised, however, arguably the
most valuable features relate directly to physiological changes
in the heart, commonly analysed by the parameters P,Q,R,S,T,
each relating to a process in the heartbeat cycle. Features
derived from these parameters, such as the QRS complex,
Beats-Per-Minute (BPM), and HRV, are inherently linked to
human psychology as they originate from involuntary changes
due to the ANS, as explored by Poly-Vagal theory.
Recent approaches have favoured deep learning method-
ologies [19], achieving significant accuracies on multi-class
classifications. However, older studies focusing on linear and
quadratic discriminant analysis remain relevant, achieving
suitable accuracy for their respective classifications [5], [22].
Furthermore, adapted ML classifiers and combinations of
ML classifiers forming ensembles have demonstrated potential
for binary classifications in emotion detection [23], [24]. In
comparison to other studies, [19] achieved the highest accuracy
for multiple emotion detection from ECG data and reported
setting the new state of the art for ECG emotion detection.
D. Affective PPG Analysis
PPG analysis provided by COTS devices has typically
focused on tracking medical conditions, physical activity, and
stress. The detrimental effects of stress on human health is a
significant motivator for physiological analysis, and preventa-
tive healthcare research [25].
Instances of PPG have demonstrated similar noise levels to
ECG, in addition skin tone and environmental light effects
can impact the signal quality. PPG, EDA (a measure of the
electrical potential of the skin) and acceleration were recorded
in [25] to create a multi-modal stress detection model. Heart
rate features such as inter-beat intervals, HRV and frequency
measures alongside EDA were leveraged with a range of ML
classifiers to achieve 88.20% across all subjects.
Further stress detection studies focused on the WESAD
dataset, [26] utilising raw PPG data, with the aim of re-
moving the requirement for hand-crafted features. In their
“Neutral, Stress, Amusement” experiment, an LDA classifier
reached 65.3% accuracy classifying 60-second windows of
PPG, demonstrating the utility of raw PPG for stress detection.
A recent approach [27] reports 99% accuracy on WESAD
detecting baseline, amusement, meditation and stress. This
approach conducts extensive signal processing techniques,
including windowing the PPG data into 5-second windows.
Furthermore, a complex feature extraction stage method was
adopted, relying on autoencoder features and recursive feature
elimination contributing to the high accuracy achieved.
TABLE I: Details per dataset utilised
Dataset ECG kHz PPG kHz # Windows Label
CASE 1000 1000 14650 Arousal/Valence
WESAD 700 64 9106 B S A M
B: Baseline, S: Stress, A: Amusement, M: Meditation
A. Datasets
For the purposes of this work, the focus was narrowed to RG
physiological signals due to a lack of publicly available data
for COTS devices. “The Dataset of Continuous Affect Anno-
tations and Physical Signals for Emotion Analysis” (CASE)
[28] and “The Wearable Stress and Affect Detection Dataset”
(WESAD) [17]see Table I) were the datasets utilised in this
work. The datasets were selected due to their inclusion of
ECG and PPG with psychological annotations, these signals
were recorded using RG devices in a laboratory environment.
CASE incorporates Arousal and Valence annotations, achieved
by collecting joystick movement resulting from emotionally
stimulating video clips. WESAD focuses on stress detection
with limited affective states: a baseline state elicited from
“neutral reading”, amusement caused by comedic video clips,
a Trier Social Stress Test [18] to provoke stress, and a med-
itation stage aimed at “de-exciting” the individual following
the amusement and stress stages.
B. Pre-Processing
ECG and PPG signals recorded per subject within these
datasets span the duration of the experiment resulting in
approx 91/58 minutes for WESAD/CASE, utilising a signal
processing method known as windowing, 10-second segments
of data were isolated. A 10-second duration was selected due
to efficient performance demonstrated in [19], additionally, this
duration enables low latency as classification occurs every 10-
seconds and contains adequate data for feature computation.
A Butterworth-Bandpass filter is used to reduce signal
noise, facilitating the extraction of selected features while
maintaining a degree of “rawness” in the signal. A simplistic
filter is used as it more closely aligns with COTS devices and
their reduced computational power.
Once filtered and windowed, the data is aligned with its
psychological annotations. For WESAD, annotations were
numeric values sampled at 700Hz. Each value from 0-4 is
associated with the psychological states: Transient, Baseline,
Stress, Amusement and Meditation. Annotations 5-7, and
Transient data are omitted as per the author’s instructions [17].
Certain windows may include multiple emotive annotations;
hence to identify the most pertinent emotion, the mean of all
annotation values per window is calculated and rounded to the
nearest annotation (1-4) using Euclidean distance. Alternative
approaches [24] omit these windows and the neighbouring
segments to prevent confusion from mixed emotions.
A similar procedure is required for CASE; the raw annota-
tion data is provided as values on an x and y-axis representing
Arousal and Valence [28], these values are normalised to a
TABLE II: Cardiac features extracted from ECG and PPG
Feature Abbrv.
Beats Per Minute BPM
Interbeat Interval IBI
Std dev. of RR Intervals SDNN
Std dev. of successive differences SDSD
RMSE of successive differences RMSSD
Proportion of differences above (20ms)/(50ms) (pNN20)/(pNN50)
Median absolute deviation of RR intervals MAD
Low frequency spectrum 0.05-0.15Hz LF
High frequency spectrum 0.15-0.5Hz HF
High/Low frequency ratio HF/LF
Estimated breathing rate BR
range of 0.5 to 9.5, and subsequently converted to discrete
representations, resulting in low (0.5-3.5), neutral (3.5-6.5) and
high (6.5-9.5) Arousal and Valence for each window.
C. Emotion Characteristics in Cardiac Signals
Cardiac signals provide a wide array of features. Many
exhibit high efficacy indicating emotional information from
the source signal. The python toolkit HeartPy [29] extracts
the features seen in Table II from 10-second windows of data.
D. Signal Variation Analysis
This work measures the absolute difference in extracted
features BPM, IBI and breathing rate from each signal as
a means of evaluating ECG and PPG variance. The method
utilises the features extracted from 60-second windows of data
for each signal.
Identifying the most influential cardiac features for psy-
chological classifications provides valuable insights for signal
choice. Furthermore, a variance in the importance of the
cardiac features between ECG and PPG may indicate higher
suitability in one signal over another. A game theory approach
for ML explanations known as “Shapley Additive exPlana-
tions“ [30] is adopted to identify feature importance. This
approach uses SHAP values which represent the degree of
change on the model output caused by each individual feature,
the magnitude of change and number of samples affected
indicate the impact factor of a given feature.
E. Automated Emotion Detection
Comparing performance measures across ten ML classifiers
provides a method for identifying the suitability of ECG and
PPG for automated affective state detection. The annotated
features for each signal are loaded individually on a per dataset
basis. 20% of the data acts as a hold-out set, essentially
un-seen data to evaluate the final classifier. Five-fold cross-
validation is utilised, transforming the remaining 80% of data
into “folds”, enabling a per fold classification. Subsequently,
comparing the best inter-fold classifier enables the identifica-
tion of the more robust and performant classifiers. Finally, the
selected classifier is trained on the initial 80% of data and
tested on the hold-out set.
Fig. 1: Five-Fold cross-validation of selected models detecting
multiple affective states
Fig. 2: Absolute difference for the IBI, BPM and BR feature
values across ECG and PPG
A. Cardiac Feature Variance
The wearables’ sample rate disparity (See Table I) is ev-
ident in the feature level results displayed in Figure 2. The
reduced sample rates in WESAD result in slightly decreased
granularity of ECG data and significantly in PPG data com-
pared to CASE. Furthermore, it leads to a visible impact
on feature computation, as demonstrated by IBI and BPM.
In CASE, IBI and BPM contain a small degree of variance
with substantial spikes relative to the average. These occur
in isolated data segments, likely caused by electrode discon-
nection, movement, or subject-specific factors. However, a
significantly higher fluctuation occurs in WESAD in terms
of magnitude and frequency, likely stemming from the high
sample rate disparity and reduced PPG granularity. Notably,
a similar degree of variance occurs in BPM and IBI due to
their inherent links to heart rate. Interestingly, the BR feature
exhibits a high deviation between ECG and PPG in both
datasets. This deviation indicates that one of the signals is
unreliably computing BR, likely caused by the wrist and finger
Fig. 3: Classification impact of ECG and PPG features from
WESAD indicated by Shapley Additive exPlanations (SHAP)
placement of the PPG sensors. Furthermore, the previously
identified variance spikes of IBI and BPM in CASE are present
in BR, further demonstrating that specific data segments may
benefit from additional signal processing to reduce noise and
increase classification accuracy.
B. Cardiac Feature Importance
Twenty data windows failed feature extraction due to a
lack of discernible heart rate. These problematic windows
demonstrated significant signal spikes and sporadic behaviour,
akin to electrode disconnection, motion artefacts, and high
signal noise. For this analysis, the occurrence has minimal
impact. However, it will occur more frequently in ambulatory
analysis, requiring further signal processing to overcome.
Analysing the SHAP values per feature indicates that IBI,
BPM, and BR have the most impact on classification. The
remaining features, most notably those relating to frequency,
exhibit inconsistent influence between the signals. Further-
more, the adopted feature importance approach enables the
evaluation of feature influence per affective state, as demon-
strated in Figure 3. The disparity in the influence of the
same feature across multiple affective states indicates a higher
utility for certain features to represent specific affective states,
specifically shown with RMSSD in PPG for stress detection.
C. Automated Affective State Classification Variance
Random Forest (RF) was selected as the most perfor-
mant classifier and subsequently evaluated on the hold-out
set, where ECG outperforms PPG consistently across both
datasets. The full model comparison is shown in Figure 1.
In contrast with the state of the art [19], [24] the performance
achieved is much lower for ECG and PPG; however, this
work focuses on the variance between the signals for affective
analysis rather than achieving high classification accuracy.
Analysing the ROC curves from RF demonstrates the true
and false positive rates per signal for each affective state, see
Figure 4. On average, ECG demonstrates increased capabilities
for affective classification by achieving a higher ROC area than
PPG, varying with a range of 0.02-0.10. Interestingly, ECG
(b) CASE
Fig. 4: ROC Curves representing the OVR classification vari-
ance between ECG and PPG
and PPG reach an identical area value for Stress classifications
in WESAD data. Further investigations identify a potential
trade-off between true and false positives using PPG for Stress
The identified variance of BPM and IBI in WESAD demon-
strates the importance of sampling rates to prevent inconsistent
computations of features due to signal granularity differences.
Furthermore, feature variance analysis enables the identifica-
tion of abnormal signal activity from sensor disconnection or
motion artefacts, valuable for ambulatory analysis.
Feature importance identifies IBI, BPM and BR as the most
impactful features for affective classification across ECG and
PPG. Notably, the remaining features exhibit inconsistent im-
pacts, specifically SD1/SD2 and RMSSD, which demonstrate
a greater impact in PPG, warranting the exploration of signal-
specific features. Moreover, a variance per affective state indi-
cates that certain features provide a greater degree of emotion-
specific information beneficial for tailored applications.
Standard ML classifiers achieve moderate classification
accuracy for detecting multiple affective states using heart-
activity features extracted from minimally filtered ECG and
PPG signals. Therefore providing a baseline for automated
multi-class affective state detection and demonstrating the
validity of heart-activity features from ECG and PPG.
[1] Rosalind W. Picard. Affective computing: challenges. International
Journal of Human-Computer Studies, 59(1):55–64, 2003.
[2] Eduardo Jos´
e da S. Luz et al. Ecg-based heartbeat classification for
arrhythmia detection: A survey. CMPB, 127:144–164, 2016.
[3] Jonghwa Kim and E. Andr´
e. Emotion recognition based on physiological
changes in music listening. IEEE PAMI, 30(12):2067–2083, 2008.
[4] Andrius Dzedzickis, A. Kaklauskas, and V. Bucinskas. Human emotion
recognition: Review of sensors and methods. Sensors, 20(3), 2020.
[5] Foteini Agrafioti et al. Ecg pattern analysis for emotion detection. IEEE
Trans. Affective Comput., 3(1):102–115, 2012.
[6] Samantha Dockray, Siobh´
an O’Neill, and Owen Jump. Measuring the
psychobiological correlates of daily experience in adolescents. Journal
of Research on Adolescence, 29(3):595–612, 2019.
[7] Shadi Mahdiani et al. Is 50 hz high enough ecg sampling frequency for
accurate hrv analysis? In EMBC, pages 5948–5951, 2015.
[8] Paul Ekman. Basic emotions. Handbook of cognition and emotion,
pages 45–60, 1999.
[9] Jonathan Posner et al. The circumplex model of affect. Development
and Psychopathology, 17(3):715–734, 2005.
[10] Lin Shu et al. A review of emotion recognition using physiological
signals. Sensors, 18(7):2074, Jun 2018.
[11] Huseyin Uyarel, Ertan O, Necati C, Ahmet K, and Nese C. Effects of
anxiety on qt dispersion in healthy young men. Acta Cardio., 2006.
[12] Can Yekta Said et al. Continuous stress detection using wearable sensors
in real life: Algorithmic programming contest case study. Sensors, 2019.
[13] A. K. Johnson and E. A. Anderson. Stress and arousal. In J. T. Cacioppo
and L. G. Tassinary (Eds.), Principles of psychophysiology: Physical,
social, and inferential elements. Cambridge University Press, 1990.
[14] Porges S W et al. Vagal tone and the physiological regulation of emotion.
Monogr. Soc. Res. Child Dev., 59(2-3):167–186, 1994.
[15] Nabeel Saghir et al. A comparison of manual electrocardiographic
interval and waveform analysis in lead 1 of 12-lead ecg and apple watch
ecg: A validation study. Cardiovascular Digital Health Journal, 2020.
[16] David Hernando, S. Roca, J. Sancho, ´
A. Alesanco, and R. Bail´
Validation of the apple watch for heart rate variability measurements
during relax and mental stress in healthy subjects. Sensors, 18(8), 2018.
[17] Philip at al. Schmidt. Introducing wesad, a multimodal dataset for
wearable stress and affect detection. In ICMI 20. ACM, 2018.
[18] Melissa A Birkett. The trier social stress test protocol for inducing
psychological stress. J. Vis. Exp., October 2011.
[19] Pritam Sarkar and Ali Etemad. Self-supervised ECG representation
learning for emotion recognition. IEEE Trans. Affective Comput., 2021.
[20] Stamos Katsigiannis and Naeem Ramzan. Dreamer: A database for
emotion recognition through eeg and ecg signals from wireless low-cost
off-the-shelf devices. IEEE Journal of Bio. and Health Info., 2018.
[21] Saskia Koldijk, Maya Sappelli, Suzan Verberne, Mark A. Neerincx, and
Wessel Kraaij. The SWELL knowledge work dataset for stress and user
modeling research. In ICMI, pages 291–298. ACM, 2014.
[22] Mimma Nardelli, G. Valenza, A. Greco, A. Lanata, and P. Scilingo.
Recognizing emotions induced by affective sounds through heart rate
variability. IEEE Trans. Affective Comput., 6(4):385–394, 2015.
[23] Yu-Liang Hsu, Jeen-Shing Wang, Wei-Chun Chiang, and Chien-Han
Hung. Automatic ECG-based emotion recognition in music listening.
IEEE Trans. Affective Comput., 11(1):85–99, 2020.
[24] Theekshana Dissanayake, Y. Rajapaksha, R. Ragel, and I. Nawinne. An
ensemble learning approach for electrocardiogram sensor based human
emotion recognition. Sensors, 19(20), 2019.
[25] Yekta Said Can, Niaz Chalabianloo, Deniz Ekiz, and Cem Ersoy. Con-
tinuous stress detection using wearable sensors in real life: Algorithmic
programming contest case study. Sensors, 19(8):1849, 2019.
[26] Aneta Lisowska, Szymon Wilk, and Mor Peleg. Catching patient’s
attention at the right time to help them undergo behavioural change.
In Artificial Intelligence in Medicine, pages 72–82. Spr. Int. Pub., 2021.
[27] Nilava Mukherjee, Sumitra Mukhopadhyay, and Rajarshi Gupta. Real-
time mental stress detection technique using neural networks towards a
wearable health monitor. Meas. Sci. Technol., 33(4):044003, 2022.
[28] Sharma Karan et al. A dataset of continuous affect annotations and
physiological signals for emotion analysis, 2018.
[29] Paul van Gent et al. Heartpy: A novel heart rate algorithm for the
analysis of noisy signals. Transportation Research Part F: Traffic
Psychology and Behaviour, 66:368–378, 2019.
[30] Christoph Molnar. Interpretable Machine Learning., 2022.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Background The Apple Watch Series 4 (AW) can detect atrial fibrillation and perform a single-lead electrocardiogram (ECG), but the clinical accuracy of AW ECG waveforms compared to lead 1 of a 12-lead ECG is unclear. Objective The purpose of this study was to assess the accuracy of interval measurements on AW ECG tracings in comparison to lead 1 on a 12-lead ECG. Methods We obtained ECGs at a university hospital of healthy volunteers age >18 years. ECG waveforms were measured with calipers to the nearest 0.25 mm. When possible, 3 consecutive waveforms in lead 1 were measured. Waveform properties, including intervals, were recorded. Concordance correlation coefficients and Bland-Altman plots were used to assess level of agreement between devices. Results Twelve-lead (n = 113) and AW (n = 129) ECG waveforms from 43 volunteers (mean age 31 years; 65% female) were analyzed. Sinus rhythm interpretation between devices was 100% concordant. No arrhythmias were recorded. Mean difference (d) for heart rate was 1.16 ± 4.33 bpm (r = 0.94); 3.83 ± 113.54 ms for RR interval (r = 0.79); 5.43 ± 17 ms for PR interval (r = 0.83); –6.89 ± 14.81 ms for QRS interval (r = 0.65); –11.27 ± 22.9 ms for QT interval (r = 0.79); and –11.67 ± 27 ms for QTc interval (r = 0.57). There was moderate (d <40 ms) to strong (d <20 ms or < 5 bpm) agreement between devices represented by Bland-Altman plots. Conclusion The AW produces accurate ECGs in healthy adults with moderate to strong agreement of basic ECG intervals.
Full-text available
Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human–robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.
Full-text available
Recently, researchers in the area of biosensor based human emotion recognition have used different types of machine learning models for recognizing human emotions. However, most of them still lack the ability to recognize human emotions with higher classification accuracy incorporating a limited number of bio-sensors. In the domain of machine learning, ensemble learning methods have been successfully applied to solve different types of real-world machine learning problems which require improved classification accuracies. Emphasising on that, this research suggests an ensemble learning approach for developing a machine learning model that can recognize four major human emotions namely: anger; sadness; joy; and pleasure incorporating electrocardiogram (ECG) signals. As feature extraction methods, this analysis combines four ECG signal based techniques, namely: heart rate variability; empirical mode decomposition; with-in beat analysis; and frequency spectrum analysis. The first three feature extraction methods are well-known ECG based feature extraction techniques mentioned in the literature, and the fourth technique is a novel method proposed in this study. The machine learning procedure of this investigation evaluates the performance of a set of well-known ensemble learners for emotion classification and further improves the classification results using feature selection as a prior step to ensemble model training. Compared to the best performing single biosensor based model in the literature, the developed ensemble learner has the accuracy gain of 10.77%. Furthermore, the developed model outperforms most of the multiple biosensor based emotion recognition models with a significantly higher classification accuracy gain.
Full-text available
From a computational viewpoint, emotions continue to be intriguingly hard to understand. In research, a direct and real-time inspection in realistic settings is not possible. Discrete, indirect, post-hoc recordings are therefore the norm. As a result, proper emotion assessment remains a problematic issue. The Continuously Annotated Signals of Emotion (CASE) dataset provides a solution as it focusses on real-time continuous annotation of emotions, as experienced by the participants, while watching various videos. For this purpose, a novel, intuitive joystick-based annotation interface was developed, that allowed for simultaneous reporting of valence and arousal, that are instead often annotated independently. In parallel, eight high quality, synchronized physiological recordings (1000 Hz, 16-bit ADC) were obtained from ECG, BVP, EMG (3x), GSR (or EDA), respiration and skin temperature sensors. The dataset consists of the physiological and annotation data from 30 participants, 15 male and 15 female, who watched several validated video-stimuli. The validity of the emotion induction, as exemplified by the annotation and physiological data, is also presented.
Full-text available
The negative effects of mental stress on human health has been known for decades. High-level stress must be detected at early stages to prevent these negative effects. After the emergence of wearable devices that could be part of our lives, researchers have started detecting extreme stress of individuals with them during daily routines. Initial experiments were performed in laboratory environments and recently a number of works took a step outside the laboratory environment to the real-life. We developed an automatic stress detection system using physiological signals obtained from unobtrusive smart wearable devices which can be carried during the daily life routines of individuals. This system has modality-specific artifact removal and feature extraction methods for real-life conditions. We further tested our system in a real-life setting with collected physiological data from 21 participants of an algorithmic programming contest for nine days. This event had lectures, contests as well as free time. By using heart activity, skin conductance and accelerometer signals, we successfully discriminated contest stress, relatively higher cognitive load (lecture) and relaxed time activities by using different machine learning methods
Full-text available
Heart rate variability (HRV) analysis is a noninvasive tool widely used to assess autonomic nervous system state. The market for wearable devices that measure the heart rate has grown exponentially, as well as their potential use for healthcare and wellbeing applications. Still, there is a lack of validation of these devices. In particular, this work aims to validate the Apple Watch in terms of HRV derived from the RR interval series provided by the device, both in temporal (HRM (mean heart rate), SDNN, RMSSD and pNN50) and frequency (low and high frequency powers, LF and HF) domain. For this purpose, a database of 20 healthy volunteers subjected to relax and a mild cognitive stress was used. First, RR interval series provided by Apple Watch were validated using as reference the RR interval series provided by a Polar H7 using Bland-Altman plots and reliability and agreement coefficients. Then, HRV parameters derived from both RR interval series were compared and their ability to identify autonomic nervous system (ANS) response to mild cognitive stress was studied. Apple Watch measurements presented very good reliability and agreement (>0.9). RR interval series provided by Apple Watch contain gaps due to missing RR interval values (on average, 5 gaps per recording, lasting 6.5 s per gap). Temporal HRV indices were not significantly affected by the gaps. However, they produced a significant decrease in the LF and HF power. Despite these differences, HRV indices derived from the Apple Watch RR interval series were able to reflect changes induced by a mild mental stress, showing a significant decrease of HF power as well as RMSSD in stress with respect to relax, suggesting the potential use of HRV measurements derived from Apple Watch for stress monitoring.
Motivation: In recent times, mental stress detection using physiological signals have received widespread attention from the technology research community. Although many motivating research works have already been reported in this area, the evidence of hardware implementation is occasional. The main challenge in stress detection research is using optimum number of physiological signals, and real-time detection with low complexity algorithm. Objective: In this work, a real-time stress detection technique is presented which utilises only photoplethysmogram (PPG) signal to achieve improved accuracy over multi-signal-based mental stress detection techniques. Methodology: A short segment of 5s PPG signal was used for feature extraction using an autoencoder (AE), and features were minimized using recursive feature elimination (RFE) integrated with a multi-class support vector machine (SVM) classifier. Results: The proposed AE-RFE-SVM based mental stress detection technique was tested with WeSAD dataset to detect four-levels of mental state, viz., baseline, amusement, meditation and stress and to achieve an overall accuracy, F1 score and sensitivity of 99%, 0.99 and 98% respectively for 5s PPG data. The technique provided improved performance over discrete wavelet transformation (DWT) based feature extraction followed by classification with either of the five types of classifiers, viz., SVM, random forest (RF), k-nearest neighbour (k-NN), linear regression (LR) and decision tree (DT). The technique was translated into a quad-core-based standalone hardware (1.2 GHz, and 1 GB RAM). The resultant hardware prototype achieves a low latency (~0.4 s) and low memory requirement (~1.7 MB). Conclusion: The present technique can be extended to develop remote healthcare system using wearable sensors.
We exploit a self-supervised deep multi-task learning framework for electrocardiogram (ECG) -based emotion recognition. The proposed solution consists of two stages of learning a) learning ECG representations and b) learning to classify emotions. ECG representations are learned by a signal transformation recognition network. The network learns high-level abstract representations from unlabeled ECG data. Six different signal transformations are applied to the ECG signals, and transformation recognition is performed as pretext tasks. Training the model on pretext tasks helps the network learn spatiotemporal representations that generalize well across different datasets and different emotion categories. We transfer the weights of the self-supervised network to an emotion recognition network, where the convolutional layers are kept frozen and the dense layers are trained with labelled ECG data. We show that the proposed solution considerably improves the performance compared to a network trained using fully-supervised learning. New state-of-the-art results are set in classification of arousal, valence, affective states, and stress for the four utilized datasets. Extensive experiments are performed, providing interesting insights into the impact of using a multi-task self-supervised structure instead of a single-task model, as well as the optimum level of difficulty required for the pretext self-supervised tasks.
This paper describes the functioning and development of HeartPy: a heart rate analysis toolkit designed for photoplethysmogram (PPG) data. Most openly available algorithms focus on electrocardiogram (ECG) data, which has very different signal properties and morphology, creating a problem with analysis. ECG-based algorithms generally don’t function well on PPG data, especially noisy PPG data collected in experimental studies. To counter this, we developed HeartPy to be a noise-resistant algorithm that handles PPG data well. It has been implemented in Python and C. Arduino IDE sketches for popular boards (Arduino, Teensy) are available to enable data collection as well. This provides both pc-based and wearable implementations of the software, which allows rapid reuse by researchers looking for a validated heart rate analysis toolkit for use in human factors studies.
Mapping the psychobiological correlates of social contexts, experiences, and emotional responses of adolescents in their daily lives provides insight into how adolescent well‐being shapes, and is shaped by, experience. Measures of these psychobiological correlates are enabled by devices and technologies that must be precise and suitable for adolescent participants. The present report reviews the most often used research measures, and suggests strategies for best practice, drawn from practical experience. The rapid advances in technological methods to collect attuned measures of psychological processes, social context, and biological function indicate the promise for multimodal measures in ecological settings. Attaining these methodological goals will support research to secure comprehensive, quality data, and advance the understanding of psychobiological function in ambulatory settings.