Content uploaded by Eiman Kanjo
Author content
All content in this area was uploaded by Eiman Kanjo on Mar 06, 2020
Content may be subject to copyright.
ON-DEV IC E TRANSFER LEARNING FOR PERSONALISING PSYCHOLOGICAL
STRESS MODELLING USING A CONVOLUTIONAL NEURAL NETWORK
Kieran Woodward 1Eiman Kanjo 1David J. Brown1T.M. McGinnity 1
ABSTRACT
Stress is a growing concern in modern society adversely impacting the wider population more than ever before.
The accurate inference of stress may result in the possibility for personalised interventions. However, individual
differences between people limits the generalisability of machine learning models to infer emotions as people’s
physiology when experiencing the same emotions widely varies. In addition, it is time consuming and extremely
challenging to collect large datasets of individuals’ emotions as it relies on users labelling sensor data in real-time
for extended periods. We propose the development of a personalised, cross-domain 1D CNN by utilising transfer
learning from an initial base model trained using data from 20 participants completing a controlled stressor
experiment. By utilising physiological sensors (HR, HRV EDA) embedded within edge computing interfaces that
additionally contain a labelling technique, it is possible to collect a small real-world personal dataset that can be
used for on-device transfer learning to improve model personalisation and cross-domain performance.
1 INTRODUCTION
Modern lifestyles are contributing heavily to increased lev-
els of daily stress with more adults experiencing work-
related stress (Perkbox,2018) and more students experienc-
ing poor mental health (The Physiological Society,2017)
than ever before making it more vital than ever to manage
such pressures. The ability to infer emotions is an exciting
proposition as it could enable better management of well-
being by providing real-time interventions. Electrodermal
Activity (EDA) is often used to infer stress as it directly cor-
relates to the sympathetic nervous system (Schumm et al.,
2010), along with Heart Rate Variability (HRV). The lat-
ter represents the variation in time between heartbeats and
when HRV is reduced the user is more likely to be stressed
(Wijsman et al.,2011). Non-invasive physiological sen-
sors that can easily be embedded within wearables or edge
computing interfaces including those that measure Heart
Rate (HR), HRV and EDA present a significant opportunity
to be widely adopted unlike other invasive approaches us-
ing Electrocardiography (ECG) and Electroencephalogram
(EEG) which cannot be widely adopted outside of controlled
experiments.
1
School of Science and Technology, Nottingham Trent
University, Nottingham, United Kingdom. Correspondence
to: Kieran Woodward
<
kieran.woodward@ntu.ac.uk
>
,
Eiman Kanjo
<
eiman.kanjo@ntu.ac.uk
>
, David Brown
<
david.brown@ntu.ac.uk
>
, T.M. McGinnity
<
mar-
tin.mcginnity@ntu.ac.uk>.
Proceedings of the
2nd
SysML Conference, Palo Alto, CA, USA,
2019. Copyright 2019 by the author(s).
Sensor streams paired with sufficiently trained Artificial
Intelligence (AI) models may make the possibility of ubiq-
uitous emotion inference a reality. However, labelled emo-
tion data can be challenging to collect as unlike images
which can be crowdsourced (Google,2019) emotion data
must be collected and labelled in real-time. Longitudinal
data collection poses even greater challenges as it relies on
multiple users continually recording their emotions while
simultaneously wearing sensors for extended periods. These
drawbacks result in limited labelled data which significantly
impacts model accuracy as deep learning requires a vast
labelled training dataset. Furthermore, emotions are always
personal and individualistic with individual users experi-
encing large variation in physiological parameters when
experiencing the same emotions thus making a generalised
model additionally challenging.
Recent advancements in wearable EDA and HRV sensors
have enabled new research using machine learning clas-
sifiers to infer stress (Healey & Picard,2005). However,
this does not consider the personalisation of models or the
possibility of the models being used in different domains
for example a model developed using a controlled experi-
ment dataset may not perform as accurately when used in
real-world environments. It is essential to develop subject-
dependent models for stress detection as when working with
a heterogeneous population there are numerous physiolog-
ical factors such as age, gender and diet that result in a
variation of physiological data (Picard et al.,2001).
Transfer learning has most commonly been used for object
recognition (Oquab et al.,2014), human activity recogni-
Submission and Formatting Instructions for SysML 2019
tion (Sargano et al.,2017) and speech recognition (Wang
& Zheng,2016). However it may also be used to address
the challenges of domain adaption and personalisation by
using a pre-trained model from a different domain and trans-
ferring the learned knowledge to the target domain. Caru-
ana (Caruana,1997) introduced multi-task learning that
uses domain information contained in the training signals
of related tasks. Convolutional Neural Networks (CNN)
are commonly used in transfer learning approaches where
they are initially trained on a vast dataset and then the last
fully-connected layer is removed and further trained on the
smaller target dataset. A pre-trained base model removes the
requirement for a large target dataset while simultaneously
decreasing the time required to train the model.
The possibility for transfer learning to be used to personalise
affective models has previously been explored. However
most work has focused on personalising EEG signals, where
transfer learning approaches have improved model accuracy
by 19% (Zheng & Lu) and 12.72% (Li et al.,2019) while
also reducing the amount of data required to train the mod-
els. Transfer learning can be used to help alleviate scarce
data as by using decision trees, data from similar subjects
can be used to improve accuracy by around 10% although if
data from dissimilar subjects is used it can have a negative
impact on the model accuracy (Maxhuni et al.,2016). To
ensure negative transfer learning that degrades the perfor-
mance of the model does not occur, a conditional transfer
learning framework has been developed that assesses indi-
vidual’s transferability against all other individual’s data
within the dataset. When tested the conditional transfer
learning model identified up to 16 individuals who could
benefit from an original dataset of 18 individuals’ EEG sig-
natures, improving classification accuracy by around 15%
(Lin & Jung,2017).
With the rise of edge computing it is now possible to train
models using transfer learning and fine-tune models on-
device. iSelf (Sun et al.,2015) applied a local transfer
learning approach within a smartphone application to infer
emotions by learning smartphone usage habits. Users pro-
vided only a few sample labels and when utilising transfer
learning the model achieved 75% accuracy. Applying a
transfer learning approach on-device greatly improves pri-
vacy as no data is externally transmitted which is vital when
collecting personal wellbeing data.
To achieve a personalised model the development of an ini-
tial base model to infer emotion trained using HRV, HR
and EDA data from a controlled experiment is proposed,
followed by an on-device transfer learning approach to per-
sonalise the model using real-world labelled data collected
from individuals using edge computing interfaces.
2 EXPERIMENTAL SETUP
2.1 Controlled Stressor Experiment
Model adaption is a personalisation and cross-domain trans-
fer learning problem. Here, a model trained on a dataset
collected from a controlled experiment is adapted to per-
form the same task in a different situation where real-world
physiological data will be collected from target participants.
The controlled experiment used the Montreal stress test
(Dedovic et al.,2005) to induce stress in 20 participants.
Participants wore HR, HRV and EDA sensors on their hand
each sampled at 30Hz to collect physiological data when
relaxing and experiencing stress. All participants were ini-
tially briefed before completing a 3-minute rest period, fol-
lowed by 3 minutes of training how to use the system to
answer maths questions before another 3-minute rest period.
Participants then completed the stressor experiment where
the time limit for each question was the average time taken
during training reduced by 10%. The time pressure along
with a progress bar showing their progress compared with
an artificially inflated average were both designed to induce
stress during the 10-minute experiment. Finally, participants
completed a final 3-minute rest session to ensure sufficient
stressed and relaxed data was collected.
Figure 1. Comparison of stressed and relaxed data collected
A similar number of samples was collected of both relaxed
and stressed data with 420,000 records collected of stressed
data and 400,000 records of relaxed data helping to reduce
bias in the classification model. Figure 1shows a sample of
400 records representing around 13 seconds of relaxed and
Submission and Formatting Instructions for SysML 2019
stressed data during the experiment. While time-series data
is traditionally challenging to classify by sight there are clear
trends displayed such as higher heart rate (stressed average-
81.2, SD-12.4; relaxed average-76.8, SD-10.7) and lower
EDA (stressed average-317.2, SD-148.3; relaxed average-
351.9, SD-152.8). However, this was not true of all 20
participants with some experiencing more pronounced phys-
iological changes than others indicating the need for person-
alised models.
2.2 On-device model personalisation
While a transfer learning approach can assist personalising
models and improve models’ performance across multiple
domains, it has traditionally relied on collecting the target
domain data in advance. The training is then completed
externally before the model can be exported to devices.
We have devised a new approach where the target user pro-
vides labelled samples over a short period and transfer learn-
ing is then used to personalise the model on-device. Three
participants were provided with physical prototypes con-
taining the same HR, HRV and EDA sensors as used in the
controlled experiment as well as two buttons to enable the
real-time self-labelling of stressed and relaxed emotions.
For processing the device utilised a Raspberry Pi 3, with
1GB of RAM and a 1.2GHz quad core CPU, as it is com-
pact so can be embedded within the prototypes, sufficiently
powerful to perform transfer learning and affordable.
Transfer learning can then be applied on-device to further
train the base model with the new target user’s data. This
approach personalises the model and improves real-world
domain accuracy, as the target data is the real-world data
collected in the ”wild” in comparison to the controlled ex-
periment data used to train the base model.
3 NETWORK ARCHITECTURE
The classification of stress is a time-series classification task
which takes the raw physiological signals as input and out-
puts a label (stressed or relaxed) for each sequence. Deep
learning particularly CNNs present many opportunities for
the classification of emotions. Traditionally CNNs have
been used to classify two-dimensional data but 1 dimen-
sional (1D) CNNs can learn from raw time-series data with-
out feature extraction being first required. The raw input
data is first divided into segments of fixed lengths with
sliding windows used to avoid semantic segmentation.
The stressed and relaxed data collected from the experiment
was used to train a 1D CNN over 50 epochs using 10-fold
cross validation achieving 82.5% accuracy. The network ar-
chitecture consists of multiple 1 dimensional convolutional
layers followed by max pooling operations. A dropout layer
with a rate of 0.3 is included to prevent overfitting before
the softmax activation function as shown in figure 2.
Figure 2. 1D CNN model architecture
During the self-labelling period each of the 3 target users
labelled similar levels of data in comparison with 1 user
participating in the controlled experiment. Thus, each of
the 3 target users provided sufficient data to personalise the
models and ensure the models function in the real-world
target domain. The base model was re-trained after the data
collection period where transfer learning was performed
on-device using the real-world data. In order to achieve the
transfer the fully connected layer in the base pre-trained
model was removed from the network and two fully con-
nected layers were added forming an adaption layer with
the first having a size of 160 and the second having a size
of 2; for the stressed and relaxed classes.
Submission and Formatting Instructions for SysML 2019
The on-device processing was slower than expected taking
an average of 25 minutes to train the model using the transfer
learning approach due to the limited specifications of the
Raspberry Pi. However, as this training only needs to be
completed once it is not a major limitation. This approach
remains simpler than returning the devices to train the model
externally and then provide users with the device embedding
the personalised model.
4 RE SULTS
The final model accuracy of the three targets were very
similar with a variance of 0.1% and an average accuracy
of 93.9% a significant improvement over the base model
accuracy as shown in table 1. The f1-scores of the 3 models
are lower and vary slightly although they consistently show
high precision and recall.
Table 1.
Accuracy of model when personalised for individual users
and tested with all three target users’ data
USE R 1 USER 2 USER 3
ACC URAC Y 93.9% 93.8% 93.9%
F1-SCORE 0.91 0.85 0.9
Figure 3shows the confusion matrix after transfer learning
had been performed for the three target users demonstrating
that few misclassification errors occurred during testing.
Figure 3. Confusion matrix for target 1, target 2 and target 3
To ensure the transfer learning approach had successfully
learned the new domain the original base model trained
with the controlled experiment data was tested with the
three target users’ real-world data. The accuracy achieved
for the three users was 55% with an f1-score of 0.58, 52%
with an f1-score of 0.53 and 40% with an f1-score of 0.41.
These results show the transfer learning approach used has
greatly improved the classification accuracy of the target
real-world domain.
The aim of using transfer learning was to additionally per-
sonalise the model. To demonstrate the model personalisa-
tion each target user’s model was tested with all target users’
data as shown in 2. The results confirm a transfer learning
approach has developed a cross-domain personalised model
as there is a significant accuracy improvement when the
personalised model is tested using the same user’s data.
Table 2.
Accuracy of model when personalised for individual users
and tested with all three target users’ data
MODEL 1MODEL 2MODEL 3
USE R 1DATA 93.9% 67% 70%
USE R 2DATA 54% 93.8% 63%
USE R 3DATA 48% 88% 93.9%
Overall a transfer learning approach has shown to improve
model performance, demonstrating its ability to personalise
the affective model and work across domains. The 11%
accuracy increase over the base model confirms the benefits
of a personalised model when inferring stress and the ability
to perform transfer learning on-device simplifies the process
of developing a personalised model.
5 CO NCLUS ION AND FUTURE WORK
We have proposed a novel method for personalising an af-
fective 1D CNN model on-device with transfer learning
techniques. Edge computing interfaces have been used to
collect a small amount of real-world personalised labelled
data for 3 target subjects where transfer learning is then used
to personalise a 1D CNN trained using data collected from
a controlled stressor experiment. As only a small sample
of labelled data is required it saves time, labour and money
while also improving model personalisation and perform-
ing across different domains. Edge computing has enabled
the transfer learning model to be trained and ran on-device
allowing for the real-time inference of stress. The transfer
learning approach improved model accuracy from 82.5% to
an average of 93.9% for the three target subjects.
In future work, the inference from the personalised models
developed will be evaluated against self-reports to confirm
real-world accuracy. Additionally, it would be beneficial to
explore other devices than the Raspberry Pi that may signifi-
cantly reduce the time required to apply the transfer learning
approach. Interventional feedback (Woodward et al.,2018)
should also be used in future iterations of the tangible in-
terfaces enabling feedback to be issued as soon as stress is
inferred, improving wellbeing in real-time.
Submission and Formatting Instructions for SysML 2019
REFERENCES
Caruana, R. Multitask Learning. Machine Learning, 28
(1):41–75, 1997. ISSN 08856125. doi: 10.1023/A:
1007379606734.
Dedovic, K., Renwick, R., Mahani, N. K., and Engert, V.
The Montreal Imaging Stress Task : using functional
imaging to investigate the ... Psychiatry & Neuroscience,
30(5):319–325, 2005. ISSN 1180-4882.
Google. reCAPTCHA: Easy on Humans, Hard on
Bots, 2019. URL
https://www.google.com/
recaptcha/intro/v3.html.
Healey, J. and Picard, R. Detecting Stress During Real-
World Driving Tasks Using Physiological Sensors. IEEE
Transactions on Intelligent Transportation Systems, 6(2):
156–166, jun 2005. ISSN 1524-9050. doi: 10.1109/TITS.
2005.848368.
Li, J., Qiu, S., Shen, Y.-Y., Liu, C.-L., and He, H. Multi-
source Transfer Learning for Cross-Subject EEG Emotion
Recognition. IEEE Transactions on Cybernetics, 2019.
ISSN 2168-2267. doi: 10.1109/tcyb.2019.2904052.
Lin, Y. P. and Jung, T. P. Improving EEG-based emotion
classification using conditional transfer learning. Fron-
tiers in Human Neuroscience, 2017. ISSN 16625161. doi:
10.3389/fnhum.2017.00334.
Maxhuni, A., Hernandez-Leal, P., Sucar, L. E., Osmani,
V., Morales, E. F., and Mayora, O. Stress modelling
and prediction in presence of scarce data. Journal of
Biomedical Informatics, 63:344–356, oct 2016. ISSN
15320464. doi: 10.1016/j.jbi.2016.08.023.
Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning
and transferring mid-level image representations using
convolutional neural networks. In Proceedings of the
IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, 2014. ISBN 9781479951178.
doi: 10.1109/CVPR.2014.222.
Perkbox. THE 2O18 UK WORKPLACE STRESS SUR-
VEY. 2018.
Picard, R. W., Vyzas, E., and Healey, J. Toward machine
emotional intelligence: Analysis of affective physiolog-
ical state. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 23(10):1175–1191, oct 2001. ISSN
01628828. doi: 10.1109/34.954607.
Sargano, A. B., Wang, X., Angelov, P., and Habib, Z.
Human action recognition using transfer learning with
deep representations. In Proceedings of the International
Joint Conference on Neural Networks, volume 2017-May,
pp. 463–469. Institute of Electrical and Electronics En-
gineers Inc., jun 2017. ISBN 9781509061815. doi:
10.1109/IJCNN.2017.7965890.
Schumm, J., Zurich, E., Ehlert, U., Setz, C., Arnrich, B.,
Marca, R. L., and Tr
¨
oster, G. Discriminating stress from
cognitive load using a wearable EDA device. Discrim-
inating Stress From Cognitive Load Using a Wearable
EDA Device. IEEE TRANSACTIONS ON INFORMA-
TION TECHNOLOGY IN BIOMEDICINE, 14(2), 2010.
doi: 10.1109/TITB.2009.2036164.
Sun, B., Ma, Q., Zhang, S., Liu, K., and Liu, Y.
ISelf: Towards cold-start emotion labeling using trans-
fer learning with smartphones. In Proceedings - IEEE
INFOCOM, volume 26, pp. 1203–1211. Institute of
Electrical and Electronics Engineers Inc., aug 2015.
ISBN 9781479983810. doi: 10.1109/INFOCOM.2015.
7218495.
The Physiological Society. Stress in modern Britain Making
Sense of Stress. 2017.
Wang, D. and Zheng, T. F. Transfer learning for speech
and language processing. In 2015 Asia-Pacific Signal
and Information Processing Association Annual Summit
and Conference, APSIPA ASC 2015, pp. 1225–1237. In-
stitute of Electrical and Electronics Engineers Inc., feb
2016. ISBN 9789881476807. doi: 10.1109/APSIPA.
2015.7415532.
Wijsman, J., Grundlehner, B., Hao Liu, Hermens, H., and
Penders, J. Towards mental stress detection using wear-
able physiological sensors. In 2011 Annual Interna-
tional Conference of the IEEE Engineering in Medicine
and Biology Society, pp. 1798–1801. IEEE, aug 2011.
ISBN 978-1-4577-1589-1. doi: 10.1109/IEMBS.2011.
6090512.
Woodward, K., Kanjo, E., Umir, M., and Sas, C. Harnessing
Digital Phenotyping to Deliver Real-Time Interventional
Bio-Feedback. 2018. doi: 10.1145/1234567890.
Zheng, W.-L. and Lu, B.-L. Personalizing EEG-Based Af-
fective Models with Transfer Learning. Technical report.
URL http://bcmi.sjtu.edu.cn/.
