ThesisPDF Available

Automatic Pain Assessment by Learning from Multiple Biopotentials

Thesis

Automatic Pain Assessment by Learning from Multiple Biopotentials

Abstract

Accurate pain assessment plays an important role in proper pain management, especially among hospitalized people experience acute pain. Pain is subjective in nature which is not only a sensory feeling but could also combine affective factors. Therefore self-report pain scales are the main assessment tools as long as patients are able to self-report. However, it remains a challenge to assess the pain from the patients who cannot self-report. In clinical practice, physiological parameters like heart rate and pain behaviors including facial expressions are observed as empirical references to infer pain objectively. The main aim of this study is to automate such process by leveraging machine learning methods and biosignal processing. To achieve this goal, biopotentials reflecting autonomic nervous system activities including electrocardiogram and galvanic skin response, and facial expressions measured with facial electromyograms were recorded from healthy volunteers undergoing experimental pain stimulus. IoT-enabled biopotential acquisition systems were developed to build the database aiming at providing compact and wearable solutions. Using the database, a biosignal processing flow was developed for continuous pain estimation. Signal features were extracted with customized time window lengths and updated every second. The extracted features were visualized and fed into multiple classifiers trained to estimate the presence of pain and pain intensity separately. Among the tested classifiers, the best pain presence estimating sensitivity achieved was 90% (specificity 84%) and the best pain intensity estimation accuracy achieved was 62.5%. The results show the validity of the proposed processing flow, especially in pain presence estimation at window level. This study adds one more piece of evidence on the feasibility of developing an automatic pain assessment tool from biopotentials, thus providing the confidence to move forward to real pain cases. In addition to the method development, the similarities and differences between automatic pain assessment studies were compared and summarized. It was found that in addition to the diversity of signals, the estimation goals also differed as a result of different study designs which made cross dataset comparison challenging. We also tried to discuss which parts in the classical processing flow would limit or boost the prediction performance and whether optimization can bring a breakthrough from the system’s perspective.
Turku Centre for Computer Science
TUCS Dissertations
No 248, November 2019
Mingzhe Jiang
Automatic Pain Assessment
by Learning from Multiple
Biopotentials
Automatic Pain Assessment by
Learning from Multiple Biopotentials
Mingzhe Jiang
To be presented, with the permission of the Faculty of Science and
Engineering of the University of Turku in MED D1024 Säätiö-Sali on
November 28, 2019, at 12 noon
University of Turku
Department of Future Technologies
2019
Supervisors
Professor Pasi Liljeberg
Department of Future Technologies, University of Turku, Finland
Adjunct Professor Amir M. Rahmani
Department of Future Technologies, University of Turku, Finland
Assistant Professor
University of California, Irvine, USA
Reviewers
Associate Professor Raquel Bailón
Department of Electronic Engineering and Communications
University of Zaragoza, Spain
Associate Professor Steven Su
School of Biomedical Engineering
University of Technology Sydney, Australia
Opponent
Adjunct Professor Heli Koskimäki
Biomimetics and Intelligent Systems Group
University of Oulu, Finland
Senior Data Scientist
OURA Health Ltd, Finland
Painosalama Oy, Turku, Finland
ISBN 978-952-12-3889-5
ISSN 1239-1883
The originality of this thesis has been checked in accordance with the Univer-
sity of Turku quality assurance system using the Turnitin Originality Check
service
Abstract
Accurate pain assessment plays an important role in proper pain manage-
ment, especially among hospitalized people experience acute pain. Pain is
subjective in nature which is not only a sensory feeling but could also com-
bine affective factors. Therefore self-report pain scales are the main assess-
ment tools as long as patients are able to self-report. However, it remains
a challenge to assess the pain from the patients who cannot self-report. In
clinical practice, physiological parameters like heart rate and pain behav-
iors including facial expressions are observed as empirical references to infer
pain objectively. The main aim of this study is to automate such process by
leveraging machine learning methods and biosignal processing.
To achieve this goal, biopotentials reflecting autonomic nervous system
activities including electrocardiogram and galvanic skin response, and fa-
cial expressions measured with facial electromyograms were recorded from
healthy volunteers undergoing experimental pain stimulus. IoT-enabled biopo-
tential acquisition systems were developed to build the database aiming at
providing compact and wearable solutions. Using the database, a biosig-
nal processing flow was developed for continuous pain estimation. Signal
features were extracted with customized time window lengths and updated
every second. The extracted features were visualized and fed into multi-
ple classifiers trained to estimate the presence of pain and pain intensity
separately. Among the tested classifiers, the best pain presence estimating
sensitivity achieved was 90% (specificity 84%) and the best pain intensity
estimation accuracy achieved was 62.5%.
The results show the validity of the proposed processing flow, especially in
pain presence estimation at window-level. This study adds one more piece
of evidence on the feasibility of developing an automatic pain assessment
tool from biopotentials, thus providing the confidence to move forward to
real pain cases. In addition to the method development, the similarities and
differences between automatic pain assessment studies were compared and
summarized. It was found that in addition to the diversity of signals, the
estimation goals also differed as a result of different study designs which
made cross dataset comparison challenging. We also tried to discuss which
parts in the classical processing flow would limit or boost the prediction
performance and whether optimization can bring a breakthrough from the
system’s perspective.
i
ii
Tiivistelmä
Kivun täsmällinen arviointi on tärkeää kivunhallinnassa, erityisesti sairaan-
hoitoa vaativille kipupotilaille. Kipu on subjektiivista, sillä se ei ole pelkästään
aistituntemus, vaan siihen saattaa liittyä myös tunnekokemuksia. Tällöin it-
searviointiin perustuvat kipuasteikot ovat tärkein työkalu, niin kauan kun
potilas pystyy kokemuksensa arvioimaan. Arviointi on kuitenkin haasteel-
lista potilailla, jotka eivät itse pysty kertomaan kivustaan. Kliinisessä hoito-
työssä kipua pyritään objektiivisesti arvioimaan esimerkiksi havainnoimalla
fysiologisia muuttujia kuten sykettä ja käyttäytymistä esimerkiksi potilaan
kasvonilmeiden perusteella. Tutkimuksen päätavoitteena on automatisoida
arviointiprosessi hyödyntämällä koneoppimismenetelmiä yhdessä biosignaalien
prosessointnin kanssa.
Tavoitteen saavuttamiseksi mitattiin autonomista keskushermoston toim-
intaa kuvastavia biopotentiaaleja: sydänsähkökäyrää, galvaanista ihoreak-
tiota ja kasvolihasliikkeitä mittaavaa lihassähkökäyrää. Mittaukset tehtiin
terveillä vapaaehtoisilla, joille aiheutettiin kokeellista kipuärsykettä. Jär-
jestelmän kehittämiseen tarvittavaa tietokantaa varten rakennettiin biopo-
tentiaaleja keräävä Internet of Things -pohjainen tallennusjärjestelmä. Koost-
etun tietokannan avulla kehitettiin biosignaaleille prosessointimenetelmä jatku-
vaan kivun arviointiin. Signaaleista eroteltiin piirteitä sekuntitasoon mukaute-
tuilla aikaikkunoilla. Piirteet visualisoitiin ja tarkasteltiin eri luokittelijoilla
kivun ja kiputason tunnistamiseksi. Parhailla luokittelumenetelmillä saavutet-
tiin kivuntunnistukseen 90% herkkyyskyky (sensitivity) ja 84% erottelukyky
(specificity) ja kivun voimakkuuden arviointiin 62,5% tarkkuus (accuracy).
Tulokset vahvistavat kyseisen käsittelytavan käyttökelpoisuuden erityis-
esti tunnistettaessa kipua yksittäisessä arviointi-ikkunassa. Tutkimus vahvis-
taa biopotentiaalien avulla kehitettävän automatisoidun kivun arvioinnin to-
teutettavuuden kokeellisella kivulla, rohkaisten etenemään todellisen kivun
tutkimiseen samoilla menetelmillä. Menetelmää kehitettäessä suoritettiin
lisäksi vertailua ja yhteenvetoa automaattiseen kivuntunnistukseen kehitet-
tyjen eri tutkimusten välisistä samankaltaisuuksista ja eroista. Tarkastelussa
löytyi signaalien eroavaisuuksien lisäksi tutkimusmuotojen aiheuttamaa eroa
arviointitavoitteisiin, mikä hankaloitti tutkimusten vertailua. Lisäksi pohdit-
tiin mitkä perinteisten prosessointitapojen osiot rajoittavat tai edistävät en-
nustekykyä ja miten, sekä tuoko optimointi läpimurtoa järjestelmän näkökul-
masta.
iii
iv
Acknowledgements
When I stepped on the land of Finland for the first time, I had no idea how
amazing the coming years would be. This peaceful nature land seems to
have the power of amplifying joy and healing sorrows. A degree is only one
of the milestones throughout life. The growth gained while reaching there is
the real treasure to me. All my academic growth and achievements cannot
be made without supports from my supervisors and the inter-disciplinary
research team that I’m in.
I would like to express my sincere thanks to my principal supervisor
Prof. Pasi Liljeberg for the inclusiveness, flexibility, academic knowledge
and resources always provided with wise guidance. I would like to thank my
supervisor Adjunct Prof. Amir M. Rahmani, who offered sharp publication
and polishing ideas with generosity in time and wisdom sharing. I want to
thank Prof. Sanna Salanterä in the team from the Nursing Science depart-
ment. Her strong supports throughout my doctoral study helped me filling
my knowledge gap in the pain assessment field and in clinical research. I
want to thank the doctor experts, Prof. Riku Aantaa and Adjunct Prof.
Nora Hagelberg, who were in the team and brought inspiring and intellec-
tual discussions. Also, I would like to express my gratitude for receiving
help and encouragement, especially before and at the beginning of my study,
from Research Prof. Geng Yang, Adjunct Prof. Tomi Westerlund, and Prof.
Hannu Tenhunen.
I’m grateful to have Associate Prof. Raquel Bailón and Associate Prof.
Steven Su as dissertation reviewers, for both valuable comments and accep-
tance with honours. My appreciation is also extended to Adjunct Prof. Heli
Koskimäki for consent to act as the opponent in the public examination.
This study was supported by the Academy of Finland, China Scholar-
ship Council, and HPY Research Foundation. The academic trips during
my doctoral study were partially supported by Turku University Founda-
tion, MATTI travel grant from the doctoral program, and IEEE Sensors
Applications Symposium. I want to thank these organizations for support-
ing young researchers in building their early careers.
Both of the presented thesis work and related publications cannot be
them without the creative and precious contributions from my talented and
v
hardworking colleague co-authors. I want to express my deep gratitude to
Tuan Nguyen Gia, Victor Kathan Sarker, and Arman Anzanpour, who al-
ways have impressive solutions to small everyday challenges in research. I
would especially thank Elise Syrjälä, who brought me lots of encourage-
ment with positive attitude, self-motivation, and beneficial discussions on
research during my toughest doctoral time in the middle. My thanks are
also for my co-workers with different experiences and backgrounds in the
Smart Pain Assessment team, Hanna-Maria Matinolli, Riitta Mieronkoski,
Mikko Koivumäki, and Virpi Terävä. There were many moments we cover
each other’s back, and I do appreciate the mutual support and knowledge
sharing at each moment. I would like to thank all the colleagues in the
same office environment and across the corridor in different periods. The
interesting stories, enthusiasm in hobbies, and positive attitudes colored my
doctoral life.
I would like to thank all my great friends, those who were in Turku, and
those who are still in Turku, especially Ping Wang, who lightened the dark
days of winter in Finland like bright sunshine. It is also my luck to have
Xueying Ma and Wei Yang, who inspired me with their passion to life. I’m
thankful to have many lovely friends from different parts of China sharing
laughs and worries, playing sports, and cooking together. Last but not least,
endless thanks to my parents, and my husband Hao Niu, for backing me up
unconditionally and always.
vi
List of original publications
Publications related to this thesis
1. Jiang, M., Mieronkoski, R., Syrjälä, E., Anzanpour, A., Terävä, V., Rah-
mani, A.M., Salanterä, S., Aantaa, R., Hagelberg, N., Liljeberg, P., 2019.
Acute pain intensity monitoring with the classification of multiple physio-
logical parameters, Journal of Clinical Monitoring and Computing, 33(3),
pp.493-507. [study 1]
2. Jiang, M., Mieronkoski, R., Rahmani, A.M., Hagelberg, N., Salanterä, S.,
Liljeberg, P., 2017, November. Ultra-short-term analysis of heart rate vari-
ability for real-time acute pain monitoring with wearable electronics, In 2017
IEEE International Conference on Bioinformatics and Biomedicine (BIBM)
(pp. 1025-1032). IEEE. [study 2]
3. Jiang, M., Gia, T.N., Anzanpour, A., Rahmani, A.M., Westerlund, T.,
Salanterä, S., Liljeberg, P. and Tenhunen, H., 2016, April. IoT-based remote
facial expression monitoring system with sEMG signal. In 2016 IEEE Sensors
Applications Symposium (SAS) (pp. 1-6). IEEE.
4. Jiang, M., Rahmani, A.M., Westerlund, T., Liljeberg, P. and Tenhunen,
H., 2015, October. Facial Expression Recognition with sEMG Method. In
2015 IEEE International Conference on Computer and Information Technol-
ogy; Ubiquitous Computing and Communications; Dependable, Autonomic
and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/
DASC/PICOM) (pp. 981-988). IEEE.
5. Syrjälä, E., Jiang, M., Pahikkala, T., Salanterä, S. and Liljeberg P., 2019,
July. Skin conductance response to gradual-increasing experimental pain.
In 2019 41st Annual International Conference of the IEEE Engineering in
Medicine and Biology Society. (Accepted) [study 3]
6. Yang G., Jiang, M., Ouyang, W., Ji, G., Rahmani, A.M., Liljeberg, P.,
Tenhunen, H., 2018. IoT-based remote pain monitoring system: from device
to cloud platform, IEEE Journal of Biomedical and Health Informatics, 22(6),
pp.1711-1719.
7. Sarker, V.K., Jiang, M., Gia, T.N., Anzanpour, A., Rahmani, A.M. and
Liljeberg, P., 2017, March. Portable multipurpose bio-signal acquisition and
wireless streaming device for wearables. In 2017 Sensors Applications Sym-
posium (SAS) (pp. 1-6). IEEE.
Other publications
8. Gia, T.N., Jiang, M., Sarker, V.K., Rahmani, A.M., Westerlund,
T., Liljeberg, P. and Tenhunen, H., 2017, June. Low-cost fog-assisted
health-care IoT system with energy-efficient sensor nodes. In 2017
vii
13th International Wireless Communications and Mobile Computing
Conference (IWCMC) (pp. 1765-1770). IEEE.
9. Gia, T.N., Jiang, M., Rahmani, A.M., Westerlund, T., Liljeberg, P.
and Tenhunen, H., 2015, October. Fog computing in healthcare in-
ternet of things: A case study on ecg feature extraction. In 2015
IEEE International Conference on Computer and Information Tech-
nology; Ubiquitous Computing and Communications; Dependable, Au-
tonomic and Secure Computing; Pervasive Intelligence and Computing
(CIT/IUCC/DASC/PICOM) (pp. 356-363). IEEE.
10. Negash, B., Gia, T.N., Anzanpour, A., Azimi, I., Jiang, M., West-
erlund, T., Rahmani, A.M., Liljeberg, P. and Tenhunen, H., 2018.
Leveraging Fog Computing for Healthcare IoT. In Fog Computing in
the Internet of Things (pp. 145-169). Springer International Publish-
ing.
11. Rahmani, A.M., Gia, T.N., Negash, B., Anzanpour, A., Azimi, I.,
Jiang, M. and Liljeberg, P., 2018. Exploiting smart e-health gateways
at the edge of healthcare internet-of-things: A fog computing approach.
Future Generation Computer Systems, 78, pp.641-658.
12. Yang, G., Deng, J., Pang, G., Zhang, H., Li, J., Deng, B., Pang, Z.,
Xu, J., Jiang, M., Liljeberg, P. and Xie, H., 2018. An IoT-Enabled
Stroke Rehabilitation System Based On Smart Wearable Armband
And Machine Learning. In IEEE Journal of Translational Engineering
in Health and Medicine, vol. 6, pp. 1-10.
viii
List of Abbreviations
µSmicro-Siemens
ABP arterial blood pressure
AC alternating current
ACC accuracy
AdaBoost adaptive boosting
ADC analog-to-digital converter
Ag silver
AgCl silver chloride
ANFIS adaptive-network-based fuzzy inference system
ANI analgesia/nociception index
ANS autonomic nervous system
AR autoregressive model
AU action unit
AUC area under the GSR/ROC curve
Bag bagging, bootstrap aggregation
BIAS bias drive output in biopotential acquisition
BPAT Behavior Pain Assessment Tool
BPS Behavioral Pain Scale
BVP blood volume pulse
CH channel
CM common-mode voltage
CMRR common-mode rejection ratio
CNAP continuous noninvasive arterial pressure
cor facial muscle-corrugator supercilii
CPOT Critical-Care Pain Observation Tool
DAMV difference absolute mean value
dB decibel
DC direct current
E4 E4 wristband from empatica
ECG electrocardiogram
EDA Electrodermal activity
EDR ECG-derived respiration
EEG electroencephalogram
EMG electromyogram
ix
FACS Facial Action Coding System
FFT fast Fourier transform
FLACC Face, Legs, Activity, Cry, and Consolability Behavioral Assessment
Tool
FMD median frequency
FMN mean frequency
fMRI functional magnetic resonance imaging
FPK peak frequency
GSR galvanic skin response
H124SG KendallTMECG electrodes, round 24mm
HF HRV-the high-frequency power component of NN series (0.15-0.40 Hz)
HRV heart rate variability
IASP International Association for the Study of Pain
ICU intensitive care unit
INxN negative input in biopotential acquisition
INxP positive input in biopotential acquisition
IoT Internet-of-Things
knn k-nearest neighbor
LA ECG measurement-left arm
LD log detector
lev facial muscle-levator labii superioris
LF HRV-the low-frequency power component of NN series (0.04-0.15 Hz)
LL ECG measurement-left leg
MAE mean absolute error
MAV mean absolute value
MAVS mean absolute value slope
MDS multidimensional scaling
MEG magnetoencephalography
MPV maximum absolute value
MSE mean square error
NAN nociception/antinociception
NaN not a number
NN normal-to-normal
NPAT Nonverbal Pain Assessment Tool
NRS numeric rating scale
NSCF number of skin conductance fluctuations
NVPS Non-verbal Pain Scale
x
orb facial muscle-orbicularis oculi
PBAT Pain Behavioral Assessment Tool
PET positron emission tomography
PPG photoplethysmogram
PPGA PPG amplitude
PSPI Prkachin and Solomon Pain Intensity Metric
R2R squared
RA ECG measurement-right arm
REF common reference in biopotential acquisition
ris facial muscle-risorius
RMS root mean square
RMSE root mean square error
RMSSD HRV-root mean square of the successive differences
RNN recurrent convolutional neural network
ROC receiver operating characteristic
RUSBoost random undersampling boosting
S1 the first step classification
S2 the second step classification
SaO2 arterial oxygen saturation
SBP systolic blood pressure
SCL skin conductance level
SCR skin conductance response
SD standard deviation
SDNN HRV-standard deviation of all NN intervals
sEMG surface electromyogram
SMNA SudoMotor Nerve Activity
SNR signal-to-noise ratio
SPA Smart Pain Assessment
SpaExp Smart Pain Assessment - Experimental Pain Database
SPI surgical pleth index
SpO2 Peripheral oxygen saturation
SPS samples per second
SSC slope sign change
SSI surgical stress index
SVM support vector machine
TNR true negative rate
TPR true positive rate
VAS visual analog scale
VRS verbal categorical rating scale
xi
WL wave length
ZC zero crossing
zyg facial muscle-zygomaticus major
xii
List of Figures
2.1 Remote pain monitoring in hospital for inpatients and ICU
patients c
2018 IEEE . . . . . . . . . . . . . . . . . . . . . . 23
3.1 Mechanisms of difference interference caused by the electric
field in ECG measurement [1, 2] . . . . . . . . . . . . . . . . . 28
3.2 Driven right leg circuit to decrease common-mode voltage . . 28
3.3 Part of the ADS1299 configurations including lead-off detec-
tion, common reference, and bias drive . . . . . . . . . . . . . 30
3.4 Comparison of power line interference in ADS1299’s different
working modes . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.5 The block diagram of the designed sEMG acquisition device . 32
3.6 Software workflow . . . . . . . . . . . . . . . . . . . . . . . . 34
3.7 The firmware structure of the AVR processor and its commu-
nication with the software . . . . . . . . . . . . . . . . . . . . 35
3.8 File operations in the software . . . . . . . . . . . . . . . . . . 37
3.9 A piece of ECG signal collected in Section 3.1.1 . . . . . . . . 38
3.10 The envelopes of the sEMG signals acquired by the system de-
veloped in this study (UTU-BASD) and by ME6000 Biomonitor 38
3.11 The respiratory modulation of the waveform amplitude . . . . 42
3.12 Two GSR exosomatic direct voltage recording circuits . . . . 43
4.1 The timeline and pain stimulus intensity in one test . . . . . 47
4.2 The waveform of the electrical stimulation . . . . . . . . . . . 48
4.3 The potential scald injury caused by heat and the temperature
curve of the heat pain stimulation in the test . . . . . . . . . 49
4.4 The distribution of the reported VAS scores at t2in all tests
and the tests without an "intolerable" report . . . . . . . . . 51
4.5 Biosignals acquisition software platform . . . . . . . . . . . . 52
4.6 GSR electrode sites on the volar surface . . . . . . . . . . . . 54
4.7 The signals in one electrical test . . . . . . . . . . . . . . . . . 56
5.1 Adaptive noise cancellation to denoise electrical pulses . . . . 61
5.2 ECG and sEMG denosing, signal processing flow . . . . . . . 62
xiii
5.3 The frequency response of the filters and their time response
on ECG signal . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.4 An illustration of ECG waves [3] . . . . . . . . . . . . . . . . 64
5.5 Some manual corrections for the wrong or missed R peak de-
tections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.6 Signal processing flow in 1) R peak detection and 2) respira-
tory cycles extraction from the detected R peaks . . . . . . . 67
5.7 Visualization of the key steps in EDR extraction (a-c). Ca-
pacitive pressure sensor waveform (d). The comparison of the
estimated breathing rate and heart rate between the reported
values by Bioharness and the extracted values in this study
(e-f) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.8 The distribution of the mean difference between the reported
values and the extracted values in all the study subjects . . . 69
5.9 Decompose GSR into tonic SCL and phasic SCR . . . . . . . 74
6.1 The GSR feature list and the linear correlation (Pearson’s r)
between every two features . . . . . . . . . . . . . . . . . . . . 81
6.2 The tonic component of all the tests in each pain category
normalized to t1and their average curve (a-c); a comparison
of the average curves (d). . . . . . . . . . . . . . . . . . . . . 82
6.3 Area under the tonic component (t_auc) . . . . . . . . . . . . 82
6.4 Standard deviation of the tonic component (t_std) in each
pain category and a comparison of their averages . . . . . . . 83
6.5 Number of phasic driver peaks per minute (p_num_pks) . . . 83
6.6 The ECG feature list and the linear correlation (Pearson’s r)
between every two features . . . . . . . . . . . . . . . . . . . . 85
6.7 Heart rate (ecg_hr) rescaled with λ1. . . . . . . . . . . . . . 86
6.8 SDNN (ecg_nsdnn) rescaled with λ2. . . . . . . . . . . . . . 86
6.9 RMSSD (ecg_nrmssd) rescaled with λ2. . . . . . . . . . . . 87
6.10 LF (ecg_nlf ) rescaled with λ3. . . . . . . . . . . . . . . . . . 87
6.11 HF (ecg_nhf ) rescaled with λ3. . . . . . . . . . . . . . . . . 88
6.12 LF/HF (ecg_lf/hf ) normalized to t1. . . . . . . . . . . . . . 88
6.13 Sample entropy (ecg_sampen) normalized to t1. . . . . . . . 89
6.14 Stimulus intensity distribution in each pain category . . . . . 89
6.15 ECG derived respiration rate (respr_edr ) rescaled with λ4. . 91
6.16 Bioharness respiration rate (respr_bha) rescaled with λ4. . . 91
6.17 Mean absolute Spearman’s ρof the subjects between adap-
tively filtered sEMG signals, std(0.13, 0.22). . . . . . . . . . 92
6.18 Mean absolution Spearman’s ρof the subjects between fifteen
sEMG features of muscle corrugator and muscle zygomaticus
major, std(0, 0.25). . . . . . . . . . . . . . . . . . . . . . . . 93
6.19 Root mean square of corrugator sEMG (emg_cor_rms) . . . 94
xiv
6.20 The first coefficient in the 4th autoregressive model of corru-
gator sEMG (emg_cor_ar_1 ) . . . . . . . . . . . . . . . . . 94
6.21 The first coefficient in the 4th autoregressive model of zygo-
maticus major sEMG (emg_zyg_ar_1 ) . . . . . . . . . . . . 95
6.22 Multidimensional scaling plot of the Euclidean distance be-
tween the representative features . . . . . . . . . . . . . . . . 95
7.1 Study design interpretation . . . . . . . . . . . . . . . . . . . 99
7.2 Signal processing & feature extraction flow summary and the
composition of the feature matrix . . . . . . . . . . . . . . . . 100
7.3 Confusion matrix in S1 binary classification . . . . . . . . . . 102
7.4 S1 - relative importance of the features in tree-based classifiers
and linear discriminant analysis classifiers, with average test
sensitivity and specificity . . . . . . . . . . . . . . . . . . . . . 106
7.5 S1 - Model performance without hyperparameter optimization 107
7.6 S1 - The performance in each leave-subject-out cross-validation
test fold (subjects in ensemble_RUSBoost sensitivity descend-
ing order) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.7 S1 - Classification performance with default model settings
and optimized settings . . . . . . . . . . . . . . . . . . . . . . 109
7.8 Two S1 RUSBoost estimation examples . . . . . . . . . . . . 112
7.9 S2 - relative importance of the features in tree-based classifiers
with average train accuracy and test accuracy . . . . . . . . . 114
7.10 S2 - Model performance without hyperparameter optimization 115
7.11 S2 - Training accuracy and test accuracy change after hyper-
parameter optimization . . . . . . . . . . . . . . . . . . . . . 116
7.12 S2 - Pain intensity labels and predictions in optimized_svm_linear
3-class classification . . . . . . . . . . . . . . . . . . . . . . . . 119
xv
xvi
List of Tables
2.1 The biosignals reflecting ANS activities and some NAN in-
dexes [4, 5] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 The description on the facial expressions/emotion in the be-
havioural pain assessment tools . . . . . . . . . . . . . . . . . 16
2.3 The facial action units involved in pain facial expressions in
adults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 The description on the movements/guarding/muscle tension
in the behavioural pain assessment tools . . . . . . . . . . . . 19
2.5 The description on the vocalization/compliance with ventila-
tion/consolability/phsiological signs in the behavioural pain
assessment tools . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.6 Example devices and systems providing raw data for research 22
3.1 The lead wire configuration and key registers configuration in
the four modes . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Firmware’s responses to the software commands . . . . . . . . 36
3.3 Frequency and amplitude characteristics of the EEG, ECG
and EMG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.1 Information lookup table, item list . . . . . . . . . . . . . . . 50
4.2 Timestamp lookup table, item list . . . . . . . . . . . . . . . 51
4.3 Electrodes placement and their connections to the device . . . 53
4.4 Signal summary . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.1 Common biosignal contaminants [6, 7] . . . . . . . . . . . . . 58
5.2 R peak detection results and manual correction . . . . . . . . 65
5.3 HRV features and their definitions . . . . . . . . . . . . . . . 70
5.4 sEMG features and their descriptions . . . . . . . . . . . . . . 72
5.5 GSR features and their descriptions . . . . . . . . . . . . . . . 73
5.6 The recommended minimum time window length in feature
extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.1 A summary of features of interest . . . . . . . . . . . . . . . . 79
xvii
6.2 The RMSE and Spearman’s ρbetween the respiration rate cal-
culated from difference sources, mean (std). One extremely
large value is excluded in the averaging. . . . . . . . . . . . . 90
7.1 Cost matrix for binary classification . . . . . . . . . . . . . . 103
7.2 Specifications of the classifiers . . . . . . . . . . . . . . . . . . 104
7.3 S1 - Four selected models . . . . . . . . . . . . . . . . . . . . 108
7.4 S1 - The best sensitivity and the corresponding model for each
test subject fold and the sensitivity standard deviation among
all the 26 models (subjects in best sensitivity descending order)111
7.5 S2 - The best test accuracy and the corresponding model for
each test subject fold and the test accuracy standard devia-
tion among the implemented 26 models (subjects in best test
accuracy descending order) . . . . . . . . . . . . . . . . . . . 117
7.6 A list of representative studies on automatic pain assessment
using machine learning . . . . . . . . . . . . . . . . . . . . . . 120
7.7 Algorithm and performance comparison (studies in Table 7.6) 130
xviii
Contents
1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Research focus . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Aim and objectives . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5 Thesis content summary . . . . . . . . . . . . . . . . . . . . . 5
2 Labels and Signals in Pain Assessment 7
2.1 The complex and subjective nature of pain . . . . . . . . . . . 7
2.2 Pain scales in clinical practice and algorithms . . . . . . . . . 9
2.3 Signals in pain intensity recognition . . . . . . . . . . . . . . . 10
2.3.1 Autonomic nervous system (ANS) based signals . . . . 11
2.3.2 Respiration and oxygen saturation . . . . . . . . . . . 14
2.3.3 Behavioural signals . . . . . . . . . . . . . . . . . . . . 14
2.3.4 Neuroimaging signals . . . . . . . . . . . . . . . . . . . 21
2.4 Wearable devices and IoT-enabled systems . . . . . . . . . . . 21
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 Biosignals Acquisition 25
3.1 Facial surface electromyography . . . . . . . . . . . . . . . . . 25
3.1.1 The core of the acquisition system . . . . . . . . . . . 26
3.1.2 sEMG/Biopotential acquisition system design . . . . . 32
3.1.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.2 Electrocardiography and respiration . . . . . . . . . . . . . . 40
3.2.1 Wearable device - Bioharness 3 . . . . . . . . . . . . . 40
3.2.2 Patient monitor . . . . . . . . . . . . . . . . . . . . . . 41
3.2.3 Respiration derived from other measurements . . . . . 41
3.3 Galvanic skin response . . . . . . . . . . . . . . . . . . . . . . 43
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4 Study Design and the SpaExp Database 45
4.1 Experiment protocol . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Study design . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
xix
4.2.1 Experimental pain stimulation . . . . . . . . . . . . . 47
4.2.2 Biosignals and the acquisition system . . . . . . . . . . 52
4.3 Summary of the collected signals . . . . . . . . . . . . . . . . 55
5 Biosignal Processing and Feature Extraction 57
5.1 Signal denoising . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.1.1 Motion artifact . . . . . . . . . . . . . . . . . . . . . . 57
5.1.2 Instrumentation contaminants . . . . . . . . . . . . . . 58
5.1.3 Interference . . . . . . . . . . . . . . . . . . . . . . . . 60
5.1.4 Implementation of denoising . . . . . . . . . . . . . . . 61
5.2 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . 63
5.2.1 R peak detection in ECG . . . . . . . . . . . . . . . . 64
5.2.2 ECG derived respiration . . . . . . . . . . . . . . . . . 66
5.2.3 Heart rate variability . . . . . . . . . . . . . . . . . . . 69
5.2.4 sEMG . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2.5 GSR . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.2.6 Window length for feature extraction . . . . . . . . . . 73
5.2.7 Inter-subject and intra-subject variability, and signal
or feature normalization . . . . . . . . . . . . . . . . . 76
5.3 Summary and future work . . . . . . . . . . . . . . . . . . . . 77
6 Features and Visualization 79
6.1 GSR features . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6.1.1 Tonic component . . . . . . . . . . . . . . . . . . . . . 81
6.1.2 Phasic component . . . . . . . . . . . . . . . . . . . . 83
6.2 ECG features . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.2.1 Heart rate . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.2.2 Heart rate variability . . . . . . . . . . . . . . . . . . . 86
6.2.3 Respiration rate . . . . . . . . . . . . . . . . . . . . . 90
6.3 sEMG features . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.4 Distance in representative features . . . . . . . . . . . . . . . 96
6.5 Summary and discussion . . . . . . . . . . . . . . . . . . . . . 96
6.5.1 GSR features . . . . . . . . . . . . . . . . . . . . . . . 96
6.5.2 ECG features . . . . . . . . . . . . . . . . . . . . . . . 96
6.5.3 sEMG features . . . . . . . . . . . . . . . . . . . . . . 97
7 Pain Estimation from Physiological Features 99
7.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.1.1 Leave-subject-out cross-validation . . . . . . . . . . . . 101
7.1.2 Performance metrics . . . . . . . . . . . . . . . . . . . 101
7.1.3 Hyperparameter or Model tuning . . . . . . . . . . . . 102
7.1.4 Misclassification cost matrix in cost-sensitive learning 102
7.1.5 Feature importance . . . . . . . . . . . . . . . . . . . . 103
xx
7.1.6 Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.2.1 Step 1 - Estimating the presence of the pain or inade-
quate pain control . . . . . . . . . . . . . . . . . . . . 105
7.2.2 Step 2 - Estimating pain level when pain is present or
pain control is inadequate . . . . . . . . . . . . . . . . 113
7.3 Discussion of automatic pain assessment algorithms . . . . . . 118
7.3.1 Signal processing time window and the pain assess-
ment frequency . . . . . . . . . . . . . . . . . . . . . . 123
7.3.2 Labels and proxy/objective pain assessment . . . . . . 124
7.3.3 Data exclusion . . . . . . . . . . . . . . . . . . . . . . 126
7.3.4 Machine learning algorithms, cross-validation and per-
formance evaluation . . . . . . . . . . . . . . . . . . . 127
7.3.5 Future work . . . . . . . . . . . . . . . . . . . . . . . . 128
8 Conclusions 131
8.1 Main findings . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8.2 Significance of the study . . . . . . . . . . . . . . . . . . . . . 133
8.3 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . 134
References 135
xxi
xxii
Chapter 1
Introduction
1.1 Background
Proper pain management plays an important role in assessing the quality of
care among hospitalized people experience acute pain. Inadequately man-
aged pain negatively affects patients’ physical and psychological health as
well as hospital’s performance, while overtreatment may lead to serious and
even life-threatening consequences. Efforts to avoiding both undertreatment
and overtreatment of acute pain are being made from the perspective of anal-
gesic medications and techniques, and both analgesic efficacy and safety are
being improved [8]. Meanwhile, pain assessment by physicians and nurses
is a critical part of reaching optimal pain management. Pain assessment
by physicians and nurses is also a critical part of achieving good pain man-
agement. Pain assessment mainly refers to an assessment of pain intensity
which helps to decide the type of intervention that will be used including
the type of analgesic to be administered and the dosage [9]. In addition to
intensity, the location and quality (e.g., aching and burning) of pain are also
the assessment aspects.
Pain is defined as "an unpleasant sensory and emotional experience asso-
ciated with actual or potential tissue damage, or described in terms of such
damage" by the International Association for the Study of Pain (IASP)1. The
definition of pain reveals that pain has a sensory dimension and an affective
or emotional dimension. Although pain, especially acute pain, originates
from the sensory neurons in response to damaging or potentially damaging
stimuli, the pain sensation is actually processed by the brain [10, 11]. Due
to the subjective nature of pain, self-reporting is considered to be the gold
standard when assessing pain. Pain intensity is assessed with a pain scale
in one of several forms. For example, in acute postoperative pain, a score 4
in the 11-point scale from 0 to 10 or between 30 and 40 in a 100 mm scale
is clinically important borderline of receiving adequate pain control [12, 13].
1https://www.iasp-pain.org/Education/Content.aspx?ItemNumber=1698#Pain
1
Pain assessment is usually documented several times a day for patients with
acute pain and the document frequency is among the indicators of pain
management quality [14]. Depending on the defined nursing protocol, the
assessment of postoperative acute pain could be regularly every 4 h plus 1
h after the intervention [15], or every 15 min immediately after surgery [16].
However, such manual inquiries can never produce continuous monitoring
and would have "missing values" when a patient cannot self-report (e.g., in
sleep).
Moreover, several patient groups cannot communicate by any means (e.g.,
critically ill patients and people with limited cognitive ability) and thus can-
not self-report. In these cases, the alternative tools in clinical practice are
pain behavior observation tools [17–22] where the levels of pain behaviors
are observed by a trained caregiver. The pain behaviors cover pain facial
expression, body movement, vocalization or compliance with ventilation.
Additionally, in one tool, some physiological signs such as systolic blood
pressure, respiration rate, heart rate, and oxygen saturation rate are consid-
ered nonverbal signs of pain.
The previous observations and work on pain behaviors and physiological
signs of pain built the foundation and intuition for the later studies that tried
to quantitate the reactions to pain. Then more studies arise with the prolifer-
ation of miniaturized sensing systems and machine learning techniques. Now,
several of the pain behaviors and physiological signals are within the scope
of automatic pain assessment studies aiming at pain intensity estimation in
a pain event or in a time-continuous manner. The automatic assessment
"machine" is expected to work as accurately and reliably as self-report, and
at least as well as expert human observers. This study is part of the efforts
to reach this ultimate goal. The study was initialized and conducted within
the Smart Pain Assessment (SPA) multidisciplinary research group.
The biopotentials included in this study are several physiological signals
including
electrocardiogram (ECG) - a graph of the electrical activity of the
heart,
galvanic skin response (GSR) - the skin resistance change due to the
activity of sweat glands in the skin,
facial electromyogram (EMG) - a record of the electrical activity pro-
duced by facial skeletal muscles which are also a pain behavior indicator
corresponding to facial expression.
Additionally, the respiration rate is also part of the analysis. The reactions
of these signals in response to experimental pain were recorded from healthy
volunteers and were analyzed. Despite using a simulated scenario with exper-
imental pain, a broader scope of knowledge on pain (e.g., postoperative pain
and surgical stress) and pain reactions were used to ascertain information
and discussed. We believe this study will add value back to this field.
2
1.2 Research focus
This study focuses on modeling self-report pain from chosen physiological
signals to provide a continuous pain estimation, by recognizing the pattern of
the signals. The rationale behind the study is twofold. One is the autonomic
nervous system (ANS) activities [23], which act unconsciously and regulate
the body functions to fight-or-flight or rest-and-digest. Such regulations are
reflected in the physiological parameters such as heart rate, respiration rate,
and sweat. The other one is the pain behaviors introduced above.
One difficulty in achieving an automatic pain assessment tool is the in-
dividual difference in pain perception which is influenced by the inherent
past and the present states such as genetic, physical, social and psychologi-
cal factors. The fact that any single potential objective pain indicator does
not respond only to pain also makes the development difficult. For example,
heart rate may vary due to age, depth of anesthesia, medications, and emo-
tions, and this is the same as regards pain behaviors. Usually, an assessment
method is narrowed down to a specific type of pain and scenario and must
be validated before moving to a new one (e.g., surgical stress index). More-
over, among the behavior observation tools, there are indications that a tool
developed for one patient group may not apply or be generalized to another
group (e.g., from children to adults) [24].
From the technical point of view, the main challenge is how to adapt the
existing biosignal processing and machine learning techniques to the case
of pain measurement. In a classic pattern recognition flow, the techniques
such as signal acquisition, noise cancellation, feature extraction, and machine
learning models have been and continue to be developed for either a general
purpose or specific applications. For example, EMG feature extraction and
selection have been studied for myoelectric control applications, and heart
rate variability analysis is used for mental stress assessment or as a risk factor
for sudden cardiac death. Meanwhile, it would be interesting to see how far
these techniques can aid the progress of this application. Finally, this study
was inspired by the upsurge in wearable technology, the growing number of
published databases for automatic pain assessment using machine learning
methods1, as well as the finding that multiple parameters or a multimodal
is better than one [30, 31].
11) EmoPain [25] (2015), chronic pain; 2) UNBC-McMaster [26] (2011), shoulder pain;
3) Infant COPE [27] (2007), neonatal pain; 4) BioVid [28] (2013), experimental heat pain;
5) X-ITE [29] (2019), experimental heat and electrical pain.
3
1.3 Aim and objectives
The study aims to model self-report pain from ECG, GSR, and facial sur-
face EMG signals in order to provide a continuous estimation of pain with
the Smart Pain Assessment - Experimental Pain Database (SpaExp). With
adjustable stimulus intensity and duration, experimental pain stimuli evoke
pain in a controllable way to unify the subjective pain report (i.e., pain
threshold 1, pain tolerance 2, and pain intensity) and are commonly used
in pain studies. As a study learning from experimental pain, this work is
expected to provide a system design and answer questions regarding validity,
reliability, and the limitations of the approach. The results will give some
indications regarding real acute pain cases such as postoperative pain and
the pain in intensitive care unit (ICU). To reach the study goal, the sys-
tem design is divided into several sub-designs linked to one another but also
relatively independent. The step-by-step objectives are
To identify the target biosignals and assistive pain assessment tools
that have been used or potentially can provide noninvasive and con-
tinuous monitoring of pain;
To build and evaluate a biosignal acquisition system for data collection
and build the SpaExp database by collaborating with medical doctor
and nurse researchers;
To develop a processing flow for continuous pain estimation based on
the attribute of each signal or feature and the SpaExp study design
whilst also considering the compatibility to different databases or study
design with the same signal;
To examine the potential capability of each extracted feature on esti-
mating pain by visualizing and observing the average response to pain;
To assess pain (intensity) by multimodal machine learning;
To evaluate the importance of the features and the model performance,
and compare the performances across multiple classifier models.
1.4 Contributions
Developing automatic pain assessment methods is an interdisciplinary topic.
The existing studies either involve experts in clinical medicine and computer
science or in psychology and computer science due to the affective dimension
of pain. The increasing number of pain databases with different signals or
study designs encouraged the desire of many computer science researchers to
improve the estimation performance within a database from the perspective
1https://www.iasp-pain.org/terminology?navItemNumber=576#Painthreshold
2https://www.iasp-pain.org/terminology?navItemNumber=576#
Paintolerancelevel
4
of algorithm optimization (i.e., trying designs with different face descriptors,
features extractions, and machine learning or deep learning models). within
the same database (i.e., study design), the performance could be improved
through algorithm optimization. However, the steps forward have been sub-
tle rather than revolutionary to make a valid and reliable tool. Moreover,
few studies analyzed data across different databases. The similarity and
difference among the study designs have hardly been discussed, especially
between the pain in hospital and the stimulated pain in an experimental
environment. This study widely reviewed the studies from clinical assess-
ment tools to the efforts made to develop automatic pain assessment tools.
Untangling the similarities among the study designs and finding the gaps is
the first contribution of this study.
Secondly, the endeavors of this study and my other works using the same
database [32–34] were aimed at answering the questions of: 1) which parts in
the classical processing flow for window-level pain classification limit the pre-
diction performance, and 2) whether it is valuable to put significant efforts
on optimizing signal processing, feature extraction as well as machine learn-
ing so as to make a better prediction within one database. Multiple attempts
and efforts were made on analysis time window, biosignal features, feature
normalization, machine learning models, performance evaluation, and study
design interpretation. The answers although they may be limited to the
database, do however contribute some new thinking to the field.
1.5 Thesis content summary
As pain estimation is defined as a pattern recognition problem, Chapter 2
first reviews the corresponding outputs/labels and input signals. The po-
tential labels are reviewed from the existing pain scales in clinical practice
and the study designs in automatic pain assessment studies. The pain scales
cover the rating scales used in self-reporting, objective observation, and noci-
ception/antinociception (NAN) balance. The reviewed signals include ANS
system based signals, other biosignals including respiration, oxygen satura-
tion, and neuroimaging signals, as well as pain behaviors summarized from
seven behavioral pain assessment tools. At the end of Chapter 2, some ex-
ample devices or systems which can continuously record one or several of the
mentioned biosignals are briefly introduced.
Chapter 3 provides detailed technical specifications of the devices for
the SpaExp data collection and the next phase of the study. The devices
include the developed biopotential acquisition system for multi-channel facial
surface electromyogram (sEMG) recording [35], Bioharness 3 for ECG and
respiration recording, and the eHealth v2.0 platform for GSR recording. In
addition, a more advanced patient monitor and wearable GSR solution - E4
wristband from empatica (E4) - are presented.
5
Chapter 4 presents the study design and the SpaExp database specifi-
cations. The details of the study design are experimental protocol, pain
stimulation, and biosignal measurements. The collected data and biosig-
nals/parameters are summarized and listed.
Chapter 5 reviews, discusses, and presents the implementation of signal
processing and feature extraction with the collected ECG, GSR, and sEMG
signals. A total of 103 features are derived as a result of signal denoising,
processing, feature extraction from sliding time windows with customized
lengths, and feature rescaling.
Chapter 6 visualized sixteen of the total 103 features. They are chosen
due to their weak correlation with the others. Each feature is a time series
and is plotted as a line graph. In total 120 pain tests are divided into three
groups according to the self-reported pain intensities. In each group, the
feature time series of the tests are aligned to the time when the gradual-
increasing pain stimulus started to be perceived as pain. The average line in
each group is observed and compared so as to answer the three questions: 1)
How does the feature react to the pain stimulus since stimulation start? 2)
How does the feature react to the sensation of pain after the pain stimulus is
perceived as pain? 3) Does the feature have the potential to differentiate pain
intensities across subjects? At the end of the chapter, the similarity among
all the representative features is checked with multidimensional scaling.
Chapter 7 introduces the final machine learning models. By analyzing
and interpreting the study design, the pain intensity recognition is designed
as a two-step estimation. The first step is binary classification estimating
whether the pain was present or whether the pain control was inadequate
as a clinical significance. The second step was an estimation of three classes
of pain intensity - when the pain is present or pain control is inadequate.
More than twenty models are built in both classification steps, and the top
best models are selected to observe any performance change if the hyper-
parameters of one model are optimized. The results in each step are sepa-
rately presented and discussed. Another important part of this chapter is a
review of automatic pain assessment studies. The review lists the similarity
and difference between the related studies and databases from four perspec-
tives, and the difficulties in the generalization of the methods and results are
discussed. Finally, Chapter 8 concludes the study.
6
Chapter 2
Labels and Signals in Pain
Assessment
The aim of this study is to build a model mapping from non-self-reported
signs of pain as a substitute for pain assessment in self-reporting or check-
lists. This chapter gives a comprehensive review of the methods used for
pain assessment, first from the angle of assessment targets - labels of pain,
and then from the perspective of observed signs of pain - potential input
signals. The labels and signals indicate the learning aim and the materials
for learning, respectively, when using machine learning methods.
2.1 The complex and subjective nature of pain
The perception of pain results from the interplay of the three motivational
components: sensory, affective and cognitive, based on Melzack and Casey’s
model of pain [36]. In addition to the physical, social and psychological
factors that could influence the individual pain experience, other sensory
perceptions and visions could also impact the perceptual experience of pain
because they are integrated into the brain and nervous system.
Due to the complex and subjective nature of pain, self-report is consid-
ered to be the gold standard for assessing pain and is widely applied to assess
pain in clinical trials and pain management. Valid and reliable assessment
of pain is essential in pain management decision-making. The assessment
of pain location and pain intensity is sufficient for acute pain that lasts less
than three months and in cases where the pain is a symptom of trauma
or disease [37]. While in the cases of chronic pain and some specific dis-
eases or pain conditions, some corresponding tests and questionnaires are
designed for each case assessing the impact of pain on the patients’ physical
and emotional functioning as well as their life.
7
However, self-report is not applicable to the patients who cannot commu-
nicate verbally, in writing or by other means such as finger span or blinking
eyes to answer yes or no questions. The patient groups are i) older adults
with advanced dementia, ii) infants and preverbal toddlers, iii) critically
ill/unconscious patients, iv) persons with intellectual disabilities and v) pa-
tients at the end of the life [24]. In clinical practice, patient behaviors are
observed as an indication of pain and various behavioral pain assessment
tools have been designed for each population of patients. Unfortunately, no
behavioral pain assessment tool applies to all because the patient popula-
tion and given context differs. The reliability and validity of a tool should
be ensured in each case.
Considering the complex nature of pain as described above, the scope
of the study regarding pain is narrowed down to the intensity of acute pain
derived from the healthy adult volunteers, which is expected to be trans-
ferred to postoperative patients in the future and ultimately to the critically
ill/unconscious adults. Although this study only includes data from experi-
mental pain with healthy volunteers, studies with a broad scope of pain and
patient groups (e.g., intensity recognition of chronic pain) are also involved in
the review due to the insight they give on algorithm design and the scarcity
of related works on the topic. Properly managed acute pain could prevent
pain from becoming chronic [38]. It is therefore essential to deal with acute
pain such as postoperative pain effectively. One important reason to only
focus on acute pain is that the cause of acute pain is mainly tissue damage
while for chronic pain it could also be nerve damage or a consequence of de-
pression/anxiety. For this reason, the pain management strategies for acute
and chronic pain are different. Another important reason is that the pattern
of the body’s response to acute pain and chronic pain could be different. For
example, acute pain may elevate heart rate, blood pressure, and respiration
rate, while chronic pain is not usually accompanied by behavioral changes
except exacerbation [39].
The motivations for developing automatic pain monitoring systems are
twofold. First, pain assessment and the adjustments on pain medication
in a hospital requires routine checks so that adequate pain management
is highly reliant on manpower. Second, a gap may exist between patients
and healthcare providers when inferring pain in others and interpreting pain
intensity scores, which could result in undertreatment or overtreatment of
pain [40, 41]. In other words, automatic pain assessment could be a solution
optimizing the existing pain assessment approaches and pain management
in terms of efficiency, reliability, and validity.
8
2.2 Pain scales in clinical practice and algorithms
Pain intensity, as one of the dimensions in pain assessment, is commonly
used in clinical practice to characterize pain and evaluate the effects of pain
treatment. In the research on pain intensity pattern recognition, the existing
pain assessment tools and concepts act as the ground truth in the system
development and validation.
The two well-known pain intensity scales are the visual analog scale
(VAS) [42] and the numeric rating scale (NRS) [43]. The main difference
between VAS and NRS is that VAS [44] is in continuous numbers within a
range between no pain and the worst imaginable pain, usually, one point on
a 100 mm ruler, while NRS is a scale of integers between 0 and 10. The
third scale is more coarse-grained, and is called the verbal categorical rating
scale (VRS), in a four-point VRS: "mild" is equal to the numbers from 1
to 3 in NRS; "moderate" is equal to the numbers in NRS from 4 to 6; and
"severe" is equal to the numbers larger than 6 in NRS. Among these three
scales, NRS and VAS are considered to be the most practical to use. In
contrast, pain can be underestimated when using VRS [37]. In addition to
the descriptive words "mild," "moderate," and "severe," other words such
as "unbearable", "intense pain" and "maximum pain" are also found in the
literature to describe pain intensity [45, 46].
For the patients who are unable to self-report, the pain scores of some
behavioral tools are derived from adding up the scores of several indicators.
Therefore, the range of the scale could be different from the standard pain
intensity rating. In this case, the scores in different scales are incompara-
ble. For example, in the Critical-Care Pain Observation Tool (CPOT) [45],
facial expression, body movements, muscle tension and compliance with the
ventilator/vocalization are scored separately from 0 to 2, so the range of
the overall score is between 0 and 8. Another similar example is in the
UNBC-McMaster shoulder pain expression archive database [26]. The score
in Prkachin and Solomon Pain Intensity Metric (PSPI) [47] is the sum of
the intensities of four action units (AUs) each intensity ranging between 0
and 4. In the same database, ratings in two other scales, VAS (0-10) and
observed pain intensity (0-5), are documented. UNBC-McMaster database
is one of the open access databases for researchers to develop automatic pain
assessment systems.
In addition to the description of pain intensities introduced above, the
two terminologies relating to both the pain experience and the pain-inducing
stimulus are pain threshold and pain tolerance. They are commonly found in
pain sensitivity studies or/and the studies involving experimental pain stim-
ulation such as electrical stimulation and thermal stimulation [28, 33, 48, 49].
According to the definitions by IASP, pain threshold is "the minimum in-
tensity of a stimulus that is perceived as painful", and pain tolerance is "the
9
maximum intensity of a pain-producing stimulus that a subject is willing
to accept in a given situation." The notations following the definitions also
emphasized that a pain threshold and pain tolerance are the subjective expe-
riences of the individual rather than the stimulus level itself. The definitions
of pain threshold and pain tolerance are separate from the definition of pain
intensity scales. However, the pain threshold is considered as the moment
when the VAS exceeds 0 [50] and just reached 3 or 4 [33] respectively for the
first time during a continuous increase of stimulation intensity.
Another perspective to pain intensity assessment could be NAN balance
[51]. However, the index measuring it is mainly applied to cases under general
anesthesia, e.g., in an operation, during emergency and postoperative care.
2.3 Signals in pain intensity recognition
Pain is a subjective feeling generated by the brain to protect ourselves. How-
ever, pain observations tools, pain intensity recognition algorithms coded
from measurable signals, and pain reported by parents or family members
are proxy measures and are objective in essence. Although the current tech-
nologies already enable us to observe brain activities and the change of many
physiological signals, assessing pain objectively is still challenging from sev-
eral aspects. On one hand, the body responds to pain through numerous and
interconnected physiological processes involving multiple body systems in-
cluding cardiovascular, respiratory, immune, endocrine, gastrointestinal, uri-
nary, musculoskeletal, nervous and brain [39]; nevertheless, there is no one
signal(s) reflecting all the responses; On the other hand, each measurable
signal does not only react to pain. To ensure the reliability of a developed
pain assessment tool, an ideal one should be sensitive and specific to pain.
It needs to be observer-independent, not reliant on the patient’s ability to
communicate and not influenced by disease characteristics [5]. Therefore,
the fusion of multiple signals is one trend in developing automatic pain as-
sessment methods.
There are several rationale dimensions found in the literature that au-
tomatic pain assessment tools or nociception/antinociception balance tools
have. One of them is based on the assumption that pain induces alterations
in the sympathetic nervous system [52]. The sympathetic nervous system
is one of the two main divisions of the ANS which regulates the body’ s
unconscious actions. The sympathetic nervous system stimulates the body
to "fight-or-flight", while the other main division of ANS, the parasympa-
thetic nervous system stimulates the body to "feed and breed" and "rest
and digest". Another approach to recognizing pain is to recognize part of
pain behaviors such as facial expressions of pain [26]. A third dimension
is visualizing the brain in pain through neuroimaging, where both physical
10
and cognitive influences on pain can be directly observed [53–55]. A fourth
dimension could be biomarkers at cellular level measuring genetic or protein
responses, or metabolic products [5]. The developed pain assessment tools
usually cover one or several of the first three dimensions. These tools and
their signals are reviewed in the following part of this subsection. Besides
the signals within the dimensions mentioned above, two physiological pa-
rameters, respiration and blood oxygen saturation are also introduced below
because they are both found being discussed in pain assessment studies and
breathing techniques are used to alleviate pain.
2.3.1 Autonomic nervous system (ANS) based signals
The literature showed that many existing NAN balance tools are based on
analyzing the reactions of ANS during general anesthesia. Even though there
are already several NAN balance measurement commercialized solutions, no
gold standard exists [4]. The ANS-based signals used in the NAN balance
tools are also commonly found in a bedside monitor such as blood pressure,
heartbeat interval and photoplethysmogram (PPG). PPG is also referred to
as blood volume pulse (BVP). However, it is different in the case of pain
assessment of patients who are unable to self-report as the ANS-based sig-
nals are poor in specificity when differentiating pain from other sources of
distress. These indexes could be influenced by numerous other physiological
and psychological conditions as well such as age, co-morbidities, depth of
anaesthesia, surgical stimulation, medications, and emotions. Moreover, the
absence of a change in vital signs does not indicate the absence of pain. For
these reasons, the ANS-based signals are not recommended in assessing the
pain of patients who are unable to self-report in clinical practice [24]. In
spite of this, ANS-based signals, especially the fusion of them, are active in
the discussion of automatic pain assessment studies [28, 30, 33, 56].
Inter-beat interval
The signals reflecting ANS activities are listed in Table 2.1 together with
some NAN balance indexes. Among them, the pulse interval is commonly
used in the NAN balance models either as original or transformed parame-
ters. With the help of heart rate variability (HRV) analysis, the sympathetic
activity is traceable from the sympathetic cardiac regulation and the same
applies to parasympathetic activity. The inter-beat intervals are extracted
from electrocardiography mostly and from PPG in some cases. The normal
inter-beat intervals are called normal-to-normal (NN) series.
Further analysis, which is called HRV analysis, can be made within a time
period in the time domain, frequency domain and in other forms. When HRV
is used to trace sympathetic or parasympathetic activity, the analysis is usu-
11
Table 2.1: The biosignals reflecting ANS activities and some NAN indexes [4, 5]
Signal Index name Parameter(s) Measure Model
Pupil dilation pupillometry
pupillary diameter and the
light-induced pupillary di-
latation reflex
sympathetic activity -
PPG Surgical pleth index
(SPI)
pulse wave amplitude
(PPGA) and heart beat
interval (HBI)
sympathetic activity SPI=100-
(0.7×PPGAnorm+0.3×HBInor m)
GSR skin conductance
number & amplitude of
SCR fluctuations (NSCF &
ASCF)
sympathetic activity painful stimuli induce an immedi-
ate increase in peaks per second-
ECG Analgesia/Nociception
index (ANI)
0.15Hz-0.4Hz bandpassed
RR seriesnorm
parasympathetic activity
ANI=100×(5.1×AUCmin+1.2)/12.8,
where AUC is the area between
the envelope of local maxima and
local minima
ECG and continu-
ous blood pressure
CARdiovascular
DEpth of ANalge-
sia(CARDEN)
RR under curve and sys-
tolic blood pressure under
curve
sympathetic activity a minor elevation in blood pressure
followed by minor tachycardia
12
ally within less than 5 minutes or several tens of seconds depending on the
study design. Among the HRV features, NN50, the number of NN interval
differences that larger than 50 ms and HRV-the high-frequency power com-
ponent of NN series (0.15-0.40 Hz), the high-frequency power component of
NN series (HF: 0.15 to 0.40 Hz), are considered as indices of parasympathetic
activity. In addition, the ratio of LF/HF (LF: 0.04 to 0.15 Hz) is an index of
sympathetic activity [57]. As Table 2.1 shows, HF is the primary parameter
to observe in an analgesia/nociception index (ANI) [58].
PPG amplitude
In addition to the inter-beat intervals, the PPG amplitude (PPGA) is also
part of the surgical pleth index (SPI) or surgical stress index (SSI) by another
name [59, 60]. The up or down of PPGA can be regulated by anesthesia,
sympathetic activation, arterial blood pressure (ABP) increase and some
other factors. The sympathetic activation could lower PPG, while anesthet-
ics could increase PPGA, and the rise of ABP may lead to either PPGA up
or down due to different causes [61].
Blood pressure
In the CARDEAN index [62], the change of continuous systolic blood pres-
sure (SBP, in mmHg) is taken into account along with the change of NN
series. Meanwhile, it has been observed that pain can raise blood pressure
and therefore an increase in systolic blood pressure is considered one of the
signs indicating intraoperative nociception [51, 63].
Electrodermal activity/Galvanic skin response
Electrodermal activity (EDA) monitors resistance variations in the skin due
to the autonomic activation of sweat glands in the skin. As EDA reflects
activity only within the sympathetic axis of the autonomic nervous system,
it is widely used in indexing emotional processing and sympathetic activity
[64]. EDA is also referred to as skin conductance, or GSR and GSR will be
the term used throughout the thesis. GSR is recorded as the conductance in
micro-Siemens (µS) between two measuring points on hand palms, fingers,
or foot soles. The GSR signal has two components to show change: one is in
the tonic level, changing within tens of seconds to minutes; the other one is
the phasic response on top of the tonic changes where obvious peaks can be
observed within one or several seconds. The former component is the skin
conductance level (SCL). The latter component is called skin conductance
response (SCR), which is more informative and is found sensitive to emo-
tionally arousing stimulus events [65]. Similarly, SCR is also emphasized in
pain assessment studies, for example, as peaks per second [66].
13
2.3.2 Respiration and oxygen saturation
Respiration
The signals introduced above are regulated unconsciously by the body, while
respiration is different because it can be adjusted manually. Breathing tech-
niques such as slow breathing and paced slow deep breathing are commonly
applied to alleviate pain as a routine procedure in the hospital, although the
physiological mechanisms behind it are not fully known yet. On the other
hand, respiratory parameters are also taken as potential indicators of pain
and were observed in both experimental and clinical studies. According to
Jafari H. et al’s review of pain and respiration [67], the respiratory pattern
does change along with a painful procedure or pain relief. For example,
inspiratory flow is found increased in experimental studies due to sudden
cutaneous pain. The process of respiration consists of inspiration (or inhala-
tion) where air rushes into the lungs and expiration (or exhalation) where
the air is forced out. Inspiratory flow is the ratio of inspiratory volume over
inspiratory time. The increase of inspiratory flow in response to sudden cu-
taneous pain is concluded to be similar to the startle reflex. This finding is
summarized to be consistent in the cases of sustained pain in experimental
studies and painful procedures in clinical studies, where increased ventila-
tion or hyperventilation is observed as a result of deeper breathing, faster
breathing (respiration rate increase) or a combination of both. However,
these observations are concluded from the mean value out of a group of
subjects, and may not be consistent across individuals.
Oxygen saturation
In medicine, oxygen saturation is the fraction of oxygen-saturated hemoglobin
relative to total hemoglobin in the blood. Peripheral oxygen saturation
(SpO2) can be derived from PPG, and it is an approximation to arterial
oxygen saturation (SaO2). Normal SpO2 is between 95% and 100%. The
observations are inconsistent among the related studies with different patient
populations and pain scenarios. SpO2 was observed to decrease in number
or/and increase in variability during a painful procedure in some studies
[68–70], while it was not found to be useful in some other studies [71, 72].
2.3.3 Behavioural signals
...However, the appropriateness of a tool must be assessed patient by patient,
and no one tool should be an institutional mandate for all patients. For
example, a behavior pain tool developed for persons with dementia may not
be appropriate for patients in the ICU who are unable to communicate, and
tools for children are not generalizable to adults. [24]
14
There are many behavioral pain assessment tools in use for different pa-
tient populations. As critically ill/unconscious adults in ICU are the patient
population this study will eventually serve, the behavioral pain assessment
tools that can be applied to critically ill/unconscious adults according to [24],
together with a tool validated in [22] are reviewed below. The tools are: 1)
BPS: Behavioral Pain Scale [17], 2) CPOT: Critical-Care Pain Observation
Tool [73], 3) FLACC: Face, Legs, Activity, Cry, and Consolability Behav-
ioral Assessment Tool [18], 4) PBAT: Pain Behavioral Assessment Tool [19],
5) NPAT: Nonverbal Pain Assessment Tool [20], 6) NVPS: Nonverbal Pain
Scale [21], and 7) BPAT: 8-item Behavior Pain Assessment Tool [22].
Each tool observes behavioral signs from at least three perspectives.
These are facial expressions, movements, and vocalization/compliance with
ventilation. There are two other categories not rigidly fitting into these three
parts. Following the principle of proximity, the emotion category in NPAT is
merged into facial expressions in the review below and the physiologic cate-
gory in NVPS is discussed with vocalization/compliance with ventilation. In
most of the tools, the output is a final score from a sum of sub-scores from
each category. In each category/indicator/item, the sub-score is defined by
either the presence or degree of the behavior. A larger score of each tool
may indicate higher pain intensity, however, the score number and pain in-
tensity number are not highly correlated. For example, it is reported that
BPAT showed a moderate ability to discriminate severe levels of pain in-
tensity (NRS8), and there is a moderate correlation between pain distress
and behavioral scores during common procedures performed in ICU patients
which also supports the interrelation between the affective and behavioral
dimensions of pain [22].
Facial expressions
From the descriptions of facial expressions summarized in Table 2.2, it can
be seen that the minimum score is consistently described as relaxed or calm.
However, the descriptions for the middle score(s) and the maximum score
demonstrate some differences across the tools, for example, in terms of the
target facial expressions. In tools such as BPS, CPOT, and NPAT, the
presence of some facial expression determines the score number, while in
FLACC and NVPS, while in FLACC and NVPS, the concern is more about
the frequency of shown facial expressions. The most frequent description
shown in Table 2.2 is grimacing, which is followed by frown and eyelid closed.
In BPAT, the involved AUs are listed for each behavior. AUs are defined
in the Facial Action Coding System (FACS), which describes all visually
distinguishable facial activity from 44 AUs [74], and each AU is dominated
by one or two facial muscles. FACS is an observational coding scheme serving
both manually and computer-based automated coding. In BPAT, grimace is
15
Table 2.2: The description on the facial expressions/emotion in the behavioural pain assessment tools
Min score Middle score(s) Max score
BPS 1- Relaxed 2- Partially tightened (e.g., brow lowering) 4- Grimacing
3- Fully tightened (e.g., eyelid closing)
CPOT No muscular tension observed Presence of frowning, brow lowering, orbit
tightening, and levator contraction
All of the above facial movements plus
eyelid tightly closed
0- Relaxed, neutral 1- Tense 2- Grimacing
FLACC 0- No particular expression or
smile
1- Occasional grimace, frown, withdrawn or
disinterested
2- Frequent to constant frown, clenched
jaw, quivering chin
NVPS 0- No particular expression or
smile
1- Occasional grimace, tearing frown or wrin-
kled forehead
2- Frequent grimace, tearing, frown or
wrinkled forehead
NPAT 0- Relaxed, calm expression 1- Drawn around mouth and eyes; tense 2- Facial frowning, wincing, grimacing
(Emotion) 0- Smiling, calm, relaxed, exhibit-
ing no emotion
1- Anxious, irritable, withdrawn, closes eyes,
does not engage with physical environment 2- Tearful or uncooperative
PBAT Grimace, frown, wince, eyes closed, eyes wide open with eyebrows raised, looking away in opposite direction of the pain,
grin/smile, mouth wide open to expose teeth and tongue, clenched teeth exposing slightly open mouth, unable to assess, other
BPAT Neutral expression, grimace, wince, eyes closed
16
defined as a combination of AU4 brow lowering, AU6 cheek raising, AU7 lid
tightening, AU20 mouth stretching and AU43 eye closing.
Table 2.3: The facial action units involved in pain facial expressions in adults
Facial action unit Muscular basis Description/Scale
Grimace Wince PSPI CPOT
AU4 brow lower Corrugator supercilii X X X
AU6 lids tighten Orbicularis oculi X X X X
AU7 cheek raise Orbicularis oculi X X X X
AU9 nose wrinkle Levator labii superioris X X
AU10 upper lip raiser Levator labii superioris X X
AU12 lip corner pull Zygomaticus major
AU20 horizontal mouth
stretch Risorius X
AU43 eyes closed
Relaxation of Leva-
tor palpebrae superi-
oris, Orbicularis oculi
X X X
The FACS and its muscular basis are the theoretical foundations of auto-
matic emotion or pain recognition from facial expressions, as FACS provides
the possibility of quantitative measures on facial expressions. The AUs in-
volved in pain facial expressions among adults are summarized in the review
[75] and are listed in Table 2.3 together with some descriptions of pain fa-
cial expressions and the PSPI metric. In the PSPI metric, pain intensity is
mapped from the intensity (0=absent, 5=maximum) of three AUs and the
binary intensity of AU43, which is calculated based on Equation 2.1.
Pain intensity=Intensity(AU4)+(Max intensity AU6 or AU7)+
(Max intensity AU9 or AU10)+Intensity(AU43) (2.1)
As Table 2.3 shows and according to Prkachin and Solomon [47], the four
actions, brow lowering (AU4), orbital tightening (AU6 and AU7), levator
contraction (AU9 and AU10) and eye closed (AU43) are the core actions
that carry most of the information about pain. However, the use of AU43
needs to be further considered because eyes closed could be the second most
frequently observed facial expression among sedated patients at rest after
neutral. Therefore, AU43 may not be effective in indicating the presence of
pain in non-communicative critically ill patients. This may explain why the
description of eyelid tightly closed is used instead in CPOT.
The PSPI metric is implemented mainly in the UNBC-McMaster video
sequences database for researching automatic pain facial expression recogni-
tion. In the database, the video frames were coded manually with AUs and
were labeled with both self-report VAS scores and observer ratings. Mean-
while, models can be built to detect facial movements automatically so that
the pattern of each AU and its intensity could be recognized in order to
17
calculate PSPI. Another approach to recording facial expressions is facial
sEMG, where surface electrodes are placed on the muscle area of interest to
capture the electric potential generated by muscle cells during muscle con-
traction. In the detection of pain facial expressions, the facial muscles listed
in Table 2.3 are the muscle areas of interest.
Movements
Compared to facial expressions, the observed perceptions of how the body
reacts to pain are more versatile in behavioral pain assessment tools. Accord-
ing to the descriptions on movements in Table 2.4, the body’s reactions can
be summarized into three types, the first type is voluntary movements (e.g.,
legs movements in FLACC and body movements or activity in CPOT, NVPS
and NPAT); the second type is reactions to passive movements (e.g., muscle
tension in CPOT and guarding in NPAT); the third type is the posture or
the static state of the body described, for example, rigid,clenched fists, and
fetal position. To date, studies on automatic pain assessment by recogniz-
ing pain movements are rarely found in the literature. However, hand and
body gesture recognition using a camera has been under development dur-
ing the past decades for human-computer interaction (e.g., [76]and [77]). An
endeavor was especially made in [76] to recognize the pattern of emotions
with a fusion of facial expression, body gesture, and acoustic analysis, which
could be used as a reference to pain assessment studies in the near future.
Vocalization/Compliance with ventilation
In ICU, the patients could be intubated or mechanically ventilated and the
equipment does not allow them to make verbal sounds. In this case, com-
pliance with the ventilator or not is observed instead of vocalization, which
is described as the level of patient-ventilator asynchrony or dyssynchrony
as Table 2.5 shows. Descriptions such as tolerating movement and coughing
but tolerating are also found in the minimum and middle scores of BPS and
CPOT. In terms of vocalization, the middle scores are described as sighing,
moaning and whimpering, while the maximum scores are described including
sobbing,crying out and screaming. In addition, verbal complains of pain is
also considered.
Among the discussed assessment tools, NVPS is the only one considering
physiological signs, and it covers most of the signals mentioned in Section
2.3.1 and Section 2.3.2. The relative changes on vital signs and respiratory
parameters are observed, either within a long time-window of 4 hours or
comparison with the defined baseline.
18
Table 2.4: The description on the movements/guarding/muscle tension in the behavioural pain assessment tools
Min score Middle score(s) Max score
BPS 1- No movement 2- Partially bent 4- Permanently retracted
(Upper limbs) 3- Fully bent with finger flexion
CPOT (Body)
Does not move at all (does
not necessarily mean absence
of pain)
Slow, cautious movements, touching or
rubbing the pain site, seeking attention
through movements
Pulling tube, attempting to sit up, mov-
ing limbs/thrashing, not following com-
mands, striking at staff, trying to climb
out of bed
0- Absence of movements 1- Protection 2- Restlessness
(Muscle tension) No resistance to passive move-
ments Resistance to passive movements Strong resistance to passive movements,
inability to complete them
0- Relaxed 1- Tense, rigid 2- Very tense or rigid
FLACC (Legs) 0- Normal position or relaxed 1- Uneasy, restless, or tense 2- Kicking, or legs drawn up
(Activity) 0- Lying quietly, normal posi-
tion, moves easily
1- Squirming, shifting back and forth, or
tense Arched, rigid or jerking
NVPS (Activity) 0- Lying quietly, normal posi-
tion
1- Seeking attention through movement of
slow cautious movements
2- Restless activity and/or withdrawal re-
flexes
NPAT (Body) 0- None, sleeping comfortably 1- Restless or slow, decreased movement 2- Immobile, afraid to move or increased
motion
(Positioning
/Guarding) 0- Relaxed body 1- Guarding, tense 2- Fetal position, jumpy when touched,
withdraws when touched
PBAT No movement, rigid, arching, clenched fists, shaking, withdrawing, splinting, flailing, picking/touching site, restlessness,
Rubbing/massaging, repetitive movements, defensive grabbing, pushing, guarding, unable to assess, other
BPAT Rigid, clenched fists
19
Table 2.5: The description on the vocalization/compliance with ventilation/consolability/phsiological signs in the be-
havioural pain assessment tools
Min score Middle score(s) Max score
BPS (compliance with venti-
lation) 1- Tolerating movement 2- Coughing but tolerating ventila-
tion for the most of the time 4- Unable to control ventilation
3- Fighting ventilator
CPOT (compliance with the
ventilator)
Alarms not activated, easy
ventilation Alarms stop spontaneously Asynchrony: blocking ventila-
tion, alarms frequently activated
0- Tolerating ventilator or
movement 1- Coughing but tolerating 2- Fighting ventilator
(Vocalization) Talking in normal tone or no
sound Sighing, moaning Crying out, sobbing
0- Talking in normal tone or
no sound 1- Sighing, moaning 2- Crying out, sobbing
FLACC (Cry) 0- No cry 1- Moans, whimpers, or occasional
complaint
2- Crying steadily, screams or
sobs, frequency complaints
NVPS (Vital signs) 0- Stable vital signs, no
change in past 4 hours
1- Change over past 4 hours in any
of the following: SBP>20, HR>20,
RR>10
2- Change over past 4 hours in
any of the following: SBP>30,
HR>25, RR>20
(A. Respiratory, 0- Baseline RR/SpO2 1- RR>10 above baseline or SpO2 de-
crease 5%
2- RR>20 above baseline or
SpO2 decrease 20%
compliant with ventilator) Complaint with ventilator Mild asynchrony with ventilator Severe asynchrony with ventila-
tor
(B. Physiological signs) 0- Warm, dry skin 1- Dilated pupils, perspiring, flushing 2- Diaphoretic, pallor
NPAT 0- Intubated, no verbalization 1- Whimpering, moaning, sighing 2- Screaming, crying out
PBAT Moaning, screaming, whimpering, crying, using protest words,
verbal complaints of pain, none, unable to assess, other
BPAT Moaning, verbal complains of pain
20
2.3.4 Neuroimaging signals
It is natural to think about assessing pain directly from brain as it dominates
all the dimensions of pain processing. However, due to the nature of each
signal, there are limitations on either ability constraints or device size and
expense that impede it from being studied and validated in a large scale.
In each study, the pattern of pain is learned from a small group of subjects
with chronic pain or healthy volunteers with experimental pain stimulus
(e.g., [78] and [79]). As is concluded in [55], among the functional imaging
techniques, the hemodynamic methods positron emission tomography (PET)
and functional magnetic resonance imaging (fMRI), are utilized to ascertain
specific points of cerebral activation due to a better spatial resolution. While
the electrophysiological methods, electroencephalogram (EEG) and magne-
toencephalography (MEG), are often used to observe the brain’s temporal
response to pain due to their good temporal resolution.
2.4 Wearable devices and IoT-enabled systems
Although the potential indicators of pain have been enumerated in Section
2.3, not all of them are applicable in continuous pain monitoring. For ex-
ample, most of the neuroimaging methods are expensive with cumbersome
machines which cannot be used every day and by patients in beds with var-
ious tubes from other machines. Exceptionally, the EEG and near-infrared
spectroscopy methods have wearable solutions where a sensing device is made
into a headset or a cap.
Regarding the behavioral signals, they could be videoed and recorded as
they can all be checked visually. However, as they are measured indirectly
from the video, computer vision techniques are needed to recognize each
sign of pain. Moreover, videos, especially those including the facial area,
violate the privacy of the patients in hospital. The existing data sources
of facial expression videos are mainly from healthy people or people with
chronic pain rather than patients in hospital. By contrast, the behavioral
signs of pain other than facial expressions are rarely found in automatic pain
assessment studies. In addition to the video recording, behavioral signs of
pain could also be determined from other sensing methods with small and
wearable sensing devices. For example, the muscle tension and facial muscle
contraction can be recorded with sEMG; body movements and body position
could be calculated from a set of accelerometers and gyroscopes when placed
in proper positions. The development of a compact facial sEMG device is
presented in this study with details in Chapter 3.
Compared to the rest signs of pain, most of the physiological signals are
much easier to record for later analysis. Many of them, including ECG, PPG,
blood pressure, respiration and oxygen saturation, are vital signs which are
21
routinely monitored by a bedside monitor in a recovery room or an ICU.
Meanwhile, such wearable devices already exist in the market to monitor
ECG, PPG, respiration, oxygen saturation or GSR. One key point the re-
searchers in this study concern is that whether the data at original sampling
rate is accessible. From the data accessibility point of view, some devices and
systems are listed as examples in Table 2.6. E4 wristband and Bioharness
3 chest strap are wearable devices measuring multiple physiological parame-
ters. Both of them support real-time signal waveform display and on-device
signal recording. The other two systems in the table are bedside or patient
monitors used in hospitals. IntelliVue MMS X2 represents the bedside mon-
itors with comprehensive functions. In terms of noninvasive blood pressure,
it could only be measured discretely every 15 minutes whereas CNAP R
can
provide continuous monitoring.
Table 2.6: Example devices and systems providing raw data for research
Device Sensors Platform
E4 wristband PPG, Recording:
(E3 version [80]) GSR, wristband-computer-cloud
3-axis accelerometer, Streaming:
infrared thermopile, wristband-mobile device-cloud
internal real-time clock.
Bioharness 3 ECG Lead I, device-computer
device & chest strap respiratory cycles,
(version 1 [81, 82]) 3-axis accelerometer.
CNAP R
monitor double finger sensor, for
monitoring continuous noninvasive arterial pressure (CNAP)
IntelliVue MMS X2 up to 12-Lead ECG,
patient monitor Noninvasive blood pressure,
Invasive arterial blood pressure,
PPG, etc.
Similar to the central patient monitoring systems in hospitals, wear-
able devices can be integrated into a remote monitoring system within an
Internet-of-Things (IoT) architecture. As shown in the example described
in [83], the architecture could be in three layers. The bottom layer con-
tains smart devices with the wireless communication function. The middle
layer contains gateways and each gateway acts as a bridge between a group
of smart devices and the top cloud layer. Finally, the central and remote
monitoring is realized through cloud services.
Two demo systems concretizing the proposed architecture were designed
and implemented in [32] and [84]. Figure 2.1 presents the remote pain moni-
toring idea, where the biosignals are transmitted through an IoT network and
can be accessible to caregivers at a distance after being interpreted. On the
basis of the three-layer IoT architecture especially, a mobile web application
that can run inside the web browser of most operating systems was devel-
22
oped in [84] for caregiver end-users. In the developed web application, the
biosignal data stream can be synchronized from the cloud server, processed
and presented as signal waveforms.
Figure 2.1: Remote pain monitoring in hospital for inpatients and ICU pa-
tients c
2018 IEEE
The methods, results, and discussions to be presented in the following
chapters are about how accurate and reliable the automatic pain assessment
could be from the point of view of the signal quality, processing algorithms,
and the inherent data pattern. These aspects are considered to define the
boundaries of the application’s feasibility. Furthermore, merging the appli-
cation into IoT systems to realize real-time monitoring and assist decision-
making in pain management is also worthy of exploration. On the one hand,
IoT systems can provide flexible processing capabilities from the resource-
constrained bottom sensor layer to the less resource-constrained top cloud
computing layer that can support deep learning streaming [85]. On the other
hand, challenges pertaining to technology as well as regarding human factors
are to be solved before the concepts could turn into reality [86].
2.5 Summary
This chapter presents the rationale for research on automatic pain assess-
ment tool development using machine learning methods, mainly supervised
learning methods. The review covers pain indexes as scales, checklists, and
automatic measures in a broad scope. The following presents the signal se-
lection process from the angle of wearable devices and IoT-enabled systems.
In the end, the vision of remote pain monitoring within an IoT architecture
was introduced, where using wearable devices is one of the original motiva-
tions when this study was initialized. The remote pain monitoring could be
an application in the future.
23
24
Chapter 3
Biosignals Acquisition
In this study, a set of the key signs of pain are observed from healthy vol-
unteers. Among the signs introduced in Section 2.3, facial sEMG, ECG,
respiratory cycle and GSR are included in the data collection and analysis.
In this chapter, the chosen or developed biosignal acquisition systems are
presented and discussed.
3.1 Facial surface electromyography
Facial sEMG is an alternative approach to face images or videos in read-
ing facial expressions. Compared to face images or videos, it conceals facial
identity so that privacy is protected. However, on the other hand, the de-
velopment of its technologies and applications is not as fast as the video
approach at this stage due to its comparatively more intrusiveness and the
resulting small database. Another challenge in generalizing facial sEMG
data is the difference in the different electrode placements covering different
muscle areas.
Most of the sEMG studies follow Fridlund and Cacioppo’s guidelines
[19] on electrode placements where electrode pair positions for ten muscle
areas on one side of the face are recommended. The guidelines also recom-
mended miniature electrodes for facial sEMG sites with 0.25 cm diameter
silver (Ag)/silver chloride (AgCl) detection surfaces and 1 cm inter-electrode
spacing, because facial muscles are small muscles compared to the big trunk
muscles. However, the most commonly used disposable pre-gelled surface
electrode H124SG has 1 cm diameter detection surface and 2.4 cm total di-
ameter with solid gel and adhesive collar. Smaller electrodes exist, such as
0.1 cm EEG cup electrodes, however, they are expensive to dispose off and
need extra conductive paste and tape to fix in place. In the measurement,
monopolar electrode configuration is adopted instead of bipolar where in
each channel on electrode is placed on the target muscle. It is to cover more
muscular areas with fewer electrodes.
25
The multi-channel sEMG measurement, Table 2.3, shows that at least
five facial muscles contribute to pain facial expressions, these are: corrugator
supercilli ,orbicularis oculi,levator labii superioris,zygomatic major and
risorius. However, existing commercial wearable or wireless sEMG devices
each can only measure up to two channels (CHs) (e.g., Shimmer3 EMG unit,
2-CH, @512 samples per second (SPS) per channel; Delsys Trigno TMWireless
EMG sensor, 1-CH, @2000 SPS per channel; BIONOMADIX-EMG, 2-CH,
@1000 SPS per channel). Therefore, in this study, we also sought to find a
compact solution to multi-channel sEMG measurement. On the basis of the
device, a wearable facial sEMG sensor can enhance the device’s easiness-to-
use.
In summary, the aims of the facial sEMG acquisition system design were:
To have one electrode per muscle area to reduce measurement intru-
siveness;
To develop a compact multi-channel sEMG measurement wireless de-
vice with wearable solutions;
To provide a continuous recording of at least 5-CH sEMG for at least
2 hours.
The designed sEMG acquisition system and its test results are presented in
the following parts.
3.1.1 The core of the acquisition system
The developed facial sEMG system was built on a Texas Instruments low-
noise analog-to-digital converter (ADC) for biopotential measurements, ADS
1299[87]. ADS 1299 is 24-bit ADC with a flexible configuration on analog
amplifier gain (up to 12), sampling rate (250 - 16k SPS), analog inputs and
outputs. However, this flexibility complicates its configuration of analog in-
puts and outputs on the other hand. In addition, some terms (e.g., reference
or reference electrode) may refer to different settings in different contexts,
and some different terms refer to the same setting. Therefore, the analog
configuration of ADS1299 and its comparison with other sensors or systems
are introduced below.
Monopolar electrode configuration, single-ended measurement, or
referential montage
In a biopotential measurement, especially multi-channel measurement, the
electrical potential on each site can be measured by the reference of the same
measurement site or an individual paired measurement site. The three terms
in this part all describe the first setting of the two mentioned. Monopolar is
used to describe a electrode configuration, where a single electrode is placed
on the skin above the muscle [88]. Single-ended is used to describe the analog
26
inputs to a differential amplifier where the voltage between the input signal
and the ground signal is measured. Referential montage is mainly used in the
context of EEG measurement. This mode reduces the number of connectors
used because only one physical connection is required to one input. How-
ever, the common point should be carefully chosen to avoid measurement
errors. Meanwhile, the signals captured in this mode could be more suscep-
tible to external noise as the external influence coupled to the two separate
wires could be non-identical and thus cannot be canceled at a differential
amplification [88, 89].
Bipolar electrode configuration, differential measurement, or se-
quential montage
The terms in this part respectively correspond to the three in the former
part. In sEMG measurement, the terms mean the voltage is measured from
the electrode pair on the skin above the muscle. Differential measurement
is more commonly employed compared to single-ended measurement due
to a better spatial resolution and a better signal-to-noise ratio (SNR) in
the measurement by suppressing the common mode noise. However, even
the high common-mode rejection ratio (CMRR) of a differential amplifier
entirely suppress power line noise due to the impedance mismatches in a
pair of electrodes and lead wires. Such mismatches convert common-mode
noise voltage into differential interference in the measurement, and thus more
solutions occur [1].
Ground, reference, bias drive and right leg drive
These terms are about the methods to reduce the captured interference in
recording small biopotential signals; therefore some interference mechanisms
caused by the electric field are introduced first.
As the model in Figure 3.1 shows, when the skin-electrode impedance
Aand Bis denoted as Zea and Zeb and the currents induced in the wires
are iaand ib, the differential interference voltage caused by the interference
currents in to the measurement cables is Vab_cable =iaZea ibZeb, which
could be 120µV (when iaib=6nA with 9 m cables, Zea Zeb = 20
k). The interference Vab_cable can be minimized by shielding the leads and
grounding each shield [2].
Another differential interference caused by the electric field is because of
the common-mode voltage on the body generated by the current flows from
the power line through the body and ground impedance. This common-
mode voltage Vcm equals to iiZGin Figure 3.1 and could be 10mV (0.2µV
×50k) and even larger [2], and Vab_body =Vcm(Zin
Zin+Zea Zin
Zin+Zeb ) =
Vcm(Zeb Zea
Zin )because Zeis much less than input impedance Zin. A typical
27
Figure 3.1: Mechanisms of difference interference caused by the electric field
in ECG measurement [1, 2]
value of Vab_body is 40 µV. Therefore, to depress power line interference,
lowering input imbalance and raising input impedance are two critical factors
because the common-mode voltage is always present.
It is also possible to diminish Vcm by adding a driven right leg circuit
instead of grounding the patient in ECG measurement, as shown in Figure
3.2. The negative feedback provided by the driven right leg circuit drives
the common-mode voltage to a low value, in which way the interference is
reduced [1, 2, 90]. Meanwhile, it forces the patient’s common voltage close
to a dc voltage level (tied to the ground in Figure 3.2) so as to maximize the
input dynamic range.
Figure 3.2: Driven right leg circuit to decrease common-mode voltage
Areference lead usually appears in the analog sensing part of a biopo-
tential acquisition system. In the monopolar mode, the reference lead is
connected to the negative inputs of all the channels so that the biopoten-
28
tial of each channel is measured as the difference between the positive input
electrode and the reference electrode. Reference could also have a different
meaning, especially in a bipolar system, where the potential at each electrode
in an electrode pair is first detected with respect to the reference electrode
[91, 92]. In this way, the same alternating current (AC) and direct current
(DC) noises are subtracted before the differential amplification in the next
stage so as to improve CMRR. Such reference is also referred to as ground
in the systems. In addition to these, reference could mean the right leg drive
in Figure 3.2, which is called bias drive in a system based on ADS1299 (e.g.,
Shimmer EMG [93]).
Active electrode and preamplifier
The active electrode is used to distinguish from the passive electrode. A
passive electrode is the metal discs connected to the amplification circuit
through a long and unshielded lead wire. The impedance of a differential
amplifier is designed to be high in order to reach a high ratio of amplifier
input impedance to electrode impedance in capturing the small biopotential.
The capturing sensitivity, on the other hand, makes the unshielded lead wires
susceptible to power line noise and any movement artifact (skin-electrode
movement and cable movements). One solution to this problem is to place
the differential amplifier to the electrode as close as possible [94], which is
called the active electrode especially when the biopotential is amplified locally
on the electrode site. As the output impedance of the amplifier could be as
low as 10, the noises have a much smaller influence on the subsequential
lead wire. Most of the active electrodes are reusable (e.g., TrignoTMsensors
and g.GAMMAsys). In the cases where disposable electrodes are used, the
term preamplifier is used instead of the active electrode (e.g., in Motion Lab
Systems and ME6000 biomonitor).
The analog configuration of ADS1299
ADS1299 was designed for biopotential acquisition, especially for EEG appli-
cations. Its working mode can be switched between monopolar and bipolar
electrode configuration by changing its configuration in firmware. The elec-
trode connection should also be changed accordingly when switching the
analog input mode. Driven right leg circuit is integrated on the chip and
the bias drive can be taken into use or kept separate from the amplification
circuit in firmware configuration. Part of the ADS1299 configurations are
presented in Figure 3.3, including lead-off detection, common reference, and
bias drive.
The analog configurations of ADS1299 in different modes are presented
here as examples of ECG lead I measurement (the potential between left
29
Figure 3.3: Part of the ADS1299 configurations including lead-off detection,
common reference, and bias drive
arm, LA and right arm, RA). To compare the power line interference levels
in different working modes, the ECG signals (sampled at 500 SPS from CH1)
after detrending are analyzed in the frequency domain. The data acquisition
system used in the test is the improved version of our prototype presented in
[35]. The design of the system in use will be introduced in detail in the next
part. Unshielded 1 meter long lead wires and H124SG electrodes are used
in the test. The lead wire configuration and the key register configuration
in the four working modes are listed in Table 3.1. The input REF is the
common reference buffered from SRB1 pin of ADS1299. The SRB1 pin
can be configured to connect the negative input of all channels through
register MISC1. The driven bias circuit on-chip can be powered on or off
by changing the register CONFIG3. The inputs to the drive bias circuit
can be configured through the registers BIAS_SENSP and BIAS_SENSN
and BIAS_SENSP/N, 0x01 means both the positive and negative inputs of
channel one are connected to the drive bias circuit.
As the test took place in a normal office room having many wall-powered
computers, a considerable amount of power line noise can be observed in
the time domain waveform in any mode. The frequency analysis of the
30
Table 3.1: The lead wire configuration and key registers configuration in the
four modes
Mode LA RA RL Key registers
single-ended, CH1+ REF open CONFIG3, 0xF0
bias drive powered-off MISC1,0x20
single-ended, CH1+ REF BIAS CONFIG3, 0xFF
bias drive connected MISC1,0x20
BIAS_SENSP/N,0x01
differential, CH1+ CH1- open CONFIG3, 0xF0
bias drive powered-off
differential, CH1+ CH1- BIAS CONFIG3, 0xFF
bias drive connected BIAS_SENSP/N,0x01
captured 10-second ECG signals in Figure 3.4 shows that the differential
mode has superior performance than single-ended mode without bias drive
(93dB@50Hz versus 101dB@50Hz). But when bias drive is connected, the
influence of power line decreases in both modes, and the improvement is more
remarkable in single-ended mode (from 101dB@50Hz to 76dB@50Hz). The
overall frequency performance improvement in the whole bandwidth in the
single-ended mode with bias may indicate the necessity of using an analog
buffer at the analog input end.
Figure 3.4: Comparison of power line interference in ADS1299’s different
working modes
31
3.1.2 sEMG/Biopotential acquisition system design
The designed sEMG acquisition system includes two main parts, 1) the de-
vice, i.e., the hardware which captures, pre-conditions, digitizes, and wire-
lessly transmits the digitized signals; 2) the software, which receives the
digitized signals, applies digital signal processing, displays the signals in
waveform for checking in real-time, and saves the recorded data in data files.
In addition to the signal flow in the system, the software can also change the
ADS1299 setting (e.g., the sampling rate and the input mode) by sending
defined commands.
Figure 3.5: The block diagram of the designed sEMG acquisition device
Hardware
The device introduced here is an updated version of our work in [35]. In
this version, the power supply part is optimized by adding charging circuits
and improving the power supply efficiency. The functional block diagram of
the designed device is presented in Figure 3.5. The analog part and digital
part of ADS1299 are powered separately by 5 V and 3.3 V and both of them
are generated from a 3.7 V lithium battery. The ADS1299 is controlled
by an AVR-based 8-bit micro-controller, which meanwhile controls an HC-
05 Bluetooth module. Regarding the analog part, there are in total 18 lead
wire connectors including 8-channel positive and negative input pairs (INxP,
INxN), a buffered common reference (REF), and a bias drive output (BIAS).
The REF is analog buffered to reduce the leakage current in the single-ended
mode where the currents from all the channels are added at REF [90].
The device is 56 mm ×40 mm ×10 mm in size. When powered at 4.2 V,
the working current of the device was on average 58 mA when sending data
32
continuously (@500 SPS) and was 30 mA when Bluetooth was connected
and idle. In addition to the collected biopotential data at the rate of 96
kbit/s, two-byte data flags were inserted as either an indicator of the start of
one 8-channel sample (sample_flag) or the start of the first sample in each
second (start_flag). The data flags are used to verify the correctness of data
transmission in the software. Including the data flags, the total data rate in
the transmission is 10.4016 kbit/s. The sampling rate of the system can be
configured to be 250 SPS and 500 SPS and this functions well, however, the
data transmission can be unstable when the sampling rate is raised to 1000
SPS.
Software
The software in the system is developed with LabVIEV in dataflow and
graphical programming. As continuous biopotential data (data_recording)
receiving, presenting and recording are the core functions of the software,
the classic producer-consumer design pattern is employed in realizing the
core functions. In the producer loop, the data at the serial port is taken in
continuously and added to a queue. In parallel, the elements in the queue are
read out in the consumer loop to be further interpreted, filtered and written
into a signal file.
Next, in order to achieve a flexible control of the start and stop record-
ing, a four-state machine structure is added on top of the producer-consumer
loops, which is the data acquisition state machine in the software workflow
(Figure 3.6). As shown in the software workflow, there are four states (S0-
S3) in the state machine. The state transitions are triggered by three click
buttons on the front panel, Ctl_1-read continuously, Ctl_2-quit the pro-
gram, and Ctl_3-stop reading continuously. The start state (S0) is the
default state waiting for the action of reading continuously or quit the
program. In this state, the estimated value of the battery output volt-
age (data_batteryVoltage) can be acquired by sending a request command
(cmd_batteryVoltage). By clicking Ctl_1, a start command (cmd_ record-
Start) is sent to the device and the device starts sending, meanwhile, the
current computer time is captured and saved as the start time of the record-
ing session. Similarly, when the stop reading continuously action is taken
from the control panel, the state transits from S1 to S2. In S2, a stop read-
ing command (cmd_recordEnd) is sent to the device, the current time is
marked as the end time of the recording session and it is saved at the end of
the signal file together with the session start time.
Outside the data acquisition state machine, as shown in Figure 3.6, is
a sequence structure where the state machine is one frame in it. Before
entering the state machine, the device can be configured in terms of the
sampling rate and the input mode according to the preset parameters in the
33
software. After sending the corresponding commands to the device, the data
acquisition state machine can be entered only after a successful handshake
(receiving data_handshake).
Figure 3.6: Software workflow
The communication between hardware and software
As described above, the software in the system sends commands to the de-
vice so that the device can respond by either changing configurations or
sending the requested data. The firmware structure is shown in Figure
3.7. After initializing the sub-modules on the device and enabling global
34
interruptions, it enters the main loop of the firmware, which contains a
three-state machine, Bluetooth connection check state (BT_CON_CHK),
ADS1299 configuration state (ADS_CONFIG) and biopotential data send-
ing state (ADS_DATA). The full list of input commands and output data
are listed in Figure 3.7 as well as in Table 3.2. The state transitions be-
tween BT_CON_CHK and two other states are based on the values of two
Boolean flags (flag_recordEnable and flag_initialization). Table 3.2 shows
how the flags are assigned in response to each command and what actions
are taken inside the corresponding state.
Figure 3.7: The firmware structure of the AVR processor and its communi-
cation with the software
In addition to the signal file that is generated in one recording sessions,
two other data files are generated as well within the state machine, which
are a time point file and a time stamp file. The processes of data saving
are presented in Figure 3.8. The time point file is a record of time labels
given to the signal in the second resolution. As mentioned above, a 2-byte
sample_flag is added in the streaming data to denote the start of an 8-
channel sample. In the S1 state in the software, this flag is recognized to
cut the sample from the continuous data. Meanwhile, the number of the
35
samples is counted from 0 when second_flag is received; this is to check the
reliability of the received data in case of any data loss. The time at which the
second_flag is received and recognized, together with the final count number
before being reset, is also saved in the initialized time point file. The time
stamp file is independent of the signals. It is to mark the external events
during a recording session. Time marks and their numbers in sequence are
added when clicking the add time mark button on the front panel of the
software.
Table 3.2: Firmware’s responses to the software commands
cmd_ flag_ flag_ State in Action
initialization recordEnable Figure 3.7
Initial state False False BT_CON_CHK Differential,
SampleRate500
batteryVoltage - - BT_CON_CHK data_batteryVoltage
configComplete True - ADS_CONFIG
report_config,
data_handshake,
flag_initializationF
inputMode - - BT_CON_CHK Differential/
Single-ended
recordStart - True ADS_DATA data_recording
recordEnd - False BT_CON_CHK exit ADS_DATA
sampleRate - - BT_CON_CHK SampleRate250/500
3.1.3 Discussion
Signal quality
The quality of the acquired signal is among the main concerns in a biopoten-
tial acquisition system. First, half of the sampling rate should cover the main
frequency range of the sampled signal due to the Nyquist theorem. Although
the biopotentials share the same mechanism, their frequency and amplitude
characteristics have some differences, as shown in Table 3.3. Generally, EMG
needs a higher sampling frequency than the other two and the optimal set-
ting is 1000 Hz. However, a sampling rate of 500 Hz in this system is high
enough to cover the dominant energy range of EMG between 50 Hz and 150
Hz, and is sufficient for collecting EEG and ECG as well. The communica-
tion baud rate is set to be 921.6 kbps (ATmega fosc=14.7456 MHz). It is in
theory high enough to handle full channel data at 1000 SPS. However, the
upper limit of the sampling rate in the system is restrained by the wireless
data transmission provided by the Bluetooth module HC-05. In the system,
as each signal channel can be individually configured, the unused channels
can be shut down and excluded in the data transmission to reach a higher
sampling rate of the data received in the software.
36
Figure 3.8: File operations in the software
Table 3.3: Frequency and amplitude characteristics of the EEG, ECG and
EMG
Biopotential Amplitude (µV) Frequency (Hz) Dominant energy range /
[95] [95] Suggest sampling rate (Hz)
EEG 1 - 10 0.5 - 40 - / 128-1024 [91]
ECG 103-5×1030.05 - 100 0-50 / 100 [96]
EMG 103-10420 - 2000 50-150 / 1000 [94]
The captured interference in the acquired signal should be low so as
to reflect enough biopotential details for analysis, which means a high SNR.
Although the subsequent digital signal processing afterward can help improve
SNR, the complexity of digital signal processing increases when the raw
signals are too poor in quality and it is not a panacea for all the cases with
signal distortion. For example, part of the raw and digital filtered ECG
signals described in Section 3.1.1 are plotted in Figure 3.9. These signals
were not collected under optimal conditions, and skin preparation was not
preformed nor was the recording isolated from line interference. The signal
with the lowest noise (Figure 3.9 (b)) shows the clearest ECG waves after
filtering when same digital filters (2nd Butterworth 0.5-90Hz bandpass and
2nd Butterworth 50Hz notch) are applied to the four signals. However, the
same processing is not sufficient for the raw signal in Figure 3.9 (a), where
the whole frequency range of the signal is contaminated by noise as well.
37
Figure 3.9: A piece of ECG signal collected in Section 3.1.1
Figure 3.10: The envelopes of the sEMG signals acquired by the system
developed in this study (UTU-BASD) and by ME6000 Biomonitor
This system (UTU-BASD) was compared to the ME6000 Biomonitor sys-
tem in terms of signal quality. The sEMG signals of the two systems were
collected from the symmetrical sites of the face on corrugator and zygomati-
cus muscle areas. ME6000 Biomonitor has a pre-amplifier near the electrodes
and a bandpass filter on the device. The data were sampled at 1000 SPS.
UTU-BASD was set as differential mode, and the bias drive was placed on
38
the bony area behind the left ear. The sampling rate of UTU-BASD was
500 SPS and the digital signal processing included a 20 Hz highpass filter
and several notch filters. The envelope of each sEMG signal from the two
systems is presented in Figure 3.10. In addition to some difference in the
envelope shape between the two systems, the zygomaticus sEMG signal in
UTU-BASD system shows a higher level of baseline noise. The baseline is
the recorded electrical noise when the muscle is not contracting. It could
include powerline interference and reflect the stability of the skin-electrode
interface. The high baseline noise would hinder the reliable separation of
sEMG data from noise. The acceptable baseline noise is concluded to be
smaller than 15 µV in Delsys’ documentation [97].
To obtain high-quality sEMG signals and rely less on the use of digital
filters, one improvement that can be done in the system is to use active
electrodes or add preamplifiers in the analog part of the device, as introduced
in Section 3.1.1. In addition, proper preparation work can help ensure the
sEMG quality. According to Delsys’ recommendations, the location of the
electrode pair is suggested to be aligned along the muscle fibers directions
and to be far from tendon origins and innervation zones. This aims to
maximize the signal amplitude in improving SNR. The other approaches
are to minimize the noise amplitude. Proper skin preparation is suggested
where the skin surface is wiped with an alcohol swab to remove oil and
debris. The firm attachment of electrodes can help avoid the noise caused
by the movement between the electrode and the skin. Finally, keeping the
environment clear from power line devices is suggested if the signal contains
excessive power line interference.
Further hardware improvements
ADS1299 is equipped with a lead-off detection function, which is to verify a
proper the electrode-skin and connection. In this function, a fixed current
is injected between a channel input and the power supply. The voltage at
the input is monitored to check the lead connection. The lead-off detection
function can provide a debugging reference when checking the signal quality
especially in a long-term recording where the patient electrode impedance
decay over time. However, the validity of the function itself needs to be con-
firmed by choosing the proper setting among multiple configurable options.
Additionally, its influence on the target signals needs to be evaluated.
Possibilities of being wearable
The signals collected by UTU-BASD were measured with off-the-shelf elec-
trodes and lead wires. The data collection device is compact for portable
usage, nevertheless the whole system is not an integrated wearable solution.
39
Further development on the wearable solution may take the using scenario
into consideration in the design (e.g., the motion extent and frequency the
wearable device should tolerate). The material to be attached to the skin
may also need to be tested in terms of effectiveness and safety. Some ref-
erential wearable solutions are clear facial mask with embedded H124SG
electrodes [84] and the adhesive electrode sensor strip used in bispectral in-
dex measurement [98]. The wearable solutions will also help improve signal
quality when the signals are amplified close to the measurement sites and
movement artifact is unlikely to happen.
3.2 Electrocardiography and respiration
Pulse rate and respiration rate are both among the vital signs in medical
monitoring. Their equipment and techniques for their continuous monitor-
ing are relatively mature. In addition to the advanced patient monitoring
systems in the hospital, there are wearable solutions as well. Regarding res-
piration, it can also be derived from the waveform of other measurements
due to the interactions between the heart and lungs in the body system.
3.2.1 Wearable device - Bioharness 3
The wearable device Bioharness 3 consists of a washable chest strap with a
monitoring device (BioModule) attached to it. Bioharness 3 integrates sev-
eral sensors including 1) two electrode sensors housed within the chest strap
to measure ECG, 2) a capacitive pressure sensor detecting circumference
expansion and contraction of the torso to measure respiratory cycles, 3) a
3-axis accelerometer to measure activity and 4) an inclinometer to measure
posture [81].
Bioharness 3 has software that can check the measurements in real-time.
Meanwhile, BioModule continuously logs the collected data in the chosen
format on its memory. By choosing the General and ECG logging format,
breathing waveform at 18 Hz and ECG waveform at 250 Hz together with
other extracted parameters are saved on the device memory for up to 140
hours due to its memory capacity. Another logging format, Summary and
Waveform, can record the raw data of accelerations at 100 Hz, breathing
waveform at 25 Hz and ECG waveform at 250 Hz. In this logging format,
the memory can hold up to 55 hours of recordings.
As a wearable device in a system that could be used during exercise and
sports, the validity and reliability of the first version Bioharness device were
tested in both a laboratory environment [81, 82] and a field-based environ-
ment [99]. The device showed good data precision and reproducibility at a
low velocity in sports (4-6 km/h) in both scenarios. Comparatively, the pre-
cision of heart rate and breathing frequency decreased at a higher velocity
40
in the field based environment. In this pain assessment study, Bioharness 3
should be adequate when collecting ECG and respiratory cycles data because
the users are basically in a relatively static state.
3.2.2 Patient monitor
A patient monitor is usually used at a patient’s bedside to provide continuous
monitoring of multiple vital signs and alarms during the monitoring. It is
a precise and reliable data source to collect raw physiological signals in the
hospital for analysis. Usually, the bedside monitor has a screen showing the
waveforms, parameters, and alarms in real-time. In addition to the one fixed
in one position with a big screen, there are portable solutions as well in the
monitoring system supporting a patient being transferred from one place to
another without interruption in the monitoring. For example, the portable
Philips IntelliVue MMS X2 can be connected to Philips IntelliVue MX800
which has an integrated computer.
Regarding respiration, there are multiple approaches to monitor it. The
direct approaches measure air flow into and out of the lungs with devices
such as spirometers and nasal thermocouples. Meanwhile, there are ap-
proaches measuring respiration indirectly and the capacitive pressure sensor
that Bioharness uses is among them. Taking IntelliVue Patient Monitor as
an example, the respiration rate can be calculated from the carbon dioxide
waveform, which is the airway respiration rate. Meanwhile, respiratory cycles
can be monitored by measuring the thoracic impedance between two ECG
electrodes, which are RA and LL in standard ECG electrode placement.
Open source solution acquiring raw data for research from a patient mon-
itor are scarce, but openICE is one. The openICE project (open-source In-
tegrated Clinical Environment) 1is actually aimed to provide a platform
networking medical devices by enabling the interoperability in the platform
[100]. With a Java-based computer running an openICE Device Adapter
from a serial to the Ethernet, a series of medical devices can be visualized
and recorded in a computer running an openICE Supervisor. If reading from
a single device, direct serial communication could be directly built between
a device and a computer where the Device Adapter and Supervisor are on
the same host computer.2
3.2.3 Respiration derived from other measurements
The indirect respiration monitoring approaches are mostly based on the phe-
nomenon that short-term changes in thoracic impedance reflect the filling
and emptying of the lungs. Impedance pneumography is such an approach.
1https://www.openice.info/index.html
2https://www.openice.info/docs/6_example-setups.html
41
Texas Instruments integrates respiration impedance measurement in some
ECG measurement ADCs (ADS129xR), where respiratory cycles can be mea-
sured between RA and LA by injecting a high-frequency ac current [101].
However, RA and LA cannot be used to measure ECG when they are con-
nected to the internal modulation clock output for respiration measurement
[102]. In contrast, respiration can be alternatively derived from ECG, ABP,
and PPG waveforms without an external current excitation by applying sig-
nal processing techniques.
Figure 3.11: The respiratory modulation of the waveform amplitude
The respiration is reflected in ECG as a modulation of QRS amplitude
[103]. It is explained as the modulation in the direction of the mean cardiac
electrical axis in the dipole model of cardiac electrical activity. Since the
heart rate is usually much higher than twice the respiration rate, the respi-
ration rate is measurable from the time series derived from the area under
the QRS complexes from Nyquist sampling theorem point of view. In the
ECG waveform, the fluctuation of the baseline could also reflect respiration
due to the motion of the chest, however, it usually only seen in deep or ex-
aggerated breathing [104]. Similar peak amplitude modulation is also found
in ABP and PPG waveforms. The respiratory changes in ventricular stroke
volumes and arterial pressure is induced by a series of cardiopulmonary in-
teractions [105]. Figure 3.11 shows the synchronously measured ECG, PPG
and ABP without any movement artifact. The respiratory modulation of
the waveform amplitude can be observed from the waveform envelope, and
the respiration rate can be then estiimated.
42
3.3 Galvanic skin response
The measurement of GSR commonly uses exosomatic direct current or volt-
age recording. Figure 3.12(a) shows a GSR measurement circuit struc-
ture with an operational amplifier for exosomatic direct voltage recording
(eHealth v2.0 platform). The given constant voltage is 0 V, and the bias
voltage to the positive input of the amplifier is 0.5 V which is divided from
Vref . A capacitor is connected in parallel with the feedback resistor and acts
as a low-pass filter (cutoff frequency fc= 15.9 Hz). The skin conductance
(1/Rskin) can be calculated according to Equation 3.1. One problem with
this circuit is the different amplification requirements SCRs and SCL, where
SCRs fluctuate in a small amplitude difference while stacking on SCL. SCL
is in a wide range due to inter- and intra individual differences [106]. An-
other problem with the eHealth v2.0 platform when working with Arduino
Uno is the skin conductance resolution, which is 0.098 µS (where the ADC
resolution, Vout =5V
1023 ) and the SCRs fluctuations smaller than that could
not be captured. To cope with the first issue, Poh M. et al [107] proposed a
solution by adding an integrator circuit (shown in Figure 3.12(b)) so that the
bias voltage to the positive input of the amplifier (Vb) can change along with
amplifier output (Vo). In this solution, the skin conductance is calculated
from Equation 3.2.
(a) eHealth v2.0 (b) Poh M. et al [107]
Figure 3.12: Two GSR exosomatic direct voltage recording circuits
0.5V
Rskin
=Vout 0.5V
100k(3.1)
43
Vcc Vb
Rskin
=VoVb
1M(3.2)
In terms of GSR measurement sites, the middle phalanges of the index
and middle fingers for bipolar recording are commonly used. As palmar
sites showed distinguished electrodermal activity [108], other similar sites
are acceptable if electrodes can be fixed easily. In addition to the palmar
sites, the distal forearm sites were also proved to be a viable alternative and
have been implemented in the E4 wristband [80].
3.4 Summary
This chapter presents the mechanisms of electronics devices and systems,
for biopotentials recording mainly. Further improvement possibilities are
discussed
44
Chapter 4
Study Design and the SpaExp
Database
Healthy volunteers are commonly involved in the studies to develop an au-
tomatic pain intensity indicator or an equivalent tool. Experimental pain
models provide the possibility to explore the pain system under controlled
settings. The experimental stimuli could be mechanical, thermal, electrical
or chemical [109]. The stimuli can be applied to skin, muscles, and viscera
for assessing different pain pathways and pain mechanisms. In this study,
two experimental pain modules, thermal and electrical, were employed. The
signals recorded in the database included some ANS-based signals (ECG,
respiration, GSR) and one behavioral signal, facial expressions using facial
sEMG. The analysis was done in a supervised learning fashion, where the
samples were firstly labeled before the pattern of the labels were learned by
machine learning algorithms. Thirty-one volunteers were recruited by the
co-workers in the Department of Nursing Science at the University of Turku.
The study was initially designed by Prof. Sanna Salanterä, Prof. Riku Aan-
taa, Prof. Pasi Liljeberg, and Adjunct Professor Amir M. Rahmani. The
previous study results were reported in [33], [34] and [110].
4.1 Experiment protocol
The study subjects recruited to the study were working-age people in a gen-
erally healthy condition without any chronic, acute somatic or mental illness.
In addition, no regular medication was taken during or in the two weeks prior
to the data collection. All the study subjects’ cardiovascular parameters were
within normal range. They had normal sensations and healthy skin on the
face and upper extremities. Pregnant women and individuals with a body
mass index larger than 30 kg/m2 were excluded from the study. This study
was approved by the Ethics Committee of the Hospital District of Southwest
45
Finland (ETMK:83/1801/2015).
The data collection took place in a quiet room where powerline interfer-
ence was avoided as much as possible. One technician and one research nurse
were also in the room to set the signal acquisition system and instruct the
study subject in the Finnish language, respectively. The research nurse first
briefly introduced the processes of the study to the study subject, then the
study subject read and signed an informed consent. In the next preparation
phase, the study subject put on the Bioharness 3 before sitting on an arm-
chair with a footstool. Thr Bioharness 3 was turned on for recording before
being put on. After placing the pre-gelled surface electrodes on the left side
of the study subject’s face, the sEMG recording started.
It was planned that each study subject took four tests, in each of which
the pain stimulation increased gradually in intensity. Two experimental pain
models were employed in the study. Each type of stimulus was applied once
to each symmetrical side of the body. The relative order of the four pain
stimulations was defined beforehand in a side-switching manner (i.e., 1-right
heat, 2-left heat, 3-right electrical, 4-left electrical). However, the order of
the four tests was randomized by choosing the pain stimulation to start
with. Before the pain stimulation was put on each time, enough rest time
was ensured for each study subject to return to a relatively steady heart rate
level. This was to check whether the heart rate at rest was in the normal
range at the very beginning and to ensure enough recovery from the previous
excitation. The signal recordings ended after the end of the last test.
4.2 Study design
Each of the four tests was designed as presented in Figure 4.1. The pain
stimulation at intensity 0 is attached to the skin surface before the pain
stimulus start time t0. From t0, the intensity of the pain stimulus increases
almost linearly until the study subject reports the pain as "intolerable" which
is denoted as t2on the timeline. The pain stimulation is either stopped or
detached from the skin surface after t2, and the intensity at this point (i2)
is marked down. Immediately after t2, the study subject was asked for a
the self-report VAS score. To protect the study subject from tissue damages
such as skin burn, the upper limit of the stimulus intensity was set. In cases
where the maximum intensity was reached without an "intolerable" report,
the pain stimulation was ceased and the maximum intensity is recorded as i2.
Between t0and t2, the study subject was also instructed to indicate the time
t1when the sensation turned into "uncomfortable." The self-report time is
defined by the study subject when pressing a button that triggered a beeper.
The description "uncomfortable" is interpreted as intensity 3 or 4 in
NRS/VAS from a clinical nursing point of view which represents a clinical
46
importance in pain severity indicating patients’ perception of adequate pain
control [12, 13]. The self-reported t1was instructed as "start to perceive it
as pain", and therefore t1and t2could respectively also be interpreted as
personalized pain threshold and pain tolerance.
Figure 4.1: The timeline and pain stimulus intensity in one test
4.2.1 Experimental pain stimulation
Acute pain is generated when a stimulus is presented to the body. The
perception of pain, called nociception, depends on specifically dedicated re-
ceptors and pathways. The nerve cell endings that initiate the sensation of
pain are called nociceptors, and they have two types of axons, Aδfibers and
C fibers. The two categories of pain perception are described as a sharp
first pain and a more delayed, diffuse and longer-lasting second pain [111].
The tingling sensation or a sharp first pain is caused by the activation of Aδ
fibers, while the longer-lasting second pain is sensed via the slowly conduct-
ing C fiber axons. Two experimental pain models in the skin were employed
in this study to cover more than one pain sensation. It is easier to access and
activate the nociceptors in the skin compared to muscle, bone, and viscera.
The details of the two pain stimulation settings are described below.
Electrical stimulation of the skin
The advantage of electrical stimulation is that it is easy to control the stim-
ulation device output with a variety of patterns in terms of waveform, fre-
47
quency, intensity, and duration. The population and activation of the nerve
fibers depend on the stimulus intensity where C fibers have a higher acti-
vation threshold than Aδfibers [109]. A muscle stimulation device, Sanitas
SEM43-Digital EMS/TENS was used as the electrical pain stimulation device
in this study. The pair of electrode pads were resized and placed vertically
on the fingertip of the ring finger. The electrodes were fixed on the positions
with medical tape to prevent them from falling off.
The Sanitas device outputs biphasic rectangular pulses with configurable
pulse width, pulse frequency, and pulse intensity. The maximum output
current of the device is 200 mA peak-to-peak at 500, and the intensity is
adjustable in a scale between 0 and 50. Before the start of the data collection,
different settings on the pulse width and frequency were evaluated within the
research group regarding the induced pain. The final chosen pulse width and
frequency were 250 µs and 100 Hz because the pulses in this combination
could evoke noticeable pain sensations. In the test, the pulse intensity was
set manually by pressing the "intensity up" button on the device in every 3
seconds. The waveform of the electrical pulses is visualized in Figure 4.2.
Figure 4.2: The waveform of the electrical stimulation
Thermal stimulation of the skin
Thermal stimulation includes several approaches in practice, which are cold,
contact heat and laser. The stimulation used in this study was contact heat
where a heating thermode evoked the heat pain. Rapid heating activates
first Aδfibers within less than 0.5 s, which is followed by a C-fiber-mediated
second pain, while slow heating less than 1C/s gives a preferential activation
of C fibers [49]. The slowing heating that activates C fibers was the second
48
experimental pain stimulation model used in the study. Long exposure to
hot tap water may cause scald injury. It is a synthetic effect of temperature
and exposure time, and the relationships of both of these to a scald injury
are presented in Figure 4.3(a) and (b) [112]. For example, it is concluded
that 55C would lead to second-degree burn in 17 s and third-degree burn
in 30 s.
Figure 4.3: The potential scald injury caused by heat and the temperature
curve of the heat pain stimulation in the test
The thermode system was developed in the laboratory. The manufac-
tured thermode has a round contact surface of metal with a diameter of 3
cm. The heating speed of the thermode was regulated through setting the
switching frequency of the solid state relay between the heating element and
its power supply. A digital temperature sensor was used in the thermode for
temperature monitoring in real time. The position of the temperature sensor
and the contact surface were vertically symmetric to the heating element in
order to estimate the temperature of the contact surface. The temperature
curve of the thermode in one test is plotted in Figure 4.3(c). It shows the
temperature increasing approximately linearly before 50C. The heating pro-
cessing slows down after that and the temperature starts to drop from 55C.
In the heat pain tests, the thermode was placed on the study subject’s inner
49
forearm. A cool pad was placed on the heated skin spot right after the end
of the test.
Data summary
In total, 122 tests were taken in the data collection. Sixty of them were with
electrical pain stimulation, and sixty-two were with heat pain stimulation.
Among the thirty-one study subjects that took part in the data collection,
thirty of them (fifteen male and fifteen female) took all the four tests. One
study subject did not take the two electrical pain stimulation tests. In the
other two heat pain stimulation tests, most of the GSR recording of this
study subject were invalid either due to the high skin impedance where the
skin conductance was too low to read. Therefore, these two heat tests were
also be excluded from the analysis that relates to GSR, and the total number
of tests was 120 in this case.
The data collected in this part were saved into two lookup tables. The
first one, is the information lookup table. Table 4.1 lists the name and data
range of each column in the information lookup table file. The second one
is the timestamp lookup table. Table 4.2 lists the name and format of the
timestamps. The rest start and rest end took place before test1 started. In
both tables, inapplicable values are each denoted as not a number (NaN).
Table 4.1: Information lookup table, item list
No. Item Data range/unit
1 subject number 1, 2,..., 31
2 subject gender male, female
3 test number 1, 2, 3, 4
4 experimental stimulus elec, heat
5 side of the body left, right
6 self report VAS score at t2[4,10]
7 whether pain tolerance was reached at t2yes, no
8 thermode temperature at t1C
9 thermode temperature at t2C
10 TENS device output intensity at t10, 1,..., 50
11 TENS device output intensity at t20, 1,..., 50
The distributions of the VAS scores reported at t2in Figure 4.1 in the two
types of pain stimulation are presented in Figure 4.4. This figure shows that,
within the intensity range, the high-frequency electrical stimulation evoked
higher pain intensity than the slowly increasing heat on average (avg.VAS:
electrical - 7.3, heat - 6.6). Moreover, pain tolerance was less readily reached
within the upper-intensity limit in the tests with heat pain stimulation. The
difference between the perceived pain caused by the two experimental pain
stimuli is reflected not only in the distribution variety of the overall reported
VAS, but also in the variety of the self-reports within the same study subject.
50
Table 4.2: Timestamp lookup table, item list
No. Item Data format
1 record start
HH:MM:SS.FFF
2 rest start
3 rest end
4-6 test1 t0,t1,t2
7-9 test2 t0,t1,t2
10-12 test3 t0,t1,t2
13-15 test4 t0,t1,t2
16 record end
Among the 11 study subjects without an "intolerable" report, the reported
VAS scores in the heat tests were 1.6 lower than the electrical ones on average.
Comparatively, this difference was only 0.2 among the subjects that reported
"intolerable" in the heat tests. All these differences among the study subjects
validate the subjectivity in pain perception in terms of pain tolerance.
Figure 4.4: The distribution of the reported VAS scores at t2in all tests and
the tests without an "intolerable" report
Regarding the stimulus intensity at t1and t2, the electrical tests had a
greater variety among the subjects than the heat tests. The average electrical
intensity at t1is 5/50 (among all tests, SD=3) and is 16/50 (among the
"intolerable" reported tests, SD=9) at t2. In the heat tests, the average
heating temperature at t1is 49C (among all tests, SD=3C) and is 53C
(among the "intolerable" reported tests, SD=2C) at t2. The temperature
reported at pain threshold and pain tolerance are slightly higher than some
51
other studies on average with nearly the same variation. With similar or
different stimulation trial design (i.e., temperature rise speed and plateau
duration if there is), the average reported pain threshold temperature was
45.7C in [56], 44.3C in [113], and 42.4C among subjects with dark hair
in [48]. While the average reported pain tolerance temperature was 47.7C
among subjects with dark hair in [48] and 45.9C in [114] although the
average self-report intensity was close to this study (avg.VAS 6.4 [114] v.s.
avg. VAS in this study 6.6). In the electrical tests, the electrical current
is estimated as 0.1 mA per intensity when converting the electrical pulses
to an average current per second: 250 µs*2(biphasic)*100 Hz*(200/50 mA).
Then both the intensity at pain threshold and pain tolerance are lower than
the 5 Hz biphasic sinusoid alternating current among subjects with dark hair
(threshold - 2.2 ±1.1 mA [48] v.s. 1.0 ±0.6 mA in this study, tolerance -
3.3 ±1.9 mA [48] v.s. 3.2 ±1.8 mA in this study).
4.2.2 Biosignals and the acquisition system
The biosignal waveforms collected in this database included: 1) 6-channel
facial sEMG, 2) ECG Lead I, 3) respiration and 4) GSR. As introduced in
Chapter 3, a breathing waveform can be derived from an ECG waveform,
which can be an alternative when extracting breathing rate. The collected
waveforms and parameters were visualized on the laptop in real-time. The
signal acquisition software platform working in one data collection is pre-
sented in Figure 4.5.
Figure 4.5: Biosignals acquisition software platform
52
Facial sEMG
The facial sEMG signals were collected with the daughter board in the
ADS1299 performance demonstration kit [115]. The data recording soft-
ware was an earlier version of the LabView software described in Chapter
3. It has the function of continuous data receiving from the hardware via a
serial port by clicking the buttons Ctl_1-read continuously and Ctl_3-stop
reading continuously on the front panel. The start time and end time of
each recording session were saved for adding time labels to the samples. The
daughter board was configured in single-ended mode by an Arduino Mega
controller, and the bias drive or any bias circuit on the daughter board was
not in use.
The facial muscle areas were chosen based on Table 2.3. The single
electrode for each area was placed on the right side of the face following the
facial EMG electrode placement guidelines [116]. The connections between
electrodes and the analog inputs of the daughter board are explained in
Table 4.31. Muscle frontalis is not involved in pain facial expressions in
existing literature. Its signal was taken as a noise reference to all the other
sEMG signals due to the electrical pulse interference which was caused by
the pain stimulation in the electrical tests. Before attaching the H124SG
electrodes to the face, the electrode sites were wiped with an alcohol pad as
skin preparation.
Table 4.3: Electrodes placement and their connections to the device
Electrode site Analog input Muscle/Head area
CH1+ Frontalis
CH2+ Corrugator supercilii
CH3+ Orbicularis oculi
CH4+ Levator labii superioris
CH5+ Zygomatic major
CH6+ Risorius
REF Bony area behind ear
Facial sEMG signals were sampled at 1000 Hz. The ADC resolution of
ADS1299 is 24-bit. Its analog input range was between -4.5 V/gain and
4.5V/gain as the internal reference 4.5 V was used and the gain was set to
be 24 in the data collection. Therefore, the input voltage range was between
-187.5 mV and 187.5 mV. The laptop powered the data acquisition hardware
through a USB cable which was also the data transmission channel. The
laptop ran on battery power during the data collection to avoid bringing
additional powerline interference to sEMG signals.
1The analog inputs are denoted as AIN1P, ANI1N...REF_ELEC in ADS1299EEG-FE
53
ECG and respiration
ECG waveform and breathing waveform were collected with Bioharness 3
chest strap. The device’s logging format was configured to be General +
ECG, where the log files reported extracted parameters such as heart rate
and breathing rate at 1 Hz, breathing waveform at 18 Hz, and ECG waveform
at 250 Hz. The ADC on the device has a resolution of 12-bit. The device has
a maximum log duration of 35 hours with new batteries, and the maximum
memory capacity of the device can store 140 hours of data with the chosen
logging format1which are beyond the protocol’s requirements. The two
sensor pads were moistened before use to aid conductivity, and the strap
should be adjusted to be a snug fit2.
GSR
The GSR acquisition system was developed from the eHealth v2.0 platform.
Its analog circuits for GSR acquisition is plotted in Figure 3.12(a). The
eHealth shield is compatible with the Arduino Uno controller which digitizes
the analog signal at 10-bit resolution.
Figure 4.6: GSR electrode sites on the volar surface
Palms and volar surfaces of the fingers are active sites for electrodermal
recording. Medial phalanges of the index and middle fingers were recom-
mended for GSR recordings because they have larger areas for electrodes
and are less prone to scarring and movements compared to the other finger
areas [106]. In this study, the medial phalanges of the middle and ring fingers
were chosen instead (as shown in Figure 4.6) as the index finger was occupied
by a pulse oximeter for checking heart rate in real time. The same H124SG
1Bioharness 3, Log Data Descriptions
2Bioharness 3, User Manual
54
electrodes were used for GSR measurement. The electrodes were placed on
the opposite side of the body to the pain stimulation. For example, as a test
on the right always followed a test on the left side, the GSR electrodes were
moved from the right side to the left side between the two tests.
4.3 Summary of the collected signals
This chapter documents how the database was built and the related technical
specifications. The signals collected in the database included waveform sig-
nals from which interpretable features could be extracted from (e.g., sEMG,
ECG and breathing waveform) and the extract parameters could be provided
by Bioharness 3 (e.g., heart rate and breathing rate). Table 4.4 lists all the
waveform time series and part of the parameter time series in the database.
The six channels of sEMG are named fro-frontalis, cor-corrugator supercilii,
orb-orbicularis oculi, lev-levator labii superioris, zyg-zygomatic major and
ris-risorius. The digitization resolution of the sEMG signal was 0.54 µV
approximately. The display resolution of the GSR signal was 10 nS (nano
Siemens), which is adequate to display the digitization resolution of the sig-
nal (98 nS). The signal time series before and during one electrical test were
plotted with their data labels in Figure 4.7. The sEMG and ECG signals in
the figure have been processed.
Table 4.4: Signal summary
No. Signal Time Data type Data unit
resolution
0 Heat temperature. 1 s integer C
1-6 sEMG 1 ms [-223+1,223 -1] 4.5/223 *106µV
7 ECG 4 ms [0 4095] (x-2048)*6.25 mV
8 breathing 1/18 s [0 4095] -
9 GSR 1 s integer 10 nS
10 heart rate 1 s integer beats per minute (bpm)
11 breathing rate 1 s integer breaths per minute
12 acceleration 1 s [0 16] VMU1of max.acc (g)
13 activity 1 s [0 16] VMU of avg.acc (g)
14 posture 1 s [-180 180] degree (vertical = 0)
1VMU=paccx2+accy2+accz2, vector magnitude unit
55
Figure 4.7: The signals in one electrical test
56
Chapter 5
Biosignal Processing and
Feature Extraction
Before feeding the collected biosignals into a classifier for pattern recogni-
tion, the signals need first to be processed for three reasons. First, the
contaminations in the raw signal could be prominent or even cover the use-
ful information in the signal and therefore should be removed. Second, the
parameters in a multi-parameter model could be sampled at different time
intervals and unifying the time interval via feature extraction can simplify
the modeling process. Third, the constructed model is more interpretable
when the inputs are meaningful features instead of waveform segments. The
principles and methods used in the signal denoising and feature extraction
are explained below.
5.1 Signal denoising
The signal denoising strategies are defined based on the characteristics of
the noise sources. Therefore the noise sources are first introduced in this
section before explaining their implementation. The common biosignal con-
taminants are categorized in Table 5.1.
5.1.1 Motion artifact
Motion artifact is caused by voluntary or involuntary patient/sub<