![Optical flow estimated with median flow tracker [20] in a 5 × 5 grid (black). Valid flow vectors (blue) are used to calculate a movement vector for each grid cell (green, scaled for improved visibility). For each frame, the maximum vector magnitude (of all 25 regions) is saved to create a time series that estimates facial activity.](https://www.researchgate.net/profile/Philipp-Werner-2/publication/319316942/figure/fig1/AS:555898108366853@1509547845969/Optical-flow-estimated-with-median-flow-tracker-20-in-a-5-5-grid-black-Valid-flow_Q320.jpg){kind=tracking}
Content uploaded by Philipp Werner
Author content
All content in this area was uploaded by Philipp Werner on Jan 18, 2018
Content may be subject to copyright.
Content uploaded by Philipp Werner
Author content
All content in this area was uploaded by Philipp Werner on Nov 01, 2017
Content may be subject to copyright.
2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)
Analysis of Facial Expressiveness During Experimentally Induced Heat Pain
Philipp Werner, Ayoub Al-Hamadi
University of Magdeburg, Germany
Email: {Philipp.Werner, Ayoub.Al-Hamadi}@ovgu.de
Steffen Walter
Ulm University, Germany
Email: Seffen.Walter@uni-ulm.de
Abstract—To develop automatic pain monitoring systems, we
need a deep understanding of pain expression and its influ-
encing factors and we need datasets with high-quality labels.
This work analyzes the variation of facial activity with pain
stimulus intensity and among subjects. We propose two distinct
methods to assess facial expressiveness and apply them on the
BioVid Heat Pain Database. Experimental results show that
facial response is rare during low intensity pain stimulation
and that the proposed measures can successfully identify highly
expressive individuals, for whom pain stimuli can be classified
reliably, and non-expressive individuals, who may have felt less
pain than intended and encoded in labels.
1. Introduction
Facial expression is a valuable cue for pain assessment
[1], [2] and can be exploited to recognize pain automatically
with computer vision and machine learning techniques [3],
[4], [5], [6]. However, pain expression does not directly
reflect the pain experience, but is known to be influenced
by personal factors and social context [2], [7]. This is a
challenge for pain recognition research, which relies on
well-labeled and representative datasets.
Several pain studies reported that a part of the subjects,
who are sometimes called stoics, displayed no facial re-
sponse to pain [7], [8]. Further, some works in automatic
pain recognition mention differences in expressiveness [9],
[10], [11], but do not analyze further. Werner et al. [12]
includes an experiment showing the difference in pain
recognition performance between more and less expressive
subjects, which motivated the more detailed analysis we
conduct in this work.
Many papers in automatic pain recognition avoid the
expressiveness problem by using an observer-based pain
definition [6], [13], [14], [15], i.e. videos or single frames
are labeled according to visible pain reactions. Stoics are
correctly labeled with this approach (no pain), but the la-
beling may fail in other cases, in which subject reported to
feel pain (see discussion in [12]).
Contributions: In this work, we propose methods to
estimate facial expressiveness (Sec. 3) that are independent
of each other. They do neither consider the features used
for automatic recognition nor the classification results. We
apply the methods on the BioVid Heat Pain Database [9],
[16] and assess the facial activity in the pain intensity classes
of the dataset (Sec. 4). Further, we show the correlation of
the proposed measures and their capability to estimate the
expressiveness of a subject (Sec. 5). Subjects are categorized
using the measures and the plausibility of the subject subsets
is supported by agreement of the measures and classification
rates that we achieve on these subsets with an independent
recognition system. We discuss the results in Sec. 6.
2. BioVid Heat Pain Database
The BioVid Heat Pain Database [9], [16] (online: http:
//www.iikt.ovgu.de/BioVid.html) was collected in a study
with 90 participants. Pain was induced experimentally by
a thermal stimulator (Medoc PATHWAY) at the right arm
and pain responses were recorded with video cameras and
physiological sensors. The temperature applied for pain
stimulation was recorded synchronously, which provides
fine-grained information in time and value domain.
Before the data recording was started, the participant’s
individual pain threshold and tolerance were determined
based on self-report. The goal was to compensate for indi-
vidual pain sensitivities, i.e. to select person-specific stim-
ulation temperatures that elicit pain experiences with same
severity across subjects.
In the main experiment, pain was stimulated in four
intensities. The highest pain intensity PA4 was stimulated
by applying the person-specific temperature that the subject
selected as pain tolerance (highest acceptable pain intensity);
the lowest pain intensity PA1 was defined by the person-
specific pain threshold (lowest temperature that the subject
identified as being painful). PA2 and PA3 were defined as
intermediate intensities by linear interpolation between PA1
and PA4. Each pain intensity was stimulated 20 times. At
each time temperature was held for 4 seconds followed by
a pause of 8-12 seconds. Pauses were used to extract the
non-painful baseline samples (BLN).
The BioVid database consists of several parts. In this
paper we use part A, which consists of 8,700 samples of 87
subjects, each with 5.5 seconds video and some time series.
The 100 samples per subject include 20 samples of each of
the 4 pain intensities (PA1 to PA4) and 20 samples of the
pain-free baseline condition (BLN). We further use part C,
which is the continuous superset of part A, to extract more
baseline samples.
P. Werner, A. Al-Hamadi, S. Walter, "Analysis of Facial Expressiveness During Experimentally Induced Heat Pain", in International Conference on Affective
Computing and Intelligent Interaction Workshops and Demos (ACIIW), 2017.
This is the accepted manuscript. The final, published version is available on IEEE Xplore.
(C) 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to
servers or lists, or reuse of any copyrighted component of this work in other works.
Figure 1. Optical flow estimated with median flow tracker [20] in a 5×5
grid (black). Valid flow vectors (blue) are used to calculate a movement
vector for each grid cell (green, scaled for improved visibility). For each
frame, the maximum vector magnitude (of all 25 regions) is saved to create
a time series that estimates facial activity.
3. Measures to Estimate Facial Expressiveness
In this section we propose methods to measure facial
activity and expressiveness.
PSPI: First, we consider the Prkachin and Solomon
Pain Intensity (PSPI) [6], [17], a pain expression score based
on the facial action coding system (FACS) [18]. It is widely
used in facial expression based pain recognition [6], [13],
[15]. We use the FACS coded subset of the BioVid heat
pain database (1 sequence per intensity and subject = 435
videos = 60,030 frames of part A) [12], [19] to calculate
PSPI. The PSPI is a frame based measure; since we work
on video level, we use maximum PSPI per video as it has
been done to validate the score [17]. FACS and PSPI are
highly objective, but require a lot of work of a trained FACS
coder – about two hours per minute of video. In general,
this solution is impractical for huge datasets, but the score
is useful to validate the other proposed measures.
Subjective Rating: As a cheap alternative to FACS
coding, we propose subjective rating. For each subject in
the dataset, an untrained observer viewed the highest pain
intensity videos (as they show the strongest response) and
rated it on a scale from 1 (no observed reaction) to 5 (clear
reactions in nearly all videos). This is much faster than
FACS coding, i.e. we can look at much more samples in
less time. Although it may be less objective and may miss
some subtle details, it should be sufficient for the purpose
of categorizing persons by expressiveness roughly.
Optical Flow: We further propose to assess facial ac-
tivity in a fully automatic manner using low-level computer
vision features. Based on optical flow, a frame-by-frame
activity score is calculated as follows:
1) We localize the face using the Viola-Jones face detec-
tor available in OpenCV [21]. To exclude background,
we only use the center part of the bounding box (60%
of width and 80% of height).
2) The region of interest is subdivided in a regular 5×5
grid. In each grid cell we apply the (very robust)
median flow tracker by Kalal et al. [20]: We calculate
sparse optical flow of 100 points per cell though the
Lucas-Kanade algorithm. The flow is calculated for-
ward and backward and the error between both is used
to remove outliers (points with erroneous movement
estimation) by only using the half of points with
lower error. More outliers are removed by a similar
criterion based on normalized cross correlation. The
remaining flow vectors are considered to be valid and
are combined into one movement vector per region by
calculating the median in each spacial dimension.
3) We calculate the magnitude of all 25 movement vec-
tors and select the maximum as an estimate of the
current facial activity (one point in the time series of
the video).
Fig. 1illustrates the median flow estimation. To speed up
the calculation, we scale the image down to half of the
resolution before applying the above processing steps. We
processed all videos of part A of the BioVid database
and further extracted more baseline samples from part C
(the superset of part A) to reduce influence of occasional
movements during some baseline samples. This way we got
80 baseline and 4×20 pain samples per subject, each with
a facial activity time series.
To analyze the variation in expressiveness across pain
intensities (Sec. 4), we use that all videos have the same
length and are temporally aligned with respect to the pain
stimulation. We calculate the median time series per class,
i.e. we keep the temporal resolution and calculate the median
across all subjects’ samples of that class for each instance
of time.
Further, we analyze the facial activity of individuals
(Sec. 5). For this purpose we calculate the mean flow
magnitude per video yielding one distribution for each class
per subject. Next, we check whether the subjects shows
more activity during PA4 than during BLN by applying
a permutation test [22] (which makes fewer assumptions
and provides greater accuracy than a t-test): We randomly
generate 2,000 permutations of the class labels, i.e. the class
assignments of activity scores are mixed up randomly 2,000
times. For each permutation, we calculate the difference
of the mean result of PA4 and the mean result of BLN,
yielding the distribution of the activity differences under
the assumption that both activities are equal. Then pis
the probability of differences being greater or equal to the
observed difference, determined from the aforementioned
distribution. I.e. a p-value close to zero indicates that the
subject shows significantly more facial activity during PA4
than during BLN.
4. Facial Reactions Across Stimulus Intensities
Fig. 2compares the median facial activity time series
across pain stimulation classes: baseline/no pain (BLN),
pain intensity at person-specific pain threshold (PA1), pain
intensity at person-specific pain tolerance (PA4), and two
012345
0.15
0.2
0.25
0.3
time (s)
median flow (pixel)
BLN
PA1
PA2
PA3
PA4
Figure 2. Median flow time series during pain stimulation (PA*) and base-
line, i.e. pauses (BLN). Red background illustrates the timing of the high
temperature plateau for stimulation (not present in BLN). In the majority
of cases there is no significant activity during baseline (BLN), in lowest
(PA1), and second lowest pain intensity (PA2). In highest pain (PA4),
facial activity starts about 2 s after the temperature plateau is reached. In
PA3 activity is lower and starts later.
mean PSPI: 0.36 0.43 0.76 1.37 1.95
PSPI = 0
PSPI = 1
PSPI = 2
PSPI = 3
PSPI = 4
PSPI > 4
Figure 3. Distributions of the maximum PSPI across pain intensities (PA*
and baseline (BLN).
intermediate pain levels (PA2 and PA3). Each point in
these curves represents the flow magnitude that is exceeded
by 50% of the samples of that class (median). We clearly
observe activity in PA4 and PA3. Activity in PA2 is barely
above BLN at the end of the time window and unless some
noise, PA1 stays constant. As to be expected, the facial
response reduces with the pain stimulus severity. However,
at the beginning of the time windows, BLN has even higher
activity than the pain classes (with a falling tendency). This
indicates a bias in the selection of the baseline samples.
The baseline time windows follow pain stimulations, which
induce a high level of activity. This activity is still fading
away during many BLN samples, which we see in decreasing
activity. When the next pain stimulus starts, activity often
has reached a lower level than during BLN. This bias should
be avoided in future studies by applying longer pauses and
selecting baseline samples more diversely.
Fig. 2also shows the time delay between stimulation
and response, which gets shorter with higher intensity. A
higher temperature leads to faster heat conduction from the
TABLE 1. CORRELATION BETWEEN FACIAL EXPRESSIVENESS
MEASURES (PEA RSO N CORR EL ATION CO EFFIC IE NTS ).
Subj. rating Flow pPSPI PA4
Flow p-0.607
PSPI PA4 0,683 -0,476
PSPI SD 0.634 -0.465 0.876
thermal stimulator onto the skin, i.e. the skin is heated up
earlier. Further, higher intensity nociception results in faster
reactions to escape the noxious stimulation.
Fig. 3illustrates difference between classes in his-
tograms of PSPI scores. It conforms with Fig. 2: Facial
reactions are rare in low pain intensities – far less than
half of the samples have a maximum PSPI pain score
greater than zero in PA1 and PA2. Further, PSPI=1 also
often corresponds to no activity, since several subjects had
closed eyes permanently (see [12] for more details). Greater
scores occur more often with higher pain intensity. They
correspond to intensity of facial pain response; the higher
the score, the more pronounced the expression.
The majority of PA1 and PA2 samples do not comprise
observable pain response, since the stimulation did not
exceed the critical thresholds to trigger a facial response for
many subjects. If we only consider facial expression and
use all subjects, this induces a lot of label noise which is
a heavy burden for machine learning algorithms used for
automatic assessment systems. That is why we suggest to
focus on PA3 and PA4 and/or to work with a subject subset
excluding the non-responding subjects.
5. Facial Reactions Across Individuals
In Sec. 3we proposed to assess expressiveness of indi-
viduals by (1) subjective rating and (2) optical flow statistics.
This section reports experiments to validate those measures.
We compare them with FACS-based PSPI measures: (3)
PSPI of the subject in the coded PA4 sample and (4) stan-
dard deviation (SD) of the PSPI across the coded samples
off all classes. Table 1lists the correlations of the measures
(1-4). The subjective rating is strongly related to all other
measures. Flow pis moderately correlated to the PSPI
measures. PSPI, subjective rating, and flow phave been
determined with very distinct methods, so their correlations
support the validity of the measures.
In another experiment we apply several thresholds on
subjective rating and flow pto split the dataset into a more
and a less expressive subject subset. We compare the subsets
regarding mean values of the measures (1-4) and regarding
the classification accuracy that the pain recognition method
proposed in [12] achieves if it is trained and tested on
the subsets. We evaluated two classification problems with
leave-one-subject-out cross validation on each subject set:
BLN vs PA4 (2 classes) and BLN vs PA3 vs PA4 (3 classes).
To reduce training time, we learned random forest ensembles
of 100 instead of 1,000 trees.
TABLE 2. SPLITTING SUBJECTS IN A LESS EXPRESSIVE (LE) AND A M ORE E XP RES SIV E (ME) SUBSET BY SEVERAL FACIAL EXPRESSIVENESS
CRITERIA. FO R EACH S UB SET,T HE TABL E RE PORT S TH E NUM BER O F SU BJE CTS I N TH E SET,THE MEAN VALUES OF EXPRESSIVENESS MEASURES,
AN D CLA SSI FIC ATION A CCU RAC IES .
# Split criterion Subject count Subj. rating Flow pPSPI PA4 PSPI SD CA 2-class CA 3-class
LE ME LE ME LE ME LE ME LE ME LE ME LE ME
1 No split 87 2.5 0.29 2.0 1.0 71.8 50.5
2 Subj. rating >480 7 2.3 5.0 0.32 0.02 1.7 4.4 0.9 2.1 70.2 92.5 48.1 70.7
3 Subj. rating >368 19 2.0 4.4 0.37 0.04 1.5 3.7 0.8 1.8 66.6 89.7 44.6 68.7
4 Subj. rating >250 37 1.6 3.7 0.46 0.07 0.9 3.4 0.5 1.7 59.2 87.4 39.6 63.9
5 Subj. rating >120 67 1.0 2.9 0.59 0.21 0.3 2.4 0.2 1.3 49.3 78.5 31.7 55.5
6 Flow p < 0.01 56 31 2.0 3.4 0.46 0.00 1.2 3.3 0.7 1.6 63.3 88.2 41.0 67.3
7 Flow p < 0.148 39 1.9 3.3 0.53 0.01 1.1 3.0 0.6 1.5 60.7 85.5 40.4 63.5
8 Flow p < 0.240 47 1.7 3.2 0.61 0.03 0.9 2.9 0.5 1.4 58.0 83.5 38.2 61.5
9 Flow p < 0.334 53 1.6 3.0 0.67 0.05 1.0 2.6 0.5 1.3 60.5 80.7 40.2 59.0
10 Flow p(67 lowest) 20 67 1.6 2.8 0.80 0.14 0.8 2.3 0.4 1.2 56.1 76.2 38.8 55.0
CA 2-class: Classification accuracy BLN vs PA4 in percent (chance is 50%)
CA 3-class: Classification accuracy BLN vs PA3 vs PA4 in percent (chance is 33%)
LE: less expressive subset ME: more expressive subset
The results are shown in Table 2. The differences be-
tween the more and less expressive subject groups are
significant for all splits and measures. Differences in rows
(3-10) are significant with p < 0.001; differences in row
(2) are significant with p < 0.05 (due to low sample size
in the more expressive group). The differences illustrate
the high variation of subjects’ expressiveness and its big
impact on the predictive performance of automatic pain
assessment systems. Row (2) shows that recognition works
very well with highly expressive subjects (classification rate
of more than 92% in 2-class and more than 70% in 3-class
problem). The split in row (5) is also remarkable, since
the classification rates of the less expressive 20 subjects
are below chance. This indicates that we have found a
group of stoic subjects, who did not react visibly to the
induced pain stimuli. Very low PSPI and high flow psupport
this conclusion. Excluding these subjects from experiments
is reasonable, since they introduce noise that may con-
fuse machine learning and lead to suboptimal recognition
models. To compare subjective rating and flow p, we split
the subjects based on the ranking of flow pto get the
same group sizes as (5), see (10). PSPI and classification
accuracies indicate a less clear split than (5), which suggests
that subjective rating is more suitable for identifying stoic
subjects than the fully automatic flow-based method.
6. Discussion
We analyzed the facial expressiveness across stimulus in-
tensities and subjects in the BioVid Heat Pain Database. Low
intensity pain does not alway trigger facial response [7],
[23]. We have observed this in two independent measures for
the two lowest pain intensities PA1 and PA2, since the ma-
jority of samples and subjects did not show facial response
in these intensities. From our perspective, it is reasonable
to exclude these classes from facial expression based pain
recognition experiments whenever we want to reduce the
burden for machine learning in interest of better models.
This also moves the focus towards high pain intensities,
which are more important for clinical applications.
Further, we proposed to assess facial expressiveness
of individuals with subjective rating and an optical flow
based measure. A high agreement was found between the
two proposed measures and two FACS-based measures. By
thresholding the measures we were able to successfully
split the dataset in a more and a less expressive group,
which differed significantly regarding all measures and the
accuracy that was achieved by a pain recognition system
on the subject groups. The subsets induced by subjective
rating were more distinct than those induced by flow p. So
subjective rating seems to be superior for finding highly
expressive or stoic subjects, but flow palso yields good
results and can be calculated fully automatically.
Pain expression is known to be influenced by personal
factors [2], [7]. Among others, there is a person-specific
pain severity threshold that must be exceeded to trigger a
facial response [7], [23]. I.e. a reason for a lack of facial
response may be too low pain intensity. Experimental pain
studies offer highly controlled pain stimulation (regarding
both intensity and time), which is valuable as a high quality
ground truth. However, due to individual differences in pain
sensitivity some kind of self-report is needed to assess
pain experience, at least to calibrate the pain stimuli in an
preceding experiment. Although self-report is the current
“gold standard" in pain assessment, it has its weaknesses
[1]. It is a controlled and goal-oriented response to pain [2],
which might be affected by reporting bias and variances in
memory and verbal ability [1]. In an experimental study,
the (paid) participant may not want to feel severe pain, so
he may underestimate his pain and/or tolerance threshold
(intentionally or unintentionally) during stimulus calibration.
Further, some subjects have low pain sensitivity resulting in
a high tolerance threshold that cannot be stimulated without
causing tissue damage, which also leads to less painful
stimulation. We observed such problems in the study with
experimentally induced heat pain, in which we recorded the
BioVid Heat Pain Database [9], [16]. They probably have
caused the lack of facial pain response for some subjects.
The split criterion in row (5) of Table 2identifies such
subjects, which we call stoics. Since these non-responding
subjects, who probably experienced less pain than intended
and encoded in the labels, are not representative for the
planned application of clinical assessment of acute pain,
we propose to exclude them from future pain recognition
experiments.
Acknowledgments
Funded by German Research Foundation proj. AL 638/3-2.
References
[1] K. D. Craig, “The facial expression of pain Better than a thousand
words?” APS Journal, vol. 1, no. 3, pp. 153–162, 1992. 1,4
[2] K. D. Craig, K. M. Prkachin, and R. E. Grunau, “The facial ex-
pression of pain,” in Handbook of Pain Assessment, D. C. Turk and
R. Melzack, Eds. Guilford Press, 2011. 1,4
[3] P. Werner, A. Al-Hamadi, and R. Niese, “Pain Recognition and Inten-
sity Rating based on Comparative Learning,” in IEEE International
Conference on Image Processing (ICIP), 2012, pp. 2313–2316. 1
[4] ——, “Comparative learning applied to intensity rating of facial
expressions of pain,” International Journal of Pattern Recognition
and Artificial Intelligence, vol. 28, no. 05, p. 1451008, Jun. 2014. 1
[5] M. Kächele, P. Thiam, M. Amirian, P. Werner, S. Walter,
F. Schwenker, and G. Palm, “Multimodal Data Fusion for Person-
Independent, Continuous Estimation of Pain Intensity,” in Engineer-
ing Applications of Neural Networks, Springer, 2015, pp. 275–285,
DOI: 10.1007/978-3-319-23983-5_26. 1
[6] P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, S. Chew, and
I. Matthews, “Painful monitoring: Automatic pain monitoring using
the UNBC-McMaster shoulder pain expression archive database,”
Image and Vision Computing, vol. 30, no. 3, pp. 197–205, 2012.
1,2
[7] K. M. Prkachin and K. D. Craig, “Expressing pain: The communica-
tion and interpretation of facial pain signals,” Journal of Nonverbal
Behavior, vol. 19, no. 4, pp. 191–205, Dec. 1995. 1,4
[8] M. Kunz and S. Lautenbacher, “The faces of pain: a cluster analysis
of individual differences in facial activity patterns of pain,” European
Journal of Pain, vol. 18, no. 6, pp. 813–823, Jul. 2014. 1
[9] P. Werner, A. Al-Hamadi, R. Niese, S. Walter, S. Gruss, and H. C.
Traue, “Towards Pain Monitoring: Facial Expression, Head Pose, a
new Database, an Automatic System and Remaining Challenges,”
in Proceedings of the British Machine Vision Conference (BMVC).
BMVA Press, 2013, pp. 119.1–119.13. 1,5
[10] ——, “Automatic Pain Recognition from Video and Biomedical Sig-
nals,” in International Conference on Pattern Recognition, 2014, pp.
4582–4587. 1
[11] M. Kächele, M. Amirian, P. Thiam, P. Werner, S. Walter, G. Palm, and
F. Schwenker, “Adaptive confidence learning for the personalization
of pain intensity estimation systems,” Evolving Systems, vol. 8, no. 1,
pp. 71–83, 2017. 1
[12] P. Werner, A. Al-Hamadi, K. Limbrecht-Ecklundt, S. Walter, S. Gruss,
and H. Traue, “Automatic Pain Assessment with Facial Activity
Descriptors,” IEEE Transactions on Affective Computing, vol. PP,
no. 99, pp. 1–1, 2016. 1,2,3
[13] S. Kaltwang, O. Rudovic, and M. Pantic, “Continuous Pain Intensity
Estimation from Facial Expressions,” in Advances in Visual Comput-
ing, Springer, 2012, pp. 368–377. 1,2
[14] K. Sikka, A. Dhall, and M. S. Bartlett, “Classification and weakly
supervised pain localization using multiple segment representation,”
Image and Vision Computing, vol. 32, no. 10, pp. 659–670, 2014. 1
[15] O. Rudovic, V. Pavlovic, and M. Pantic, “Context-Sensitive Dynamic
Ordinal Regression for Intensity Estimation of Facial Action Units,”
IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 37, no. 5, pp. 944–958, 2015. 1,2
[16] S. Walter, P. Werner, S. Gruss, H. Ehleiter, J. Tan, H. C. Traue, A. Al-
Hamadi, A. O. Andrade, G. Moreira da Silva, and S. Crawcour,
“The BioVid Heat Pain Database: Data for the Advancement and
Systematic Validation of an Automated Pain Recognition System,” in
IEEE Int’l Conf. on Cybernetics (CYBCONF), 2013, pp. 128–131. 1,
5
[17] K. M. Prkachin and P. E. Solomon, “The structure, reliability and
validity of pain expression: Evidence from patients with shoulder
pain,” PAIN, vol. 139, no. 2, pp. 267–274, 2008. 2
[18] P. Ekman and W. V. Friesen, Facial Action Coding System: A
technique for the measurement of facial movements. Palo Alto:
Consulting Psychologist Press, 1978. 2
[19] K. Limbrecht-Ecklundt, P. Werner, H. C. Traue, A. Al-Hamadi, and
S. Walter, “Mimische Aktivität differenzierter Schmerzintensitäten,”
Der Schmerz, vol. 30, no. 3, pp. 248–256, Apr. 2016. 2
[20] Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-Backward Error:
Automatic Detection of Tracking Failures,” in International Confer-
ence on Pattern Recognition (ICPR), 2010, pp. 2756–2759. 2
[21] R. Lienhart, A. Kuranov, and V. Pisarevsky, “Empirical Analysis of
Detection Cascades of Boosted Classifiers for Rapid Object Detec-
tion,” in DAGM Pattern Recognition Symposium, 2003, pp. 297–304.
2
[22] D. S. Moore, G. P. McCabe, W. M. Duckworth, and S. L. Sclove,
The practice of business statistics: using data for decisions. Wh
Freeman, 2003. 2
[23] M. Kunz, V. Mylius, K. Schepelmann, and S. Lautenbacher, “On the
relationship between self-report and facial expression of pain,” The
Journal of Pain, vol. 5, no. 7, pp. 368–376, Sep. 2004. 4
