Content uploaded by Caterina Nahler
Author content
All content in this area was uploaded by Caterina Nahler on Sep 20, 2018
Content may be subject to copyright.
Exploring the usage of Time-of-Flight Cameras for
contact and remote Photoplethysmography
Caterina Nahler∗, Bernhard Feldhofer∗, Matthias Ruether†, Gerald Holweg∗and Norbert Druml∗
∗Infineon Technologies Austria AG, Graz, Austria
{caterina.nahler2, bernhard.feldhofer, gerald.holweg, norbert.druml}@infineon.com
†Institute for Computer Graphics and Vision, Graz University of Technology, Austria
{ruether}@icg.tu-graz.ac.at
Abstract—The heart beat is one of the basic vital signs,
but the pulse wave can communicate much more information
than the beat frequency of the heart. Through the heart rate
variability (HRV) among other things, stress level and drowsiness
can be inferred. Reliable HRV measurements are commonly
obtained by electrocardiography (ECG). In this paper we analyse
the correlation between HRV and pulse rate variability (PRV)
obtained from contact or remote photoplethysmography (PPG)
on a Time-of-Flight (ToF) camera from PMD Technologies AG.
The ToF camera is independent of passive illumination as it uses
an infrared (IR) light source, which is invisible to the human
eye. Therefore, potential use cases include driver monitoring or
convenient heart rate measurement with smart phones. We will
demonstrate methods for using a ToF camera for contact and
remote PPG. For contact PPG the sensor is mounted directly
on the subjects body. The pulse wave is calculated as the mean
amplitude intensity of all pixels. For remote PPG the sensor is
mounted to measure the subjects face. The pulse wave cannot
be found in every part of exposed skin. Therefore, the part
of the skin with the clearest signal is localised and the pulse
wave is recovered from there, by means of passband filtering
and blind source separation. Our results show that contact PPG
on a ToF camera has high accuracy and highly correlates with
a commercial IR pulse oximetry device, as well as with an ECG
grade chest belt. For remote PPG our method shows correlation
with the ToF contact PPG signal.
Index Terms—Time-of-Flight, ToF, remote, contact, photo-
plethysmography, PPG, rPPG, infrared, pulse, heart beat
I. INTRODUCTION
The standard method for heart rate and vital sign observa-
tions is the electrocardiography (ECG). An alternative method
is the photoplethysmography (PPG). The blood oxygenation
and volume changes affect the light absorption characteristics
of living tissue. PPG is based on non-invasive measurement
of these light absorption changes. Contact sensors produce
a measurement, either by means of reflected or transmitted
light, while being loosely attached to the subject’s body [1].
Alrick Hertzman first described a setup to photoelectrically
measure the blood volume on fingers and toes in 1937 [2]. In
contrast to contact PPG, remote PPG relies mostly on reflective
measurement. One of the first versions of remote PPG to
measure the face was developed in 2008 [3]. Earlier imple-
mentations focused the camera on other parts of the human
body, such as the arm [4]. Nowadays PPG has already been
explored for many applications, including the measurement of
blood pressure and oxygenation [4], breathing cycles, mental
Fig. 1. A visualisation of the pulse wave analysis on the ToF amplitude video
data. Every pixel in the video is analysed for the likelihood of containing a
clear pulse wave. It can be seen that the carotid artery becomes visible as a
bright spot. This indicates, that a clear pulse wave signal is contained in that
region.
stress [5], drowsiness detection [6], sleep monitoring and heart
rhythm disturbances [7]. A recent area of research includes the
usage of contact and remote PPG for driver monitoring [6],
[8].
The Time-of-Flight (ToF) camera is independent of passive
illumination as it uses an active infrared (IR) light source,
that is invisible to the human eye. The camera, therefore has
potential for remote PPG also during night time. Most of
the mentioned use cases can benefit from this property. PPG
further enables convenient heart rate measurement with smart-
phones. Remote PPG variants using the front facing camera
exist [9]. Recently, remote PPG has also been integrated into
augmented reality. An application was developed which uses
the HoloLens and RGB data to measure and visualize another
human’s heart rate [10].
In this paper, we will explore the usage of the Pico Flexx
ToF camera from PMD Technologies AG for contact and
remote PPG. We contribute two methods which are, subse-
quently shown, to work on the data provide by the Pico Flexx
ToF camera. For contact PPG, the pulse wave is calculated as
the mean amplitude intensity of all pixels. For remote PPG
the sensor is mounted to measure the subjects face. The pulse
wave cannot be found in every part of exposed skin. Therefore,
we propose a novel combination approach to select the part of
the skin with the clearest signal. An example visualisation of
the signal localisation metrics is shown in Fig. 1. The pulse
wave is recovered from the selected skin region, by means of
passband filtering and blind source separation. The remainder
of the paper is structured as follows. Background information
will be provided in Section II. The overview on related work
is given in Section III. The method will be described in detail
in Section IV and the results including a short discussion will
be provided in Section V. A summary of the findings will
conclude this paper in Section VI.
II. BACKGROU ND
A. Pulse Measurement
Commonly known ways to measure the heart rate include
ECG and PPG. The typical heart beat signal which is known
from ECG is not the same as in PPG. The optically measured
signal is called a pulse wave. The R-R intervals (peak to
peak distances) in ECG measurements are closely correlated
with the N-N intervals (peak to peak distances) in PPG. The
Heart rate variability (HRV) is one of the most promising
markers of the autonomic nervous system. It is the variation
of time intervals between successive peaks, obtained from an
ECG signal [11]. However, the variation of peak intervals
obtained from a PPG signal is the pulse rate variability (PRV).
HRV measures were shown to highly correlated with PRV
measures [12]–[14]. To evaluate the accuracy of our methods,
several measures are extracted from the pulse wave. The av-
erage heart rate (AHR) is useful to extrapolate the heart beats
up to a minute from a short measurement. For this calculation
it is necessary to measure several peaks and calculate the
time interval between the peaks. The average value ∆tis
then formed over all the extracted time intervals and with the
resulting value a prediction up to a whole minute
BPM = ∆t·60,(1)
where BPM means beats per minute, is possible.
The standard deviation of R-R intervals (SDRR) or respec-
tively N-N intervals (SDNN) can be calculated as
SDNN [s] = v
u
u
t
1
M−1·
M
X
i=1
(∆t
i−∆t)2,(2)
where Mis the number of intervals counted in the measure-
ment and ∆t
iare the time intervals between the peaks [7].
Another measure of the HRV is the coefficient of variation
(CV) [7]. In this paper, we are using the CV obtained from
the PPG signal (CVP RV ) to compare our method with the
CV obtained from a Polar H7 (CVH RV ) which claims to have
ECG quality. We calculate the CV according to the standard
equation. Therefore, CVP RV is calculated as
CVPRV [%] = SDNN [s]
∆t[s]·100[%].(3)
Among other information, it is possible to infer stress
level [5], breathing cycles, heart rhythm disturbances [7] and
risk for sudden heart failure, from the HRV. These values can
only be calculated if the measurement is very accurate. In
healthy subjects at rest, HRV measures and PRV measures
have a high correlation. However as PPG is quite vulnerable
to motion artefacts, the reliability of the PRV measurement
under motion is limited in comparison to ECG.
B. Heart Rate
The normal resting heart rate of a healthy human adult was
traditionally defined as in the range of [60, 100]beats/min.
Bradycardia (bellow 60 beats/min) and Tachycardia (above
100 beats/min) are considered pathological heart rates [15].
The range of possible heart rates is not straightforward to
define. Athletes can show a resting heart beat as low as
[30, 40]beats/min, which is known as athlete’s heart syn-
drome [16]. The highest heart rate a human can achieve during
excising, is strongly correlated with age. In young adults heart
rates up to 220 beats/min were measured [17]. Given all this
considerations, in conjunction with the filter limitations, this
work is considering a range of [45, 200]beats/min.
III. REL ATED WO RK
A. Contact PPG
PPG measurements with blue, red and near-IR lights are
possible [18]. Different light wavelengths perform differently
for different skin tones and situations (for example: measure-
ments under motion). Smartphone and smartwatch manufac-
turers are also using PPG technology [19], [20]. Contact PPG
is widely used in smartwatches to obtain the heart rate. For
example, smartwatches as the Mio Alpha and forearm devices
as Schosche Rhythm are using green or IR light to detect the
pulse on the wrist and arm [19]. Jihyoung Lee and his team
claim that green light gives best results in case of motions [18].
Typical devices for blood oxygen saturation measurements are
using a combination of red and IR light due to the deep tissue
penetration [21]. Many of these devices use the transmission
plethysmography, i.e. emitter and detector are located opposite
each other. In our approach we are using a ToF camera with
reflective plethysmography, i.e. emitter and detector are on the
same side [1]. Bone, various tissues, arterial and venous blood
contribute amongst other influences in reflecting the emitted
active IR illumination back into the ToF photodiodes [22].
Typical placements for measuring the pulse are the fingers,
toes, wrist and ear lobes [23].
B. Remote PPG
In remote PPG the sensor is not attached to the subject
but measures parts of the skin or the face remotely. A
lot of research was done on extracting the heart rate from
facial colour videos (rgbPPG). Extensive reviews on the used
rgbPPG methods exist [24]–[26]. The reviews report methods
using multi spectral, additional wavelengths such as orange
and cyan [24], high quality cameras to consumer grade web
cams [26]. The mentioned methods differ in the selection of
regions of interest (ROI). The forehead, cheeks, lips and other
parts up to the full face were tried as ROI [25]. In addition,
methods for skin region selection in colour videos exist [26].
To reduce motion artefacts and noise and reconstruct the PPG
signal more truthfully, many of the reported methods com-
bine the colour channels. For combining the colour channels
derived weighting schemes or blind source separation (BSS)
were used. In the reviews mentioned BSS methods include
versions of the Independent Component Analysis (ICA) or
Principal Component Analysis (PCA). As another method
of denoising, passband filters, stochastic filters and wavelet
analysis are used, in combination with BSS or separate.
Depending on the method, filters were applied before and
after the colour channel combination. The heart rate was
then derived either by peak extraction or by spectral density
estimates such as fast Fourier transform (FFT) [26]. General
video magnification algorithm were also suggested to be used
to determine the heart rate [27], or amplify the signal before
further processing [28]. A video magnification algorithm was
used for real-time visualization of the heart rate on Microsoft
Kinect camera data [29]. For extraction of the pulse wave
rgbPPG is not limited to skin reflection changes in the video.
It is shown that heart rate can be extracted from slight
head movements due to the pressure variations in the carotid
arteries [30]. However rgbPPG methods which rely on passive
illumination, are prone to errors due to spatial and temporal
illumination variations. Several approaches were evaluated to
algorithmically improve the temporal illumination variation
resistance of rgbPPG [24]–[26].
Nevertheless, methods that use passive illumination, will not
work in the dark. Thermal cameras and IR cameras are known
to work well in the dark. Methods using thermal cameras [31],
thermal cameras in combination with IR cameras [32] and
solely IR cameras have been used for remote heart rate
estimation. An approach was presented which uses a single
wavelength IR camera to estimate the heart rate in the face by
spectrum analysis [33]. The benefit of rgbPPG is the availabil-
ity of more than one wavelength to reduce noise by putting
the different measurements into reference. A similar approach
was explored by using three different IR wavelengths [34].
Remote pulse oxygenation measurement on the face is also
demonstrated using orange and IR wavelengths [35]. It was
further shown that real time remote PPG measurement with
peak detection using an IR camera is possible on a FPGA [36].
The Microsoft Kinect RGBD camera has also been explored
for its use in heart rate measurement. Several approaches
utilize the RGB camera of the Kinect for this purpose. The
dominant frequencies in the spectral density estimate are
used to estimate the heart rate, which is not uncommon for
remote PPG algorithm [37]–[39]. More heart rate estimation
approaches were presented using the amplitude data of the
Kinect depth camera and a spectral density estimate [40], [41].
As our approach aims at recovering the pulse peak locations,
including N-N intervals, this heart rate estimation technique
is not enough. Another method uses the head movement to
extract the heart rate [42], but in comparison to rgbPPG
methods the depth data is used for motion tracking. Our
method also uses solely a depth camera, but in contrast, we
utilize the amplitude data to track the skin absorption changes.
Further methods where proposed which track the heart rate
and the respiratory rate simultaneously. An early method
required the subjects to lie down and expose the chest during
measurement. The measurement regions included the carotid
artery, however the regions were selected manually [43]. In
contrast our method selects the region automatically and does
not necessarily choose the carotid artery. A later approach
estimated both vital signs simultaneously from a spectral
density estimate [40]. One of the most recent methods tracks
both vital signs using RGB or IR data, depending on the
lightning conditions, for abnormality monitoring during sleep.
In contrast to our method the vital sings are extracted from
the abdominal-thoracic region which has to be visible. While
we extract the heart rate from the head and throat.
IV. MET HO D
In this section we will describe the contact and remote
methods used to measure the pulse wave with a Pico Flexx
ToF camera. This ToF camera uses the phase difference
between pulsed IR light and its reflection to measure depth.
To remove depth ambiguities, several phase shifted measure-
ments (phases) are used. ToF cameras can operate on different
use-case settings. A use-case setting consists of a predefined
framerate, illumination time and number of phases used to
calculate the depth and amplitude image. The quality of the
signal improves with higher illumination time. However, if the
phases or the illumination time go up, the framerate has to go
down, to adhere the laser skin and eye safety requirements.
Additionally it depends on the framerate how accurately a
peak of the pulse wave can be sampled. Accurate sampling
of the pulse peak is important for the calculation of the HRV.
Therefore a trade-off between framerate and illumination time
has to be found. For our experiments we have chosen to
evaluate a framerate 45 frames per second and the maximal
illumination time of 500 microseconds. With these settings,
we trade a higher peak sampling accuracy for slightly more
noise.
A. Contact PPG
The IR light of the ToF camera penetrates the tissue and is
partially reflected and absorbed. The absorbed part of the light
can be divided into two fractions, pulse independent absorption
and the pulse dependent absorption. The first fraction is caused
Fig. 2. Behaviour of oxygenated blood (O2Hb) and oxygen depleted blood
(HHb) under different wavelengths. Obtained with changes from [44].
(a) (b) (c)
Fig. 3. Examples to demonstrate how the amplitude image changes without
a finger (a), with a finger having an increased blood volume (b) and a finger
having a decreased blood volume (c). Pseudo colours are used to highlight
the amplitude intensity changes.
Fig. 4. Measurement with contact PPG over ten seconds. The yellow area
highlights a single pulse cycle. Up to the first peak it is called systolic phase.
The rest of the wave is the diastolic phase.
by the absorption of tissue, bone and venous blood [45].
This fraction has close to no variations and is disregarded.
The second fraction varies, depending on whether a pulse has
occurred or not. If a pulse takes place, the pulsating arterial
blood absorbs more IR light, as the blood volume increases.
Oxygenated blood absorbs more IR light than oxygen de-
pleted blood at 850 nanometres wavelength. This absorption
behaviour of different wavelengths is shown in Fig. 2. Because
of this effect we have inverted the signal and now consider the
high points instead low points as it is more intuitive. Fig. 3
shows how the light is reflected from the vessels and bones
in a human finger. For better visualization we use a pseudo
colouring scheme to make the intensity changes better visible.
In (a) the amplitude image without a finger is illustrated. In
(b) and (c) a finger is placed on the camera and covers the
whole sensor. A pulse with fresh, oxygen enriched blood from
the heart causes a higher blood volume in the arteries, which
results in increased absorption. More absorption means that
less light is reflected and arrives at the sensor, which results
in a darker image. When the oxygen depleted blood volume
lowers at the end of the pulse cycle, less IR light is absorbed,
which results in a slightly brighter image.
As our ToF camera is limited to reflective PPG, the tissue
has to be placed right on top of the illumination unit and the
receiver. Important for good results is a complete coverage of
the receiver with the tissue. Several body locations were tested,
and several possible placements were found. As reported in
literature, a good signal was found in body parts such as
toes, fingers and ear lobes. However, it was found to be the
most stable configuration to use one finger to cover both, the
illumination unit and the receiver. The finger has a rich arterial
blood supply and the sensor can be fixated with relative ease,
which supports to reduce motion errors. We are using the
ToF amplitude data for our method. It can be described as a
grayscale image that is derived from the active IR illumination
and independent of passive illumination.
Given the amplitude image, the pulse wave is calculated
as the mean intensity of all pixels in every frame. The result
of a 10 seconds lasting measurement is illustrated in Fig. 4.
The pulse wave is centred as the absolute intensity values are
less important for heart rate and PRV analysis as the relative
changes at each time stamp. The ECG heart beat intervals are
similar to the peak distances of the pulse wave. The AHR,
SDNN and CVP R V can be derived from the pulse wave. To
calculate these measures, the peaks need to be detected in the
pulse wave.
B. Peak Detection
The yellow area in Fig. 4 highlights a single pulse cycle. The
first part of the signal up to the first line is the systolic phase.
The rest is part of the diastolic phase [46]. It can be seen that
the pulse cycle has two characteristic peaks, the first and most
prominent in the systolic phase and the second smaller one in
the diastolic phase. Only the peak of the systolic phase is used
for the calculation of AHR, SDNN and CVP R V . To suppress
the peak of the diastolic phase, an empirically determined
factor of 0.4 is multiplied to the average peak high of the
systolic phase. This determines the threshold for the final peak
detection. In most cases, the small peaks of the diastolic phase,
will not reach 40% of the height of a systolic peak.
Within the first run over the whole signal, the potential high
points are marked which passes the threshold. For a more
stable detection also a threshold on time stamps is used. Each
incoming frame has time stamp. Within a second run over the
signal, an arithmetic mean value ∆ttemp is calculated over all
delta time stamps
∆t
i=ti+1 −ti,(4)
where tiis the time of the current peak and ti+1 of the
next peak. With this temporal mean value and the dynamic
amplitude threshold, we sort out unwanted peaks in our PPG
signal. The time difference between the last peak and the
new potential peak must be at least 60% of the temporal
mean value. After sorting out unwanted peaks, the real mean
value (∆t) can be calculated from the remaining peaks. With
the remaining peaks and the corresponding time stamps, the
AHR, SDNN and CVP R V are calculated. The equations for
these quantitative measures are given in Section II. Once the
pulse wave is extracted with the remote PPG method the
same detection can be applied and the measurement values
calculated.
C. Remote PPG
In order to extract the pulse wave from a ToF amplitude
video of a subject’s face, a multi-step process is used. The
amplitude data contains regions in which a pulse wave can be
located. These regions consist of subsections on the human
Fig. 5. Sample pipeline of the remote PPG signal extraction algorithm. The input consist of a ToF video and will be spatially pre-filtered to remove the
background and camera noise. For the extraction of the pulse wave the amplitude data will be used. Every pixel in the pre-filtered video will be regarded as
a signal over time and analysed for its potential to contain a pulse wave. Periodicity and pulse wave shape characteristics are used to create three heat maps.
By edge suppression, energy equalization followed by combining of the three heat maps, one heat map is created. By thresholding and iterative adjustment of
the threshold, the final mask is extracted. The signal is recovered by BSS or averaging. The final pulse wave is selected by periodicity and pulse wave shape
characteristics analysis.
skin but not the whole. It can be explained by light absorption
properties being skin region dependent [47]. By considering
every pixel in the video to be a signal over time and analysing
it for its potential to contain a sufficiently clear pulse wave,
a mask is created. The analysis is based on signal periodicity
and pulse wave shape characteristics. Passband signal filtering
is used to limit the influence of noise and movement artefacts.
This calculated mask is used for signal localisation and the
selection of a subset of time series for further processing. To
recover the pulse wave from the bundle of selected signals,
BSS is used. As the best pulse signal does not need to be
contained in the first component, the signal with the highest
score on signal periodicity and pulse wave shape characteris-
tics, is selected for peak detection and heart rate calculation.
A summary of the algorithms pipeline is shown in Fig. 5.
1) Image Filtering: To counteract the inherently noisy ToF
image, spatial filtering is used. The box filter with a kernel
size of 5x5 was selected. To reduce calculation time, the ToF
depth, noise and confidence data is used to pre-filter the video.
The noise and confidence data is used to filter the frames
for regions with high confidence and low noise. Additionally
the median of the depth value of all non-zero measurements
is calculated and half a meter around is selected to be the
subject. This simplification is justified by the requirement that
the subject has to be close enough to the ToF camera to fill
most of the image. Alternatively, face detection could be used
to determine the average position of the subject.
2) Signal Filtering: As previously discussed, PPG is very
prone to motion errors. To further reduce sensor noise and
motion influences, temporal filtering will be used to improve
the signal. As discussed in Section II, the heart rate range
of interest is limited to [45, 200]beats/min which translates
to [0.75, 3]Hz. In [30], it is mentioned that frequencies
lower than 0.75 Hz negatively affect the algorithm. It was
observed, that those frequencies tend to correlate with slow
Fig. 6. Periodicity analysis of a passband filtered contact PPG pulse wave and
comparison to a sawtooth signal with same average peak width. The upper
plot visualizes the analysis by autocorrelation. The middle plot visualizes the
analysis with the spectral power density estimate. The lowest plot shows the
reference pulse wave and sawtooth signal.
movements of the subject, such as breathing. It is further
suggested to choose a passband in the range of [0.75, 5]Hz
as higher frequencies contain information needed for peak
detection [30]. Given this argument it was decided to filter
with an ideal filter and a passband of [0.75, 5].
3) Signal Periodicity Analysis: For final pulse wave selec-
tion and pulse localisation analysis, the signals are analysed for
certain properties. One property is periodicity. A measure for
periodicity is calculated, by using the autocorrelation function.
To reduce the function to one descriptive value the maximum
value after the first zero crossing is selected. When further
analysing the pulse wave, a similarity to a sawtooth signal
can be seen. For these kinds of signals, a strong frequency
response at several harmonic positions is characteristic. Due
to this property, the energy contained in the main frequency
component and its second harmonic, is used as a measure for
a pulse wave like signal [30], [42]. Two derived descriptive
values are used, once this contained energy is put into relation
to the overall contained peak energies, and once into relation
of the rest of the peak energies. It has to be noted that,
in order for both analysis to give good results, it is best
to centre the signal and filter out low, movement related,
frequencies. A visualisation of the analysis principle on a
passband filtered contact PPG pulse wave, in comparison to
an artificial sawtooth signal, can be seen in Fig. 6.
4) Signal Localisation: As it is not a sufficient approach to
select all or random parts of the subject’s skin, a pulse wave
signal localisation analysis is performed. As mentioned before,
every pixel in the video is considered to be a signal over time.
The signal periodicity analysis metrics are used to create three
heat maps. An example of the heat maps can be seen in Fig. 5.
A strong false positive response of the heat maps was observed
near edge regions. These regions are prone to strong changes
even due to slight motion. To suppress this error source the
Sobel gradient of the mean image is subtracted from the heat
maps. Before edge suppression and combining of the heat
maps, the energy contained in every heat map is adjusted to
be distributed around the mean. As the used metrics will also
respond to other signals than the pulse wave, all three resulting
heat maps are used as a way of false positive suppression.
The combined heat maps are thresholded to create the final
mask. To remove outliers from the heat maps, a median filter
is used before thresholding. To further address the problem of
threshold choice, a threshold of 95% is applied, and lowered
gradually until at least the required number of pixels was
selected, even after minor components were cleared from the
masks by morphological operations. The number of required
pixels is the number of desired components to be extracted
by the BSS. This results in the best region being accepted as
final mask.
5) Signal Extraction: Even after region selection and pre-
filtering to the range of intended frequencies, the signal is not
necessarily clear enough to be extracted directly by averaging
the selected signals. To recovery the pulse wave from the noisy
signals BSS is used. As BSS algorithm, PCA was selected.
However cases were observed in which the signal was clearly
recovered by averaging the selected signals, while BSS failed.
In other cases it was observed that with only the average, no
pulse wave could be extracted, while BSS was able to recover
it. In most cases of a successful extraction, the signal was
observed to be contained in the first four components. As the
pulse wave is not always contained in the first component,
the previously explained signal periodicity analysis metrics are
used to select the best pulse wave. It was decided to regard
the first six components as potential pulse waves. In order
to address the previously described problem the average of
the selected signals is included in the pool of potential pulse
waves. This emphasises the need for post selection of the
signal with the highest score on periodicity and pulse wave
shape characteristics. The BSS algorithm happens to extract
the correct and sometimes the inverted pulse wave. Prior to
peak detection and heart rate calculation the selected signal is
analysed for its characteristic pulse wave shape. If the inverted
signal scores higher it is selected as the final pulse wave.
V. RE SU LTS
Both approaches were designed in Matlab, while the contact
PPG approach was ported to Android. Also the designed
remote PPG method can be migrated to other platforms.
Results for both methods are provided in this section. The
tested ToF camera is a Pico Flexx model of PMD Technologies
AG. To show the accuracy of our methods AHR, SDNN and
CVP RV are used as a measure. Contact and remote PPG are
very sensitive to motion. Therefore the test subjects were asked
to remain relatively still while the measurements were taken.
For the contact PPG experiments the ToF camera is mounted
on the finger of the subject. Note that it is important not to
attach the sensor too firmly to the subjects finger, as pressure
influences the measurement. We use eight recordings of seven
subjects. The selected test subjects consisted of four males and
three females. The measurement was taken for an average of
one minute. As a reference the Polar H7 chest belt (with ECG
quality) and a pulse oximeter PO-250 from Pulox is used. In
Fig. 7 a comparison of a ToF pulse wave and a pulse oximeter
wave is given. It can be seen that the curves are a close match.
However for detailed evaluation in regard to AHR, SDNN and
CVH RV only the Polar H7 chest belt is used as a reference.
For the remote PPG experiments we evaluate a total of eigh-
teen recordings. The selected test subjects consisted of four
males and four females. An additional limitation for remote
PPG is, that the subjects need to remain close to the camera,
in a range of about 30cm to 40cm. An additional limitation
in relation to background illumination, such as sunlight, was
not found to be necessary. This can be explained by the
close distance to the sensor and the background illumination
suppressing properties of this ToF measurement principle. The
measurements are recorded for ten seconds, simultaneously
with two ToF cameras. One ToF camera records the face,
another ToF camera is securely mounted on the finger. The IR
light source of the second ToF camera is completely covered
by the finger, therefore interference between the ToF cameras
is not to be expected. The finger mounted ToF camera is
used to provide a ground truth pulse wave. The choice of
the reference measurement is based on several factors. As
the same light wavelength is used, the pulse wave signal
appearance is expected to be similar. For evaluation of pulse
wave form extraction quality we intended to utilize a sensor
Fig. 7. Comparison between pulse oximeter and ToF camera over 10 seconds.
The orange line shows the result of the pulse oximeter, the blue line shows
the ToF signal.
(a) AHR Error. (b) SDNN Error. (c) PRV Error.
Fig. 8. The comparison of the measured AHR, SDNN and PRV in relation to their selected reference measurement.
(a)
(b)
(c)
Fig. 9. Examples of three remote pulse waves (blue) in comparison to contact
pulse waves (orange). Depending on quality, the results are sorted into three
categories. In (a) good quality and in (b) medium quality wave is shown. In
(c) an example of a failure case is given.
providing a reference pulse wave, which the Polar H7 chest
belt does not simply provide. Additionally good accuracy
of the contact PPG method in reference to the provided
measurements of the chest belt, was shown. Good results were
achieved when analysing the amplitude data. For the sake of
understanding we categorized the results into three categories,
good quality, medium quality and failure case. In the test set
seven samples were selected to be good quality, eight more to
be medium quality and three failure cases. An example of the
three different quality stages is shown in Fig. 9. The results
show that an accurate heart rate can be extracted from the good
and medium quality signal. While mesures derived from the
PRV suffer much sooner from inaccuracies and also medium
quality signals will not provide sufficient results.
A detailed plot of the results for both methods is given in
Fig. 8. The average error and standard deviation, is given in
TABLE I
AVERA GE (AVG. ) ER ROR A ND STA NDAR D DE VIAT ION (SD) O F THE E RRO R
contact PPG remote PPG
Error Avg. SD Avg. SD
AHR [BPM] 0.33 0.26 3.46 5.22
SDNN [ms] 0.95 0.46 79.15 63.08
CVH RV [%] 0.13 0.09 8.77 7.37
Table I. From the quantitative measurements it can be seen
that the quality of the contact PPG method is sufficient for
personal assistance applications. However the remote PPG
method still requires improvements in the areas of motion
resistance and pulse peak extraction. Especially the PRV
measures derived from the remote PPG signals are currently
still quite erroneous and require improvement in the temporal
peak location extraction. From the results it can be seen that
the pulse wave is a lot harder to extract from remote ToF data,
than from contact data. The pulse wave cannot be found in
every part of the exposed skin. It is argued that several factors
may influence the signal to noise ration. Factors found to have
an influence are, the distance to the camera, the incident angle
on the skin, the exposed skin regions and the skin of the
subject itself even amongst subjects with similar skin colour. A
difference in light wavelengths efficiency in rgbPPG in relation
to different skin colours was reported [48]. However it was
suggested that differences in optical properties can even be
subject dependent or temperature related [49]. To the best of
our knowledge no study exists which examines the differences
in remote PPG response between subjects of the same skin
colour. Further research is needed. Additionally, it can be
observed that, depending on the strength of the N-N interval
variations the spectral power analysis may display more than
one major frequency peak. This leads to a less discriminative
results, while the autocorrelation is not as strongly affected.
Nevertheless, in case the signal localisation or pulse wave
selection fails, it can either be that no signal was found in
the subjects skin or periodicity analysis failed in favour of
other periodic movements such as repeated swallowing or hair
moment in the wind.
In fifteen out of eighteen measurements, the method was
observed to be able to localise a clear enough signal. The
method works best, if one carotid artery is exposed, as can be
seen Fig. 1. However, signal localisation analysis was observed
to find a signal up until the superficial temporal artery and also
in the cheeks and the forehead.
VI. SUMMARY
This paper has explored the use of a Pico Flexx ToF camera
of PMD Technologies AG for contact and remote PPG. Two
methods are introduced, to successfully measure the human
heart rate with contact and remote PPG. Contact PPG further
shows high accuracy in respect to HRV measures. Our findings
show that contact and remote PPG on a ToF camera are
feasible but also very sensitive to motion. For remote PPG, an
additional limitation is that the subjects need to remain close
to the camera. It is shown that it works best if one carotid
artery is exposed, but the approach is not limited to it. One of
the largest benefits of IR base PPG methods is the usability
in the dark without the need of visible light. This qualifies
the method to be used for night time driver monitoring and
sleep monitoring. Improvements that we may address in future
work include the increase of motion robustness, either in
the described method or in a derived method. We further
intend to explore different approaches for calculating the PRV
measures, which are less susceptible to small errors in the
remote measurement. Additionally, we aim at a subsequent
study with a larger number of subjects including different skin
tones and an evaluation of processing time.
ACK NOW LE DG ME NT S
The authors would like to thank the European Commission
and its Horizon 2020 program, which funded the TrustVehicle
project under the grant agreement n◦723324.
REFERENCES
[1] K. H. Shelley, “Photoplethysmography: Beyond the Calculation of
Arterial Oxygen Saturation and Heart Rate,” Anesth. Analg., vol. 105,
no. 6, pp. 31–36, Dec. 2007.
[2] A. B. Hertzman, “Photoelectric Plethysmography of the Fingers and
Toes in Man,” Proc. Soc. Exp. Biol. Med., vol. 37, no. 3, pp. 529–534,
Dec. 1937.
[3] W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmo-
graphic imaging using ambient light.” Opt. Express, vol. 16, no. 26, pp.
21 434–21 445, Dec. 2008.
[4] F. P. Wieringa, F. Mastik, and A. F. van der Steen, “Contactless multiple
wavelength photoplethysmographic imaging: a first step toward SpO 2
camera technology,” Ann. Biomed. Eng., vol. 33, no. 8, pp. 1034–1041,
Aug. 2005.
[5] R. Zangrniz, A. Martnez-Rodrigo, M. T. Lpez, J. M. Pastor, and
A. Fernndez-Caballero, “Estimation of Mental Distress from Photo-
plethysmography,” Appl. Sci., vol. 8, no. 1, p. 69, Jan. 2018.
[6] G. Li and W.-Y. Chung, “Detection of Driver Drowsiness Using Wavelet
Analysis of Heart Rate Variability and a Support Vector Machine
Classifier,” Sensors, vol. 13, no. 12, pp. 16 494–16 511, Dec. 2013.
[7] F. Shaffer and J. P. Ginsberg, “An overview of heart rate variability
metrics and norms,” Frontiers in Public Health, vol. 5, p. 258, 2017.
[8] H. Park, S. Oh, and M. Hahn, “Drowsy Driving Detection Based on
Human Pulse Wave by Photoplethysmography Signal Processing,” in
Proc. 3rd Int. Univer. Commun. Symp., Dec. 2009, pp. 89–92.
[9] M.-C. Li and Y.-H. Lin, “A real-time non-contact pulse rate detector
based on smartphone,” in International Symposium on Next-Generation
Electronics (ISNE). IEEE, May 2015, pp. 1–3.
[10] D. McDuff, C. Hurter, and M. Gonzalez-Franco, “Pulse and Vital Sign
Measurement in Mixed Reality Using a HoloLens,” in Proc. ACM Symp.
Virtual Real. Softw. Technol. ACM, Nov. 2017, pp. 34:1–34:9.
[11] V. Jeyhani, S. Mahdiani, M. Peltokangas, and A. Vehkaoja, “Comparison
of hrv parameters derived from photoplethysmography and electrocar-
diography signals,” in Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug.
2015, pp. 5952–5955.
[12] M. Elgendi, “On the Analysis of Fingertip Photoplethysmogram Sig-
nals,” Curr. Cardiol. Rev., vol. 8, no. 1, pp. 14–25, Feb. 2012.
[13] G. Lu and F. Yang, “Limitations of oximetry to measure heart rate
variability measures,” Cardiovasc. Eng., vol. 9, no. 3, pp. 119–125, 2009.
[14] N. Pinheiro, R. Couceiro, J. Henriques, J. Muehlsteff, I. Quintal,
L. Gonc¸alves, and P. Carvalho, “Can PPG be used for HRV analysis?”
in Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug. 2016, pp. 2945–2949.
[15] P. Babington, Praktische Elektrokardiographie und Elektrophysiologie
des Herzens: ein diagnostischer und therapeutischer Leitfaden f¨
ur
Studenten und ¨
Arzte, 1st ed. Stuttgart: Gustav Fischer, 1988.
[16] W. Micheo, E. Amy, and J. Correa, “Laboratory Tests and Diagnostic
Imaging,” in Clinical Sports Medicine: Medical Management and Re-
habilitation, Frontera, Walter R. and Herring, Stanley A. and Micheli,
Lyle J. and Silver, Julie K. and Young, Timothy P., Ed. Saunders,
Philadelphia, PA, 2007, pp. 187–191.
[17] H. Tanaka, K. D. Monahan, and D. R. Seals, “Age-predicted maximal
heart rate revisited,” J. Am. Coll. Cardiol., vol. 37, no. 1, pp. 153–156,
Jan. 2001.
[18] J. Lee, K. Matsumura, K.-i. Yamakoshi, P. Rolfe, S. Tanaka, and
T. Yamakoshi, “Comparison between red, green and blue light reflection
photoplethysmography for heart rate monitoring during motion,” in Conf.
Proc. IEEE Eng. Med. Biol. Soc., Jul. 2013, pp. 1724–1727.
[19] J. Parak and I. Korhonen, “Evaluation of wearable consumer heart rate
monitors based on photopletysmography,” in Conf. Proc. IEEE Eng.
Med. Biol. Soc., Aug. 2014, pp. 3670–3673.
[20] D. Grimaldi, Y. Kurylyak, F. Lamonaca, and A. Nastro, “Photo-
plethysmography detection by smartphone’s videocamera,” in IEEE
International Conference on Intelligent Data Acquisition and Advanced
Computing Systems (IDAACS), vol. 1, Sep. 2011, pp. 488–491.
[21] J. Liu, B. P.-Y. Yan, W.-X. Dai, X.-R. Ding, Y.-T. Zhang, and N. Zhao,
“Multi-wavelength photoplethysmography method for skin arterial pulse
extraction,” Biomed. Opt. Express, vol. 7, no. 10, pp. 4313–4326, 2016.
[22] N. Constant, T. Wang, and K. Mankodiya, “Pulseband: A hands-on
tutorial on how to design a smart wristband to monitor heart-rate,”
in IEEE Virtual Conference on Applications of Commercial Sensors
(VCACS), Mar. 2015, pp. 1–3.
[23] Y. Sun and N. Thakor, “Photoplethysmography revisited: from contact to
noncontact, from point to imaging,” IEEE Trans. Biomed. Eng., vol. 63,
no. 3, pp. 463–477, Mar. 2016.
[24] D. J. McDuff, J. R. Estepp, A. M. Piasecki, and E. B. Blackford, “A
survey of remote optical photoplethysmographic imaging methods,” in
Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug. 2015, pp. 6398–6404.
[25] M. Hassan, A. Malik, D. Fofi, N. Saad, B. Karasfi, Y. Ali, and
F. Meriaudeau, “Heart rate estimation using facial video: A review,”
Biomed. Signal Process. Control., vol. 38, pp. 346–360, Sep. 2017.
[26] P. V. Rouast, M. T. P. Adam, R. Chiong, D. Cornforth, and E. Lux,
“Remote heart rate measurement using low-cost RGB face video: a
technical literature review,” Front. Comput. Sci., Dec. 2017.
[27] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. T.
Freeman, “Eulerian Video Magnification for Revealing Subtle Changes
in the World,” ACM Trans. Graph, vol. 31, no. 4, pp. 1–8, Jul. 2012.
[28] A. Al-Naji and J. Chahl, “Detection of Cardiopulmonary Activity and
Related Abnormal Events Using Microsoft Kinect Sensor,” Sensors,
vol. 18, no. 3, p. 920, Mar. 2018.
[29] T. Smilkstein, M. Buenrostro, A. Kenyon, M. Lienemann, and G. Larson,
“Heart rate monitoring using Kinect and color amplification,” in Proc.
IEEE Healthcare Innovation Conf. (HIC). IEEE, Oct. 2014, pp. 60–62.
[30] G. Balakrishnan, F. Durand, and J. Guttag, “Detecting Pulse from Head
Motions in Video,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis.
Pattern. Recognit., Jun. 2013, pp. 3430–3437.
[31] M. Garbey, N. Sun, A. Merla, and I. Pavlidis, “Contact-Free Measure-
ment of Cardiac Pulse Based on the Analysis of Thermal Imagery,”
IEEE Trans. Biomed. Eng., vol. 54, no. 8, pp. 1418–1426, Aug. 2007.
[32] M. Hu, G. Zhai, D. Li, Y. Fan, H. Duan, W. Zhu, and X. Yang,
“Combination of near-infrared and thermal imaging techniques for the
remote and simultaneous measurements of breathing and heart rates
under sleep situation,” PloS one, vol. 13, no. 1, pp. 1–14, Jan. 2018.
[33] M. Lindelw and A. Lindqvist, “Remote heart rate extraction from near
infrared videos - An approach to heart rate measurements for the smart
eye head tracking system,” M.S. Thesis, Department of Signals and
Systems, Chalmers University of Technology, Gteborg, 2016.
[34] M. van Gastel, S. Stuijk, and G. de Haan, “Motion Robust Remote-PPG
in Infrared,” IEEE Trans. Biomed. Eng., vol. 62, no. 5, pp. 1425–1433,
May 2015.
[35] D. Shao, C. Liu, F. Tsow, Y. Yang, Z. Du, R. Iriya, H. Yu, and N. Tao,
“Noncontact Monitoring of Blood Oxygen Saturation Using Camera and
Dual-Wavelength Imaging System,” IEEE Trans. Biomed. Eng., vol. 63,
no. 6, pp. 1091–1098, Jun. 2016.
[36] Y.-C. Lin, N.-K. Chou, G.-Y. Lin, M.-H. Li, and Y.-H. Lin, “A Real-Time
Contactless Pulse Rate and Motion Status Monitoring System Based on
Complexion Tracking,” Sensors, vol. 17, no. 7, 2017.
[37] I. Bosi, C. Cogerino, and M. Bazzani, “Real-time monitoring of heart
rate by processing of Microsoft Kinect 2.0 generated streams,” in
International Multidisciplinary Conference on Computer and Energy
Science (SpliTech), Jul. 2016.
[38] L. Zhou, M. Yin, X. Xu, X. Yuan, and X. Liu, “Non-contact detection
of human heart rate with Kinect,” Cluster Comput., pp. 1–8, Jan. 2018.
[39] E. Gambi, A. Agostinelli, A. Belli, L. Burattini, E. Cippitelli, S. Fioretti,
P. Pierleoni, M. Ricciuti, A. Sbrollini, and S. Spinsante, “Heart Rate De-
tection Using Microsoft Kinect: Validation and Comparison to Wearable
Devices,” Sensors, vol. 17, no. 8, p. 1776, Aug. 2017.
[40] A. Proch ´
azka, M. Sch¨
atz, O. Vyˇ
sata, and M. Valiˇ
s, “Microsoft kinect
visual and depth sensors for breathing and heart rate analysis,” Sensors,
vol. 16, no. 7, p. 996, Jun. 2016.
[41] I. Bosi, C. Cogerino, and M. Bazzani. (2017, June) Real-Time
Monitoring of Heart Rate by Processing of Near Infrared Generated
Streams. Accessed on: Apr. 03, 2018. [Online]. Available: https://
www.researchgate.net/publication/318402304 Real-Time Monitoring
of Heart Rate by Processing of Near Infrared Generated Streams
[42] C. Yang, G. Cheung, and V. Stankovic, “Estimating Heart Rate and
Rhythm via 3D Motion Tracking in Depth Video,” IEEE Trans. Multi-
media, vol. 19, no. 7, pp. 1625–1636, Jul. 2017.
[43] N. Bernacchia, L. Scalise, L. Casacanditella, I. Ercoli, P. Marchionni,
and E. P. Tomasini, “Non contact measurement of heart and respiration
rates based on kinect,” in IEEE Int. Symp. Med. Meas. Appl., Jun. 2014,
pp. 1–5.
[44] E. D. Chan, M. M. Chan, and M. M. Chan, “Pulse oximetry: under-
standing its basic principles facilitates appreciation of its limitations,”
Respir. Med., vol. 107, no. 6, pp. 789–799, 2013.
[45] T. Tamura, Y. Maeda, M. Sekine, and M. Yoshida, “Wearable Photo-
plethysmographic SensorsPast and Present,” Electronics, vol. 3, no. 2,
pp. 282–302, 2014.
[46] B. M. Jayadevappa and M. S. Holi, “Photoplethysmography: Design,
Development, Analysis and Applications in Clinical and Physiological
Measurement - A Review,” Int. j. innov. res. sci. eng. technol., vol. 5,
no. 3, Mar. 2016.
[47] A. V. Moc¸o, S. Stuijk, and G. de Haan, “Skin inhomogeneity as a source
of error in remote ppg-imaging,” Biomed. Opt. Express, vol. 7, no. 11,
pp. 4718–4733, Nov. 2016.
[48] A. Kr ´
olak, “Influence of Skin Tone on Efficiency of Vision-Based
Heart Rate Estimation,” in Polish Conference on Biocybernetics and
Biomedical Engineering. Springer, Aug. 2017, pp. 44–55.
[49] T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human
skin,” J. Biomed. Opt., vol. 17, no. 9, pp. 1–14, 2012.