Conference PaperPDF Available

Exploring the Usage of Time-of-Flight Cameras for Contact and Remote Photoplethysmography

Authors:

Abstract and Figures

The heart beat is one of the basic vital signs, but the pulse wave can communicate much more information than the beat frequency of the heart. Through the heart rate variability (HRV) among other things, stress level and drowsiness can be inferred. Reliable HRV measurements are commonly obtained by electrocardiography (ECG). In this paper we analyse the correlation between HRV and pulse rate variability (PRV) obtained from contact or remote photoplethysmography (PPG) on a Time-of-Flight (ToF) camera from PMD Technologies AG. The ToF camera is independent of passive illumination as it uses an infrared (IR) light source, which is invisible to the human eye. Therefore, potential use cases include driver monitoring or convenient heart rate measurement with smart phones. We will demonstrate methods for using a ToF camera for contact and remote PPG. For contact PPG the sensor is mounted directly on the subjects body. The pulse wave is calculated as the mean amplitude intensity of all pixels. For remote PPG the sensor is mounted to measure the subjects face. The pulse wave cannot be found in every part of exposed skin. Therefore, the part of the skin with the clearest signal is localised and the pulse wave is recovered from there, by means of passband filtering and blind source separation. Our results show that contact PPG on a ToF camera has high accuracy and highly correlates with a commercial IR pulse oximetry device, as well as with an ECG grade chest belt. For remote PPG our method shows correlation with the ToF contact PPG signal.
Content may be subject to copyright.
Exploring the usage of Time-of-Flight Cameras for
contact and remote Photoplethysmography
Caterina Nahler, Bernhard Feldhofer, Matthias Ruether, Gerald Holwegand Norbert Druml
Infineon Technologies Austria AG, Graz, Austria
{caterina.nahler2, bernhard.feldhofer, gerald.holweg, norbert.druml}@infineon.com
Institute for Computer Graphics and Vision, Graz University of Technology, Austria
{ruether}@icg.tu-graz.ac.at
Abstract—The heart beat is one of the basic vital signs,
but the pulse wave can communicate much more information
than the beat frequency of the heart. Through the heart rate
variability (HRV) among other things, stress level and drowsiness
can be inferred. Reliable HRV measurements are commonly
obtained by electrocardiography (ECG). In this paper we analyse
the correlation between HRV and pulse rate variability (PRV)
obtained from contact or remote photoplethysmography (PPG)
on a Time-of-Flight (ToF) camera from PMD Technologies AG.
The ToF camera is independent of passive illumination as it uses
an infrared (IR) light source, which is invisible to the human
eye. Therefore, potential use cases include driver monitoring or
convenient heart rate measurement with smart phones. We will
demonstrate methods for using a ToF camera for contact and
remote PPG. For contact PPG the sensor is mounted directly
on the subjects body. The pulse wave is calculated as the mean
amplitude intensity of all pixels. For remote PPG the sensor is
mounted to measure the subjects face. The pulse wave cannot
be found in every part of exposed skin. Therefore, the part
of the skin with the clearest signal is localised and the pulse
wave is recovered from there, by means of passband filtering
and blind source separation. Our results show that contact PPG
on a ToF camera has high accuracy and highly correlates with
a commercial IR pulse oximetry device, as well as with an ECG
grade chest belt. For remote PPG our method shows correlation
with the ToF contact PPG signal.
Index Terms—Time-of-Flight, ToF, remote, contact, photo-
plethysmography, PPG, rPPG, infrared, pulse, heart beat
I. INTRODUCTION
The standard method for heart rate and vital sign observa-
tions is the electrocardiography (ECG). An alternative method
is the photoplethysmography (PPG). The blood oxygenation
and volume changes affect the light absorption characteristics
of living tissue. PPG is based on non-invasive measurement
of these light absorption changes. Contact sensors produce
a measurement, either by means of reflected or transmitted
light, while being loosely attached to the subject’s body [1].
Alrick Hertzman first described a setup to photoelectrically
measure the blood volume on fingers and toes in 1937 [2]. In
contrast to contact PPG, remote PPG relies mostly on reflective
measurement. One of the first versions of remote PPG to
measure the face was developed in 2008 [3]. Earlier imple-
mentations focused the camera on other parts of the human
body, such as the arm [4]. Nowadays PPG has already been
explored for many applications, including the measurement of
blood pressure and oxygenation [4], breathing cycles, mental
Fig. 1. A visualisation of the pulse wave analysis on the ToF amplitude video
data. Every pixel in the video is analysed for the likelihood of containing a
clear pulse wave. It can be seen that the carotid artery becomes visible as a
bright spot. This indicates, that a clear pulse wave signal is contained in that
region.
stress [5], drowsiness detection [6], sleep monitoring and heart
rhythm disturbances [7]. A recent area of research includes the
usage of contact and remote PPG for driver monitoring [6],
[8].
The Time-of-Flight (ToF) camera is independent of passive
illumination as it uses an active infrared (IR) light source,
that is invisible to the human eye. The camera, therefore has
potential for remote PPG also during night time. Most of
the mentioned use cases can benefit from this property. PPG
further enables convenient heart rate measurement with smart-
phones. Remote PPG variants using the front facing camera
exist [9]. Recently, remote PPG has also been integrated into
augmented reality. An application was developed which uses
the HoloLens and RGB data to measure and visualize another
human’s heart rate [10].
In this paper, we will explore the usage of the Pico Flexx
ToF camera from PMD Technologies AG for contact and
remote PPG. We contribute two methods which are, subse-
quently shown, to work on the data provide by the Pico Flexx
ToF camera. For contact PPG, the pulse wave is calculated as
the mean amplitude intensity of all pixels. For remote PPG
the sensor is mounted to measure the subjects face. The pulse
wave cannot be found in every part of exposed skin. Therefore,
we propose a novel combination approach to select the part of
the skin with the clearest signal. An example visualisation of
the signal localisation metrics is shown in Fig. 1. The pulse
wave is recovered from the selected skin region, by means of
passband filtering and blind source separation. The remainder
of the paper is structured as follows. Background information
will be provided in Section II. The overview on related work
is given in Section III. The method will be described in detail
in Section IV and the results including a short discussion will
be provided in Section V. A summary of the findings will
conclude this paper in Section VI.
II. BACKGROU ND
A. Pulse Measurement
Commonly known ways to measure the heart rate include
ECG and PPG. The typical heart beat signal which is known
from ECG is not the same as in PPG. The optically measured
signal is called a pulse wave. The R-R intervals (peak to
peak distances) in ECG measurements are closely correlated
with the N-N intervals (peak to peak distances) in PPG. The
Heart rate variability (HRV) is one of the most promising
markers of the autonomic nervous system. It is the variation
of time intervals between successive peaks, obtained from an
ECG signal [11]. However, the variation of peak intervals
obtained from a PPG signal is the pulse rate variability (PRV).
HRV measures were shown to highly correlated with PRV
measures [12]–[14]. To evaluate the accuracy of our methods,
several measures are extracted from the pulse wave. The av-
erage heart rate (AHR) is useful to extrapolate the heart beats
up to a minute from a short measurement. For this calculation
it is necessary to measure several peaks and calculate the
time interval between the peaks. The average value tis
then formed over all the extracted time intervals and with the
resulting value a prediction up to a whole minute
BPM = ∆t·60,(1)
where BPM means beats per minute, is possible.
The standard deviation of R-R intervals (SDRR) or respec-
tively N-N intervals (SDNN) can be calculated as
SDNN [s] = v
u
u
t
1
M1·
M
X
i=1
(∆t
it)2,(2)
where Mis the number of intervals counted in the measure-
ment and t
iare the time intervals between the peaks [7].
Another measure of the HRV is the coefficient of variation
(CV) [7]. In this paper, we are using the CV obtained from
the PPG signal (CVP RV ) to compare our method with the
CV obtained from a Polar H7 (CVH RV ) which claims to have
ECG quality. We calculate the CV according to the standard
equation. Therefore, CVP RV is calculated as
CVPRV [%] = SDNN [s]
t[s]·100[%].(3)
Among other information, it is possible to infer stress
level [5], breathing cycles, heart rhythm disturbances [7] and
risk for sudden heart failure, from the HRV. These values can
only be calculated if the measurement is very accurate. In
healthy subjects at rest, HRV measures and PRV measures
have a high correlation. However as PPG is quite vulnerable
to motion artefacts, the reliability of the PRV measurement
under motion is limited in comparison to ECG.
B. Heart Rate
The normal resting heart rate of a healthy human adult was
traditionally defined as in the range of [60, 100]beats/min.
Bradycardia (bellow 60 beats/min) and Tachycardia (above
100 beats/min) are considered pathological heart rates [15].
The range of possible heart rates is not straightforward to
define. Athletes can show a resting heart beat as low as
[30, 40]beats/min, which is known as athlete’s heart syn-
drome [16]. The highest heart rate a human can achieve during
excising, is strongly correlated with age. In young adults heart
rates up to 220 beats/min were measured [17]. Given all this
considerations, in conjunction with the filter limitations, this
work is considering a range of [45, 200]beats/min.
III. REL ATED WO RK
A. Contact PPG
PPG measurements with blue, red and near-IR lights are
possible [18]. Different light wavelengths perform differently
for different skin tones and situations (for example: measure-
ments under motion). Smartphone and smartwatch manufac-
turers are also using PPG technology [19], [20]. Contact PPG
is widely used in smartwatches to obtain the heart rate. For
example, smartwatches as the Mio Alpha and forearm devices
as Schosche Rhythm are using green or IR light to detect the
pulse on the wrist and arm [19]. Jihyoung Lee and his team
claim that green light gives best results in case of motions [18].
Typical devices for blood oxygen saturation measurements are
using a combination of red and IR light due to the deep tissue
penetration [21]. Many of these devices use the transmission
plethysmography, i.e. emitter and detector are located opposite
each other. In our approach we are using a ToF camera with
reflective plethysmography, i.e. emitter and detector are on the
same side [1]. Bone, various tissues, arterial and venous blood
contribute amongst other influences in reflecting the emitted
active IR illumination back into the ToF photodiodes [22].
Typical placements for measuring the pulse are the fingers,
toes, wrist and ear lobes [23].
B. Remote PPG
In remote PPG the sensor is not attached to the subject
but measures parts of the skin or the face remotely. A
lot of research was done on extracting the heart rate from
facial colour videos (rgbPPG). Extensive reviews on the used
rgbPPG methods exist [24]–[26]. The reviews report methods
using multi spectral, additional wavelengths such as orange
and cyan [24], high quality cameras to consumer grade web
cams [26]. The mentioned methods differ in the selection of
regions of interest (ROI). The forehead, cheeks, lips and other
parts up to the full face were tried as ROI [25]. In addition,
methods for skin region selection in colour videos exist [26].
To reduce motion artefacts and noise and reconstruct the PPG
signal more truthfully, many of the reported methods com-
bine the colour channels. For combining the colour channels
derived weighting schemes or blind source separation (BSS)
were used. In the reviews mentioned BSS methods include
versions of the Independent Component Analysis (ICA) or
Principal Component Analysis (PCA). As another method
of denoising, passband filters, stochastic filters and wavelet
analysis are used, in combination with BSS or separate.
Depending on the method, filters were applied before and
after the colour channel combination. The heart rate was
then derived either by peak extraction or by spectral density
estimates such as fast Fourier transform (FFT) [26]. General
video magnification algorithm were also suggested to be used
to determine the heart rate [27], or amplify the signal before
further processing [28]. A video magnification algorithm was
used for real-time visualization of the heart rate on Microsoft
Kinect camera data [29]. For extraction of the pulse wave
rgbPPG is not limited to skin reflection changes in the video.
It is shown that heart rate can be extracted from slight
head movements due to the pressure variations in the carotid
arteries [30]. However rgbPPG methods which rely on passive
illumination, are prone to errors due to spatial and temporal
illumination variations. Several approaches were evaluated to
algorithmically improve the temporal illumination variation
resistance of rgbPPG [24]–[26].
Nevertheless, methods that use passive illumination, will not
work in the dark. Thermal cameras and IR cameras are known
to work well in the dark. Methods using thermal cameras [31],
thermal cameras in combination with IR cameras [32] and
solely IR cameras have been used for remote heart rate
estimation. An approach was presented which uses a single
wavelength IR camera to estimate the heart rate in the face by
spectrum analysis [33]. The benefit of rgbPPG is the availabil-
ity of more than one wavelength to reduce noise by putting
the different measurements into reference. A similar approach
was explored by using three different IR wavelengths [34].
Remote pulse oxygenation measurement on the face is also
demonstrated using orange and IR wavelengths [35]. It was
further shown that real time remote PPG measurement with
peak detection using an IR camera is possible on a FPGA [36].
The Microsoft Kinect RGBD camera has also been explored
for its use in heart rate measurement. Several approaches
utilize the RGB camera of the Kinect for this purpose. The
dominant frequencies in the spectral density estimate are
used to estimate the heart rate, which is not uncommon for
remote PPG algorithm [37]–[39]. More heart rate estimation
approaches were presented using the amplitude data of the
Kinect depth camera and a spectral density estimate [40], [41].
As our approach aims at recovering the pulse peak locations,
including N-N intervals, this heart rate estimation technique
is not enough. Another method uses the head movement to
extract the heart rate [42], but in comparison to rgbPPG
methods the depth data is used for motion tracking. Our
method also uses solely a depth camera, but in contrast, we
utilize the amplitude data to track the skin absorption changes.
Further methods where proposed which track the heart rate
and the respiratory rate simultaneously. An early method
required the subjects to lie down and expose the chest during
measurement. The measurement regions included the carotid
artery, however the regions were selected manually [43]. In
contrast our method selects the region automatically and does
not necessarily choose the carotid artery. A later approach
estimated both vital signs simultaneously from a spectral
density estimate [40]. One of the most recent methods tracks
both vital signs using RGB or IR data, depending on the
lightning conditions, for abnormality monitoring during sleep.
In contrast to our method the vital sings are extracted from
the abdominal-thoracic region which has to be visible. While
we extract the heart rate from the head and throat.
IV. MET HO D
In this section we will describe the contact and remote
methods used to measure the pulse wave with a Pico Flexx
ToF camera. This ToF camera uses the phase difference
between pulsed IR light and its reflection to measure depth.
To remove depth ambiguities, several phase shifted measure-
ments (phases) are used. ToF cameras can operate on different
use-case settings. A use-case setting consists of a predefined
framerate, illumination time and number of phases used to
calculate the depth and amplitude image. The quality of the
signal improves with higher illumination time. However, if the
phases or the illumination time go up, the framerate has to go
down, to adhere the laser skin and eye safety requirements.
Additionally it depends on the framerate how accurately a
peak of the pulse wave can be sampled. Accurate sampling
of the pulse peak is important for the calculation of the HRV.
Therefore a trade-off between framerate and illumination time
has to be found. For our experiments we have chosen to
evaluate a framerate 45 frames per second and the maximal
illumination time of 500 microseconds. With these settings,
we trade a higher peak sampling accuracy for slightly more
noise.
A. Contact PPG
The IR light of the ToF camera penetrates the tissue and is
partially reflected and absorbed. The absorbed part of the light
can be divided into two fractions, pulse independent absorption
and the pulse dependent absorption. The first fraction is caused
Fig. 2. Behaviour of oxygenated blood (O2Hb) and oxygen depleted blood
(HHb) under different wavelengths. Obtained with changes from [44].
(a) (b) (c)
Fig. 3. Examples to demonstrate how the amplitude image changes without
a finger (a), with a finger having an increased blood volume (b) and a finger
having a decreased blood volume (c). Pseudo colours are used to highlight
the amplitude intensity changes.
Fig. 4. Measurement with contact PPG over ten seconds. The yellow area
highlights a single pulse cycle. Up to the first peak it is called systolic phase.
The rest of the wave is the diastolic phase.
by the absorption of tissue, bone and venous blood [45].
This fraction has close to no variations and is disregarded.
The second fraction varies, depending on whether a pulse has
occurred or not. If a pulse takes place, the pulsating arterial
blood absorbs more IR light, as the blood volume increases.
Oxygenated blood absorbs more IR light than oxygen de-
pleted blood at 850 nanometres wavelength. This absorption
behaviour of different wavelengths is shown in Fig. 2. Because
of this effect we have inverted the signal and now consider the
high points instead low points as it is more intuitive. Fig. 3
shows how the light is reflected from the vessels and bones
in a human finger. For better visualization we use a pseudo
colouring scheme to make the intensity changes better visible.
In (a) the amplitude image without a finger is illustrated. In
(b) and (c) a finger is placed on the camera and covers the
whole sensor. A pulse with fresh, oxygen enriched blood from
the heart causes a higher blood volume in the arteries, which
results in increased absorption. More absorption means that
less light is reflected and arrives at the sensor, which results
in a darker image. When the oxygen depleted blood volume
lowers at the end of the pulse cycle, less IR light is absorbed,
which results in a slightly brighter image.
As our ToF camera is limited to reflective PPG, the tissue
has to be placed right on top of the illumination unit and the
receiver. Important for good results is a complete coverage of
the receiver with the tissue. Several body locations were tested,
and several possible placements were found. As reported in
literature, a good signal was found in body parts such as
toes, fingers and ear lobes. However, it was found to be the
most stable configuration to use one finger to cover both, the
illumination unit and the receiver. The finger has a rich arterial
blood supply and the sensor can be fixated with relative ease,
which supports to reduce motion errors. We are using the
ToF amplitude data for our method. It can be described as a
grayscale image that is derived from the active IR illumination
and independent of passive illumination.
Given the amplitude image, the pulse wave is calculated
as the mean intensity of all pixels in every frame. The result
of a 10 seconds lasting measurement is illustrated in Fig. 4.
The pulse wave is centred as the absolute intensity values are
less important for heart rate and PRV analysis as the relative
changes at each time stamp. The ECG heart beat intervals are
similar to the peak distances of the pulse wave. The AHR,
SDNN and CVP R V can be derived from the pulse wave. To
calculate these measures, the peaks need to be detected in the
pulse wave.
B. Peak Detection
The yellow area in Fig. 4 highlights a single pulse cycle. The
first part of the signal up to the first line is the systolic phase.
The rest is part of the diastolic phase [46]. It can be seen that
the pulse cycle has two characteristic peaks, the first and most
prominent in the systolic phase and the second smaller one in
the diastolic phase. Only the peak of the systolic phase is used
for the calculation of AHR, SDNN and CVP R V . To suppress
the peak of the diastolic phase, an empirically determined
factor of 0.4 is multiplied to the average peak high of the
systolic phase. This determines the threshold for the final peak
detection. In most cases, the small peaks of the diastolic phase,
will not reach 40% of the height of a systolic peak.
Within the first run over the whole signal, the potential high
points are marked which passes the threshold. For a more
stable detection also a threshold on time stamps is used. Each
incoming frame has time stamp. Within a second run over the
signal, an arithmetic mean value ttemp is calculated over all
delta time stamps
t
i=ti+1 ti,(4)
where tiis the time of the current peak and ti+1 of the
next peak. With this temporal mean value and the dynamic
amplitude threshold, we sort out unwanted peaks in our PPG
signal. The time difference between the last peak and the
new potential peak must be at least 60% of the temporal
mean value. After sorting out unwanted peaks, the real mean
value (t) can be calculated from the remaining peaks. With
the remaining peaks and the corresponding time stamps, the
AHR, SDNN and CVP R V are calculated. The equations for
these quantitative measures are given in Section II. Once the
pulse wave is extracted with the remote PPG method the
same detection can be applied and the measurement values
calculated.
C. Remote PPG
In order to extract the pulse wave from a ToF amplitude
video of a subject’s face, a multi-step process is used. The
amplitude data contains regions in which a pulse wave can be
located. These regions consist of subsections on the human
Fig. 5. Sample pipeline of the remote PPG signal extraction algorithm. The input consist of a ToF video and will be spatially pre-filtered to remove the
background and camera noise. For the extraction of the pulse wave the amplitude data will be used. Every pixel in the pre-filtered video will be regarded as
a signal over time and analysed for its potential to contain a pulse wave. Periodicity and pulse wave shape characteristics are used to create three heat maps.
By edge suppression, energy equalization followed by combining of the three heat maps, one heat map is created. By thresholding and iterative adjustment of
the threshold, the final mask is extracted. The signal is recovered by BSS or averaging. The final pulse wave is selected by periodicity and pulse wave shape
characteristics analysis.
skin but not the whole. It can be explained by light absorption
properties being skin region dependent [47]. By considering
every pixel in the video to be a signal over time and analysing
it for its potential to contain a sufficiently clear pulse wave,
a mask is created. The analysis is based on signal periodicity
and pulse wave shape characteristics. Passband signal filtering
is used to limit the influence of noise and movement artefacts.
This calculated mask is used for signal localisation and the
selection of a subset of time series for further processing. To
recover the pulse wave from the bundle of selected signals,
BSS is used. As the best pulse signal does not need to be
contained in the first component, the signal with the highest
score on signal periodicity and pulse wave shape characteris-
tics, is selected for peak detection and heart rate calculation.
A summary of the algorithms pipeline is shown in Fig. 5.
1) Image Filtering: To counteract the inherently noisy ToF
image, spatial filtering is used. The box filter with a kernel
size of 5x5 was selected. To reduce calculation time, the ToF
depth, noise and confidence data is used to pre-filter the video.
The noise and confidence data is used to filter the frames
for regions with high confidence and low noise. Additionally
the median of the depth value of all non-zero measurements
is calculated and half a meter around is selected to be the
subject. This simplification is justified by the requirement that
the subject has to be close enough to the ToF camera to fill
most of the image. Alternatively, face detection could be used
to determine the average position of the subject.
2) Signal Filtering: As previously discussed, PPG is very
prone to motion errors. To further reduce sensor noise and
motion influences, temporal filtering will be used to improve
the signal. As discussed in Section II, the heart rate range
of interest is limited to [45, 200]beats/min which translates
to [0.75, 3]Hz. In [30], it is mentioned that frequencies
lower than 0.75 Hz negatively affect the algorithm. It was
observed, that those frequencies tend to correlate with slow
Fig. 6. Periodicity analysis of a passband filtered contact PPG pulse wave and
comparison to a sawtooth signal with same average peak width. The upper
plot visualizes the analysis by autocorrelation. The middle plot visualizes the
analysis with the spectral power density estimate. The lowest plot shows the
reference pulse wave and sawtooth signal.
movements of the subject, such as breathing. It is further
suggested to choose a passband in the range of [0.75, 5]Hz
as higher frequencies contain information needed for peak
detection [30]. Given this argument it was decided to filter
with an ideal filter and a passband of [0.75, 5].
3) Signal Periodicity Analysis: For final pulse wave selec-
tion and pulse localisation analysis, the signals are analysed for
certain properties. One property is periodicity. A measure for
periodicity is calculated, by using the autocorrelation function.
To reduce the function to one descriptive value the maximum
value after the first zero crossing is selected. When further
analysing the pulse wave, a similarity to a sawtooth signal
can be seen. For these kinds of signals, a strong frequency
response at several harmonic positions is characteristic. Due
to this property, the energy contained in the main frequency
component and its second harmonic, is used as a measure for
a pulse wave like signal [30], [42]. Two derived descriptive
values are used, once this contained energy is put into relation
to the overall contained peak energies, and once into relation
of the rest of the peak energies. It has to be noted that,
in order for both analysis to give good results, it is best
to centre the signal and filter out low, movement related,
frequencies. A visualisation of the analysis principle on a
passband filtered contact PPG pulse wave, in comparison to
an artificial sawtooth signal, can be seen in Fig. 6.
4) Signal Localisation: As it is not a sufficient approach to
select all or random parts of the subject’s skin, a pulse wave
signal localisation analysis is performed. As mentioned before,
every pixel in the video is considered to be a signal over time.
The signal periodicity analysis metrics are used to create three
heat maps. An example of the heat maps can be seen in Fig. 5.
A strong false positive response of the heat maps was observed
near edge regions. These regions are prone to strong changes
even due to slight motion. To suppress this error source the
Sobel gradient of the mean image is subtracted from the heat
maps. Before edge suppression and combining of the heat
maps, the energy contained in every heat map is adjusted to
be distributed around the mean. As the used metrics will also
respond to other signals than the pulse wave, all three resulting
heat maps are used as a way of false positive suppression.
The combined heat maps are thresholded to create the final
mask. To remove outliers from the heat maps, a median filter
is used before thresholding. To further address the problem of
threshold choice, a threshold of 95% is applied, and lowered
gradually until at least the required number of pixels was
selected, even after minor components were cleared from the
masks by morphological operations. The number of required
pixels is the number of desired components to be extracted
by the BSS. This results in the best region being accepted as
final mask.
5) Signal Extraction: Even after region selection and pre-
filtering to the range of intended frequencies, the signal is not
necessarily clear enough to be extracted directly by averaging
the selected signals. To recovery the pulse wave from the noisy
signals BSS is used. As BSS algorithm, PCA was selected.
However cases were observed in which the signal was clearly
recovered by averaging the selected signals, while BSS failed.
In other cases it was observed that with only the average, no
pulse wave could be extracted, while BSS was able to recover
it. In most cases of a successful extraction, the signal was
observed to be contained in the first four components. As the
pulse wave is not always contained in the first component,
the previously explained signal periodicity analysis metrics are
used to select the best pulse wave. It was decided to regard
the first six components as potential pulse waves. In order
to address the previously described problem the average of
the selected signals is included in the pool of potential pulse
waves. This emphasises the need for post selection of the
signal with the highest score on periodicity and pulse wave
shape characteristics. The BSS algorithm happens to extract
the correct and sometimes the inverted pulse wave. Prior to
peak detection and heart rate calculation the selected signal is
analysed for its characteristic pulse wave shape. If the inverted
signal scores higher it is selected as the final pulse wave.
V. RE SU LTS
Both approaches were designed in Matlab, while the contact
PPG approach was ported to Android. Also the designed
remote PPG method can be migrated to other platforms.
Results for both methods are provided in this section. The
tested ToF camera is a Pico Flexx model of PMD Technologies
AG. To show the accuracy of our methods AHR, SDNN and
CVP RV are used as a measure. Contact and remote PPG are
very sensitive to motion. Therefore the test subjects were asked
to remain relatively still while the measurements were taken.
For the contact PPG experiments the ToF camera is mounted
on the finger of the subject. Note that it is important not to
attach the sensor too firmly to the subjects finger, as pressure
influences the measurement. We use eight recordings of seven
subjects. The selected test subjects consisted of four males and
three females. The measurement was taken for an average of
one minute. As a reference the Polar H7 chest belt (with ECG
quality) and a pulse oximeter PO-250 from Pulox is used. In
Fig. 7 a comparison of a ToF pulse wave and a pulse oximeter
wave is given. It can be seen that the curves are a close match.
However for detailed evaluation in regard to AHR, SDNN and
CVH RV only the Polar H7 chest belt is used as a reference.
For the remote PPG experiments we evaluate a total of eigh-
teen recordings. The selected test subjects consisted of four
males and four females. An additional limitation for remote
PPG is, that the subjects need to remain close to the camera,
in a range of about 30cm to 40cm. An additional limitation
in relation to background illumination, such as sunlight, was
not found to be necessary. This can be explained by the
close distance to the sensor and the background illumination
suppressing properties of this ToF measurement principle. The
measurements are recorded for ten seconds, simultaneously
with two ToF cameras. One ToF camera records the face,
another ToF camera is securely mounted on the finger. The IR
light source of the second ToF camera is completely covered
by the finger, therefore interference between the ToF cameras
is not to be expected. The finger mounted ToF camera is
used to provide a ground truth pulse wave. The choice of
the reference measurement is based on several factors. As
the same light wavelength is used, the pulse wave signal
appearance is expected to be similar. For evaluation of pulse
wave form extraction quality we intended to utilize a sensor
Fig. 7. Comparison between pulse oximeter and ToF camera over 10 seconds.
The orange line shows the result of the pulse oximeter, the blue line shows
the ToF signal.
(a) AHR Error. (b) SDNN Error. (c) PRV Error.
Fig. 8. The comparison of the measured AHR, SDNN and PRV in relation to their selected reference measurement.
(a)
(b)
(c)
Fig. 9. Examples of three remote pulse waves (blue) in comparison to contact
pulse waves (orange). Depending on quality, the results are sorted into three
categories. In (a) good quality and in (b) medium quality wave is shown. In
(c) an example of a failure case is given.
providing a reference pulse wave, which the Polar H7 chest
belt does not simply provide. Additionally good accuracy
of the contact PPG method in reference to the provided
measurements of the chest belt, was shown. Good results were
achieved when analysing the amplitude data. For the sake of
understanding we categorized the results into three categories,
good quality, medium quality and failure case. In the test set
seven samples were selected to be good quality, eight more to
be medium quality and three failure cases. An example of the
three different quality stages is shown in Fig. 9. The results
show that an accurate heart rate can be extracted from the good
and medium quality signal. While mesures derived from the
PRV suffer much sooner from inaccuracies and also medium
quality signals will not provide sufficient results.
A detailed plot of the results for both methods is given in
Fig. 8. The average error and standard deviation, is given in
TABLE I
AVERA GE (AVG. ) ER ROR A ND STA NDAR D DE VIAT ION (SD) O F THE E RRO R
contact PPG remote PPG
Error Avg. SD Avg. SD
AHR [BPM] 0.33 0.26 3.46 5.22
SDNN [ms] 0.95 0.46 79.15 63.08
CVH RV [%] 0.13 0.09 8.77 7.37
Table I. From the quantitative measurements it can be seen
that the quality of the contact PPG method is sufficient for
personal assistance applications. However the remote PPG
method still requires improvements in the areas of motion
resistance and pulse peak extraction. Especially the PRV
measures derived from the remote PPG signals are currently
still quite erroneous and require improvement in the temporal
peak location extraction. From the results it can be seen that
the pulse wave is a lot harder to extract from remote ToF data,
than from contact data. The pulse wave cannot be found in
every part of the exposed skin. It is argued that several factors
may influence the signal to noise ration. Factors found to have
an influence are, the distance to the camera, the incident angle
on the skin, the exposed skin regions and the skin of the
subject itself even amongst subjects with similar skin colour. A
difference in light wavelengths efficiency in rgbPPG in relation
to different skin colours was reported [48]. However it was
suggested that differences in optical properties can even be
subject dependent or temperature related [49]. To the best of
our knowledge no study exists which examines the differences
in remote PPG response between subjects of the same skin
colour. Further research is needed. Additionally, it can be
observed that, depending on the strength of the N-N interval
variations the spectral power analysis may display more than
one major frequency peak. This leads to a less discriminative
results, while the autocorrelation is not as strongly affected.
Nevertheless, in case the signal localisation or pulse wave
selection fails, it can either be that no signal was found in
the subjects skin or periodicity analysis failed in favour of
other periodic movements such as repeated swallowing or hair
moment in the wind.
In fifteen out of eighteen measurements, the method was
observed to be able to localise a clear enough signal. The
method works best, if one carotid artery is exposed, as can be
seen Fig. 1. However, signal localisation analysis was observed
to find a signal up until the superficial temporal artery and also
in the cheeks and the forehead.
VI. SUMMARY
This paper has explored the use of a Pico Flexx ToF camera
of PMD Technologies AG for contact and remote PPG. Two
methods are introduced, to successfully measure the human
heart rate with contact and remote PPG. Contact PPG further
shows high accuracy in respect to HRV measures. Our findings
show that contact and remote PPG on a ToF camera are
feasible but also very sensitive to motion. For remote PPG, an
additional limitation is that the subjects need to remain close
to the camera. It is shown that it works best if one carotid
artery is exposed, but the approach is not limited to it. One of
the largest benefits of IR base PPG methods is the usability
in the dark without the need of visible light. This qualifies
the method to be used for night time driver monitoring and
sleep monitoring. Improvements that we may address in future
work include the increase of motion robustness, either in
the described method or in a derived method. We further
intend to explore different approaches for calculating the PRV
measures, which are less susceptible to small errors in the
remote measurement. Additionally, we aim at a subsequent
study with a larger number of subjects including different skin
tones and an evaluation of processing time.
ACK NOW LE DG ME NT S
The authors would like to thank the European Commission
and its Horizon 2020 program, which funded the TrustVehicle
project under the grant agreement n723324.
REFERENCES
[1] K. H. Shelley, “Photoplethysmography: Beyond the Calculation of
Arterial Oxygen Saturation and Heart Rate,” Anesth. Analg., vol. 105,
no. 6, pp. 31–36, Dec. 2007.
[2] A. B. Hertzman, “Photoelectric Plethysmography of the Fingers and
Toes in Man,Proc. Soc. Exp. Biol. Med., vol. 37, no. 3, pp. 529–534,
Dec. 1937.
[3] W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmo-
graphic imaging using ambient light.” Opt. Express, vol. 16, no. 26, pp.
21 434–21 445, Dec. 2008.
[4] F. P. Wieringa, F. Mastik, and A. F. van der Steen, “Contactless multiple
wavelength photoplethysmographic imaging: a first step toward SpO 2
camera technology,” Ann. Biomed. Eng., vol. 33, no. 8, pp. 1034–1041,
Aug. 2005.
[5] R. Zangrniz, A. Martnez-Rodrigo, M. T. Lpez, J. M. Pastor, and
A. Fernndez-Caballero, “Estimation of Mental Distress from Photo-
plethysmography,Appl. Sci., vol. 8, no. 1, p. 69, Jan. 2018.
[6] G. Li and W.-Y. Chung, “Detection of Driver Drowsiness Using Wavelet
Analysis of Heart Rate Variability and a Support Vector Machine
Classifier,Sensors, vol. 13, no. 12, pp. 16 494–16 511, Dec. 2013.
[7] F. Shaffer and J. P. Ginsberg, “An overview of heart rate variability
metrics and norms,” Frontiers in Public Health, vol. 5, p. 258, 2017.
[8] H. Park, S. Oh, and M. Hahn, “Drowsy Driving Detection Based on
Human Pulse Wave by Photoplethysmography Signal Processing,” in
Proc. 3rd Int. Univer. Commun. Symp., Dec. 2009, pp. 89–92.
[9] M.-C. Li and Y.-H. Lin, “A real-time non-contact pulse rate detector
based on smartphone,” in International Symposium on Next-Generation
Electronics (ISNE). IEEE, May 2015, pp. 1–3.
[10] D. McDuff, C. Hurter, and M. Gonzalez-Franco, “Pulse and Vital Sign
Measurement in Mixed Reality Using a HoloLens,” in Proc. ACM Symp.
Virtual Real. Softw. Technol. ACM, Nov. 2017, pp. 34:1–34:9.
[11] V. Jeyhani, S. Mahdiani, M. Peltokangas, and A. Vehkaoja, “Comparison
of hrv parameters derived from photoplethysmography and electrocar-
diography signals,” in Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug.
2015, pp. 5952–5955.
[12] M. Elgendi, “On the Analysis of Fingertip Photoplethysmogram Sig-
nals,” Curr. Cardiol. Rev., vol. 8, no. 1, pp. 14–25, Feb. 2012.
[13] G. Lu and F. Yang, “Limitations of oximetry to measure heart rate
variability measures,Cardiovasc. Eng., vol. 9, no. 3, pp. 119–125, 2009.
[14] N. Pinheiro, R. Couceiro, J. Henriques, J. Muehlsteff, I. Quintal,
L. Gonc¸alves, and P. Carvalho, “Can PPG be used for HRV analysis?”
in Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug. 2016, pp. 2945–2949.
[15] P. Babington, Praktische Elektrokardiographie und Elektrophysiologie
des Herzens: ein diagnostischer und therapeutischer Leitfaden f¨
ur
Studenten und ¨
Arzte, 1st ed. Stuttgart: Gustav Fischer, 1988.
[16] W. Micheo, E. Amy, and J. Correa, “Laboratory Tests and Diagnostic
Imaging,” in Clinical Sports Medicine: Medical Management and Re-
habilitation, Frontera, Walter R. and Herring, Stanley A. and Micheli,
Lyle J. and Silver, Julie K. and Young, Timothy P., Ed. Saunders,
Philadelphia, PA, 2007, pp. 187–191.
[17] H. Tanaka, K. D. Monahan, and D. R. Seals, “Age-predicted maximal
heart rate revisited,” J. Am. Coll. Cardiol., vol. 37, no. 1, pp. 153–156,
Jan. 2001.
[18] J. Lee, K. Matsumura, K.-i. Yamakoshi, P. Rolfe, S. Tanaka, and
T. Yamakoshi, “Comparison between red, green and blue light reflection
photoplethysmography for heart rate monitoring during motion,” in Conf.
Proc. IEEE Eng. Med. Biol. Soc., Jul. 2013, pp. 1724–1727.
[19] J. Parak and I. Korhonen, “Evaluation of wearable consumer heart rate
monitors based on photopletysmography,” in Conf. Proc. IEEE Eng.
Med. Biol. Soc., Aug. 2014, pp. 3670–3673.
[20] D. Grimaldi, Y. Kurylyak, F. Lamonaca, and A. Nastro, “Photo-
plethysmography detection by smartphone’s videocamera,” in IEEE
International Conference on Intelligent Data Acquisition and Advanced
Computing Systems (IDAACS), vol. 1, Sep. 2011, pp. 488–491.
[21] J. Liu, B. P.-Y. Yan, W.-X. Dai, X.-R. Ding, Y.-T. Zhang, and N. Zhao,
“Multi-wavelength photoplethysmography method for skin arterial pulse
extraction,” Biomed. Opt. Express, vol. 7, no. 10, pp. 4313–4326, 2016.
[22] N. Constant, T. Wang, and K. Mankodiya, “Pulseband: A hands-on
tutorial on how to design a smart wristband to monitor heart-rate,”
in IEEE Virtual Conference on Applications of Commercial Sensors
(VCACS), Mar. 2015, pp. 1–3.
[23] Y. Sun and N. Thakor, “Photoplethysmography revisited: from contact to
noncontact, from point to imaging,” IEEE Trans. Biomed. Eng., vol. 63,
no. 3, pp. 463–477, Mar. 2016.
[24] D. J. McDuff, J. R. Estepp, A. M. Piasecki, and E. B. Blackford, “A
survey of remote optical photoplethysmographic imaging methods,” in
Conf. Proc. IEEE Eng. Med. Biol. Soc., Aug. 2015, pp. 6398–6404.
[25] M. Hassan, A. Malik, D. Fofi, N. Saad, B. Karasfi, Y. Ali, and
F. Meriaudeau, “Heart rate estimation using facial video: A review,”
Biomed. Signal Process. Control., vol. 38, pp. 346–360, Sep. 2017.
[26] P. V. Rouast, M. T. P. Adam, R. Chiong, D. Cornforth, and E. Lux,
“Remote heart rate measurement using low-cost RGB face video: a
technical literature review,” Front. Comput. Sci., Dec. 2017.
[27] H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. T.
Freeman, “Eulerian Video Magnification for Revealing Subtle Changes
in the World,ACM Trans. Graph, vol. 31, no. 4, pp. 1–8, Jul. 2012.
[28] A. Al-Naji and J. Chahl, “Detection of Cardiopulmonary Activity and
Related Abnormal Events Using Microsoft Kinect Sensor,Sensors,
vol. 18, no. 3, p. 920, Mar. 2018.
[29] T. Smilkstein, M. Buenrostro, A. Kenyon, M. Lienemann, and G. Larson,
“Heart rate monitoring using Kinect and color amplification,” in Proc.
IEEE Healthcare Innovation Conf. (HIC). IEEE, Oct. 2014, pp. 60–62.
[30] G. Balakrishnan, F. Durand, and J. Guttag, “Detecting Pulse from Head
Motions in Video,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis.
Pattern. Recognit., Jun. 2013, pp. 3430–3437.
[31] M. Garbey, N. Sun, A. Merla, and I. Pavlidis, “Contact-Free Measure-
ment of Cardiac Pulse Based on the Analysis of Thermal Imagery,
IEEE Trans. Biomed. Eng., vol. 54, no. 8, pp. 1418–1426, Aug. 2007.
[32] M. Hu, G. Zhai, D. Li, Y. Fan, H. Duan, W. Zhu, and X. Yang,
“Combination of near-infrared and thermal imaging techniques for the
remote and simultaneous measurements of breathing and heart rates
under sleep situation,” PloS one, vol. 13, no. 1, pp. 1–14, Jan. 2018.
[33] M. Lindelw and A. Lindqvist, “Remote heart rate extraction from near
infrared videos - An approach to heart rate measurements for the smart
eye head tracking system,” M.S. Thesis, Department of Signals and
Systems, Chalmers University of Technology, Gteborg, 2016.
[34] M. van Gastel, S. Stuijk, and G. de Haan, “Motion Robust Remote-PPG
in Infrared,” IEEE Trans. Biomed. Eng., vol. 62, no. 5, pp. 1425–1433,
May 2015.
[35] D. Shao, C. Liu, F. Tsow, Y. Yang, Z. Du, R. Iriya, H. Yu, and N. Tao,
“Noncontact Monitoring of Blood Oxygen Saturation Using Camera and
Dual-Wavelength Imaging System,” IEEE Trans. Biomed. Eng., vol. 63,
no. 6, pp. 1091–1098, Jun. 2016.
[36] Y.-C. Lin, N.-K. Chou, G.-Y. Lin, M.-H. Li, and Y.-H. Lin, “A Real-Time
Contactless Pulse Rate and Motion Status Monitoring System Based on
Complexion Tracking,Sensors, vol. 17, no. 7, 2017.
[37] I. Bosi, C. Cogerino, and M. Bazzani, “Real-time monitoring of heart
rate by processing of Microsoft Kinect 2.0 generated streams,” in
International Multidisciplinary Conference on Computer and Energy
Science (SpliTech), Jul. 2016.
[38] L. Zhou, M. Yin, X. Xu, X. Yuan, and X. Liu, “Non-contact detection
of human heart rate with Kinect,” Cluster Comput., pp. 1–8, Jan. 2018.
[39] E. Gambi, A. Agostinelli, A. Belli, L. Burattini, E. Cippitelli, S. Fioretti,
P. Pierleoni, M. Ricciuti, A. Sbrollini, and S. Spinsante, “Heart Rate De-
tection Using Microsoft Kinect: Validation and Comparison to Wearable
Devices,Sensors, vol. 17, no. 8, p. 1776, Aug. 2017.
[40] A. Proch ´
azka, M. Sch¨
atz, O. Vyˇ
sata, and M. Valiˇ
s, “Microsoft kinect
visual and depth sensors for breathing and heart rate analysis,” Sensors,
vol. 16, no. 7, p. 996, Jun. 2016.
[41] I. Bosi, C. Cogerino, and M. Bazzani. (2017, June) Real-Time
Monitoring of Heart Rate by Processing of Near Infrared Generated
Streams. Accessed on: Apr. 03, 2018. [Online]. Available: https://
www.researchgate.net/publication/318402304 Real-Time Monitoring
of Heart Rate by Processing of Near Infrared Generated Streams
[42] C. Yang, G. Cheung, and V. Stankovic, “Estimating Heart Rate and
Rhythm via 3D Motion Tracking in Depth Video,” IEEE Trans. Multi-
media, vol. 19, no. 7, pp. 1625–1636, Jul. 2017.
[43] N. Bernacchia, L. Scalise, L. Casacanditella, I. Ercoli, P. Marchionni,
and E. P. Tomasini, “Non contact measurement of heart and respiration
rates based on kinect,” in IEEE Int. Symp. Med. Meas. Appl., Jun. 2014,
pp. 1–5.
[44] E. D. Chan, M. M. Chan, and M. M. Chan, “Pulse oximetry: under-
standing its basic principles facilitates appreciation of its limitations,”
Respir. Med., vol. 107, no. 6, pp. 789–799, 2013.
[45] T. Tamura, Y. Maeda, M. Sekine, and M. Yoshida, “Wearable Photo-
plethysmographic SensorsPast and Present,” Electronics, vol. 3, no. 2,
pp. 282–302, 2014.
[46] B. M. Jayadevappa and M. S. Holi, “Photoplethysmography: Design,
Development, Analysis and Applications in Clinical and Physiological
Measurement - A Review,Int. j. innov. res. sci. eng. technol., vol. 5,
no. 3, Mar. 2016.
[47] A. V. Moc¸o, S. Stuijk, and G. de Haan, “Skin inhomogeneity as a source
of error in remote ppg-imaging,” Biomed. Opt. Express, vol. 7, no. 11,
pp. 4718–4733, Nov. 2016.
[48] A. Kr ´
olak, “Influence of Skin Tone on Efficiency of Vision-Based
Heart Rate Estimation,” in Polish Conference on Biocybernetics and
Biomedical Engineering. Springer, Aug. 2017, pp. 44–55.
[49] T. Lister, P. A. Wright, and P. H. Chappell, “Optical properties of human
skin,” J. Biomed. Opt., vol. 17, no. 9, pp. 1–14, 2012.
... ToF sensing is independent of passive illumination as it uses an active IR light source that is invisible to the human eyes. Due to this flexibility, the ToF camera can operate accurately during the nighttime [140]. The Microsoft Kinect ToF sensor has been widely used in recent research for the collection of real-time data. ...
... The time interval between the heartbeats is a potential indicator of myocardial infarction or a heart attack. Pico Flexx ToF camera was used by Nahler et al. [140] in their proposed study for HRV monitoring. In their framework, the pulse wave was recovered from the skin through blind source separation passband filtering. ...
... These studies used video cameras and ToF cameras for vital signs monitoring. A technique by Nahler et al. [140] used a ToF camera to estimate HRV, which achieved the accuracy as compared with a commercial IR pulse oximetry device. A ToF camera is independent of passive illumination because it uses an active IR light source, which is invisible to the human eye. ...
Article
The process of monitoring mental health has relied on methods such as invasive sensing and self-reporting. The use of these methods has been limited because of the invasiveness of sensing devices, or the subjective nature of patients' responses. Recent research focuses on the contactless sensing methods used to objectively monitor mental health issues. These methods allow continuous collection of real-time data in a non-disruptive manner. Machine learning methods are then applied to the sensed data to predict information such as physical activity, gestures, and heart rate. This information can be then used to assess mental health issues such as depression, stress, anxiety, among others. This paper presents a comprehensive review of contactless sensing methods for mental health monitoring. It investigates the published research that focuses on contactless sensing methods to predict mental health condition. Moreover, this review categorizes the applications of contactless sensing methods into detection, recognition, and monitoring of vital signs. Furthermore, a comparison of recent studies on contactless sensing methods is presented, which shows the effectiveness and reliability of these methods. This study also highlights the existing challenges in contactless sensing methods and provides future research directions to mitigate these challenges.
... Due to the benefits of being independent of background illumination and users' skin color, remote HR and RR monitoring systems using a near-infrared light source can be a valuable alternative. [26,27]; +/=/-indicates system has advantage/neutral/disadvantage in certain category; / indicates system has certain capability Over the past few years, HR/RR measurement systems using near-infrared cameras or even RGB/Depth cameras (such as Microsoft Kinect) [19][20][21][22][23][24]28,29] have been studied to show their capability of measuring HR/RR. In this study, we propose a system that leverages the existing ToF cameras in many consumer electronics and vehicles to perform HR and RR monitoring. ...
Article
Full-text available
This study describes a contactless vital sign monitoring (CVSM) system capable of measuring heart rate (HR) and respiration rate (RR) using a low-power, indirect time-of-flight (ToF) camera. The system takes advantage of both the active infrared illumination as well as the additional depth information from the ToF camera to compensate for the motion-induced artifacts during the HR measurements. The depth information captures how the user is moving with respect to the camera and, therefore, can be used to differentiate where the intensity change in the raw signal is from the underlying heartbeat or motion. Moreover, from the depth information, the system can acquire respiration rate by directly measuring the motion of the chest wall during breathing. We also conducted a pilot human study using this system with 29 participants of different demographics such as age, gender, and skin color. Our study shows that with depth-based motion compensation, the success rate (system measurement within 10% of reference) of HR measurements increases to 75%, as compared to 35% when motion compensation is not used. The mean HR deviation from the reference also drops from 21 BPM to −6.25 BPM when we apply the depth-based motion compensation. In terms of the RR measurement, our system shows a mean deviation of 1.7 BPM from the reference measurement. The pilot human study shows the system performance is independent of skin color but weakly dependent on gender and age.
Chapter
To make a L3AD vehicle trustworthy and make the user feel comfortable using such a vehicle is crucial for the adoption of automated vehicles. Such an automated vehicle cannot be just a vehicle it has to be a TrustVehicle. Therefore, the TrustVehicle project presents an approach to define how a vehicle should behave in order to be a TrustVehicle by collecting requirements in the form of user concerns and expectations as well as expert opinions. Based on these results, Key Performance Indicators are defined to measure trustworthy vehicle behaviour divided in human, technology and vehicle category. Finally, the defined trustworthy vehicle behaviour needs to be benchmarked. This is done by a variety of use cases, dealing with complex scenarios and different demonstrator vehicles to reflect diverse aspects of complex traffic situations and conditions.
Chapter
Autonomous Vehicles (AVs) have the potential to enhance the road safety, while reducing congestion, and GHG emissions. It is expected that vehicles with the different level of autonomy (Level 2+) will hit the road in the near future. This chapter discusses briefly the state of art of the three main building blocks of the automated driving (AD), to develop a reliable, safe, and trustable AV. In this framework, AV sensors, planning, and tracking controller techniques are reviewed. Moreover, the challenges, some promising technologies (e.g. self-cleaning sensors), technical solutions, and shortfalls that need to be alleviated are described considering different vehicle classes ranging from passenger up to heavy duty vehicle.
Chapter
Automated cars bring not only technical but also non-technical challenges that influence acceptance such as legal and ethical questions, social and economic implications, or road safety effects. In general, if technical innovations are in line with societal values and behavioural intentions, they can gain acceptance. Based on a literature review, issues that influence the acceptance of automated cars are identified. However, not only the macro aspects are important for acceptance, but also the design of the technology for potential users, especially the HMI design. As the division of tasks between humans and machines changes, the implementation of automated driving systems changes the task of the driver fundamentally, posing new challenges for HMI design. Uses cases involving vulnerable road users (VRUs) are the focus of the BRAVE project (EU-funded) when developing and testing HMIs. Research on the interaction between automated vehicles and VRUs is needed, not only to increase acceptance among road users but also to increase safety for VRUs. The development of interfaces and products must be carried out in an iterative process, to allow users to be involved at an early stage. Adjusting concepts in the beginning of the process is more cost-efficient compared to making changes later in the process and ensures a better adaptation of the developed technical solutions to the expectations of end-users, thus favouring the acceptance of automated vehicles.
Article
Full-text available
Monitoring of cardiopulmonary activity is a challenge when attempted under adverse conditions, including different sleeping postures, environmental settings, and an unclear region of interest (ROI). This study proposes an efficient remote imaging system based on a Microsoft Kinect v2 sensor for the observation of cardiopulmonary-signal-and-detection-related abnormal cardiopulmonary events (e.g., tachycardia, bradycardia, tachypnea, bradypnea, and central apnoea) in many possible sleeping postures within varying environmental settings including in total darkness and whether the subject is covered by a blanket or not. The proposed system extracts the signal from the abdominal-thoracic region where cardiopulmonary activity is most pronounced, using a real-time image sequence captured by Kinect v2 sensor. The proposed system shows promising results in any sleep posture, regardless of illumination conditions and unclear ROI even in the presence of a blanket, whilst being reliable, safe, and cost-effective.
Article
Full-text available
Non-contact detection of heart rate has been addressed by researchers from very different fields. However, the low accuracy of measuring results in the difficulties in methodology deployment. This paper introduces the principle of heartbeat detection. A detection scheme by using Kinect is proposed. Further, the signal processing approach based on JADE algorithm is developed to efficiently remove the clutter in mixture signals, and it enables accurate transforming via Z-score normalization. Due to the significances presented in this work, the detection error is 1.79%, when processed with the proposed algorithm. Experimental results are statistically analyzed, which makes it a promising basis for the realization of heart rate detection.
Article
Full-text available
This paper introduces the design of a new wearable photoplethysmography (PPG) sensor and its assessment for mental distress estimation. In our design, a PPG sensor obtains blood volume information by means of an optical plethysmogram technique. A number of temporal, morphological and frequency markers are computed using time intervals between adjacent normal cardiac cycles to characterize pulse rate variability (PRV). In order to test the efficiency of the developed wearable for classifying distress versus calmness, the well-known International Affective Picture System has been used to induce different levels of arousal in forty-five healthy participants. The obtained results have shown that temporal features present a single discriminant power between emotional states of calm and stress, ranging from 67 to 72%. Moreover, a discriminant tree-based model is used to assess the possible underlying relationship among parameters. In this case, the combination of temporal parameters reaches 82.35% accuracy. Considering the low difficulty of metrics and methods used in this work, the algorithms are prepared to be embedded into a micro-controller device to work in real-time and in a long-term fashion.
Article
Full-text available
To achieve the simultaneous and unobtrusive breathing rate (BR) and heart rate (HR) measurements during nighttime, we leverage a far-infrared imager and an infrared camera equipped with IR-Cut lens and an infrared lighting array to develop a dual-camera imaging system. A custom-built cascade face classifier, containing the conventional Adaboost model and fully convolutional network trained by 32K images, was used to detect the face region in registered infrared images. The region of interest (ROI) inclusive of mouth and nose regions was afterwards confirmed by the discriminative regression and coordinate conversions of three selected landmarks. Subsequently, a tracking algorithm based on spatio-temporal context learning was applied for following the ROI in thermal video, and the raw signal was synchronously extracted. Finally, a custom-made time-domain signal analysis approach was developed for the determinations of BR and HR. A dual-mode sleep video database, including the videos obtained under environment where illumination intensity ranged from 0 to 3 Lux, was constructed to evaluate the effectiveness of the proposed system and algorithms. In linear regression analysis, the determination coefficient (R²) of 0.831 had been observed for the measured BR and reference BR, and this value was 0.933 for HR measurement. In addition, the Bland-Altman plots of BR and HR demonstrated that almost all the data points located within their own 95% limits of agreement. Consequently, the overall performance of the proposed technique is acceptable for BR and HR estimations during nighttime.
Article
Full-text available
Healthy biological systems exhibit complex patterns of variability that can be described by mathematical chaos. Heart rate variability (HRV) consists of changes in the time intervals between consecutive heartbeats called interbeat intervals (IBIs). A healthy heart is not a metronome. The oscillations of a healthy heart are complex and constantly changing, which allow the cardiovascular system to rapidly adjust to sudden physical and psychological challenges to homeostasis. This article briefly reviews current perspectives on the mechanisms that generate 24 h, short-term (~5 min), and ultra-short-term (<5 min) HRV, the importance of HRV, and its implications for health and performance. The authors provide an overview of widely-used HRV time-domain, frequency-domain, and non-linear metrics. Time-domain indices quantify the amount of HRV observed during monitoring periods that may range from ~2 min to 24 h. Frequency-domain values calculate the absolute or relative amount of signal energy within component bands. Non-linear measurements quantify the unpredictability and complexity of a series of IBIs. The authors survey published normative values for clinical, healthy, and optimal performance populations. They stress the importance of measurement context, including recording period length, subject age, and sex, on baseline HRV values. They caution that 24 h, short-term, and ultra-short-term normative values are not interchangeable. They encourage professionals to supplement published norms with findings from their own specialized populations. Finally, the authors provide an overview of HRV assessment strategies for clinical and optimal performance interventions.
Article
Full-text available
Contactless detection is one of the new frontiers of technological innovation in the field of healthcare, enabling unobtrusive measurements of biomedical parameters. Compared to conventional methods for Heart Rate (HR) detection that employ expensive and/or uncomfortable devices, such as the Electrocardiograph (ECG) or pulse oximeter, contactless HR detection offers fast and continuous monitoring of heart activities and provides support for clinical analysis without the need for the user to wear a device. This paper presents a validation study for a contactless HR estimation method exploiting RGB (Red, Green, Blue) data from a Microsoft Kinect v2 device. This method, based on Eulerian Video Magnification (EVM), Photoplethysmography (PPG) and Videoplethysmography (VPG), can achieve performance comparable to classical approaches exploiting wearable systems, under specific test conditions. The output given by a Holter, which represents the gold-standard device used in the test for ECG extraction, is considered as the ground-truth, while a comparison with a commercial smartwatch is also included. The validation process is conducted with two modalities that differ for the availability of a priori knowledge about the subjects’ normal HR. The two test modalities provide different results. In particular, the HR estimation differs from the ground-truth by 2% when the knowledge about the subject’s lifestyle and his/her HR is considered and by 3.4% if no information about the person is taken into account.
Conference Paper
Full-text available
This paper presents a novel solution for non-invasive real-time heart rate monitoring by processing of Near Infrared (NIR) generated streams, provided by a low-cost, easy-to-use sensor, such as Microsoft Kinect™. The standard method to monitor physiological information exploits photoplethysmographic images. In fact, the changes in blood volume can be determined from the spectra of radiation reflected from (or transmitted through) body tissues. Using a mathematical processing, the study shows how it is possible to real-time estimate the heart rate of people sitting for 1 minute in front of the sensor at distance 1 meter by analysing the NIR stream and without wearing any other sensors. In order to prove the correctness of the method proposed, 35 different subjects are involved in the test phase. During the tests, each subject wears also a pulse oximeter for comparing the values calculated by our method.
Article
Full-text available
Subject movement and a dark environment will increase the difficulty of image-based contactless pulse rate detection. In this paper, we detected the subject’s motion status based on complexion tracking and proposed a motion index (MI) to filter motion artifacts in order to increase pulse rate measurement accuracy. Additionally, we integrated the near infrared (NIR) LEDs with the adopted sensor and proposed an effective method to measure the pulse rate in a dark environment. To achieve real-time data processing, the proposed framework is constructed on a Field Programmable Gate Array (FPGA) platform. Next, the instant pulse rate and motion status are transmitted to a smartphone for remote monitoring. The experiment results showed the error of the pulse rate detection to be within −3.44 to +4.53 bpm under sufficient ambient light and −2.96 to + 4.24 bpm for night mode detection, when the moving speed is higher than 14.45 cm/s. These results demonstrate that the proposed method can improve the robustness of image-based contactless pulse rate detection despite subject movement and a dark environment.
Conference Paper
Non-contact methods of human heart rate (HR) facilitate medical assessment of this most important vital sign and increase patient comfort. Videoplethysmograpy (VPG) can be applied not only in healthcare units but also at homes and remote locations. In this paper the efficiency of an algorithm for pulse rate detection based on face image is analyzed. The correspondence between patient ethnicity and various color components used for HR estimation is examined. The results suggest that analysis of color components related to red provide better performance in case of darker skin tone, while green-related components are more accurate for persons of Caucasian origin.