ChapterPDF Available

An Overview of Remote Photoplethysmography Methods for Vital Sign Monitoring

Authors:
An Overview of Remote
Photoplethysmography Methods for Vital
Sign Monitoring
Ruchika Sinhal, Kavita Singh and M. M. Raghuwanshi
Abstract Vital signs such as heart rate, respiratory rate, blood pressure, body tem-
perature, and oxygen saturation are essential for early detection of any related signifi-
cant illness. Many of the existing methods that are used to monitor the aforementioned
vital signs are camera-based. In these methods, sensors are fixed to the body that are
not sturdy to the motion of the subject. Another method to monitor vital signs is
photoplethysmography (PPG), an emerging noncontact technique that maps, spatial
blood volume variation in living tissue from the images captured through a video.
Most of the camera-based methods are driven by three remote photoplethysmography
algorithms. The camera-based methods are useful for detecting vital signs with an
objective AAA, i.e., anyone, anywhere, and anytime. However, there exist few chal-
lenges in r-ppg methods and make it an open research problem. This paper presents an
overview of the signal processing challenges faced by remote photoplethysmography
for calculating the vital signs with a focus on heart rate estimation.
Keywords Heart rate ·Remote photoplethysmography ·Vital signs ·Signal
processing ·Video processing
R. Sinhal (B)
Department of CSE, Datta Meghe Institute of Engineering, Technology & Research, Wardha, India
e-mail: ruchisinhal04@gmail.com
K. Singh
Department of Computer Technology, Yeshwantrao Chavan College of Engineering, Nagpur,
India
e-mail: singhkavita19@yahoo.co.in
M. M. Raghuwanshi
Department of Information Technology, Yeshwantrao Chavan College of Engineering, Nagpur,
India
e-mail: m_raghuwanshi@rediffmail.com
© Springer Nature Singapore Pte Ltd. 2020
M. Gupta et al. (eds.), Computer Vision and Machine Intelligence
in Medical Image Analysis, Advances in Intelligent Systems and Computing 992,
https://doi.org/10.1007/978-981- 13-8798- 2_3
21
22 R.Sinhaletal.
1 Introduction
The human body has many integral signs such as heart rate, respiratory rate, blood
pressure, body temperature, and oxygen saturation, to name a few that helps in
deciding the fitness of the human body. Measurement of these vital signs needs a
device, and therefore, it is not possible to measure them for the examination of body
fitness at home. The heart rate is one of the important vital signs, as it plays a vital
role in checking the fitness level of a human body. Monitoring of the heart rate even
helps in detecting any growing heart problem. Heart rate is measured as beats per
minute (bpm) that ranges from 60 to 100 bpm in healthy adults.
Respiratory rate (RR) is another vital sign, defined as the frequency of breaths per
minute which ranges from 12 to 20 breaths per minute in a normal person. Monitoring
respiration can give valuable information about neural and pulmonary conditions.
Checking human RR regularly is an important task of examining the health of a person
[1]. Measuring these mentioned vital signs needs some medical devices and a doctor
to examine them. Generally, an individual has to visit the clinic and record the signs.
For instance, medical professionals examine heart rate through ECG, which is the
most commonly used contact-based or invasive method that induces skin irritation.
The noninvasive healthcare monitoring system involves either capturing the image
or recording the video through a digital camera. One of the systems is photoplethys-
mography (PPG) that is used to detect blood volume variations in the microvascular
tissue. PPG devices have been competent to monitor relevant signs that include pulse
rate, respiration rate, and body temperature [2]. Verkruysse et al., in [3], described
image photoplethysmography (i-PPG) technique. In this paper, the authors recorded
the facial skin affected by the port-wine dye after laser therapy. The resulting data
were used to obtain maps of the amplitude and phase of the spatially differing PPG
signals. The main drawback of the proposed technique is that the PPG signals are
sensitive to variations induced due to motion when camera-based phones are used.
Many a time, the noninvasive vital sign monitoring is also termed as remote photo-
plethysmography (r-PPG). The name r-PPG has been coined due to the monitoring
of remotely captured video through digital camera. Therefore, in the forthcoming
paragraph, we present an overview of the work presented by various authors related
to PPG vital sign monitoring.
Independent component analysis (ICA), a blind source separation method pro-
posed by Poh et al., in [4] removed the noise from PPG signal face imaging. The
measuring standards are recommended for the use of ECG sensors for measuring
the heart rate variability (HRV) in [5]. However, it has been shown that PPG-derived
heart rate variability can be a good substitute for HRV at rest [6]. In [7,8], Sun et al.
compared the performance of a low-cost web camera and a high-sensitivity camera
to evaluate the variability of the heart rate and pulses. The authors stated that the 30
fps webcam function is similar to the 100 fps camera when signals are incorporated
to improve the time-domain resolution [8]. HRV has been used to detect real-time
changes in the workload to evaluate and index the autonomous nervous system [9].
Its spectral analysis can provide a sympathetic balance, a ratio that reflects mutual
An Overview of Remote Photoplethysmography Methods … 23
changes in sympathetic and vagal outflows [10]. HRV tends to be rhythmic and emo-
tionally positive, followed by a phenomenon known as respiratory sinus arrhythmia.
HRV, on the other hand, tends to chaotic, angry, anxiety, or sadness. These rhythmic
variations create a condition of cardiac coherence [11,12]. Detection of physiological
signals using noncontact equipment is especially beneficial in emotional computing,
where emotions like stress or fear are induced. Contact sensors can create a bias
in these physiological experiments by interfering with the user, which results in
an erroneous emotion classification [13]. Currently, published methods effectively
recover spectral components of HR, BR, and HRV over a certain period. However,
there have been a few attempts, instant HR measurement with a webcam, particularly
considering artifacts of head motion [14].
The r-PPG algorithms proposed in the literature have been developed on videos
under constrained environments. However, there are many challenging issues faced
during the development of algorithms under uncontrolled environment. Therefore,
it is a potential field of research for researchers who are willing to work in the field
of r-PPG for vital sign monitoring. These challenges have been described in Sect. 3.
This paper organization has three sections. The introductory section describes the
vital signs and its importance. Section 2discusses various r-PPG methods used for
estimating the heart rate. Section 3describes the different challenges faced in r-PPG.
Finally, concluding section presents a summary of the review of techniques.
2 r-PPG Methods for Estimating Heart Rate
r-PPG is a remote photoplethysmography technique that measures, small changes
in skin color caused by variations, in volume and oxygen saturation while heart
pumping. All r-PPG techniques developed so far have used the videos captured
from a digital camera for analyzing the pulse or heart rate. Recently, several r-PPG
algorithms are developed for extracting the heart signal from videos. In this section,
we present the study of each approach developed by researchers for estimating heart
rate. Broad categories of the various approaches are blind source separation, model-
based methods, and design-based methods. Each category and algorithm is further
discussed in the forthcoming section.
2.1 Blind Source Separation (BSS)
The generalization of time series data as an alternative representation in the fre-
quency domain is also important. This representation enables the understanding of
the signals and the filtering or interpolation of the data. In particular, the singular
value decomposition (SVD) [4] and independent component analysis (ICA) [15]
techniques for the principal component analysis (PCA) have been examined. Both
these PCA and ICA techniques use statistical data representation rather than time or
24 R.Sinhaletal.
frequency domain. In other words, data are projected on a new set of axes that fulfill
certain statistical criteria, which implies independence, instead of a set of axes repre-
senting discrete frequencies such as the Fourier transformation, where independence
is assumed. The criterion depends on the structure of the data and the axes on which
the data is projected. The projection direction increases the signal-to-noise ratio,
which allows us to observe the important structural signals. For example, the power
spectrum of the data can be calculated to make the peaks of certain signals visible
and to separate the noise from the signal. Such unwanted signals can be filtered using
PCA and ICA. Most important, BSS techniques are analytical and computational for
general problems of signal processing. It does not benefit from the unique character-
istic of skin reflections used to solve the r-PPG problem. The ICA-based approach,
in particular, normalizes the standard deviation of RGB signals, ignoring the fact that
the PPG signal induces distant yet known relative amplitudes in the particular RGB
channels.
2.2 Model-Based Method
The BSS method discussed above has limitations of assumption on the colors asso-
ciated with source signals. In the blind source separation method, the colors are
considered independently for the signal estimation. To overcome the limitations of
BSS, model-based methods use different components of color vectors to control
de-mixing. Therefore, model-based methods eliminate the dependency of colors on
skin color reflection including light color. In addition, the model-based methods are
also motion tolerance. Model-based method includes PBV and CHROM techniques
proposed by De Haan in [15,16], respectively. The PBV technique is based on
blood volume pulse that retrieves the pulse directly from the pulsatile components
restricting all color variations to possible direction. The PBV is also a motion robust
improved method which uses blood volume pulse signature as mentioned. Further,
the CHROM technique is robust to motion based on the standardized assumption
of skin tone. The CHROM is different from PBV because it reduces sensitivity by
eliminating the specular component and reducing the size. The CHROM algorithm
assumes a standardized skin tone vector that allows white images to be balanced.
From the literature, it has been observed that the CHROM is robust to mono-white
illuminations and so it is categorized as best model-based algorithm. In addition,
Wenjing Wang proposed a new method, the orthogonal skin plane (POS) [17]. This
method resembles CHROM but alters the order in which the expected color distor-
tions are reduced using different priors. In this new algorithm, authors developed a
skin tone orthogonal plane in a temporarily normalized RGB environment.
Compared to multistep CHROM and POS, PBV is a one-step process and requires
an accurate knowledge of the signature of the blood volume pulse. With regard to
movement and stationary parameters, CHROM and POS perform well in stationary
and motion situations when the alpha tuning is driven either by pulse or by large
distortions, while PBV is specifically designed for movement. In addition, CHROM
An Overview of Remote Photoplethysmography Methods … 25
and POS are not as restrictive as PBV. In addition to all the above comparative
analysis, one more similarity between CHROM and POS is that these two methods
use soft priors to define a projection plane for alpha tuning in blood volume pulsation
(i.e., channel ranking).
2.3 Design-Based Method
Model-based methods are good as they are robust to motion. They perform better
than non-model-based methods. As discussed earlier, the limitation in CHROM is
that it uses the vector for white skin reference, whereas PBV depends on the blood
signals and diverts the signal to that side. However, apart from these limitations, the
vital sign measurement is best for model-based methods as these methods are motion
robust.
In the recently developed spatial subspace rotation (2SR) method [18], the RGB
values are quantified as spatial representation. In the temporal domain, pulsatile
blood causes variation in RGB channels, thus changing subspace of skin pixels. The
algorithm creates a subject-dependent skin color space and tracks the tone change
over time to measure the pulse in which the instant tone is determined on the basis of
the statistical distribution of the skin pixels in the image. The idea of using the hue as
a basic pulse extraction parameter is supported by the analysis of the use of different
color spaces to measure the pulse [19]. Since then, the tone drives measurement, the
method at an early stage eliminates all variations in intensity. In this sense, 2SR is a
skin approach that defines a temporarily normalized orthogonal projection plane in
the RGB pulse extraction space. The subspace axes built by 2SR are, however, com-
pletely data-driven without physiological consideration. This presents performance
problems in practice when spatial measurements are not reliable, i.e., when the skin
mask is noisy or poorly selected. A new lock-in technique is proposed in [20]for
extracting pulse rate which when compared with gold standards differed only by four
beats [20].
In this section, we presented different approaches proposed by various researchers
in last few decades. From the detailed literature, it has been observed that estimation
of heart rate using r-PPG from captured video still struggles with many challenges.
These challenges are further discussed in the next forthcoming section.
3 Challenges in r-PPG
From the literature, it is evident that r-PPG focuses on extracting the pulse signal from
video to estimate the heart rate of a person. However, r-PPG has many challenges due
to various factors like subject motion, ambient light illumination, image optimiza-
tion, spectrum analysis, etc. These factors present challenges to the researcher for
recovering the accurate physiological data for different r-PPG methods. The men-
26 R.Sinhaletal.
Fig. 1 Factors related to
challenges in r-PPG
Motion
Stabilization
Ambient Lig ht
Tolerance
Image
Optimization
Multi-spectral
Imaging
rPPG
tioned four factors related to the main challenges in r-PPG are shown in Fig. 1.In
subsequent sections, we present each of these factors for better understanding.
3.1 Motion Stabilization
The subject motion has been studied in most of the r-PPG algorithms to analyze its
effects. The subject motion is an important and a great challenge that is faced by
r-PPG algorithms. The subject motion changes the region of interest in the video
when the subject is in motion. Early investigations of r-PPG focused on rigid and
stationary regions of interest as used in researches [3,14,16,2123]. Ideally, when
subjects are stationary, the ROI must also be constant in subsequent frames used by
r-PPG algorithms. However, this might not be the practical scenario of the application
of r-PPG algorithms. Therefore, there is a need of motion stabilization that is to be
achieved in the video. Thus, the tracking of ROI can be proposed as a solution to this
problem.
On the other hand, few of the studies introduced later focus on face imaging of
the subjects that allowed limited naturalistic motion [19,21]. The work proposed in
[19,21] used the emotion factor in unconditional environment while recording the
videos for the study.
Further studies investigated the algorithm’s performance under translational
motion which limited the use of techniques as simple region of interest (ROI). These
ROIs focused on object tracking [15], color difference, and chrominance-based sig-
nals from RGB color space [19]. Later, approaches for estimating the motion arti-
facts and correcting the obtained r-PPG signal using adaptive filtering have also been
explored [24]. As stated earlier, this motion artifact is a challenge for r-PPG, with
specific advancements in three big areas: (1) the development of an algorithm for
image processing, (2) spatial redundancy, and (3) the use of integrated multiband
techniques in visible and invisible wavelengths. While it is important to explore sys-
tematically varying motion artifacts, use cases in applied environments are included
but not limited to exercise [25,26], cognitive states [27,28], and clinical care [29,
30], with a view to transition from the laboratory environment. Finally, the effects
of movement artifacts have been extended beyond the pulse rate alone to cardiopul-
monary measures. The authors in [31] proposed a new technology sub-band r-PPG
An Overview of Remote Photoplethysmography Methods … 27
for HR measurement in fitness by increasing the pulse extraction signal dimension-
ality. During fitness exercise, they tested the algorithm on the subjects. The proposed
pre-filtering method improved r-PPG performance. The approach of selective ampli-
tude filtering filtered the r-PPG signal based on the RGB color band. The authors in
[32,33] designed a filtering method that filters the RGB signals before the pulse is
extracted.
3.2 Ambient Light Tolerance
Ambient light is the light already present in a video without any manipulation. There
is no additional light added in the video. Such ambient lighting conditions might be
considered proportionately consistent in most of the applications; nevertheless, there
are some probable use cases like a computer simulation, virtual reality, mobile screen
brightness, etc, where there can be variations in lighting conditions. The difference
in illumination intensity many a time influences the intensity of the PPG waveform
[23]. At the same time, the effects of ambient light on any other currently available
PPG methods are unknown. In a limited, uncontrolled study in [32], Li et al. used an
adaptive filtering approach, with more background region of interest. The ROI served
as the input noise reference signal which compensates for background illumination.
The results in [34] depicted improvement for varying illumination on a publicly
available video database [35]. The detailed experiments presented in [34] derived
modest reduction in heart rate error when compared to ECG recorded.
To estimate the vital signs, it is very important to deal with the ambient light toler-
ance in the background of the video. This is because the background or the region of
interest, if it is either under-illuminated or over-illuminated, might result in incorrect
signal extraction. This incorrect signal extraction will further lead to an incorrect
estimate from the video. Consequently, it is very important to keep the illumination
of a region of interest constant throughout the video acquisition process. The back-
ground illumination might be canceled using the technique developed in [34] as one
of the illumination cancelation techniques. However, identification and removal of
the effect of illumination on the region of interest still remains a challenging issue to
be explored that further bring complexity in analyzing the heart rate from the region
of interest.
3.3 Image Optimization
Image optimization specifically emphasizes on capturing the images or videos
through different sources. The varieties of image sensors or video cameras that allow
fostering of r-PPG also bring variations in a different level. For instance, a varia-
tion in the image sensor (e.g., digital camera, mobile phone cameras, etc,) brings
variations in features that are to be examined, which additionally induces variations
28 R.Sinhaletal.
in further analyses. These features could be a basic sensor type, color separation
sensors, special sensors, aspect ratio, image sizes, and the number of pixels, to name
afew.
However, from the literature, it can be seen that the image quality does not rely only
on the quality of an image sensor being used. The other factors that affect the image
quality are a type of lens (CCD and CMOS), spectral properties as an illumination
source, and image shutter speed. Thus, image quality directly or indirectly changes
the feature of an image or video under consideration. Apart from this, although the
frame rate is not directly related to image sensor properties, it is also considered
as one of the vital components for variations in features under consideration for
estimating vital signs. An image can be optimized with the use of automatic ROI
selection from the image. Recently, Wang et al. performed supervised living skin
detection using r-PPG by transforming the video into signal shape descriptor called
multiresolution iterative spectrum [36].
Thus, an image optimization deals with capturing video in different formats. One
such format for video is ‘mp4’ format that stores the frames in compressed form. The
compression of an image or a video might lose information resulting in an inaccurate
vital sign estimation from the recorded video. In this context, the image optimization
is, therefore, a challenge and is to be dealt with before measuring the vital sign from
the optimized video.
3.4 Multispectral Imaging
Multispectral imaging captures images with a specific wavelength and multiband
spectrum. There are different spectral bands used for satellite images, such as Blue,
Green, Red, near-infrared, mid-infrared, and thermal. Many contemporary r-PPG
studies focus on three spectral bands in the visible light spectrum, i.e., red, green,
and blue. While green/orange visible bands are the most pervasive of common oxygen
and de-oxyhemoglobin derivatives [37], multispectral imagery from a single image
sensor has often been used for r-PPG methods involving linear decomposition using
multiple data channels. As shown by Martinez et al., in [38] with front spectropho-
tometry, some wavebands are better for measuring r-PPG pulse rate and respiration
rate. The different spectrums have a different effect on the signal estimation. Thus,
the r-PPG is still an open challenge because the different bands lead to the creation
of different estimates for vital signs. This shows that it is difficult to identify good
signal for estimation of vital signs.
4 Conclusion
Photoplethysmography technique maps the blood volume pulsating signals into the
vital signs. The vital signs can be measured by heart rate, respiratory rate, or saturation
An Overview of Remote Photoplethysmography Methods … 29
per oxygen level (SPO2) from images or videos. In this paper, we have presented
three r-PPG methods, namely, BSS, model-based, and design-based, based on digital
video signals. Each method has its own advantages and limitations in various contexts
of r-PPG as discussed in the paper. Among all the three described methods, BSS is a
widely used r-PPG method in the literature. Further, we presented different factors,
like subject motion, ambient light illumination, image optimization, and spectrum
analysis, that make r-PPG an open research issue to be explored. Among all the
factors listed above, motion stabilization is an important factor to be dealt with as it
brings an uncontrolled environment and makes r-PPG a challenging task.
From the discussions presented in this paper, it has been observed that many
research challenges are still open for solutions in the field of r-PPG. The researchers
can initiate to pursue research and contribute to social technology.
References
1. Sinhal, R., Singh, K., Shankar, A.: Estimating vital signs through non-contact video-based
approaches: a survey. In: 2017 International Conference on Recent Innovations in Signal pro-
cessing and Embedded Systems (RISE), pp. 139–141. IEEE, Bhopal (2017)
2. Allen, J.: Photoplethysmography and its application in clinical physiological measurement.
Physiol. Meas. 28, R1–R39 (2007)
3. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient
light. Opt. Express 16, 21434 (2008)
4. Poh, M.-Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physi-
ological measurements using a webcam. IEEE Trans. Biomed. Eng. 58, 7–11 (2011)
5. Camm, A.J., et al.: Heart rate variability. Standards of measurement, physiological interpreta-
tion, and clinical use. Task force of the european society of cardiology and the north american
society of pacing and electrophysiology. Eur. Heart J. 17, 354–381 (1996)
6. Gil, E., Orini, M., Bailón, R., Vergara, J.M., Mainardi, L., Laguna, P.: Photoplethysmography
pulse rate variability as a surrogate measurement of heart rate variability during non-stationary
conditions. Physiol. Meas. 31, 1271–1290 (2010)
7. Sun, Y., Papin, C., Azorin-Peris, V., Kalawsky, R., Greenwald, S., Hu, S.: Use of ambient light
in remote photoplethysmographic systems: comparison between a high-performance camera
and a low-cost webcam. J. Biomed. Opt. 17, 037005 (2012)
8. Sun, Y., Hu, S., Azorin-Peris, V., Kalawsky, R., Greenwald, S.: Noncontact imaging photo-
plethysmography to effectively access pulse rate variability. J. Biomed. Opt. 18, 061205 (2012)
9. Appelhans, B.M., Luecken, L.J.: Heart rate variability as an index of regulated emotional
responding. Rev. Gen. Psychol. 10, 229–240 (2006)
10. Hoover, A., Singh, A., Fishel-Brown, S., Muth, E.: Real-time detection of workload changes
using heart rate variability. Biomed. Signal Process. Control 7, 333–341 (2012)
11. Servant, D., Logier, R., Mouster, Y., Goudemand, M.: Heart rate variability. Appl. Psychiatry.
L’Enceph. 35, 423–428 (2009)
12. Tiller, W.A., McCraty, R., Atkinson, M.: Cardiac coherence: a new, noninvasive measure of
autonomic nervous system order. Altern. Ther. Health Med. 2, 52–65 (1996)
13. Pavlidis, I., Dowdall, J., Sun, N., Puri, C., Fei, J., Garbey, M.: Interacting with human physi-
ology. Comput. Vis. Image Underst. 108, 150–170 (2007)
14. Bousefsaf, F., Maaoui, C., Pruski, A.: Continuous wavelet filtering on webcam photoplethys-
mographic signals to remotely assess the instantaneous heart rate. Biomed. Signal Process.
Control 8, 568–574 (2013)
30 R.Sinhaletal.
15. de Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans.Biomed.
Eng. 60, 2878–2886 (2013)
16. de Haan, G., van Leest, A.: Improved motion robustness of remote-PPG by using the blood
volume pulse signature. Physiol. Meas. 35, 1913–1926 (2014)
17. Wang, W., den Brinker, A., Stuijk, S., de Haan, G.: Algorithmic principles of remote-PPG.
IEEE Trans. Biomed. Eng. 1–1 (2016)
18. Wang, W., Stuijk, S., de Haan, G.: A novel algorithm for remote photoplethysmography: spatial
subspace rotation. IEEE Trans. Biomed. Eng. 63, 1974–1984 (2016)
19. Tsouri, G.R., Li, Z.: On the benefits of alternative color spaces for noncontact heart rate mea-
surements using standard red-green-blue cameras. J. Biomed. Opt. 20, 048002 (2015)
20. Eaton, A., Vishwanath, K., Cheng, C.-H., Paige Lloyd, E., Hugenberg, K.: Lock-in technique
for extraction of pulse rates and associated confidence levels from video. Appl. Opt. 57, 4360
(2018)
21. Humphreys, K., Ward, T., Markham, C.: Noncontact simultaneous dual wavelength photo-
plethysmography: a further step toward noncontact pulse oximetry. Rev. Sci. Instrum. 78,
044304 (2007)
22. McDuff, D.J., Estepp, J.R., Piasecki, A.M., Blackford, E.B.: A survey of remote optical pho-
toplethysmographic imaging methods. In: 2015 37th Annual International Conference of the
IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6398–6404. IEEE, Milan
(2015)
23. Shi, P., Hu, S., Zhu, Y.: A preliminary attempt to understand compatibility of photoplethysmo-
graphic pulse rate variability with electrocardiogramic heart rate variability. (2008)
24. Wieringa, F.P., Mastik, F., van der Steen, A.F.W.: Contactless multiple wavelength photo-
plethysmographic imaging: a first step toward “SpO2 camera” technology. Ann. Biomed. Eng.
33, 1034–1041 (2005)
25. Sun, Y.: Motion-compensated noncontact imaging photoplethysmography to monitor car-
diorespiratory status during exercise. J. Biomed. Opt. 16, 077010 (2011)
26. Takano, C., Ohta, Y.: Heart rate measurement based on a time-lapse image. Med. Eng. Phys.
29, 853–857 (2007)
27. McDuff, D., Gontarek, S., Picard, R.: Remote measurement of cognitive stress via heart rate
variability (2014)
28. Cennini, G., Arguel, J., Ak¸sit, K., van Leest, A.: Heart rate monitoring via remote photoplethys-
mography with motion artifacts reduction. Opt. Express 18, 4867 (2010)
29. Aarts, L.A.M., Jeanne, V., Cleary, J.P., Lieber, C., Nelson, J.S., Bambang Oetomo, S.,
Verkruysse, W.: Non-contact heart rate monitoring utilizing camera photoplethysmography
in the neonatal intensive care unit—A pilot study. Early Hum. Dev. 89, 943–948 (2013)
30. Klaessens, J.H., van den Born, M., van der Veen, A., Sikkens-van de Kraats, J., van den Dungen,
F.A., Verdaasdonk, R.M.: Development of a baby friendly non-contact method for measuring
vital signs: First results of clinical measurements in an open incubator at a neonatal intensive
care unit (2014)
31. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Robust heart rate from fitness videos.
Physiol. Meas. 38, 1023–1044 (2017)
32. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Color-distortion filtering for remote
photoplethysmography (2017)
33. Wang, W., den Brinker, A.C., Stuijk, S., de Haan, G.: Amplitude-selective filtering for remote-
PPG. Biomed. Opt. Express. 8, 1965 (2017)
34. Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos
under realistic situations (2014)
35. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recogni-
tion and implicit tagging. IEEE Trans. Affect. Comput. 3, 42–55 (2012)
36. Wang, W., Stuijk, S., de Haan, G.: Living-skin classification via remote-PPG. IEEE Trans.
Biomed. Eng. 64, 2781–2792 (2017)
37. Zijlstra, W.G., Buursma, A., Meeuwsen-van der Roest, W.P.: Absorption spectra of human
fetal and adult oxyhemoglobin, de-oxyhemoglobin, carboxyhemoglobin, and methemoglobin.
Clin. Chem. 37, 1633–1638 (1991)
An Overview of Remote Photoplethysmography Methods … 31
38. Corral Martinez, L.F., Paez, G., Strojnik, M.: Optimal wavelength selection for noncontact
reflection photoplethysmography (2011)
... In addition, vital signs assessment usually requires close interaction between medical personnel and patients, which can increase the risk of disease transmission. Therefore, along with the development of health science and biomedical engineering, vital sign monitoring instruments are also being developed, ranging from wearable technology [9] to contactless technology [10,11]. ...
... In recent years, various developments have been made in remote vital sign monitoring technologies, including WiFi-based [19], Doppler radar systems [20], piezoelectric [21], and remote-photoplethysmography (rPPG) [11]. Despite some challenges and limitations, such as motion artefacts, variability in ambient light, skin tone variation, and signal processing complexity, research on the remote photoplethysmography (rPPG) method is advancing rapidly. ...
Article
Full-text available
Vital sign assessment is an examination that indicates changes in health. Direct contact during vital signs assessment can increase the risk of disease transmission. This research aimed to develop a contactless vital sign monitoring prototype that includes heart rate, respiratory rate, blood pressure, and oxygen saturation using a digital camera based on remote photoplethysmography with an adaptive region of interest. The adaptive regionof-interest method uses face detection and skin segmentation to generate red green-blue signals, taking only the skin pixels of the patients while also minimising the effect of motion artefacts. The hybrid processing method combines several vital sign extraction methods to filter external irrelevant factors and produce heart rate, respiratory rate, blood pressure, and blood oxygen saturation values. In addition, the prototype was tested on 50 participants using standard vital sign assessment tools for comparison. The technical specification test of the prototype concluded that the optimal distance of this prototype was up to 2 m with a processing time of 2 s for every 1-s video. The vital signs results were presented using Bland-Altman, which showed that although the Bland-Altman plots revealed a substantial variance in the limits of agreement (±15–20 mmHg for blood pressure, ±15–17 bpm for heart rate, ±4–6 bpm for respiratory rate, and ±1–3 % for blood oxygen saturation), the mean differences for all vital signs were small (±0.7–5 mmHg for blood pressure, ±0.4–0.6 bpm for heart rate, ±0.5–0.7 bpm for respiratory rate, ±0.4–0.6 for blood oxygen saturation) and most data points were within the limits. While further clinical studies are needed to assess its reliability in monitoring specific medical conditions, the prototype has shown an acceptable agreement in assessing vital signs compared to the conventional methods, making it feasible for further development into a medical device.
... The rPPG technique stands out for its ability to estimate vital signs without physical contact, making the vital sign assessment process more convenient. Consequently, non-contact measurement of physiological parameters using rPPG technology has been extensively explored in literature (Antink et al., 2019;Khanam et al., 2019;Sinhal et al., 2020;Zaunseder & Rasche, 2022). rPPG techniques are widely used for monitoring purposes, such as for monitoring dialysis patients (Villarroel et al., 2017), detecting arrhythmias (Amelard et al., 2016), and monitoring neonates (Klaessens et al., 2014), among other applications. ...
Article
Full-text available
Contactless methods are widely used to measure vital signs from recorded or live videos using remote photoplethysmography (rPPG), which takes advantage of the slight skin color variation that occurs periodically on specific body regions with each blood pulse. However, existing rPPG-based solutions are typically expensive and not suitable for daily use at home for personal healthcare. To address this issue, we have recently developed a low-cost device that allows for the real-time estimation of vital signs using rPPG and can be easily integrated into any common home environment. The device consists of a smart mirror equipped with a camera that captures facial videos and extracts rPPG signals by processing video frames. One major limitation of this solution was its high sensitivity to abrupt head movements during video acquisition. This paper presents some advancements in the development of our smart device aimed at obtaining a more robust measurement of vital signs. Experimental results on live videos show that the new version of our system overcomes the limitations of the previous version, offering a more stable performance. Moreover, the new methodology shows improved performance compared to other state-of-the-art rPPG algorithms when tested on pre-recorded in-house videos from the UBFC-RPPG database.
... Notably, select functionalities of these wearables have garnered approval from the Food and Drug Administration (FDA), marking a significant endorsement of their medical utility [12]. The predominant technology employed in these wrist-worn devices is photoplethysmography (PPG) [13,14]. A photoplethysmography (PPG) sensor comprises a light-emitting diode (LED) and a photodetector (PD). ...
Article
Full-text available
The remote monitoring of vital signs via wearable devices holds significant potential for alleviating the strain on hospital resources and elder-care facilities. Among the various techniques available, photoplethysmography stands out as particularly promising for assessing vital signs such as heart rate, respiratory rate, oxygen saturation, and blood pressure. Despite the efficacy of this method, many commercially available wearables, bearing Conformité Européenne marks and the approval of the Food and Drug Administration, are often integrated within proprietary, closed data ecosystems and are very expensive. In an effort to democratize access to affordable wearable devices, our research endeavored to develop an open-source photoplethysmographic sensor utilizing off-the-shelf hardware and open-source software components. The primary aim of this investigation was to ascertain whether the combination of off-the-shelf hardware components and open-source software yielded vital-sign measurements (specifically heart rate and respiratory rate) comparable to those obtained from more expensive, commercially endorsed medical devices. Conducted as a prospective, single-center study, the research involved the assessment of fifteen participants for three minutes in four distinct positions, supine, seated, standing, and walking in place. The sensor consisted of four PulseSensors measuring photoplethysmographic signals with green light in reflection mode. Subsequent signal processing utilized various open-source Python packages. The heart rate assessment involved the comparison of three distinct methodologies, while the respiratory rate analysis entailed the evaluation of fifteen different algorithmic combinations. For one-minute average heart rates’ determination, the Neurokit process pipeline achieved the best results in a seated position with a Spearman’s coefficient of 0.9 and a mean difference of 0.59 BPM. For the respiratory rate, the combined utilization of Neurokit and Charlton algorithms yielded the most favorable outcomes with a Spearman’s coefficient of 0.82 and a mean difference of 1.90 BrPM. This research found that off-the-shelf components are able to produce comparable results for heart and respiratory rates to those of commercial and approved medical wearables.
... By processing these image pixels over time using specialized signal processing techniques, we can extract the PPG signal and thereby predict the physiological parameters including heart rate, Heart Rate Variability (HRV) and Blood Pressure (BP). With the introduction of digital cameras, remote heart rate monitoring has become accessible across diverse fields, encompassing hospital care [2], telemedicine [3,4], fitness assessment [5,6], motion recognition [7], and the automotive industry [8,9]. This remote method has extended its applications to numerous areas including mental stress detection, cardiovascular function variations, sleep quality assessment, and drowsiness identification [10][11][12]. ...
Article
Full-text available
Heart rate estimation from face videos is an emerging technology that offers numerous potential applications in healthcare and human–computer interaction. However, most of the existing approaches often overlook the importance of long-range spatiotemporal dependencies, which is essential for robust measurement of heart rate prediction. Additionally, they involve extensive pre-processing steps to enhance the prediction accuracy, resulting in high computational complexity. In this paper, we propose an innovative solution called LGTransPPG. This end-to-end transformer-based framework eliminates the need for pre-processing steps while achieving improved efficiency and accuracy. LGTransPPG incorporates local and global aggregation techniques to capture fine-grained facial features and contextual information. By leveraging the power of transformers, our framework can effectively model long-range dependencies and temporal dynamics, enhancing the heart rate prediction process. The proposed approach is evaluated on three publicly available datasets, demonstrating its robustness and generalizability. Furthermore, we achieved a high Pearson correlation coefficient (PCC) value of 0.88, indicating its superior efficiency and accuracy between the predicted and actual heart rate values.
Article
Full-text available
The development of non-contact techniques for monitoring human vital signs has significant potential to improve patient care in diverse settings. By facilitating easier and more convenient monitoring, these techniques can prevent serious health issues and improve patient outcomes, especially for those unable or unwilling to travel to traditional healthcare environments. This systematic review examines recent advancements in non-contact vital sign monitoring techniques, evaluating publicly available datasets and signal preprocessing methods. Additionally, we identified potential future research directions in this rapidly evolving field.
Article
We present a cost-effective multi-sensor wristband that monitors vital signs, including heart rate, respiratory rate, and oxygen saturation, with very low-hardware resources, while effectively mitigating motion artifacts. Our system, which operates in real-time, optimizes resource utilization and minimizes memory footprint by intelligently managing motion artifacts during vital signs extraction. For this purpose, we introduce a new Intelligent Domain Fusion algorithm, which allows to reduce resource allocation during moderate motion artifacts by up to a 30% in memory usage, while achieving an average latency of as short as 18 ms in worst case. To further enhance efficiency, our frequency domain analysis is optimized through the utilization of a Z-chirp transform, minimizing memory utilization during Fourier transforms. Our prototype is built around a real-time operating system (freeRTOS) running on an ARM-M4 architecture, enabling Bluetooth Low Energy (BLE) 5.0 control, data processing, and communication. It incorporates an onboard 1-Gbit NAND flash memory for continuous data storage, when no wireless connection is available, offering up to 7 hours of uninterrupted sensor data recording. The sensor data collected by our prototype encompasses a comprehensive range of raw and processed information critical for monitoring various physiological and environmental parameters in a concise message format comprising a data ID, a data payload, and a timestamp. With a small 250 mAh battery, our prototype provides an autonomy of 18 hours of continuous data collection, averaging a mere 10.41 mAh of power consumption.
Article
Full-text available
Remote photoplethysmography (rPPG) is a low-cost technique to measure physiological parameters such as heart rate by analyzing videos of a person. There has been growing attention to this technique due to the increased possibilities and demand for running psychological experiments on online platforms. Technological advancements in commercially available cameras and video processing algorithms have led to significant progress in this field. However, despite these advancements, past research indicates that suboptimal video recording conditions can severely compromise the accuracy of rPPG. In this study, we aimed to develop an open-source rPPG methodology and test its performance on videos collected via an online platform, without control of the hardware of the participants and the contextual variables, such as illumination, distance, and motion. Across two experiments, we compared the results of the rPPG extraction methodology to a validated dataset used for rPPG testing. Furthermore, we then collected 231 online video recordings and compared the results of the rPPG extraction to finger pulse oximeter data acquired with a validated mobile heart rate application. Results indicated that the rPPG algorithm was highly accurate, showing a significant degree of convergence with both datasets thus providing an improved tool for recording and analyzing heart rate in online experiments.
Conference Paper
Full-text available
The human body exhibits many vital signs, such as heart rate (HR) and respiratory rate (RR) used to assess fitness and health. Vital signs are typically measured by a trained health professional and may be difficult for individuals to accurately measure at home. Clinic visits are therefore needed with associated burdens of cost and time spent waiting in long queues. The widespread use of smart phones with video capability presents an opportunity to create non-invasive applications for assessment of vital signs. Over the past decade, several researchers have worked on assessing vital signs from video, including HR, RR and other parameters such as anemia and blood oxygen saturation (SpO2). This paper reviews the different image and video processing algorithms developed for vital signs assessment through non-contact methods, and outline the key remaining challenges in the field which can be used as potential research topics. The CHROM algorithm produces highest accuracy in detecting the signals from rPPG. There are different challenges of handling large database and motion stabilization which is not provided by any algorithm, this is main area of research in rPPG.
Article
Full-text available
We investigate the practical applicability of video photoplethysmography (VPPG) to extract heart rates of subjects using noncontact color video recordings of human faces collected under typical indoor laboratory conditions using commercial video cameras. Videos were processed following three previously described simple VPPG algorithms to produce a time-varying plethysmographic signal. These time signals were then analyzed using, to the best of our knowledge, a novel, lock-in algorithm that was developed to extract the pulsatile frequency component. A protocol to associate confidence estimates for the extracted heart rates for each video stream is presented. Results indicate that the difference between heart rates extracted using the lock-in technique and gold-standard measurements, for videos with high-confidence metrics, was less than 4 beats per minute. Constraints on video acquisition and processing, including natural subject motion and the total duration of video recorded required for evaluating these confidence metrics, are discussed.
Article
Full-text available
Biometric signatures of remote photoplethysmography (rPPG), including the pulse-induced characteristic color absorptions and pulse frequency range, have been used to design robust algorithms for extracting the pulse-signal from a video. In this paper, we look into a new biometric signature, i.e., the relative pulsatile amplitude, and use it to design a very effective yet computationally low-cost filtering method for rPPG, namely “amplitude-selective filtering” (ASF). Based on the observation that the human relative pulsatile amplitude varies in a specific lower range as a function of RGB channels, our basic idea is using the spectral amplitude of, e.g., the R-channel, to select the RGB frequency components inside the assumed pulsatile amplitude-range for pulse extraction. Similar to band-pass filtering (BPF), the proposed ASF can be applied to a broad range of rPPG algorithms to pre-process the RGB-signals before extracting the pulse. The benchmark in challenging fitness use-cases shows that applying ASF (ASF+BPF) as a pre-processing step brings significant and consistent improvements to all multi-channel pulse extraction methods. It improves different (multi-wavelength) rPPG algorithms to the extent where quality differences between the individual approaches almost disappear. The novelty of the proposed method is its simplicity and effectiveness in providing a solution for the extremely challenging application of rPPG to a fitness setting. The proposed method is easy to understand, simple to implement, and low-cost in running. It is the first time that the physiological property of pulsatile amplitude is used as a biometric signature for generic signal filtering.
Conference Paper
Full-text available
In recent years researchers have presented a number of new methods for recovering physiological parameters using just low-cost digital cameras and image processing. The ubiquity of digital cameras presents the possibility for many new, low-cost applications of vital sign monitoring. In this paper we present a review of the work on remote photoplethysmographic (PPG) imaging using digital cameras. This review specifically focuses on the state-of-the-art in PPG imaging where: 1) measures beyond pulse rate are evaluated, 2) non-ideal conditions (e.g., the presence of motion artifacts) are explored, and 3) use cases in relevant environments are demonstrated. We discuss gaps within the literature and future challenges for the research community. To aid in the continuing advancement of PPG imaging research, we are making available a website with the references collected for this review as well as information on available code and datasets of interest. It is our hope that this website will become a valuable resource for the PPG imaging community. The site can be found at: http://web.mit.edu/~djmcduff/www/ remote-physiology.html.
Article
Full-text available
This paper introduces a mathematical model that incorporates the pertinent optical and physiological properties of skin reflections with the objective to increase our understanding of the algorithmic principles behind remote photoplethysmography (rPPG). The model is used to explain the different choices that were made in existing rPPG methods for pulse extraction. The understanding that comes from the model can be used to design robust or application-specific rPPG solutions. We illustrate this by designing an alternative rPPG method where a projection plane orthogonal to the skin-tone is used for pulse extraction. A large benchmark on the various discussed rPPG methods shows that their relative merits can indeed be understood from the proposed model.
Article
Full-text available
In this paper, we propose a conceptually novel algorithm, namely “Spatial Subspace Rotation” (2SR), that improves the robustness of remote photoplethysmography. Based on the assumption of (1) spatially redundant pixel-sensors of a camera and (2) a well-defined skin mask, our core idea is to estimate a spatial subspace of skin-pixels and measure its temporal rotation for pulse extraction, which does not require skin-tone or pulse-related priors in contrast to existing algorithms. The proposed algorithm is thoroughly assessed on a large benchmark dataset containing 54 videos, which includes challenges of various skin-tones, body-motions in complex illuminance conditions, and pulse-rate recovery after exercise. The experimental results show that given a well-defined skin mask, 2SR outperforms the popular ICA-based approach and two state-of-the-art algorithms (CHROM and PBV). When comparing the pulse frequency spectrum, 2SR improves on average the SNR of ICA by 2.22 dB, CHROM by 1.56 dB, and PBV by 1.95 dB. When comparing the instant pulse-rate, 2SR improves on average the Pearson correlation and precision of ICA by 47% and 65%, CHROM by 22% and 23%, PBV by 21% and 39%. ANOVA confirms the significant improvement of 2SR in peak-to-peak accuracy. The proposed 2SR algorithm is very simple to use and extend, i.e., the implementation only requires a few lines Matlab code.
Article
Objective: This paper aims to improve the rPPG technology targeting continuous heart-rate measurement during fitness exercises. The fundamental limitation of the existing (multi-wavelength) rPPG methods is that they can suppress at most n - 1 independent distortions by linearly combining n wavelength color channels. Their performance are highly restricted when more than n - 1 independent distortions appear in a measurement, as typically occurs in fitness applications with vigorous body motions. Approach: To mitigate this limitation, we propose an effective yet very simple method that algorithmically extends the number of possibly suppressed distortions without using more wavelengths. Our core idea is to increase the degrees-of-freedom of noise reduction by decomposing the n wavelength camera-signals into multiple orthogonal frequency bands and extracting the pulse-signal per band-basis. This processing, namely Sub-band rPPG (SB), can suppress different distortion-frequencies using independent combinations of color channels. Main results: A challenging fitness benchmark dataset is created, including 25 videos recorded from 7 healthy adult subjects (ages from 25 to 40 yrs; six male and one female) running on a treadmill in an indoor environment. Various practical challenges are simulated in the recordings, such as different skin-tones, light sources, illumination intensities, and exercising modes. The basic form of SB is benchmarked against a state-of-the-art method (POS) on the fitness dataset. Using non-biased parameter settings, the average signal-to-noise-ratio (SNR) for POS varies in [-4.18, -2.07] dB, for SB varies in [-1.08, 4.77] dB. The ANOVA test shows that the improvement of SB over POS is statistically significant for almost all settings (p-value <0.05). Significance: The results suggest that the proposed SB method considerably increases the robustness of heart-rate measurement in challenging fitness applications, and outperforms the state-of-the-art method.
Article
Detecting living-skin tissue in a video on the basis of induced color changes due to blood pulsation is emerging for automatic region of interest localization in remote photoplethysmography (rPPG). However, the state-of-the-art method performing unsupervised living-skin detection in a video is rather time-consuming, which is mainly due to the high complexity of its unsupervised on-line learning for pulse/noise separation. In this paper, we address this issue by proposing a fast living-skin classification method. Our basic idea is to transform the time-variant rPPG-signals into signal shape descriptors called ``Multi-resolution Iterative Spectrum'' (MIS), where pulse and noise have different patterns enabling accurate binary classification. The proposed technique is a proof-of-concept that has only been validated in lab conditions but not in real clinical conditions. The benchmark, including synthetic and realistic (non-clinical) experiments, shows that it achieves a high detection accuracy better than the state-of-the-art method, and a high detection speed at hundreds of frames per second in Matlab, enabling real-time living-skin detection.