Figure 7 - uploaded by Christian S. Pilz
Content may be subject to copyright.
![Correlation and Bland-Altman plots for the POS [41] (upper plots) and LGI (lower plots) method over the entire data collection. POS archives a correlation of 0.35 with a RMSE of 21 BPM and LGI a correlation of 0.87 with a RMSE of 11 BPM. Many outliers can be attributed to false predictions during the gym session where the heart rate is confused with the pedal frequency.](profile/Christian-Pilz-2/publication/325645274/figure/fig2/AS:635177445371905@1528449512602/Correlation-and-Bland-Altman-plots-for-the-POS-41-upper-plots-and-LGI-lower-plots.png)
Correlation and Bland-Altman plots for the POS [41] (upper plots) and LGI (lower plots) method over the entire data collection. POS archives a correlation of 0.35 with a RMSE of 21 BPM and LGI a correlation of 0.87 with a RMSE of 11 BPM. Many outliers can be attributed to false predictions during the gym session where the heart rate is confused with the pedal frequency.
Source publication
![Figure 7. Correlation and Bland-Altman plots for the POS [41] (upper...](profile/Christian-Pilz-2/publication/325645274/figure/fig2/AS:635177445371905@1528449512602/Correlation-and-Bland-Altman-plots-for-the-POS-41-upper-plots-and-LGI-lower-plots_Q320.jpg)
We study the impact of prior knowledge about invariance for the task of heart rate estimation from face videos in the wild (e.g. in presence of disturbing factors like rigid head motion, talking, facial expressions and natural illumination conditions under different scenarios). We introduce features invariant with respect to the action of a differe...
Contexts in source publication
Context 1
... urban conversation most of the algorithms to fail completely. We observed this behavior during the entire evaluation. The results for each session are presented in Table 1. During the resting scenario the LGI method performs slightly worse. For all other sessions the LGI method archives quite robust results where the others mostly start to fail. Fig. 7 compares the estimation performance between the POS and the LGI approach. The correlation for the POS method is heavily affected by outliers. Although the LGI approach results in a better statistical performance, it shows an estimation bias of approximately 4 BPM. This also explains why the LGI method performs slightly worse during the ...
Context 2
... urban conversation most of the algorithms to fail completely. We observed this behavior during the entire evaluation. The results for each session are presented in Table 1. During the resting scenario the LGI method performs slightly worse. For all other sessions the LGI method archives quite robust results where the others mostly start to fail. Fig. 7 compares the estimation performance between the POS and the LGI approach. The correlation for the POS method is heavily affected by outliers. Although the LGI approach results in a better statistical performance, it shows an estimation bias of approximately 4 BPM. This also explains why the LGI method performs slightly worse during the ...
Similar publications
Attaining reliable profile gradients is of utmost relevance for many physical systems. In most situations, the estimation of gradient can be inaccurate due to noise. It is common practice to first estimate the underlying system and then compute the profile gradient by taking the subsequent analytic derivative. The underlying system is often estimat...
Citations
... The LGI-PGGI dataset from Pilz et al. [14] was used for the evaluation. It includes videos of the participants' faces with the referenced fingertip PPG signal. ...
... We compared our rPPG methods with the most recent rPPG methods: GREEN [13], ICA [20], PCA [21], the chrominance-base method (CHROM) [22], blood volume pulse (PBV) [23], plane orthogonal to the skin (POS) [11], local group invariance (LGI) [14], and orthogonal matrix image transformation (OMIT) [24]. All mentioned rPPG methods are implemented in the Python toolbox, pyVHR, from Boccignone et al. [25]. ...
... The LGI [14] calculates an rPPG signal with a robust algorithm using local transformations. The rPPG signal is computed by: ...
Remote photoplethysmography (rPPG) is a promising contactless technology that uses videos of faces to extract health parameters, such as heart rate. Several methods for transforming red, green, and blue (RGB) video signals into rPPG signals have been introduced in the existing literature. The RGB signals represent variations in the reflected luminance from the skin surface of an individual over a given period of time. These methods attempt to find the best combination of color channels to reconstruct an rPPG signal. Usually, rPPG methods use a combination of prepossessed color channels to convert the three RGB signals to one rPPG signal that is most influenced by blood volume changes. This study examined simple yet effective methods to convert the RGB to rPPG, relying only on RGB signals without applying complex mathematical models or machine learning algorithms. A new method, GRGB rPPG, was proposed that outperformed most machine-learning-based rPPG methods and was robust to indoor lighting and participant motion. Moreover, the proposed method estimated the heart rate better than well-established rPPG methods. This paper also discusses the results and provides recommendations for further research.
... We evaluated six traditional unsupervised learning methods on our dataset [19], [20], [17], [18], [16], [21]. In the skin tone comparison, we excluded the exercise, natural light, and walking conditions to eliminate any confounding factors and concentrate on the task at hand. ...
... As shown in Table 2, for unsupervised method comparison, the LGI [20] method performed best for relatively simple data from type 3 skin tone, while the POS [22] method had the best average performance for all conditions and robustness. For skin tone comparison, all the methods performed well on the data of skin type 3. ...
Remote photoplethysmography (rPPG) is an attractive method for noninvasive, convenient and concomitant measurement of physiological vital signals. Public benchmark datasets have served a valuable role in the development of this technology and improvements in accuracy over recent years.However, there remain gaps the public datasets.First, despite the ubiquity of cameras on mobile devices, there are few datasets recorded specifically with mobile phones cameras. Second, most datasets are relatively small and therefore are limited in diversity, both in appearance (e.g., skin tone), behaviors (e.g., motion) and enivornment (e.g., lighting conditions). In an effort to help the field advance, we present the Multi-domain Mobile Video Physiology Dataset (MMPD), comprising 11 hours of recordings from mobile phones of 33 subjects. The dataset was designed to capture videos with greater representation across skin tone, body motion, and lighting conditions. MMPD is comprehensive with eight descriptive labels and can be used in conjunction with the rPPG-toolbox. The Github repository of our dataset: {https://github.com/McJackTang/MMPD_rPPG_dataset}
... The third dataset LGI-PPGI [40] records the facial videos by Logitech HD C270 webcam (L-cam) of 20 males and 5 females in the age range of 25-42. It also provides standard PPG signals by CMS50E as ground truth. ...
... Data availability. The original video datasets are annotated by the relevant papers [38][39][40] and the datasets are available from the authors of the papers. The BVP data annotated by the quality score generated by our algorithm are available in [51]. ...
Remote photoplethysmogram (rPPG) is a low-cost method to extract blood volume pulse (BVP). Some crucial vital signs, such as heart rate (HR) and respiratory rate (RR) etc. can be achieved from BVP for clinical medicine and healthcare application. As compared to the conventional PPG methods, rPPG is more promising because of its non-contacted measurement. However, both BVP detection methods, especially rPPG, are susceptible to motion and illumination artifacts, which lead to inaccurate estimation of vital signs. Signal quality assessment (SQA) is a method to measure the quality of BVP signals and ensure the credibility of estimated physiological parameters. But the existing SQA methods are not suitable for real-time processing. In this paper, we proposed an end-to-end BVP signal quality evaluation method based on a long short-term memory network (LSTM-SQA). Two LSTM-SQA models were trained using the BVP signals obtained with PPG and rPPG techniques so that the quality of BVP signals derived from these two methods can be evaluated, respectively. As there is no publicly available rPPG dataset with quality annotations, we designed a training sample generation method with blind source separation, by which two kinds of training datasets respective to PPG and rPPG were built. Each dataset consists of 38400 high and low-quality BVP segments. The achieved models were verified on three public datasets (IIP-HCI dataset, UBFC-Phys dataset, and LGI-PPGI dataset). The experimental results show that the proposed LSTM-SQA models can effectively predict the quality of the BVP signal in real-time.
... After that, these methods carefully process these raw RGB signals to separate the physiological signals contained in the subtle variations of the skin color from the rest of the information (motion, illumination changes, or facial expressions, among others) by applying filtering and different ways of combining the RGB signals into an rPPG signal. RGB to rPPG conversion methods are based on several ideas such as signal decomposition (PCA, OMIT [27]), chrominance information (Green, CHROM [28], POS [29]), or self similarity (LGI [30]). ...
Depression is a mental illness that may be harmful to an individual's health. The detection of mental health disorders in the early stages and a precise diagnosis are critical to avoid social, physiological, or psychological side effects. This work analyzes physiological signals to observe if different depressive states have a noticeable impact on the blood volume pulse (BVP) and the heart rate variability (HRV) response. Although typically, HRV features are calculated from biosignals obtained with contact-based sensors such as wearables, we propose instead a novel scheme that directly extracts them from facial videos, just based on visual information, removing the need for any contact-based device. Our solution is based on a pipeline that is able to extract complete remote photoplethysmography signals (rPPG) in a fully unsupervised manner. We use these rPPG signals to calculate over 60 statistical, geometrical, and physiological features that are further used to train several machine learning regressors to recognize different levels of depression. Experiments on two benchmark datasets indicate that this approach offers comparable results to other audiovisual modalities based on voice or facial expression, potentially complementing them. In addition, the results achieved for the proposed method show promising and solid performance that outperforms hand-engineered methods and is comparable to deep learning-based approaches.
... Based on this model, several pulse extraction algorithms are proposed [9,18,19]. The role of a differentiable local group of local transformations was introduced by Pilz et al. [20]; they emphasized the point of view on the unsupervised learning of invariant features. To extend the utilization of rPPG sensors, Lee et al. [6] proposed an algorithm that estimates the heart rate, which can be performed in real time using vision and robot manipulation algorithms. ...
... In this section, we compare the proposed method with several state-of-the-art methods for averaging the heart rate prediction. Specifically, we analyze six well-known rPPG methods: ICA [42] and PCA [43] are used as the blind source separation methods, CHROM [9] and POS [8] are used as the skin reflection model methods, DAOMP [31] is used as the sparse representation method, and LGI [20] is used as the feature transform method. ...
In recent years, the physiological measurement based on remote photoplethysmography has attracted wide attention, especially since the epidemic of COVID-19. Many researchers paid great efforts to improve the robustness of illumination and motion variation. Most of the existing methods divided the ROIs into many sub-regions and extracted the heart rate separately, while ignoring the fact that the heart rates from different sub-regions are consistent. To address this problem, in this work, we propose a structural sparse representation method to reconstruct the pulse signals (SSR2RPS) from different sub-regions and estimate the heart rate. The structural sparse representation (SSR) method considers that the chrominance signals from different sub-regions should have a similar sparse representation on the combined dictionary. Specifically, we firstly eliminate the signal deviation trend using the adaptive iteratively re-weighted penalized least squares (Airpls) for each sub-region. Then, we conduct the sparse representation on the combined dictionary, which is constructed considering the pulsatility and periodicity of the heart rate. Finally, we obtain the reconstructed pulse signals from different sub-regions and estimate the heart rate with a power spectrum analysis. The experimental results on the public UBFC and COHFACE datasets demonstrate the significant improvement for the accuracy of the heart rate estimation under realistic conditions.
... LGI [22] The local group invariance (LGI) calculates an rPPG signal with a robust algorithm as a result of local transformations. ...
... For the evaluation, the LGI-PGGI dataset from Pilz et al. [22] was applied. It contains video recordings with the participants' faces in the center labelled with the referenced fingertip PPG signal. ...
... For the difficult video activities, Gym and Talking, LGI was one of the two best performing methods. This data makes sense because earlier studies from the group behind developing LGI [22] have shown that the rPPG method has a high level of motion and lightning resilience in heart rate estimation. CHROM appeared to perform well in the natural light of the Talking video activity. ...
The contactless recording of a photoplethysmography (PPG) signal with a Red-Green-Blue (RGB) camera is known as remote photoplethysmography (rPPG). Studies have reported on the positive impact of using this technique, particularly in heart rate estimation, which has led to increased research on this topic among scientists. Therefore, converting from RGB signals to constructing an rPPG signal is an important step. Eight rPPG methods (plant-orthogonal-to-skin (POS), local group invariance (LGI), the chrominance-based method (CHROM), orthogonal matrix image transformation (OMIT), GREEN, independent component analysis (ICA), principal component analysis (PCA), and blood volume pulse (PBV) methods) were assessed using dynamic time warping, power spectrum analysis, and Pearson’s correlation coefficient, with different activities (at rest, during exercising in the gym, during talking, and while head rotating) and four regions of interest (ROI) : the forehead, the left cheek, the right cheek, and a combination of all three ROIs. The best performing rPPG methods in all categories were the POS, LGI, and OMI methods; each performed well in all activities. Recommendations for future work are provided.
... We trained a state-of-the-art model as our user authentication target model (see Section IV-B). The threat of rPPG signals to IPI-based security protocols is explored on four datasets -PURE [38], UBFC rPPG [39] (UBFC 1, UBFC 2), LGI PPGI [40] and COHFACE [41]. We use trend-based and quantile functionbased as our target model (see Section III-C) for IPI-based security protocols. ...
... Other users are treated as non-victims. As shown in Table I To analyze the threat of rPPG for IPI-based protocols, we considered the four most commonly used rPPG datasets, PURE [38], UBFC rPPG [39], LGI PPGI [40] and CO-HFACE [41]. ...
Recently, physiological signal-based biometric systems have received wide attention. Unlike traditional biometric features, physiological signals can not be easily compromised (usually unobservable to human eyes). Photoplethysmography (PPG) signal is easy to measure, making it more attractive than many other physiological signals for biometric authentication. However, with the advent of remote PPG (rPPG), unobservability has been challenged when the attacker can remotely steal the rPPG signals by monitoring the victim's face, subsequently posing a threat to PPG-based biometrics. In PPG-based biometric authentication, current attack approaches mandate the victim's PPG signal, making rPPG-based attacks neglected. In this paper, we firstly analyze the security of PPG-based biometrics, including user authentication and communication protocols. We evaluate the signal waveforms, heart rate and inter-pulse-interval information extracted by five rPPG methods, including four traditional optical computing methods (CHROM, POS, LGI, PCA) and one deep learning method (CL_rPPG). We conducted experiments on five datasets (PURE, UBFC_rPPG, UBFC_Phys, LGI_PPGI, and COHFACE) to collect a comprehensive set of results. Our empirical studies show that rPPG poses a serious threat to the authentication system. The success rate of the rPPG signal spoofing attack in the user authentication system reached 0.35. The bit hit rate is 0.6 in inter-pulse-interval-based security protocols. Further, we propose an active defence strategy to hide the physiological signals of the face to resist the attack. It reduces the success rate of rPPG spoofing attacks in user authentication to 0.05. The bit hit rate was reduced to 0.5, which is at the level of a random guess. Our strategy effectively prevents the exposure of PPG signals to protect users' sensitive physiological data.
... The development of rPPG technology emphasizes the importance of fair comparison of different algorithms, and promotes the research of repeatability. Thus, Boccignone et al. [29] provided a systemic open framework, allowing to assess eight traditional rPPG algorithms (ICA [18], PCA [24], GREEN [1], CHROM [27], POS [26], SSR [30], LGI [31], PBV [28]), by setting parameters and metaparameters. ...
Remote photoplethysmography (rPPG), which aims at measuring heart activities without any contact, has great potential in many applications. The emergence of novel coronavirus pneumonia COVID-19 has attracted worldwide attentions. Contact photoplethysmography (cPPG) methods need to contact the detection equipment with the patient, which may accelerate the spread of the epidemic. In the future, the non-contact heart rate detection will be an urgent need. However, existing heart rate measuring methods from facial videos are vulnerable to the less-constrained scenarios (e.g., with head movement and wearing a mask). In this paper, we proposed a method of heart rate detection based on eye location of region of interest (ROI) to solve the problem of missing information when wearing masks. Besides, a model to filter outliers based on residual network was conceived first by us and the better heart rate measurement accuracy was generated. To validate our method, we also created a mask dataset. The results demonstrated that after using our method for correcting the heart rate (HR) value measured with the traditional method, the accuracy reaches 4.65 bpm, which is 0.42 bpm higher than that without correction.
... As a matter of fact, rPPG has sparked great interest by 1 Freely available on GitHub: https: //github.com/phuselab/pyVHR. fostering the opportunity for measuring PPG at distance (e.g., remote health assistance) or in all those cases where contact has to be prevented (e.g., surveillance, fitness, health, emotion analysis) (Aarts et al., 2013;McDuff, Gontarek & Picard, 2014;Ramírez et al., 2014;Boccignone et al., 2020b;Rouast et al., 2017). Indeed, the rPPG research field has witnessed a growing number of techniques proposed for making this approach more and more robust and thus viable in contexts facing challenging problems such as subject motion, ambient light changes, low-cost cameras (Lewandowska et al., 2011;Verkruysse, Svaasand & Nelson, 2008;Tarassenko et al., 2014;Benezeth et al., 2018;Wang et al., 2016;Wang, Stuijk & De Haan, 2015;Pilz et al., 2018;De Haan & Van Leest, 2014). More recently, alongside the traditional methods listed above, rPPG approaches based on deep learning (DL) have burst into this research field (Chen & McDuff, 2018;Niu et al., 2019;Yu et al., 2020;Liu et al., 2020;Liu et al., 2021;Gideon & Stent, 2021;Yu et al., 2021). ...
... In Pilz (2019) a MATLAB toolbox is presented, implementing two newly proposed methods, namely Local Group Invariance (LGI) (Pilz et al., 2018) and Riemannian-PPGI (SPH) (Pilz, 2019), and comparing them to the GREEN channel expectation (Verkruysse, Svaasand & Nelson, 2008) baseline, and two state-of-the-art methods, i.e., Spatial Subspace Rotation (SSR) (Wang, Stuijk & De Haan, 2015), and Projection Orthogonal to Skin (POS) (Wang et al., 2016). ...
... The resulting mask is employed to isolate the pixels that are generally associated to the skin. An example is shown in the left image of Fig. 3 on a subject of the LG-PPGI dataset (Pilz et al., 2018). 3 Alternatively, the Face parsing extractor computes a semantic segmentation of the subject's face. ...
Remote photoplethysmography (rPPG) aspires to automatically estimate heart rate (HR) variability from videos in realistic environments. A number of effective methods relying on data-driven, model-based and statistical approaches have emerged in the past two decades. They exhibit increasing ability to estimate the blood volume pulse (BVP) signal upon which BPMs (Beats per Minute) can be estimated. Furthermore, learning-based rPPG methods have been recently proposed. The present pyVHR framework represents a multi-stage pipeline covering the whole process for extracting and analyzing HR fluctuations. It is designed for both theoretical studies and practical applications in contexts where wearable sensors are inconvenient to use. Namely, pyVHR supports either the development, assessment and statistical analysis of novel rPPG methods, either traditional or learning-based, or simply the sound comparison of well-established methods on multiple datasets. It is built up on accelerated Python libraries for video and signal processing as well as equipped with parallel/accelerated ad-hoc procedures paving the way to online processing on a GPU. The whole accelerated process can be safely run in real-time for 30 fps HD videos with an average speedup of around 5. This paper is shaped in the form of a gentle tutorial presentation of the framework.
... In 2017, Wang et al. [20] proposed a new plane-orthogonal-to-skin (POS) algorithm that finds pulsatile signals in an RGB normalized space orthogonal to the skin tone. In 2018, Pilz et al. [21] proposed the Local Group Invariance (LGI) method, a stochastic representation of the pulse signal based on a model that leverages the local invariance of the heart rate, as a quasi-periodical process dynamics, obtained by recursive inference to remove extrinsic factors such as head motion and lightness variations. In 2020, Gudi et al. [22] proposed an unsupervised method with emphasis on motion suppression and novel filtering based the head-orientations (FaceRPPG). ...
... RGB to PPG transformation (rPPG): The Baseline pipeline includes several reference methods such as POS [20], CHROM [15], GREEN [12], PCA [14], ICA [13], SSR [18], LGI [21] and PVB [16]. We have added one method based on selecting the chroma channel a after applying a CIE Lab color space transformation. ...
... LGI-PPGI-Face-Video-Database is a database that contains 25 subjects, but only 6 were released officially. The age of the participants, primarily Caucasian, is in the range of 25-42 years [21]. It was recorded using a Logitech HD C270 webcam at a sampling rate of 25 Hz and a resolution of 640x480 pixels, in uncompressed format with auto-exposure. ...
Photoplethysmography (PPG) signals have become a key technology in many fields such as medicine, well-being, or sports. Our work proposes a set of pipelines to extract remote PPG signals (rPPG) from the face, robustly, reliably, and in a configurable manner. We identify and evaluate the possible choices in the critical steps of unsupervised rPPG methodologies. We evaluate a state-of-the-art processing pipeline in six different datasets, incorporating important corrections in the methodology that ensure reproducible and fair comparisons. In addition, we extend the pipeline by proposing three novel ideas; 1) a new method to stabilize the detected face based on a rigid mesh normalization; 2) a new method to dynamically select the different regions in the face that provide the best raw signals, and 3) a new RGB to rPPG transformation method called Orthogonal Matrix Image Transformation (OMIT) based on QR decomposition, that increases robustness against compression artifacts. We show that all three changes introduce noticeable improvements in retrieving rPPG signals from faces, obtaining state-of-the-art results compared with unsupervised, non-learning-based methodologies, and in some databases, very close to supervised, learning-based methods. We perform a comparative study to quantify the contribution of each proposed idea. In addition, we depict a series of observations that could help in future implementations.
