Conference PaperPDF Available

Remote Photoplethysmography: Evaluation of Contactless Heart Rate Measurement in an Information Systems Setting

Authors:

Abstract and Figures

As a source of valuable information about a person's affective state, heart rate data has the potential to improve both understanding and experience of human-computer interaction. Conventional methods for measuring heart rate use skin contact methods, where a measuring device must be worn by the user. In an Information Systems setting, a contactless approach without interference in the user's natural environment could prove to be advantageous. We develop an application that fulfils these conditions. The algorithm is based on remote photoplethysmography, taking advantage of the slight skin color variation that occurs periodically with the user's pulse. When evaluating this application in an Information Systems setting with various arousal levels and naturally moving subjects, we achieve an average root mean square error of 7.32 bpm for the best performing configuration. We find that a higher frame rate yields better results than a larger size of the moving measurement window. Regarding algorithm specifics, we find that a more detailed algorithm using the three RGB signals slightly outperforms a simple algorithm using only the green signal.
Content may be subject to copyright.
A preview of the PDF is not available
... More recent publications, i.e., in 2008 [12], show that PPG could be performed remotely (i.e., rPPG) using ambient light as the optical source. Many other rPPG focused studies were published shortly after [5,[13][14][15][16][17]. Some surveys on the state of the art of this field could be found in [1,[18][19][20]. ...
... In order to test the advantage of using a deep learning skin detection algorithm instead of a classical face detection method, a specific experiment has been performed. In particular, the heart rate estimation obtained with the method described in Section 3.3.1 has been compared to the one obtained with a classical rPPG approach [14]. In classic rPPG, an optimal face region (usually the forehead) is detected by applying a fixed proportion to a bounding box obtained with classical face detection methods (e.g., [29]). ...
... In particular, the main motivation for utilizing a segmentation method was to be able to use all the possible pixel surface related to the heart activity. As a matter of fact, using a traditional forehead region adopted in many rPPG systems [14], given the very low spatial resolution of SPAD cameras, would result in selecting very few pixels for the pulse signal estimation. The results reported in Section 5.3 show a slight increment in heart rate estimation accuracy while using the deep learning skin segmentation method instead of the forehead region obtained with traditional computer vision techniques. ...
Article
Full-text available
The problem of performing remote biomedical measurements using just a video stream of a subject face is called remote photoplethysmography (rPPG). The aim of this work is to propose a novel method able to perform rPPG using single-photon avalanche diode (SPAD) cameras. These are extremely accurate cameras able to detect even a single photon and are already used in many other applications. Moreover, a novel method that mixes deep learning and traditional signal analysis is proposed in order to extract and study the pulse signal. Experimental results show that this system achieves accurate results in the estimation of biomedical information such as heart rate, respiration rate, and tachogram. Lastly, thanks to the adoption of the deep learning segmentation method and dependability checks, this method could be adopted in non-ideal working conditions—for example, in the presence of partial facial occlusions.
... Contact photoplethysmography (PPG) is a simple technique that traces back to 1930s [9] in which a light is used to measure blood volume changes related to the pulsating nature of circulatory systems [10]. In more recent years, starting from 2008, it was demonstrated [11] that PPG could be performed remotely using ambient light and since then many studies focused on the extraction of heart rate using videocameras were published [1], [12], [13], [14], [15], [16], [17]. The goal of most recent studies is to obtain the tachogram, which is defined as a chart reporting time on the x axes and Inter Beat Interval (IBI) on the y axes, by performing remote-photoplethysmography [12]. ...
... Particular attention is commonly paid to acquisition frequency, but different works provide different values: Poh considers 15 fps [18], while others acquire at 20 to 60 fps. Some works consider regions on which to extract the pulse signal by manually choosing the pixels of the image corresponding to the skin of the subject [1], [13], but this is not a feasible choice in an automatic application. On the other hand, some modern rPPG applications involve the use of face detection and tracking algorithms [18], [20] in order to select an appropriate Region Of Interest (ROI). ...
... On the other hand, some modern rPPG applications involve the use of face detection and tracking algorithms [18], [20] in order to select an appropriate Region Of Interest (ROI). While using a RGB camera instead of a monochrome one could lead to some benefits for rPPG applications [12], V. Rouast et al. [13] arrived to the conclusion that the G channel contains enough information and also recommend the use of the single G channel in order to reduce computational costs and to implement online analysis. ...
... All the features are classified into several stages, with each stage having a definite but not specified number of features. Each stage decides if a particular subwindow is immediately discarded as not a face if it fails in any of the stages [42,43]. ...
... At this stage, the signal with a distinct periodicity obtained from the signal extraction stage is converted to the frequency domain. In the frequency domain, the frequency that matches the index with the highest spectral power is selected as the estimated heart rate frequency [43]. In order to compute the heart rate in bpm, we used the formula given in Eq. (8) [23]. ...
Article
Full-text available
Heart rate monitors operated without actual contact with the human body are gaining widespread popularity in modern healthcare systems. These monitors help medical personnel remotely and non-invasively monitor the heart rates of patients. The traditional pulse oximeters exhibit significant limitations, such as the inability to prevent random data capturing when the subject is out of focus and the need to manually initiate capture via the keyboard input. Additionally, it is pretty difficult to measure the heart rate from a prerecorded video clip of a human face and automatically track the face of a stable subject. To address these inherent limitations of the traditional pulse oximeters, we propose an enhanced, cost-effective webcam-based pulse detector to conduct remote heartbeat monitoring of a subject. Specifically, we modify the original software developed by Tristan Hearn of the NASA Glenn Study Center using a colour model based on red, green, and blue imaging to collect signals from a subject several meters away from a camera. The modification gave birth to an efficient, cost-effective and robust software for continuous heart rate monitoring via a webcam. To test the validity of the modified software, twenty subjects were employed to examine the correlation of the modified software relative to a reference hospital-approved pulse oximeter. The correlation coefficient of 0.779 indicates that the accuracy of the enhanced software compared to an existing pulse oximeter is highly favourable.
... This general framework can be used to design rPPG algorithms for specific situations. Based on this general framework, Rouast et al. implemented a heart rate monitoring application in [25,26]. Moreover, they provided the parameters which can be adjusted to increase the accuracy. ...
Conference Paper
Full-text available
In this work, a real-time unobtrusive heart rate monitoring system is proposed and implemented. The proposed system aims to monitor the heart rate of the passengers by using a low-cost camera, which can be readily embedded in the car's rear-view mirror. Additionally, we integrate this system with the main system of our test driverless car, and we propose how driverless cars should act in response to serious medical emergency situations. Moreover, we investigate how this system can benefit from the promising features of Google I/O and Google AI. Our approach is based on Remote Photoplethysmography (rPPG), in which the heart rate is extracted from the subtle tiny changes occurring in the skin color of the face during every pulsation. The face is automatically detected and tracked, then the raw signal is calculated from each frame over a 10-seconds sliding window. After that, a series of signal processing techniques are implemented on the raw signals to recover the heart rate frequency. Finally, the resultant heart rate measurements are processed and stored, then we compare it with ground truth measurements values obtained using pulse oximeter.
Conference Paper
Heart rate estimation from fitness plays an important role in the evaluation of fitness exercises. Conventional approaches use the photoplethysmography (PPG) sensor to consider the change of light absorption on the wrist skin for heart rate estimation. However, users are required to buy smartwatches for using this function. Various approaches based on video analysis are recently implemented for surveillance purpose. However, it is unstable for motion scenario such as fitness exercises due to the color distortion induced by movement. POS and CHROM are introduced to address this issue. Since the fixed projection planes from POS and CHROM are given in several sources of light, it is not widely applied for surveillance applications. Therefore, a novel projection plane that is adaptively changed with the lighting environment is proposed to estimate the heart rate from fitness videos in ambient light. Moreover, image and digital signal processing techniques are also applied to extract the clean pulse signal from a novel projection plane. From the experiments conducted, the proposed approach outperformed the existing approaches to be the best model for heart rate estimation from fitness videos with the accuracy up to 91.08%.
Article
Full-text available
Stress is a major problem in the human society, impairing the well-being, health, performance, and productivity of many people worldwide. Most notably, people increasingly experience stress during human-computer interactions because of the ubiquity of and permanent connection to information and communication technologies. This phenomenon is referred to as technostress. Enterprise systems, designed to improve the productivity of organizations, frequently contribute to this technostress and thereby counteract their objective. Based on theoretical foundations and input from exploratory interviews and focus group discussions, the paper presents a design blueprint for stress-sensitive adaptive enterprise systems (SSAESes). A major characteristic of SSAESes is that bio-signals (e.g., heart rate or skin conductance) are integrated as real-time stress measures, with the goal that systems automatically adapt to the users’ stress levels, thereby improving human-computer interactions. Various design interventions on the individual, technological, and organizational levels promise to directly affect stressors or moderate the impact of stressors on important negative effects (e.g., health or performance). However, designing and deploying SSAESes pose significant challenges with respect to technical feasibility, social and ethical acceptability, as well as adoption and use. Considering these challenges, the paper proposes a 4-stage step-by-step implementation approach. With this Research Note on technostress in organizations, the authors seek to stimulate the discussion about a timely and important phenomenon, particularly from a design science research perspective.
Article
Full-text available
Remote Photoplethysmography (rPPG) allows remote measurement of the heart rate using low-cost RGB imaging equipment. In this paper, we review the development of the field since its emergence in 2008, classify existing approaches for rPPG, and derive a framework that provides an overview of modular steps. Based on this framework, practitioners can use the classification to orchestrate algorithms to an rPPG approach that suits their specific needs. Researchers can use the reviewed and classified algorithms as a starting point to improve particular features of an rPPG algorithm.
Article
Full-text available
Environmental conditions and the interplay of cognitive and affective processes both exert influences on bidding behavior. This paper brings the above together, considering how the (external) auction environment determines the impact of (internal) cognitive and affective processes on bidding behavior, assessed in comparison to the optimal bid. Two aspects of the auction environment were considered, namely auction dynamics (low: first-price sealed-bid auction, high: Dutch auction) and value uncertainty (low, high). In a laboratory experiment, we assess bidders’ cognitive workload and emotional arousal through physiological measurements. We find that higher auction dynamics increase the impact of emotional arousal on bid deviations, but not that of cognitive workload. Higher value uncertainty, conversely, increases the impact of cognitive workload on bid deviations, but not that of emotional arousal. Taken together, the auction environment is a critical factor in understanding the nature of the underlying decision process and its impact on bids. © 2016 Institute of Applied Informatics at University of Leipzig
Conference Paper
Full-text available
Heart rate is an important indicator of people's physiological state. Recently, several papers reported methods to measure heart rate remotely from face videos. Those methods work well on stationary subjects under well controlled conditions, but their performance significantly degrades if the videos are recorded under more challenging conditions, specifically when subjects' motions and illumination variations are involved. We propose a framework which utilizes face tracking and Normalized Least Mean Square adaptive filtering methods to counter their influences. We test our framework on a large difficult and public database MAHNOB-HCI and demonstrate that our method substantially outperforms all previous methods. We also use our method for long term heart rate monitoring in a game evaluation scenario and achieve promising results.
Article
In the NeuroIS field, experimental software needs to simultaneously present experimental stimuli to participants while recording, analyzing, or displaying neurophysiological measures. For example, a researcher might record a user’s heart beat (neurophysiological measure) as the user interacts with an e-commerce website (stimulus) to track changes in user arousal or show a user’s changing arousal levels during an exciting game. In this paper, we identify requirements for a NeuroIS experimental platform that we call Brownie and present its architecture and functionality. We then evaluate Brownie via a literature review and a case study that demonstrates Brownie’s capability to meet the requirements in a complex research context. We also verify Brownie’s usability via a quantitative study with prospective experimenters who implemented a test experiment in Brownie and an alternative software. We summarize the salient features of Brownie as follows: 1) it integrates neurophysiological measurements, 2) it incorporates real-time processing of neurophysiological data, 3) it facilitates research on individual and group behavior in the lab, 4) it offers a large variety of options for presenting experimental stimuli, and 5) it is open source and easily extensible with open source libraries. In summary, we conclude that Brownie is innovative in its potential to reduce barriers for IS researchers by fostering replicability and research collaboration and to support NeuroIS and interdisciplinary research in cognate areas, such as management, economics, or human-computer interaction.
Conference Paper
Heart rate measurements contain valuable information about a person’s affective state. There is a wide range of application domains for heart rate-based measures in information systems. To date, heart rate is typically measured using skin contact methods, where users must wear a measuring device. A non-contact and easy to use mobile approach, allowing heart rate measurements without interfering with the users’ natural environment, could prove to be a valuable NeuroIS tool. Hence, our two research objectives are (1) to develop an application for mobile devices that allows for non-contact, real-time heart rate measurement and (2) to evaluate this application in an IS context by benchmarking the results of our approach against established measurements. The proposed algorithm is based on non-contact photoplethysmography and hence takes advantage of slight skin color variations that occurs periodically with the user’s pulse.
Article
Internet auction sites frequently employ images as design elements on their websites in order to either induce a sense of community or competition among the bidders. In this paper, we investigate the impact of such affective images on bidding behavior in a controlled laboratory experiment during which participants' emotional processes are assessed through psychophysiological measurements. Immediately before placing a bid in a first-price sealed-bid auction, bidders are presented a) pictures of competitive sports scenes, b) pictures of families or children, or c) a blank screen. Participants place significantly lower bids when they were exposed to pictures that induce competition emotions as opposed to pictures that induce community emotions. This relationship is moderated by the bidders' emotion regulation strategy. In particular, we find that the more participants try to suppress their emotional responses to the presented images, the more they are affected in their bidding behavior. Our results entail valuable insights about the coherence of emotional stimuli on Internet auction marketplaces and customers' decisions. They also question recent marketing strategies by the market leader.
Article
Camera-based remote photoplethysmography (rPPG) is a technique that can be used to measure vital signs contactlessly. In order to optimize the extraction of photoplethysmographic signals from video sequences, we investigate the spatial dependence of the photoplethysmographic signal. For an evaluation of the suitability of various regions of interest for rPPG measurements, we conducted a study on 20 healthy subjects. We analysed the videos using a refined pulse amplitude mapping approach. Our results show that the signal-to-noise ratio of rPPG signals can be improved by limiting the region of interest to certain regions of the face.
Conference Paper
Remote imaging photoplethysmography (RIPPG) is able to access human vital signs without physical contact. However most of the conventional RIPPG approaches are susceptive to motions of subjects or camera. Overcoming motion artifacts presents one of the most challenging problems. Focusing on the motion artifacts problem, the effects of motion artifacts on RIPPG signals were analyzed. In order to suppress motion artifacts for RIPPG, region of interest (ROI) is stabilized by using face tracking based on feature points tracking. And adaptive bandpass filter is further used to suppress the residual motion artifacts. With the addition of motion artifacts, the sorting of independent component analysis (ICA) outputs becomes more important, hence reference sine signals are generated to be correlated with ICA output components, and the cardiac pulse wave is automatically picked up from ICA output components, with the largest correlation coefficient. Fourteen subjects were enrolled to test the robustness with large motion artifacts for the proposed RIPPG method. Experimental results show that the proposed method could obtain a much better performance in accessing pulse rates for moving subjects, compared to the state-of-the-art method. The effectiveness of our method in motion artifacts suppression was verified by comparison with a commercial oximeter using Bland-Altman analysis and Pearson's correlation. With the efficient motion artifact suppression, RIPPG method has good potential in broadening the application of vital signs accesses.