Using Rear Smartphone Cameras as Sensors for Measuring Heart Rate Variability

Abstract

The measurement of heart rate variability (HRV) is the preferred method for assessing the function of the autonomic nervous system (ANS). Traditional HRV detection requires an electrocardiogram (ECG) or photoelectric sensor. In this paper, we propose a new method for HRV measurement using a rear smartphone camera as a sensor. Video signals from the fingertips of 24 college students were acquired using the rear camera of an HTC M8d smartphone. ECG signals were simultaneously recorded as reference. The video signals were converted into single-frame image sequences over time. Each image frame was transformed into point data through superpositioning of pixel color attribute values and averaging according to space. The point data were sorted by time to obtain a photoplethysmogram (PPG). Finally, the Hilbert transform was used to extract the pulse-to-pulse interval and the R-to-R interval for the PPG and ECG, respectively. Sixteen HRV parameters (mean HR, max HR, min HR, SDNN, RMSSD, NN50, pNN50, VLF, LF, HF, TP, LFnu, HFnu, LF/HF, SD1, and SD2) were analyzed. All 16 HRV parameters were highly correlated (all rs > 0.95, ps < 0.05), and the effect size (ES) differences were small (ES < 0.175) for all indices except for RMSSD, HF, and SD1. Compared with the ECG method, the errors of the 13 HRV parameters measured using this method were within acceptable ranges. The results suggest that this technique can be used as a convenient method to assess and quantify ANS activity and balance.

Full text

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
VOLUME XX, 2020 1
Date of publication xxxx 00, 2020, date of current version xxxx 00, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.Doi Number
Using Rear Smartphone Cameras as Sensors
for Measuring Heart Rate Variability
GENXUAN ZHANG1,2, SAI ZHANG1, YIMING DAI1, AND BO SHI1,2
1School of Medical Imaging, Bengbu Medical College, Bengbu, Anhui 233030, China
2Anhui Key Laboratory of Computational Medicine and Intelligent Health, Bengbu Medical College, Bengbu, Anhui 233030, China
Corresponding author: Bo Shi (shibo@bbmc.edu.cn)
This research was funded in part by the “512” Outstanding Talents Fostering Project of Bengbu Medical College under Grant 2020-42-3-12, in part by the
Scientific Research Innovation Project of Bengbu Medical College under Grant BYKC201905, and in part by the Natural Science Foundation of Bengbu
Medical College under Grant BYKY2019023ZD.
ABSTRACT The measurement of heart rate variability (HRV) is the preferred method for assessing the
function of the autonomic nervous system (ANS). Traditional HRV detection requires an electrocardiogram
(ECG) or photoelectric sensor. In this paper, we propose a new method for HRV measurement using a rear
smartphone camera as a sensor. Video signals from the fingertips of 24 college students were acquired using
the rear camera of an HTC M8d smartphone. ECG signals were simultaneously recorded as reference. The
video signals were converted into single-frame image sequences over time. Each image frame was
transformed into point data through superpositioning of pixel color attribute values and averaging according
to space. The point data were sorted by time to obtain a photoplethysmogram (PPG). Finally, the Hilbert
transform was used to extract the pulse-to-pulse interval and the R-to-R interval for the PPG and ECG,
respectively. Sixteen HRV parameters (mean HR, max HR, min HR, SDNN, RMSSD, NN50, pNN50, VLF,
LF, HF, TP, LFnu, HFnu, LF/HF, SD1, and SD2) were analyzed. All 16 HRV parameters were highly
correlated (all rs > 0.95, ps < 0.05), and the effect size (ES) differences were small (ES < 0.175) for all indices
except for RMSSD, HF, and SD1. Compared with the ECG method, the errors of the 13 HRV parameters
measured using this method were within acceptable ranges. The results suggest that this technique can be
used as a convenient method to assess and quantify ANS activity and balance.
INDEX TERMS heart rate variability, photoplethysmography, smartphone, camera, video
I. INTRODUCTION
Over the past decade, the advent of smartphones has
revolutionized daily life. Today, smartphones are no longer
simple communication tools; they also have many functions,
including photography, payment, and entertainment. With
advances in electronic technology and cloud computing,
healthcare fields will likely be transformed by smartphones.
Currently, smartphones can be used for monitoring personal
health by measuring physiological parameters, such as
glucose [1], immunoglobulin G [2], and serum bilirubin
levels [3], blood pressure [4], electrocardiograms (ECGs) [5],
and heart rate variability (HRV) [6], using built-in or external
sensors. Among these parameters, HRV, which measures
small differences in time between successive normal (sinus)
cardiac cycles, is recognized as the preferred metric for
quantitatively evaluating the function of the autonomic
nervous system (ANS) [7]. Numerous articles have reported
inverse relationships of HRV parameters to age [8], [9] and
exercise [10], [11]. HRV parameters significantly decreases
during disease states, such as diabetes [12], hypertension
[13], and cancer [14]. Therefore, monitoring HRV
parameters offers important reference values for quantitative
health assessment and management.
HRV analysis is generally based on two time series,
namely, the R-to-R interval (RRI) of an ECG and the pulse-
to-pulse interval (PPI) of a photoplethysmogram (PPG) [15].
Studies have shown that time- and frequency-domain HRV
parameters analyzed by PPI are negligibly different from
those conducted using RRI [16]-[18]. Regardless of the
approach, the use of smartphones for HRV measurements
requires external or built-in professional sensors and adds
use-related costs to the user. In recent years, high-definition
cameras have become standard issue in smartphones. Many
scholars have used smartphone cameras as sensors for

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
2
studies on heart rate (HR) monitoring and HRV
measurements using contact or noncontact video PPG (vPPG)
technology [19]-[26]. In general, the noncontact vPPG is
more convenient in the application of continuous dynamic
monitoring, but the measurement accuracy of the contact
vPPG is higher. For HR monitoring, the use of the
noncontact method meets measurement accuracy
requirements. For example, in 2010, Poh et al. [19] proposed
a method for measuring the HR using blind source separation
(BSS) of facial video imaging. When subjects were at rest,
the mean value of the HR error was only -0.05 bpm, and the
range of 95% of the limits of agreement (LoA) was (−4.6,
4.4) bpm, i.e., nearly medical-grade levels. Measuring the
HRV does not require continuous monitoring but does
require higher PPI accuracy, which often cannot be met by
the noncontact method. Peng et al. [6] used contact vPPG to
analyze 16 HRV parameters. Compared with the ECG
method, 14 parameters were highly correlated (r > 0.7, p <
0.001), and the errors of seven parameters were within
acceptable ranges. Most existing smartphone camera-based
HRV analysis methods are algorithmically complex and
have limited accuracy. In this study, a simple video
processing algorithm is proposed. Compared with the ECG
method, the analyzed HRV time- and frequency-domain
parameters were highly correlated (r > 0.95, p < 0.001).
II. METHOD
A. SUBJECTS AND DATA COLLECTION
Twenty-four college students at Bengbu Medical College,
China, participated in this study, with 13 male students and
11 female students. Their mean (± standard deviation) age
was 20.6 ± 1.0 years, their height was 1.69 ± 0.07 m, their
body weight was 61.5 ± 10.4 kg, and they had a body mass
index of 21.5 ± 2.7 kg/m2. None of the subjects had a history
of heart disease or hypertension, and they were informed of
the purpose and details of the experiment before
participating. The study was conducted in accordance with
the Declaration of Helsinki, and the protocol was approved
by the Ethics Committee of Bengbu Medical College.
Before the tests, we instructed the subjects. They were
asked to keep their fingers clean, apply moderate pressure on
the smartphone camera, and remain quiet during the test.
Approximately 30% of subjects needed to repeat the test two
to three times to obtain a satisfactory signal. Signal
satisfaction was dependent on whether 95% PPI was
extracted. During the experiment, a video signal from the
fingertip of an index finger of each subject was obtained
using the rear camera of an HTC M8d smartphone (HTC,
Taiwan, China) (Figure 1); simultaneously, the ECG signal
of the subjects was recorded synchronously using a
HeaLink-R211B micro-ECG recorder (HeaLink, Ltd.,
Bengbu, China) as reference. The rear camera of the HTC
M8d was an HTC 4 million UltraPixel camera with the
following settings: Full HD 1920 × 1080 video quality and
square (1:1) cropping. The slow-motion mode was selected
when recording. The sampling rate of the micro-ECG
recorder was set at 400 Hz, and disposable Ag/AgCl ECG
electrodes (Junkang, Ltd., Shanghai, China) were used for
signal acquisition.
FIGURE 1. A video signal of the index fingertip which was obtained by
using the rear camera of an HTC M8d smartphone.
B. VIDEO SIGNAL PROCESSING
The collected fingertip video signals were processed
according to the following steps, and a flow chart of the
entire process is shown in Figure 2.
First, the acquired video signal is converted to a single-
frame image sequence over time. The video signal
acquisition time is 120 s for each subject. X is the number of
frames of the video. Assuming that the frame rate of the
video acquisition is F frames/s, a video of 120 s will be
composed of 120*F frame images, and X will be further
scaled by a factor of 4 when filmed in 4x slow-motion mode.
The frame rate F is 25 frames/s for the HTC M8d smart
phone. Thus, X = 12,000 for a video of 120 s taken in 4x
slow-motion mode (i.e., 120*25*4).
Second, each frame image in the image sequence is
transformed into point data by pixel superposition and
averaging according to the space. For any jth frame image,
assuming N pixels and the ith pixel color attribute value is Pj
(i), the pixel values of all of points are superimposed and
averaged such that the average pixel value Pj of this frame
image is obtained. The formula is expressed as
󰇛󰇜

 (1)
In this manner, the jth frame image is converted from a
two-dimensional image to point data; then, X frames of the
image is converted to X point data, namely, P1, P2, P3 … Px.
Finally, the PPG pulse wave is obtained by sorting these
point data across time.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
3
FIGURE 2. Overall diagram of the video signal processing. The acquired video signal was converted to a single-frame image sequence over time.
Then, each frame image in the image sequence was transformed into point data by pixel superposition and averaging according to the space. Finally,
the PPG pulse wave can be obtained by sorting these point data across time.
The waveform of the pulse wave obtained through video
signal processing may experience baseline drift and various
sources of interference; therefore, further digital filtering of
this pulse wave may be necessary. Commonly used digital
filtering algorithms for pulse waves include wavelet filtering
[27], independent component analysis (ICA) [28], Kalman
filtering [29], and mathematical morphological filtering [30].
A Chebyshev type II filter [31] was used in this study.
According to the passband cutoff frequency for pulse waves,
stopband initial cutoff frequency, maximum passband
attenuation decibel, minimum stopband attenuation decibel,
and other requirements, the fluctuation coefficient and filter
order was calculated. Ultimately, the system function of the
filter was obtained. Figure 3 shows the waveform
comparison of pulse waves before and after Chebyshev II
filtering.
FIGURE 3. Comparison of waveforms before and after Chebyshev filtering.
C. FEATURE EXTRACTION
Feature value extraction of synchronously acquired ECG
signals and PPG signals is needed for the next step in the
analysis. The ECG signal is used to extract the R wave and
calculate its RRI time series; the PPG signal is used to extract
the peak values and calculate the PPI time series (Figure 4).
In the experiment, we synchronized ourselves with Beijing
time. Manual checks were then performed to ensure that each
pair of RRI and PPI was aligned.
The methods for extracting the feature values of the ECG
and PPG signals include finite difference method [32],
wavelet transform method [33], and morphological method
[34]. In this study, the Hilbert transform was used for the
extraction. The Hilbert transform is a linear transformation.
For signal x (t), the transformation [35],[36] can be defined
as

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
4
FIGURE 4. The RRI and PPI extracted from the simultaneously recorded ECG and pulse wave signals respectively. RRI is calculated based on the R to
R wave of ECG. PPI is calculated from the peak to peak of the pulse wave.
󰇛󰇜 󰇟󰇛󰇜󰇠  
󰇛󰇜

 
  (2)
Equation (2) indicates that the time-independent
components remain unchanged before and after the
transformation; thus, x (t) can be expressed as the
convolution form of x (t) and (πt)-1,
󰇛󰇜
 󰇛󰇜. (3)
By performing Fourier transformations on both sides of
equation (3), we obtain
󰇝󰇛󰇜󰇞 
󰇥
󰇦󰇝󰇛󰇜󰇞 (4)
Because
󰇥
󰇦


    (5)
where sgn f is a sign function (i.e., 1 when f > 0, 0 when f
= 0, and − 1 when f < 0), the Fourier transform of the Hilbert
transform x (t) from the original signal x (t) can be expressed
as
󰇝󰇛󰇜󰇞  󰇝󰇛󰇜󰇞 (6)
Because the Hilbert transform is an odd function, after the
Hilbert transformation of the signal, the inflection point of
the original signal corresponds to the zero-crossing point of
its Hilbert transform signal. For the zero-crossing point, the
extreme point will appear in the Hilbert transform. Using this
property of the Hilbert transform, the position of the R-wave
in the ECG signal or the position of the peak in the PPG
signal is determined such that the RRI and PPI are calculated
(Figure 5).
FIGURE 5. A typical PPI extraction instance.
D. HRV ANALYSIS
The HRV time domain indices include the mean HR,
maximum HR (Max HR), minimum HR (Min HR), the
standard deviation of all normal-to-normal (NN) intervals
(SDNN), the root mean square of the successive differences
in the adjacent NN (RMSSD), the number of pairs of
successive NN intervals that differ by more than 50 ms
(NN50), and the proportion of NN50 divided by the total
number of NN intervals (pNN50). The calculation formulas
for HR, SDNN, and RMSSD are as follows [37]:

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
5

 (7)
  
󰇛





󰇜

 (8)
  
󰇛 󰇜

 (9)
The HRV frequency domain parameters include very low
frequency power (VLF), low frequency power (LF), high
frequency power (HF), total power (TP), normalized units of
LF (LFnu), normalized units of HF (HFnu), and the ratio of
LF to HF (LF/HF), where VLF is defined as 0 - 0.04 Hz, LF
is defined as 0.04 - 0.15 Hz, HF is defined as 0.15 - 0.5 Hz,
and TP is defined as 0 - 0.5 Hz. All calculations for the power
spectrum density are based on the fast Fourier transform
(FFT).
Most studies use ellipse-fitting methods to analyze
Poincaré plots, and short-axis SD1 and long-axis SD2 are
indicators of quantitative analyses, where SD1 reflects the
difference between adjacent RRIs, representing the
instantaneous HR change and SD2 reflects the overall degree
of variation in the HR. The calculation formulas for SD1 and
SD2 are as follows [38]:
  
󰇛󰇜


 (10)
  
󰇛





󰇜


 (11)
In equations (8) – (11), N is the total number of all normal
sinus RRIs, RRIi and RRIi+1 are the ith and the i+1th RRIs,
respectively, and 





is the average of all N RRIs.
E. STATISTICAL ANALYSIS
Histograms of the HRV frequency domain parameters
showed an obvious right skew; they were expressed as
natural logarithms prior to further analyses. To assess the
consistency between the two measurement methods,
matched-pair tests, Pearson correlation analyses, Bland-
Altman analyses, and effect sizes (ES) were used to process
the data between the groups. In the Pearson correlation
analyses, the correlation coefficient r was used to
characterize the degree of correlation between the two sets
of data; a larger absolute value of r indicated a stronger
correlation. In the Bland-Altman analysis, the 95% LoA for
the two sets of data was calculated as the mean of the
difference between the two ± 1.96 standard deviation. The
magnitude of the difference between HRV parameters was
assessed using the ES with smaller values, which indicated
lower differences [39]-[41]. The ES value is expressed as the
difference between the means of the two samples divided by
the pooled variance. The criteria for defining the magnitude
of ES values included the following: the difference was small
when ES ≤ 0.2; the difference was moderate when ES ≤ 0.5;
and the difference was large when ES ≥ 0.8 [42]. p < 0.05
was considered significant for all analyses. These analyses
were performed using SPSS (ver. 23.0, SPSS, Inc., Chicago,
IL., USA) statistical software and MATLAB (MathWorks,
Natick, MA, USA) self-programming.
IV. RESULTS
The fingertip video and ECG signals of the 24 subjects were
analyzed to obtain 2114 PPIs and 2108 RRIs, respectively.
Of the 2114 PPIs, there were 6 (< 0.3%) misdetections.
Figure 6 shows a typical example of a PPI misdetection.
FIGURE 6. A representative illustration of PPI error detection.
The primary cause for the six false checks was
interference. We addressed these false checks by adding two
PPI values. In practical applications, automatic corrections
can be adopted. Here, we used manual corrections. After
manually correcting, a total of 2108 pairs of PPI and RRI
values were obtained. There was a strong correlation

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
6
between the two sets of data (r = 0.994, p = 0.000) (Figure
7). The Bland-Altman analyses showed that the mean of the
difference between the two datasets was -0.2 ms, the 95%
LoA range was (-19.8, 20.2) ms (Figure 8), and ES = 0.001,
indicating a small difference.
The time domain parameters, frequency domain
parameters, and Poincaré plot parameters of the HRV were
analyzed for their PPI and RRI values, respectively, and the
results are shown in Table 1. There was an extremely strong
correlation between the two methods for all the HRV indices
(r > 0.95, p = 0.000). Except for the RMSSD, HF, and SD1,
the ES was small (< 0.2) for all evaluated indices between
the two methods.
References [6] and [25] are more consistent with our
study. In reference [6], the 14 HRV parameters from vPPG
were highly correlated (r > 0.7, p < 0.001) with those from
ECG, and only 7 of them were in the acceptable range.
However, our study still has a little difference in accuracy
compared with reference [25].
FIGURE 7. The correlation between the PPI and RRI (r = 0.994, p = 0.000).
FIGURE 8. Bland-Altman agreement analysis between the PPI and RRI. The mean of the difference between the two datasets was -0.2 ms, and the 95%
LoA range was (-19.8, 20.2) ms.
V. DISCUSSION
The existing studies based on vPPG technology have
primarily focused on applying this technology within the
field of HR monitoring [19]-[23]. For HR monitoring, the
high PPI measurement accuracy is not required but should be
dynamic and in real time. Therefore, a noncontact method
has generally been adopted as the measurement mode. In
video signal processing, techniques such as BSS, ICA, and
FFT analyses must be used to remove noise. The HRV

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
7
TABLE 1. Comparison of HRV parameters between the video and ECG
Video
ECG
Correlation
(r, P)
Bias (LoA)
Effect size
(interpretation)
Mean RR (ms)
716.3±85.7
716.5±85.6
(1.000, 0.000)
-0.2(-1.2 to 0.8)
0.002 (small)
Mean HR (BPM)
85.0±10.1
84.9±10.1
(1.000, 0.000)
0.1(-0.5 to 0.6)
0.008 (small)
Min HR (BPM)
79.0±8.8
79.0±8.8
(0.998, 0.000)
-0.1(-1.1 to 0.9)
0.009 (small)
Max HR (BPM)
92.0±11.6
91.9±11.5
(0.999, 0.000)
0.2(-0.8 to 1.1)
0.014 (small)
SDNN (ms)
38.9±13.0
36.7±12.5
(0.996, 0.000)
2.2(-0.2 to 4.6)
0.171 (small)
RMSSD (ms)
34.4±15.5
29.6±15.7
(0.983, 0.000)
4.9(0.8 to 10.6)
0.311 (moderate)
NN50
12±12
10±11
(0.962, 0.000)
2.2(-4.1 to 8.5)
0.190 (small)
pNN50 (%)
14.6±16.2
12.0±16.0
(0.970, 0.000)
2.6(-5.1 to 10.3)
0.162 (small)
LnVLF
5.046±1.149
5.048±1.121
(0.999, 0.000)
-0.001(-0.121 to 0.112)
0.001 (small)
LnLF
5.396±0.813
5.343±0.784
(0.998, 0.000)
0.053(-0.069 to 0.174)
0.066 (small)
LnHF
5.975±1.133
5.734±1.173
(0.975, 0.000)
0.241(-0.271 to 0.753)
0.209 (moderate)
LnTP
6.866±0.848
6.723±0.847
(0.989, 0.000)
0.143(-0.107 to 0.392)
0.168 (small)
LFnu (n.u.)
37.8±22.0
41.8±21.6
(0.954, 0.000)
-4.0(-17.0 to 9.1)
0.182 (small)
HFnu (n.u.)
62.1±22.0
58.1±21.5
(0.954, 0.000)
4.0(-9.0 to 17.0)
0.184 (small)
LF/HF
1.027±1.359
1.155±1.391
(0.954, 0.000)
-0.129(-0.948 to 0.691)
0.094 (small)
SD1 (ms)
24.5±11.1
21.1±11.2
(0.983, 0.000)
3.5(-0.6 to 7.5)
0.311 (moderate)
SD2 (ms)
48.6±16.2
46.7±15.5
(0.998, 0.000)
1.9(-0.5 to 4.4)
0.121 (small)
Values are expressed as the mean (standard deviation).
measurement typically does not depend on long-term
continuous monitoring. In practice, it is usually performed
with ECG data of <=5 minutes [7]. However, HRV
measurement does require a high PPI accuracy. It is often
difficult to meet the requirements using a noncontact method;
therefore, a contact method has generally been adopted.
Unlike the traditional noncontact video acquisition
technology, BSS, ICA, and other denoising technologies are
not required for collecting fingertip video signals by contact
methods when the subject is in a quiet state. Instead, the
spatial pixel attribute values of each frame are superimposed
and averaged; then, the pulse wave signals are synthesized
from a time perspective. Although there is some noise in the
pulse wave signal, it can be processed by simple digital
filtering. The HRV analysis of these pulse waves showed that
all indices, except RMSSD, HF, and SD1, were strongly
correlated with the results of the ECG measurement with
very small differences and within acceptable ranges of error.
The results presented in this study confirmed the feasibility
of the proposed method for HRV analysis.
The periodic systolic and diastolic heart activity causes
blood to enter arteries and return from veins, forming a blood
circulation system. The pulse wave is formed by the
spreading of heart pulsation along arterial blood vessels and
by blood flow to peripheral arteries. From a temporal
perspective, the PPI lags relative to the RRI; as such, using
the PPI for HRV analysis may produce errors. However, the
existing research has shown that in a resting state, all of the
time- and frequency-domain HRV indices analyzed with the
PPI were negligibly different from those analyzed with the
RRI [16]-[18]. Therefore, there are inherent errors in using
PPI instead of RRI to analyze HRV. But such errors are
negligible.
Another error factor is the PPI accuracy. Generally, the
frame rate of a smartphone camera is approximately 25 f/s,
and the sampling rate of the corresponding pulse wave after
processing is 25 Hz. In this study, a 4x slow-motion mode
was adopted to improve the sampling rate to 100 Hz. The
sampling rate of ECG signals for HRV analysis is
recommended to be no less than 250 Hz [7]. Compared with
that of the RRI, the average difference for the PPI, as
calculated by the method proposed in this study, was 0.2 ms,
but the range for the 95% limit of agreement reached from -
19.8 to 20.2 ms. Therefore, the low sampling rate may be the
fundamental reason for the large differences in high-
frequency indicators. For undersampled signals,
interpolation is an effective method for improving the
sampling rate. We performed parabolic interpolation and
Fourier interpolation but had little success. The interpolation
method requires further study.
Among the HRV indices, SDNN and LF represent the
coregulation of sympathetic and parasympathetic nerves, and
the LF/HF ratio represents the balance between the
sympathetic and parasympathetic nerves [7]. The results of
this study show that the SDNN and LF/HF calculated by the
video technique proposed in this paper have high
correlations with the ECG method to the extent that there is
no significant difference between the two methods. This
finding demonstrates the great potential of applying video
technology in HRV analysis. To date, HRV measurement
has been used to quantitatively evaluate the overall
autonomic nerve activity and to track the balance of the
sympathetic and parasympathetic nerves. For example, in
patients with heart failure [43] and terminal illnesses [44],
many variables are involved in the HRV decrease, and thus,
HRV has become an independent prognostic indicator for
these diseases. Unlike the traditional HRV measurement
method, the method proposed in this study enables patients
to use smartphones to perform remote monitoring and
evaluation at home. Another example is that HRV is age
related and can be used as a noninvasive biological marker
for assessing aging [9]. If a big data model of HRV and age
is established with this method, the quantitative assessment
of daily aging can be performed and used for health guidance.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
8
vPPG is more sensitive to interference and noise than ECG
acquisition. Therefore, there are several obvious limitations
to this study. First, differences in skin tones, skin roughness,
and even fat thickness affected the absorption and reflection
of light to some extent, thus affecting the quality of the PPG
signal. Therefore, the influence of these factors on the
accuracy of the HRV measurement should be further
explored. Second, due to the influence of subjective and
objective factors, such as the size of the phone, the size of
the individual’s palm, and the degree of personal tension, the
pressure on the fingers of each person varied. Pressure
intensity and stability greatly influence the PPG signal
quality. In extreme cases, the pulse wave shape may not even
be obtained and thus may need to be measured again.
Therefore, further research is needed to address this issue.
Third, although we preliminarily verified that this method
can be used for quantitative evaluations of personal daily
ANS values, it is necessary to further study the ANS level of
subjects under different stress states and evaluate whether
the device can capture these changes.
VI. CONCLUSION
We proposed a new method for HRV analysis based on the
rear-facing camera of a smartphone. This method abandons
traditional signal separation and time-frequency
transformation methods; instead, this method uses the time-
space-time transformation of the video signal to obtain the
PPG and then applies the Hilbert transform to calculate the
PPI for HRV analysis. There was a very strong correlation
with the RRI and HRV parameters obtained using the ECG
method. Except for RMSSD, HF, and SD1, the differences
for all indices were very small. These findings suggest that
this method can be used to quantitatively assess the overall
activity and balance of autonomic nerves and for the
quantitative management of an individual's day-to-day
health. Future directions will include the development of
applications based on a smartphone platform.
REFERENCES
[1] M. Elsherif et al., “Wearable contact lens biosensors for continuous
glucose monitoring using smartphones,” ACS Nano, vol. 12, no. 6, pp.
5452-5462, May. 2018.
[2] Q. Liu et al., “Real-time biodetection using a smartphone-based dual-
color surface plasmon resonance sensor,” J. Biomed. Opt., vol. 23, no.
4, pp. 1-6, Apr. 2018.
[3] J.A. Taylor et al., “Use of a smartphone app to assess neonatal
jaundice,” Pediatrics., vol. 140, no. 3, Sep. 2017, Art. no. e20170312.
[4] A. Chandrasekhar et al., “Smartphone-based blood pressure
monitoring via the oscillometric finger-pressing method,” Sci. Transl.
Med., vol. 10, no. 431, Mar. 2018, Art. no. eaap8674.
[5] F. Miao et al., “A wearable context-aware ECG monitoring system
integrated with built-in kinematic sensors of the
smartphone,” Sensors., vol.15, no. 5, pp. 11465-11484, May. 2015.
[6] R. Peng et al., “Extraction of heart rate variability from smartphone
photoplethysmograms.” Comput. Math. Methods Med., vol. 2015,
Jan. 2015, Art. no. 516826.
[7] Task Force of the European Society of Cardiology and the North
American Society of Pacing and Electrophysiology. “Heart rate
variability: standards of measurement, physiological interpretation,
and clinical use,” Circulation., vol. 93, no. 5, pp. 1043-1065, Mar.
1996.
[8] A. Voss et al., “Short-term heart rate variability - age dependence in
healthy subjects,” Physiol. Meas., vol. 33, no. 8, pp. 1289-1311, Aug.
2012.
[9] A. Voss et al., “Short-term heart rate variability - influence of gender
and age in healthy subjects,” PLoS One, vol. 10, no. 3, Mar. 2015,
Art. no. e0118308.
[10] K. Shin et al., “The power spectral analysis of heart rate variability in
athletes during dynamic exercise -- Part I,” Clin. Cardiol., vol. 18, no.
10, pp. 583-586, Oct. 1995.
[11] X. Wang et al., “Does the temporal asymmetry of short-term heart
rate variability change during regular walking? A pilot study of
healthy young subjects,” Comput. Math. Methods Med., vol. 2018,
Apr. 2018, Art. no. 3543048.
[12] C.R. Cardoso et al., “Relationships between reduced heart rate
variability and pre-clinical cardiovascular disease in patients with
type 2 diabetes,” Diabetes Res. Clin. Pract., vol. 106, no. 1, pp. 110-
117, Oct. 2014.
[13] H. Mussalo et al., “Heart rate variability and its determinants in
patients with severe or mild essential hypertension,” Clin. Physiol.,
vol. 21, no. 5, pp. 594-604, Sep. 2001.
[14] S. Hu et al., “Low heart rate variability relates to the progression of
gastric cancer,” World J. Surg. Oncol., vol. 16, no. 1, pp. 49, Mar.
2018.
[15] J.L. Moraes et al., “Advances in photopletysmography signal analysis
for biomedical applications,” Sensors., vol.18, no. 6, Jun. 2018, Art.
no. 1894.
[16] N. Selvaraj et al., “Assessment of heart rate variability derived from
finger-tip photoplethysmography as compared to
electrocardiography,” J. Med. Eng. Technol., vol. 32, no. 6, pp. 479-
484, Jul. 2008.
[17] G. Lu et al., “A comparison of photoplethysmography and ECG
recording to analyse heart rate variability in healthy subjects,” J. Med.
Eng. Technol., vol. 33, no. 8, pp. 634-641, Dec. 2009.
[18] E. Gil et al., “Photoplethysmography pulse rate variability as a
surrogate measurement of heart rate variability during non-stationary
conditions,” Physiol. Meas., vol. 31, no. 9, pp. 1271-1290, Aug. 2010.
[19] M.Z. Poh, D.J. McDuff, and R.W. Picard, “Non‑contact, automated
cardiac pulse measurements using video imaging and blind source
separation,” Opt. Express, vol. 18, no. 10, pp. 10762-10774, May.
2010.
[20] K. Matsumura and T. Yamakoshi, “iPhysioMeter: a new approach for
measuring heart rate and normalized pulse volume using only a
smartphone,” Behav. Res. Methods, vol. 45, no. 4, pp. 1272-1278,
Dec. 2013.
[21] S.A. Siddiqui et al., “A pulse rate estimation algorithm using PPG
and smartphone camera,” J. Med. Syst., vol. 40, no. 5, pp.126, May.
2016
[22] J. Lomaliza and H. Park, “A highly efficient and reliable heart rate
monitoring system using smartphone cameras,” Multimed. Tools
Appl., vol. 76, no.20, pp. 21051-21071, Oct. 2016.
[23] R. Zaman et al., “Novel fingertip image-based heart rate detection
methods for a smartphone,” Sensors., vol. 17, no. 2, pp. 358-370, Feb.
2017.
[24] R.Y. Huang and L.R. Dung, “Measurement of heart rate variability
using off-the-shelf smart phones,” BioMed. Eng. Online., vol. 15, Jan.
2016, Art. no. 11.
[25] A. Bánhalmi et al., “Analysis of a pulse rate variability measurement
using a smartphone camera,” J. Healthc. Eng., vol. 2018, Feb. 2018,
Art. no. 4038034.
[26] T. Thap et al., “Real-time heart activity monitoring with optical
illusion using a smartphone,” Multimed. Tool Appl., vol. 77, no. 5, pp.
6209-6224, Mar. 2018.
[27] T.H. Fu, S.H. Liu, and K.T. Tang, “Heart rate extraction from
photoplethysmogram waveform using wavelet multi-resolution
analysis,” J. Med. Biol. Eng., vol. 28, no. 4, pp. 229-232, Dec. 2008.
[28] M. Granegger et al., “Use of independent component analysis for
reducing CPR artefacts in human emergency ECGs,” Resuscitation,
vol. 82, no. 1, pp. 79-84, Jan. 2011.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2021.3054065, IEEE Access
9
[29] M.H. Moradi et al., “ECG signal enhancement using adaptive
Kalman filter and signal averaging,” Int. J. Cardiol., vol. 173, no. 3,
pp. 553-555, May. 2014.
[30] C.H. Chu, and E.J. Delp, “Impulsive noise suppression and
background normalization of electrocardiogram signals using
morphological operators,” IEEE Trans. Biomed. Eng., vol. 36, no. 2,
pp. 262-273, Feb. 1989.
[31] L.J. Karam and J. H. McClellan. “Chebyshev digital FIR filter design,”
Signal Proces., vol. 76, no. 1, pp. 17-36, Jul. 1999.
[32] B.N. Li, M.C. Dong, and M.I. Vai, “On an automatic delineator for
arterial blood pressure waveforms,” Biomed. Signal Proces. Control,
vol. 5, no. 1, pp. 76-81, Jan. 2010.
[33] S.G. Mallat and W.L. Hwang, “Singularity detection and processing
with wavelets,” IEEE Trans. Inform. Theory., vol. 38, no. 2, pp. 617-
643, Mar. 1992.
[34] P.E. Trahanias, “An approach to QRS complexes detection using
mathematical morphology,” IEEE Trans. Biomed. Eng., vol. 40, no.
2, pp. 201-205, Feb. 1993.
[35] R.N. Bracewell, “The Fourier Transform and its Applications”,
McGraw-Hill, New York, 1978, pp. 267–274.
[36] D. Benitez et al., “The use of the Hilbert transform in ECG signal
analysis,” Comput. Biol. Med., vol. 31, no. 5, pp. 399-406, Sep. 2001.
[37] C.A.G. Martinez et al., “Time-domain analysis” in Heart rate
variability analysis with the R package RHRV. Switzerland: Springer,
2017, pp: 29-31.
[38] A.H. Khandoker et al., “Quantitative Poincaré Plot,” in Poincaré plot
methods for heart rate variability analysis. New York, NY, USA:
Springer, 2013, pp. 17-20.
[39] D.Giles, N.Draper, and W.Neil, “Validity of the Polar V800 heart rate
monitor to measure RR intervals at rest,” Eur J Appl Physiol., vol.
116, no. 3, pp. 563-571, Mar. 2016.
[40] D.A.Giles, and N.Draper, “Heart rate variability during exercise: a
comparison of artefact correction methods,” J Strength Cond Res.,
vol. 32, no. 3, pp. 726-735, Mar. 2018.
[41] P.Caminal et al., “Validity of the Polar V800 monitor for measuring
heart rate variability in mountain running route conditions,” Eur J
Appl Physiol., vol. 118, no. 3, pp. 669-677, Mar. 2018.
[42] J. Cohen, “The t test for means,” in Statistical power analysis for the
behavioral sciences, 2nd ed. Mahwah, NJ, USA: Lawrence Erlbaum
Associates, 1988, pp. 20-40.
[43] K. Imai et al., “Vagally mediated heart rate recovery after exercise is
accelerated in athletes but blunted in patients with chronic heart
failure,” J. Am. Coll. Cardiol., vol. 24, no. 6, pp. 1529-1535, Nov.
1994.
[44] C. Arab et al., “Heart rate variability measure in breast cancer patients
and survivors: A systematic review,” Psychoneuroendocrinology, vol.
68, pp. 57-68, Jun, 2016.
GENXUAN ZHANG received the M.S. degree in
circuits and systems from Anhui University, Hefei,
China in 2007.
He is currently an Associate Professor with the
School of Medical Imaging, Bengbu Medical
College, Bengbu, China. His research interests
include biomedical signal processing and analysis.
He has published over 10 research papers on these
topics.
SAI ZHANG received the B. S. degree in
electronic information science and technology
from Tianjin Chengjian University, Tianjin, China
in 2013, and the M.S. degree in circuits and
systems from Anhui University of Science and
Technology, Huainan, China in 2016.
She is currently a lecturer with the School of
Medical Imaging, Bengbu Medical College,
Bengbu, China. Her main research interests
include biomedical signal acquisition and analysis.
YIMING DAI received the B. S. degree in
electronic and information engineering from Hefei
Normal University, Hefei, China in 2012, and the
M.S. degree in electromagnetic and microwave
technology from Anhui University, Hefei, China in
2015.
He is currently a lecturer with the School of
Medical Imaging, Bengbu Medical College,
Bengbu, China. His research interests include
biomedical signal analysis.
BO SHI received the B.S. degree in electronic
science and technology from Xuzhou Normal
University, Xuzhou, China in 2004, and M.S.
degree in biomedical engineering from Nanjing
University of Science and Technology, Nanjing,
China, in 2006.
He is currently an Associate Professor with the
School of Medical Imaging, Bengbu Medical
College, Bengbu, China. His research focuses on
novel biomedical instrumentation, physiological
measurements, and biomedical signal processing.