ArticlePDF AvailableLiterature Review

Implementation of Thermal Camera for Non-Contact Physiological Measurement: A Systematic Review

Abstract and Figures

Non-contact physiological measurements based on image sensors have developed rapidlyin recent years. Among them, thermal cameras have the advantage of measuring temperature in theenvironment without light and have potential to develop physiological measurement applications.Various studies have used thermal camera to measure the physiological signals such as respiratoryrate, heart rate, and body temperature. In this paper, we provided a general overview of theexisting studies by examining the physiological signals of measurement, the used platforms, thethermal camera models and specifications, the use of camera fusion, the image and signal processingstep (including the algorithms and tools used), and the performance evaluation. The advantagesand challenges of thermal camera-based physiological measurement were also discussed. Severalsuggestions and prospects such as healthcare applications, machine learning, multi-parameter, andimage fusion, have been proposed to improve the physiological measurement of thermal camera inthe future.
Content may be subject to copyright.
sensors
Systematic Review
Implementation of Thermal Camera for Non-Contact
Physiological Measurement: A Systematic Review
Martin Clinton Tosima Manullang 1,2 , Yuan-Hsiang Lin 1,* , Sheng-Jie Lai 1and Nai-Kuan Chou 3, *


Citation: Manullang, M.C.T.; Lin,
Y.-H.; Lai, S.-J.; Chou, N.-K.
Implementation of Thermal Camera
for Non-Contact Physiological
Measurement: A Systematic Review.
Sensors 2021,21, 7777. https://
doi.org/10.3390/s21237777
Academic Editors: Lorenzo Scalise
and Maria Lepore
Received: 10 October 2021
Accepted: 19 November 2021
Published: 23 November 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1
Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology,
Taipei 10607, Taiwan; D10902809@mail.ntust.edu.tw (M.C.T.M.); m10902137@ntust.edu.tw (S.-J.L.)
2Department of Informatics, Institut Teknologi Sumatera, South Lampung Regency 35365, Indonesia
3Department of Cardiovascular Surgery, National Taiwan University Hospital, Taipei 10002, Taiwan
*Correspondence: linyh@mail.ntust.edu.tw (Y.-H.L.); nickchou@ntu.edu.tw (N.-K.C.)
Abstract:
Non-contact physiological measurements based on image sensors have developed rapidly
in recent years. Among them, thermal cameras have the advantage of measuring temperature in the
environment without light and have potential to develop physiological measurement applications.
Various studies have used thermal camera to measure the physiological signals such as respiratory
rate, heart rate, and body temperature. In this paper, we provided a general overview of the
existing studies by examining the physiological signals of measurement, the used platforms, the
thermal camera models and specifications, the use of camera fusion, the image and signal processing
step (including the algorithms and tools used), and the performance evaluation. The advantages
and challenges of thermal camera-based physiological measurement were also discussed. Several
suggestions and prospects such as healthcare applications, machine learning, multi-parameter, and
image fusion, have been proposed to improve the physiological measurement of thermal camera in
the future.
Keywords: thermal camera; contactless sensors; non-contact; physiological measurement
1. Introduction
1.1. Research Motivation
The use of thermal cameras has become very widespread in recent years as it can
be applied in various fields. Thermal cameras have the advantages of operating in an
environment without light and not being affected by changes in light. There are existing
studies illustrating that thermal camera can be used to monitor respiratory rate (RR),
heart rate (HR), and body temperature, while other studies found its use in breast cancer
diagnosis [
1
], evaluating physical condition [
2
], stress level [
3
], as well as the neonates’
health condition [4], sleep posture [5], and many more not mentioned here.
Meanwhile, vital signs data, such as blood pressure, temperature, respiration rate,
and heart rate, are critical for patient care and diagnosis. They enable physicians and other
healthcare workers to make informed decisions about a patient’s treatment options and
overall well-being. However, existing medical instruments still rely on physical touch
from existing tools to gather data about patients’ health. Most techniques for determining
respiration and heart rate include physical contact with the patients such as pulse oximeters,
ECG (electrocardiogram), monitoring systems using electrodes, or piezoelectric sensors.
During the SARS-CoV-2 19 pandemic, the whole world reduced the amount of direct
contact drastically. People are reluctant to visit health and medical institutions due to fear
of infection. Based on existing studies [
6
], there has been a change in medical services
since the outbreak of SARS-CoV-2 19. Contactless services have been implemented during
SARS-CoV-2 19, and will become commonplace [
7
] even after the pandemic. Several
developments such as measuring RR and HR using the non-contact method with radar
Sensors 2021,21, 7777. https://doi.org/10.3390/s21237777 https://www.mdpi.com/journal/sensors
Sensors 2021,21, 7777 2 of 21
sensors [
8
], blood volume pulse and vasomotion measurements [
9
], using radio frequency,
and the Doppler effect to monitor vital body objects [
10
] have been tested and researched.
Body temperature is an excellent indication of a patient’s health [
11
]. Human body
temperature can be categorized into two types: skin temperature and core body tempera-
ture. Skin temperature is the temperature of the outermost surface of the body. Average
human skin temperature varies between 33.5 and 36.9 C (92.3 and 98.4 F) while healthy
core body temperature falls within 37
C (98
F) and 37.8
C (100
F) [
12
]. According to a
study [
13
], extreme body temperature can negatively affect how a human’s body and vital
organs work. To compensate for this, the body has thermoregulation processes that enable
it to maintain a standard core internal temperature.
Non-contact systems can detect temperature through infrared thermography because
it can detect electromagnetic waves produced by anything with a temperature greater than
absolute zero Kelvin. This phenomenon is used for the development of thermal cameras.
Thermal cameras measure temperature using infrared radiation from 1 to 14
µ
m spectral
range [
14
,
15
]. This measurement procedure is known as infrared thermography (IRT).
IRT is a non-invasive technique that remotely measures the energy emitted by an entity
(i.e., human body, industrial machine, engine, and many more objects). IRT is applied as
an indirect technique for capturing the changes in surface body temperature and can be
used to measure other physiological signals [
16
]. However, the commonly used thermal
camera for medical purpose is a long-wavelength infrared (LWIR) type with a 7–14
µ
m
spectral range [17].
According to research in United States by McKinsey between March and April 2020, a
large migration to telemedicine occurred, coinciding with an over 80% drop in in-person
visits. The use of telemedicine by physicians and healthcare organizations has also ex-
panded by 50–175 times since the COVID-19 outbreak [
18
]. The increase in public demand
for indirect healthcare during the pandemic led to the rapid development of non-contact
healthcare practice and emphasized its importance. There is also a trend for non-contact
measurement technology to replace the current conventional methods without any com-
promise on performance and accuracy. However, the use of thermal cameras to capture
vital signs has its challenges in terms of image and signal processing.
1.2. Research Objective
This paper aims to systematically evaluate the use and development of thermal
cameras in its application for measuring vital signs using preferred reporting items for
systematic reviews and meta-analyses (PRISMA) to produce relevant papers from 2012 to
2021. More specifically, this paper will evaluate system capabilities, thermal camera types,
signal processing steps, system platform, and highlight system performance along with
the validation method. This systematic review contributes an evaluation of recent progress
in non-contact physiological measurement with LWIR thermal camera that can be used as
a basis for reference for other related research.
1.3. Comparison with Existing Reviews
Other previous systematic review papers that also discussed the applications of
thermal cameras are listed in Table 1.
Sensors 2021,21, 7777 3 of 21
Table 1. Existing Systematic review.
Author Published Year Difference
Mikulska D. [19] 2006 Covered studies before 2006
Lahiri et al. [20] 2012
Published in 2012 and covered studies before 2012
El et al. [21] 2015 Only covered applications related to sports
Znamenskaya et al. [
22
]
2016
Limited to human psychophysiological conditions
that are based on thermographic video
Zadeh et al. [1] 2016 Only covered breast cancer diagnostics by using
thermal imaging
Moreira et al. [23] 2017 Developed checklist guidelines to assess skin
temperature for sports and exercise medicine
Topalidou et al. [4] 2019
Database limited to EMBASE, MEDLINE, and
MIDIRS and only covered thermal camera usage
in neonatal care
Pan et al. [24] 2019 Focused on vein finder by using near
infrared (NIR)
Aggarwal et al. [25] 2020 Focused on reviewing the accuracy of handheld
thermal cameras
Foster et al. [26] 2021 Focused on assessing human core temperature
using infrared thermometry
He et al. [27] 2021 Not focused on human vital signs
There are some significant differences between this systematic review and the existing
ones. These differences mean that this study has a novelty in review. The following
studies [
19
,
20
] have similarities with this study regarding research objectives, but in these
the coverage years are different. Some studies also have a scope that focuses on specific
aspects such as sports [
21
], psychophysiological [
22
], breast cancer [
1
], neonatal [
4
], vein
finder [
24
], and human core temperature [
26
]. Some studies have a broader scope; for
example, the study conducted by He et al. [
27
] covered all aspects outside the health field.
2. System Architecture in General
The image processing carried out by each study requires a platform on which the
processing takes place. Most of the studies carried out signal processing on software such
as python for a programming language and OpenCV as a library framework. Python is
a multi-hardware programming language that runs on various hardware ranging from
personal computers to mini-PC boards such as Jetson [
28
] and Raspberry Pi. OpenCV
stands for Open-Source Computer Vision, a library that provides various kinds of image
processing functions which can be used for real-time processing directly from the camera
or by using pre-recorded image data.
Although each study that uses a thermal camera for physiological measurement
included in this systematic review is very diverse in various vital signs, the processing
stages of each study can generally be summed up in one process flow that can be seen in
Figure 1.
2.1. Thermal Camera Model and Specification
The image process stage begins with the acquisition of a thermal image from a thermal
camera. The resolution and the number of images that run in one second or what is known
as frames per second (FPS) are essential in signal processing, primarily related to images.
Several cameras are used more than once by the studies included in this systematic review,
i.e., the A315 and A325 from Teledyne FLIR LLC and the MAG62 from Magnity Electronics
Co., Ltd., Shanghai, China.
Sensors 2021,21, 7777 4 of 21
Sensors 2021, 21, x FOR PEER REVIEW 4 of 21
Figure 1. The general process stages of studies using a thermal camera that performs physiological
measurements. Several stages are depicted by dotted line boxes explaining that these stages only
apply in certain studies in general.
2.1. Thermal Camera Model and Specification
The image process stage begins with the acquisition of a thermal image from a ther-
mal camera. The resolution and the number of images that run in one second or what is
known as frames per second (FPS) are essential in signal processing, primarily related to
images. Several cameras are used more than once by the studies included in this system-
atic review, i.e., the A315 and A325 from Teledyne FLIR LLC and the MAG62 from Mag-
nity Electronics Co. Ltd., Shanghai, China.
There are several necessary specifications related to thermal cameras. In general, the
cameras used in the included studies can record video from the lowest FPS of 8.7 FPS to
60 FPS in resolution between 160 × 120 pixels to 1024 × 768 pixels. FPS and resolution are
closely related to the quality of the resulting signal [29,30]. FPS refers to the number of
thermal image frames captured in one second. The more image frames obtained, the
greater the variation of thermal information and its variability. Therefore, in this case, FPS
can be interpreted as the sampling rate of a system. Likewise, the image’s dimensions also
show the number of measurement points made by the thermal camera. The larger the
image dimensions, the easier it will be for the system to detect region of interest (ROI).
Apart from thermal image specifications, two variables are often shown regarding
the performance of thermal cameras, i.e., temperature accuracy and thermal sensitivity.
Temperature accuracy indicates how close a measurement from a thermal imager is to the
actual absolute value. Meanwhile, thermal sensitivity refers to the noise equivalent tem-
perature difference (NETD). This value specifies the most negligible temperature differ-
ence that the camera can detect.
Some research shows that NETD is a critical aspect to show the performance of a
thermal camera [31–34]. The value of NETD is also an essential variable in using low-cost
thermal cameras for applications in the medical field. For example, a thermal camera with
a NETD value of less than 50 mK is ideal for medical applications [35]. In studies related
to the measurement of respiratory rate and heart rate, the value of the NETD becomes an
essential aspect because the measurement of respiratory rate considers changes in tem-
perature rather than the value of the temperature itself. Meanwhile, the study conducted
Figure 1.
The general process stages of studies using a thermal camera that performs physiological
measurements. Several stages are depicted by dotted line boxes explaining that these stages only
apply in certain studies in general.
There are several necessary specifications related to thermal cameras. In general, the
cameras used in the included studies can record video from the lowest FPS of 8.7 FPS to
60 FPS in resolution between 160
×
120 pixels to 1024
×
768 pixels. FPS and resolution
are closely related to the quality of the resulting signal [
29
,
30
]. FPS refers to the number
of thermal image frames captured in one second. The more image frames obtained, the
greater the variation of thermal information and its variability. Therefore, in this case, FPS
can be interpreted as the sampling rate of a system. Likewise, the image’s dimensions
also show the number of measurement points made by the thermal camera. The larger the
image dimensions, the easier it will be for the system to detect region of interest (ROI).
Apart from thermal image specifications, two variables are often shown regarding
the performance of thermal cameras, i.e., temperature accuracy and thermal sensitivity.
Temperature accuracy indicates how close a measurement from a thermal imager is to
the actual absolute value. Meanwhile, thermal sensitivity refers to the noise equivalent
temperature difference (NETD). This value specifies the most negligible temperature
difference that the camera can detect.
Some research shows that NETD is a critical aspect to show the performance of a
thermal camera [
31
34
]. The value of NETD is also an essential variable in using low-cost
thermal cameras for applications in the medical field. For example, a thermal camera
with a NETD value of less than 50 mK is ideal for medical applications [
35
]. In studies
related to the measurement of respiratory rate and heart rate, the value of the NETD
becomes an essential aspect because the measurement of respiratory rate considers changes
in temperature rather than the value of the temperature itself. Meanwhile, the study
conducted by Pan et al. [
36
] made corrections to the value of body temperature readings
with correction variables in measuring body temperature, while a study conducted by
Rao et al. [
37
] developed an automatic temperature correction algorithm to calibrate the
camera according to black body reference.
Sensors 2021,21, 7777 5 of 21
The correction value from the thermal camera is obtained by calibrating it. This
calibration process is performed using a radiometric calibration method, which establishes
the relationship between the pixel signal and the temperature of the target object [
38
,
39
].
Calibrated thermal cameras minimize temperature readings that differ significantly from
reference devices and become more reliable for medical applications.
More complete details regarding the use of thermal cameras in each study are summa-
rized in Table 2.
Table 2. List of Thermal Cameras Used in Some Studies in this Systematic Review Along with The Specifications Used.
Manufacturer Model Spectral
Range
Temperature
Accuracy
Thermal
Sensitivity
(NETD)
Maximum FPS and
Resolutions Used by
Flir Systems Inc.,
Wilsonville, OR,
USA
Lepton 3.5 8 to 14 µm±5C <50 mK 8.7 FPS, 160 ×120 pixels [40,41]
A325 7 to 13.5 µm±5C <50 mK 60 FPS, 320 ×240 pixels [4245]
Thermovision
A40M 7 to 13.5 µm±2C <50 mK 60 FPS, 320 ×240 pixels [46]
A315 7.5 to 13 µm±2C <50 mK 60 FPS, 320 ×240 pixels [47,48]
P384-20 8 to 14 µm±2C <50 mK 50 FPS, 384 ×288 pixels [36]
T430sc 7.5 to 13 µm±2C <30 mK 12 FPS, 320 ×240 pixels [49]
InfraTec GmbH,
Dresden, Germany
VarioCAMR
HD 820S 7.5 to 14 µm±1C <55 mK 30 FPS, 1024 ×768 pixels [50]
Magnity
Electronics Co.,
Ltd., Shanghai,
China
MAG 62 7.5 to 14 µm±2C <60 mK 50 FPS, 640 ×480 pixels [5153]
Optris Gmbh,
Berlin, Germany Optris PI 450i 8 to 14 µm±2C <75 mK 80 FPS, 382 ×288 pixels [28]
Seek Thermal Inc.,
Santa Barbara,
CA, USA
Compact PRO
7.5 to 14 µm - <70 mK >15 FPS, 320 ×240 pixels [54]
Mobotix AG,
Winnweiler,
Germany
M16 TR 7.5 to 13 µm±10 C <50 mK 9 FPS, 336 ×252 pixels [37]
2.2. Image Pre-Processing and Feature Matching
The second stage is the pre-processing of the thermal image. Pre-processing is carried
out on the entire frame of the image. During pre-processing, gaussian filter, changes in
image dimensions or size, conversion of the number of FPS, altering color channels to
grayscale, bitmap, or using pseudocolor can be applied.
A feature matching stage is required for studies that use more than one camera or
what is known as image fusion. The fusion cameras used are also varied. Some use a
combination of near-infrared (NIR) and LWIR cameras [
40
,
53
], while the others use a com-
bination of LWIR and RGB cameras [
42
,
47
,
48
,
52
]. Most studies combined the two images,
generally thermal images and RGB images. The RGB image is used for ROI detection. The
ROI coordinates in the RGB image are matched with the coordinates on the thermal camera.
Determination of these coordinates has a diverse method, ranging from multispectral local-
ization using the dlib algorithm [
55
,
56
], and pre-trained machine learning models. These
points are then correlated with the thermal image. Some pre-processing may be required,
such as frame per second synchronization and adjustment of the image’s dimensions
(in general, the dimensions of thermal images are often smaller than RGB images). This
cross-correlation process also has several algorithms, including affine transformation [
57
],
the Oriented Fast and Rotated Brief feature [
58
,
59
], and others. This cross-correlation
Sensors 2021,21, 7777 6 of 21
process will produce an equation matrix called a homography matrix (some studies call it
a transformation matrix or correlation matrix). An illustrative simplification of this process
can be seen in Figure 2.
Sensors 2021, 21, x FOR PEER REVIEW 6 of 21
localization using the dlib algorithm [55,56], and pre-trained machine learning models.
These points are then correlated with the thermal image. Some pre-processing may be
required, such as frame per second synchronization and adjustment of the image’s dimen-
sions (in general, the dimensions of thermal images are often smaller than RGB images).
This cross-correlation process also has several algorithms, including affine transformation
[57], the Oriented Fast and Rotated Brief feature [58,59], and others. This cross-correlation
process will produce an equation matrix called a homography matrix (some studies call it
a transformation matrix or correlation matrix). An illustrative simplification of this pro-
cess can be seen in Figure 2.
Figure 2. An overview of how RGB cameras are used to assist thermal cameras in determining ROI
and the transformation process.
2.3. Determining and Tracking of ROI
Determining the ROI in a dimensional image is an important stage in thermal camera
image processing. There are several methods used by the studies selected in this system-
atic review. The first one that was used is called Viola-Jones framework [60]. Paul Viola
and Michael Jones developed this framework using the haar feature, commonly used to
detect facial parts. All studies by Negishi [42,47,48] included in this systematic review
used Viola-Jones to determine ROI. However, Hu et al. [51] claimed that their ROI detec-
tion algorithm has better performance when compared to Viola-Jones, with 98.46% accu-
racy versus 87.69%. Chen et al. [52] also argue that Viola-Jones in OpenCV is always used
to determine coarse faces’ locations but is not precise in respiratory rate measurement.
Therefore, deep learning is used as a method to determine ROI.
Movement between frames, especially when the camera is set at a low FPS, is often a
problem in signal acquisition from the thermal camera. For this reason, optical flow is
used to solve this problem. The study conducted by Lyra et al. [28] used optical flow to
quantify the thermal image so that the subtle motion in the chest area is reduced for later
extraction of the respiratory signal. Furthermore, the study conducted by Scebba et al. [40]
used a dense optical flow algorithm developed by Farneback to reduce the periodic mo-
tion of the torso.
Extracting signals from moving objects is a challenge, and therefore tracking methods
are needed to overcome them, one of which is by using The Kanade-Lucas-Tomasi (KLT)
algorithm, which is used by several studies [40,51,52]. This algorithm uses a linear coor-
dinate mapping which determine the corresponding region in the thermal video. This
tracker extracts feature points from ROI using the minimum eigenvalue algorithm and
follows those points with a single point tracker.
Figure 2. An overview of how RGB cameras are used to assist thermal cameras in determining ROI
and the transformation process.
2.3. Determining and Tracking of ROI
Determining the ROI in a dimensional image is an important stage in thermal camera
image processing. There are several methods used by the studies selected in this systematic
review. The first one that was used is called Viola-Jones framework [
60
]. Paul Viola and
Michael Jones developed this framework using the haar feature, commonly used to detect
facial parts. All studies by Negishi [
42
,
47
,
48
] included in this systematic review used
Viola-Jones to determine ROI. However, Hu et al. [
51
] claimed that their ROI detection
algorithm has better performance when compared to Viola-Jones, with 98.46% accuracy
versus 87.69%. Chen et al. [
52
] also argue that Viola-Jones in OpenCV is always used
to determine coarse faces’ locations but is not precise in respiratory rate measurement.
Therefore, deep learning is used as a method to determine ROI.
Movement between frames, especially when the camera is set at a low FPS, is often
a problem in signal acquisition from the thermal camera. For this reason, optical flow is
used to solve this problem. The study conducted by Lyra et al. [
28
] used optical flow to
quantify the thermal image so that the subtle motion in the chest area is reduced for later
extraction of the respiratory signal. Furthermore, the study conducted by Scebba et al. [
40
]
used a dense optical flow algorithm developed by Farneback to reduce the periodic motion
of the torso.
Extracting signals from moving objects is a challenge, and therefore tracking methods
are needed to overcome them, one of which is by using The Kanade-Lucas-Tomasi (KLT)
algorithm, which is used by several studies [
40
,
51
,
52
]. This algorithm uses a linear coordi-
nate mapping which determine the corresponding region in the thermal video. This tracker
extracts feature points from ROI using the minimum eigenvalue algorithm and follows
those points with a single point tracker.
2.4. Signal Extraction, Feature Extraction, and Classification
Signal extraction is an advanced stage that is carried out when the system has suc-
ceeded in identifying ROI and tracking ROI movements. Vital physiological signals are
plotted in units of time known as time-series signals. The general method for extracting this
signal is by comparing pixel-per-pixel motion in thermal images [
61
]. Not infrequently, the
extracted signal requires post-processing in a filter until the signal results can characterize
a change (signature). These changes contain data that must be extracted at the feature
extraction stage. Some algorithms can be used for extracting the feature, i.e., peak detec-
tion [
62
,
63
], fuzzy rule [
64
], one dimensional CNN [
65
], power spectral density [
66
], and
Sensors 2021,21, 7777 7 of 21
various other methods. There is also a python toolkit for quickly extracting the feature [
67
]
in a python package library.
The results of this feature extraction will show a value that can be drawn into the
system’s output. In some studies, this output cannot be easily interpreted. Using machine
learning or deep learning [
43
,
45
], a classification of the signals generated by the previous
stages is carried out based on the model trained beforehand. This output is a form of classi-
fication that users can easily interpret. Each input will be directed to two or more outputs,
either anomaly detection or multi-class output, using classification and machine learning.
3. Thermal Camera for Physiological Measurement
This systematic review reviews the use of thermal cameras for physiological measure-
ments which will be broken down into three subsections, including respiratory rate, heart
rate, and body temperature.
3.1. Respiratory Rate
3.1.1. Overview of Respiratory Rate Measurement
Monitoring RR and related variations are critical for determining an individual’s
health status [
68
]. Moreover, research [
69
] states that the RR can be used as one of the most
efficient indicators to determine whether a person is healthy. Anomalous RR is critical for
detecting significant health problems and can also be used to forecast potentially serious
clinical outcomes such as influenza classification [
42
], lung airflow limitation [
70
], and
sleep apnea screening. Additionally, monitoring variations in RR can assist in identifying a
high-risk intensive care patient up to 24-h before a medical emergency.
RR is defined clinically as the times of respiration recorded within a minute (in breaths
per minute, or bpm). In general, the RR is normal if it is 12–20 bpm for adult humans.
A study [
71
] shows that a slight increase of breaths per minute to 24–28 correlates to an
increased risk of mortality by 5%. During the COVID-19 pandemic, RR counting is essential.
RR is also a vital sign that determines the severity of SARS-CoV-2 19 infection [
72
]. This
viral outbreak has caused many ICUs to be at full capacity and high bed occupancy rates
throughout the world. Medical equipment, including instruments for measuring vital signs,
is insufficient in some countries [
73
]. The thermal camera is undoubtedly one potential
solution for developing a RR counter that works without direct contact.
While RR is a significant clinical predictor of severe events, it is often measured
manually, yielding erroneous findings. RR was often not regularly recorded, even when
the patient’s main complaint was a respiratory illness [68].
There are several well-known methods used for RR monitoring [
68
]. The first is by
using a manual human counting method. However, it might be inaccurate and time-
consuming. The second is using a spirometer. This method is considered accurate and
also measures some other respiratory parameters. However, it can interfere with natural
breathing and difficult for continuous RR monitoring. The third approach employs cap-
nometry, a highly accurate, simple, and measured continuous monitoring technique. This is
a contact approach that is not particularly pleasant and requires analysis using specialized
equipment. The last approach is impedance pneumography, which is precise, continuous,
and concurrent. However, this procedure is challenging to conduct and requires specialized
tools for analysis.
In addition to standard medical measurements used as a reference in hospitals, there
are also several affordable, contact-based methods to measure RR, such as using a nasal
temperature probe near the nostril [
74
] or using a microphone located near the nostril that
records the inhale-exhale sound noise [
75
]. The primary drawbacks of these methods stem
from their intrusive nature: they may be unpleasant and potentially disruptive to sleep
which may alter the results. Additionally, patient movement and any other signal noise
may dislodge the sensors or skew the data.
Noninvasive methods for detecting breathing include non-contact audio analysis,
vibration sensors, thermal imaging, and doppler radar sensors. The extraction of breathing
Sensors 2021,21, 7777 8 of 21
sounds from sensor data polluted by ambient noise is significant for non-contact audio
analysis. Vibration sensors impose positional and postural limitations and need the use
of costly specialized hardware. By detecting the breath as it is exhaled, thermal imaging
methods were utilized to record a breathing signal. Thermal imaging can be achieved
using a thermal camera. One of the advantages of a thermal camera is that it can be used
indirectly and reliably without affecting the light intensity, and can be used in a completely
dark room. In general, the challenge in using a thermal camera is processing thermal
images and extracting features from these images.
3.1.2. Summary of Thermal Camera Usage Related to Respiratory
Next, Table 3summarizes all the characteristics of the studies with respiration as the
main objective that were used in this systematic review.
3.1.3. Deep Learning for RR Monitoring
Several studies use deep learning to classify the breathing pattern and to determine
the ROI from the image. There are some deep learning algorithms used in the research list
above, CSPDarknet [
28
], FlowNet 2.0 [
76
] (algorithm based on deep networks), k-nearest
neighbors (k-NN) [43,45], and cascade convolutional neural network (CCNN) [40].
CSPDarknet is the backbone running on YOLOv4. Lyra et al., in their research [
28
],
used it to determine four classes that would be used as ROI, namely head, chest, patient,
and clinician. Head ROI is used to measure body temperature, chest ROI is used for RR
estimation, and clinician ROI is used to count the number of clinicians near the patient.
After the chest ROI was determined, respiratory movement was tracked using a pixel-wise
temporal mean algorithm by comparing movement between frames. The use of the neural
network to determine five facial landmarks were also used by Scebba et al. [
40
] by utilizing
CCNN on NIR images.
Meanwhile, Jagadev et al., in both of their studies [
43
,
45
], used k-NN to decide
whether the human volunteer had normal or abnormal respiration. Previously, the breath
detection algorithm (BDA), which they also developed, was used to extract respiratory
movements. In simple terms, BDA calculates the number of peaks and valleys based on
the specified ROI movement from the nostrils. Finally, the output of this BDA is forwarded
to the k-NN to classify between normal breathing, bradypnea, or tachypnea (abnormal).
The output of this system is compared with the Support-Vector Machine (SVM) method to
determine the accuracy achieved.
3.1.4. Camera Sensor Fusion: Usability and Image Fusion Method
Several studies on the list combined the two types of cameras with different measure-
ments related to respiratory. Most of them use a combination of a LWIR thermal camera
with a CMOS RGB camera or a color camera that we commonly find on smartphones,
webcams, or point-and-shoot cameras. The merging of these two cameras aims to gain the
advantages of each camera and eliminate the weaknesses of each camera. Meanwhile, light
significantly affects RGB cameras, and this type of camera cannot be used in low or no
light conditions. In contrast, thermal cameras can capture objects even in light conditions
because they work using the principle of radiation emitted by objects. Table 4summarizes
each study involving fusion cameras and their characteristics.
Sensors 2021,21, 7777 9 of 21
Table 3. Summary of Thermal Camera Usage Related to Respiratory.
Author Objectives
Thermal Camera
Model, FPS, and
Dimension Used
Image and Signal
Processing Tools Algorithm Used Validation Method Performance
Chen et al. [52] RR measurement MAG 62, 10 FPS,
640 ×480 pixels
·Open CV: Image
Processing Tools
·KLT: Coordinate
Mapping
·RSQI_dtw: score each
ROI
Compared with the
GY-6620 sleep monitor
·
Root Mean Square Error:
0.71 breaths/min and
0.76 breaths/min
Goldman et al. [46] RR measurement Thermovision A40,
50FPS, 320 ×240 pixels
·Matlab for signal
processing software ·n/a
Compared with
standard measurements
of nasal pressure
·Intraclass correlation of
0.978 (0.991–0.954 95%
CI)
Hu et al. [51] RR measurement MAG 62, 640 ×480
pixels
·All analysis conducted
with Matlab R2014A
·Viola-Jones Algorithm
for Cascade Object
Detector
·Shi-Tomasi for the
corner detection
algorithm
Compared with human
observers (manual
counting)
·Accuracy for face, nose,
and mouth: 98.46%,
95.38%, 84.62%
Hu, et al. [53]RR and HR
measurement
MAG 62, 30 FPS,
640 ×480 pixels
·Matlab R2014a for
Image Processing
·Affine Transformation
for transforming images
Compared with human
observers (manual
counting)
·Determination
Coefficient: 0.831
Jagadev et al. [45] RR measurement Flir A325, 25 FPS,
320 ×240 pixels
·k-nearest neighbors
(k-NN) Classifier
·the t-Stochastic
Neighbor Embedding
algorithm
Statistical calculation of
sensitivity, precision,
spurious cycle rate,
missed cycle rate
·Sensitivity: 98.76%
·Precision: 99.07%
·Spurious cycle rate:
0.92%
·
Missed cycle rate: 1.23%
Jagadev et al. [43]RR measurement and
classification
Flir A325, 25 FPS,
320 ×240 pixels
·Breath Detection
algorithm for counting
RR
·k-NN and SVM to
classify the
abnormalities
Statistical calculation of
sensitivity, precision,
spurious cycle rate,
missed cycle rate
·Sensitivity: 97.2%
·Precision: 98.6%
·Spurious cycle rate:
1.4%
·Missed cycle rate: 2.8%
Sensors 2021,21, 7777 10 of 21
Table 3. Cont.
Author Objectives
Thermal Camera
Model, FPS, and
Dimension Used
Image and Signal
Processing Tools Algorithm Used Validation Method Performance
Jakkaew et al. [54]
RR measurement and
body movement
detection
Compact PRO, 17 FPS,
640 ×480 pixels
·minMaxLoc OpenCV:
ROI Detection
·findContour:
programming library to
detect significant
movement
·OpenCV: image
processing framework
Compared with Go
Direct respiratory belt
·
Root Mean Square Error:
1.82 ±0.75 bpm
Lyra et al. [28] RR measurement Optris PI 450i, 4 FPS,
382 ×288 pixels
·YOLO_mark: Labelling
framework
·YOLOv4 with
CSPDarknet53 Backbone:
training framework
·YOLOv4-Tiny:
Real-Time classifier
framework
Compared with thoracic
bioimpedance based
patient monitor device
(Philips, Amsterdam,
The Netherlands)
·Intersection over unit
(IoU): 0.70
·IoU (tiny): 0.75
·Mean Absolute Errors:
2.79 bpm, 2.69 bpm
(Tiny)
Mutlu et al. [44] RR measurement Flir A325, 60 FPS,
320 ×240 pixels
·FLIR ResearchIRMax:
Video Recording
software
·
Labview: camera trigger
software
·
MATLAB: analysis tools
Compared with a
respiratory belt
transducer containing a
piezoelectric
·
Median Error Rate: 6.2%
Negishi et al. [47] RR measurement Flir A315, 15 FPS,
320 ×240 pixels
·Labview: Image
recording and analysis
·Grab cut: Extraction of
contour
·Oriented FAST and
Rotated Brief (ORB):
feature matching
·dlib: ROI detection
library
·OpenCV: Image
Processing Tools
Compared with a
respiratory effort belt
(DL-231, S&ME, Japan)
·
Root Mean Square Error:
2.52 RPM
·Correlation Coefficient
0.77
Sensors 2021,21, 7777 11 of 21
Table 3. Cont.
Author Objectives
Thermal Camera
Model, FPS, and
Dimension Used
Image and Signal
Processing Tools Algorithm Used Validation Method Performance
Negishi et al. [48]RR and HR
measurement
Flir A315, 15 FPS,
320 ×240 pixels
·Labview: Image
recording and analysis
·Grab cut: Extraction of
contour
·Oriented Fast and
Rotated Brief: feature
matching
·dlib: ROI detection
library
·OpenCV: Image
Processing Tools
Compared with a
respiratory effort belt
(DL-231, S&ME, Japan)
·
Root Mean Square Error:
1.13 RPM
·Correlation Coefficient
0.92
Negishi et al. [42]RR and HR
measurement
Flir A325, 15 FPS,
320 ×240 pixels
·dlib: ROI detection
library
·OpenCV: Image
Processing Library
·Multiple signal
classification (MUSIC)
algorithm for signal
estimation
·
Homography Matrix for
facial landmarking
Compared with a
respiratory effort belt
(DL-231, S&ME, Japan)
·Sensitivity: 85.7%
·Specificity: 90.1%
Pereira et al. [50]RR measurement for
infants
VarioCAMR HD 820S, 30
FPS, 1024 ×768 pixels
·Matlab 2017 for
Evaluation and Signal
Processing software
Compared with thoracic
effort
piezo plethysmography
belt, namely
SOMNOlab2
·
Root Mean Square Error:
(0.31 ±0.09)
breaths/min.
Scebba et al. [40]RR measurement for
apnea detection
NIR: See3cam_CU40 MV,
15 FPS, 336×190 pixels
LWIR: Flir Lepton 3.5,
8.7 FPS, 160 ×120 pixels
·Smart Signal Quality
Fusion (S2Fusion) for RR
estimation
·Cascade Convolutional
Neural Network
(CCNN) for facial
landmark
·KLT for tracking
Compared with
piezo-resistive sensors
based ezRIP module,
Philips Respironics
·Median of Root Mean
Square Error: 1.17
breaths/min
Sensors 2021,21, 7777 12 of 21
Table 4. List of Studies Involved Camera Fusion and Its Characteristic.
Authors Fusion Camera Combination Characteristic
Scebba et al. [40] NIR and LWIR Camera
LWIR camera used for nostrils
and chest ROI, NIR camera
used for chest ROI
Negishi et al. [42,47,48] RGB and LWIR Camera
RGB camera used for
determining ROI and
extracting PPG signals while
LWIR camera used for
extracting respiratory signal
Hu et al. [51] RGB and LWIR Camera
RGB camera used for
determining ROI while LWIR
camera used for extracting
respiratory signal
Chen et al. [52] RGB and LWIR Camera
RGB camera used for
determining ROI and
alternative method to extract
respiratory signal if no face
detected while LWIR camera
sued for extract respiratory
signal if any face detected
The following spectral video fusion study [
40
] combines two types of thermal cameras,
which are NIR and LWIR cameras. Scebba et al. initiated a new algorithm to calculate
the RR based on multispectral data fusion from the two cameras. The multispectral ROI
localization analyzes footage from LWIR and NIR cameras. The localized ROIs extract the
Thermal Airflow (TA) signal from the nose ROI and the respiratory motion signal from
the chest ROIs in the LWIR and NIR cameras. The RR and signal to noise ratio (SNR)
are calculated in the Signal Quality-based fusion (SQb Fusion) using the TA’s frequency
analysis and respiratory motion signal both from the LWIR camera and the NIR camera.
The weighted median generates an RR estimation by combining all RR estimations and
weighting them by their SNR. Temporal aspects and frequency characteristics of TA,
respiratory motion from NIR and LWIR cameras are used as input to the ensemble of
support-vector-machine to determine whether apnea occurs or not. The intelligent signal
quality-based fusion (S
2
Fusion) algorithm combines the findings of the SQb Fusion with
the apnea classifier (h) to produce an apnea-sensitive signal.
There is also the use of two different cameras to measure two different physiological quan-
tities. Negishi et al. used the same two cameras configuration in all
three studies [42,47,48]
.
LWIR camera is used to measure RR while RGB camera is used to measure HR. Image
fusion determines ROI based on RGB images rather than thermal images. To determine the
ROI of the nose and mouth, a feature matching analysis was performed with a homography
matrix between RGB and thermal images based on the contours of the human face. Grabcut
is used for facial contour extraction, while oriented-fast and rotated brief (ORB) algorithm
is used for feature matching and dlib for determining the region of nose and mouth. All
these algorithms and tools are available as libraries in OpenCV.
Almost the same as before, Hu et al., in their studies [
51
], also use camera fusion to
make the determination of ROI while facial objects was detected using the RGB camera. To
record thermal and visual images, an affine transformation is needed. The first step is to
pick the most correlated points in the first frame of bimodal movies to determine the thermal
image’s fixed point and RGB image’s image points. Following that, cross-correlation is
used to modify these points in order to produce the transformation matrix. After mapping
between the RGB image and the thermal image, the bounding box is determined for
the ROI object (the face, nose, and mouth) by using the Viola-Jones algorithm. The shi-
Sensors 2021,21, 7777 13 of 21
Tomasi corner detection algorithm is used to help extract the interest points to calculate the
covariance matrix, while the KLT algorithm is used to track ROI on movement.
As before, the use of RGB camera for face detection is also used by Chen et al. [
52
].
They provide an alternative method to measuring RR if no face is detected by tracking
the sticky markers that placed on the body. Meanwhile, to combine RGB and thermal
images, they use affine transformation to transform two different geometric shapes. The
Viola-Jones algorithm is used to detect faces, while the KLT algorithm is used for tracking
the ROI.
3.1.5. RR Signal Extraction Process
A thermal image only has a single information channel that is a temperature repre-
sentation converted into an image matrix. It is by utilizing this only information that a
respiratory signal can be generated. In general, as shown in Figure 3, two changes can
be observed: the first is a change in the temperature value and the second is a change in
movement. Each of these characters will be explored by each study based on the methods
and algorithms they use, respectively.
The study [
44
] conducted by Mutlu et al. used the temperature change around the
nostril to indicate respiration. After defining the ROI and excluding non-varying pixels, the
decreasing segments are identified using experimentally established criteria for a minimal
frame-to-frame decline. If a single frame exists between two possible decrease segments,
they are combined. The process of identifying temperature changes at the pixel level by
comparing frame per frame was also used in other studies.
Another way of extracting the RR signal is to consider the movement of pixels between
frames without making the nose or mouth the ROI. This method was used in the following
study [
54
] and is more reliable when used in real time and with patients in the frame
using a blanket or in a position not facing the camera. Breathing motion detection uses a
subtraction technique in the background to identify motion by computing the difference
between the current and previous frames. To be precise, the absolute difference between
the current frame
I(x,y,f)
and the previous frame
I(x,y,f1)
is computed for all pixels
where
x
,
y
are the coordinates of x and y axis respectively, and f is the frame sequence. Then,
employing thresholding, erosion, and dilation procedures, parts of the relocated region
are removed. The parameters utilized in these procedures are 5 for thresholding, which
ensures that the difference between the pixel values is smaller than 5, and 5
×
5 kernel
for opening (i.e., erosion and dilation). Following that, boundary boxes are determined
using contour detection and noise filtering. Finally, the RR is determined by the number
of bounding boxes. The concept of comparing each pixel movement between frames per
frame is also used by other studies [77].
Sensors 2021, 21, x FOR PEER REVIEW 12 of 21
[42,47,48]. LWIR camera is used to measure RR while RGB camera is used to measure HR.
Image fusion determines ROI based on RGB images rather than thermal images. To deter-
mine the ROI of the nose and mouth, a feature matching analysis was performed with a
homography matrix between RGB and thermal images based on the contours of the hu-
man face. Grabcut is used for facial contour extraction, while oriented-fast and rotated
brief (ORB) algorithm is used for feature matching and dlib for determining the region of
nose and mouth. All these algorithms and tools are available as libraries in OpenCV.
Almost the same as before, Hu et al., in their studies [51], also use camera fusion to
make the determination of ROI while facial objects was detected using the RGB camera.
To record thermal and visual images, an affine transformation is needed. The first step is
to pick the most correlated points in the first frame of bimodal movies to determine the
thermal image’s fixed point and RGB image’s image points. Following that, cross-correla-
tion is used to modify these points in order to produce the transformation matrix. After
mapping between the RGB image and the thermal image, the bounding box is determined
for the ROI object (the face, nose, and mouth) by using the Viola-Jones algorithm. The shi-
Tomasi corner detection algorithm is used to help extract the interest points to calculate
the covariance matrix, while the KLT algorithm is used to track ROI on movement.
As before, the use of RGB camera for face detection is also used by Chen et al. [52].
They provide an alternative method to measuring RR if no face is detected by tracking the
sticky markers that placed on the body. Meanwhile, to combine RGB and thermal images,
they use affine transformation to transform two different geometric shapes. The Viola-
Jones algorithm is used to detect faces, while the KLT algorithm is used for tracking the
ROI.
3.1.5. RR Signal Extraction Process
A thermal image only has a single information channel that is a temperature repre-
sentation converted into an image matrix. It is by utilizing this only information that a
respiratory signal can be generated. In general, as shown in Figure 3, two changes can be
observed: the first is a change in the temperature value and the second is a change in
movement. Each of these characters will be explored by each study based on the methods
and algorithms they use, respectively.
Figure 3. An overview of how the signal extraction process from a thermal image is carried out. In
general, there are two methods: first by measuring changes in temperature in the area around the
nostrils and mouth, and second by looking at the movement based on the comparison between
changes in pixels in each frame.
The study [44] conducted by Mutlu et al. used the temperature change around the
nostril to indicate respiration. After defining the ROI and excluding non-varying pixels,
the decreasing segments are identified using experimentally established criteria for a min-
imal frame-to-frame decline. If a single frame exists between two possible decrease seg-
ments, they are combined. The process of identifying temperature changes at the pixel
level by comparing frame per frame was also used in other studies.
Figure 3.
An overview of how the signal extraction process from a thermal image is carried out.
In general, there are two methods: first by measuring changes in temperature in the area around
the nostrils and mouth, and second by looking at the movement based on the comparison between
changes in pixels in each frame.
Sensors 2021,21, 7777 14 of 21
Negishi et al., in three of their studies [
42
,
47
,
48
], use multiple signal classification
(MUSIC) algorithms to calculate RR estimates. This algorithm is also proven to be more
accurate than FFT in time series data with a shorter window. By using this algorithm, the
correlation matrix from the time series data is calculated, and the eigenvectors are obtained.
3.1.6. Performance Validation Method on RR
Testing on RR is carried out by comparing the system output with reference equipment
such as apnea monitors, sleep diagnostic equipment, respiratory belts, and other respiratory
rate measuring devices. For example, in this study [
52
], GY-6620 (South China Medical and
Electrical Technology Co., Ltd., Zhengzhou, China) was used as a comparison.
GY-6620
is the equipment used in polysomnography or sleep tests and it can provide output
records of body activity during sleep, including RR. In another study [
50
], the SOMNOlab2
tool (Weinman GmbH, Hamburg, Germany) was used as the reference. The device is
a measuring device for body activity that records thoracic movements based on piezo
plethysmography. In his three studies [
42
,
47
,
48
], Negishi also used the same validation
system, namely the respiratory effort belt. However, the model of the tool is only listed
in one of their studies [
48
], namely DL-231 (S&ME, Tokyo, Japan). A similar force sensor-
based respiratory belt was also used by another study [
54
], the Go Direct Respiration
Belt model. This belt is set to record ten respiration samples per second for 5400 s. This
study [
44
] also uses a respiratory belt to obtain the reference value. Unfortunately, not
all studies compare the results with standard medical equipment; others use statistical
calculations as a performance test method, such as scatter plots, bland-Altman plots, or
other statistical calculations.
3.2. Heart Rate
Studies that use thermal cameras to measure heart rate tend to be less popular than
those measuring respiratory rate. Many researchers use RGB cameras rather than thermal
cameras for heart rate measurement. The heart rate measurement with the RGB camera uti-
lizes changes in skin color that can be observed in one of the three color channels in the RGB
image. While in thermal cameras, discriminant characteristics can be obtained based on
the two most popular methods: using the blood perfusion temperature changes from a par-
ticular pixel [
49
,
78
,
79
] or by analyzing the head movement based on Balistocardiography
(BCG) [80,81].
Similar to the respiratory rate process, obtaining a heart rate signal from a thermal
camera begins with capturing the images, pre-processing the image, detecting the ROI, and
tracking the ROI so the ROI will be more stable to movement.
Some studies describe the camera specification they use for the research. For example,
Bennet et al. [
79
] use a FLIR-A camera with 640
×
480 pixels and 60 FPS framerate. On
the other hand, Kim et al. [
49
] use a FLIR T430sc camera with 320
×
240 pixels and 12 FPS
framerate. Unfortunately, Gault et al. [
78
] use pre-recorded thermal video with ten subjects
without further detail about the camera specification.
For studies that are using temperature changes methods, they select some regions
such as the highest blood vessel temperature region on the skin [
49
,
78
] or chest [
79
] as the
ROI. In other parts, for the head movement-based method, they use the entire head as an
ROI and track its movement [80,81].
Some studies reported that the noise was very high on the unprocessed signal and the
discriminant characteristics are almost imperceptible. Various and multiple filters were
applied to enhance the signal quality and its characteristics. For example, Kim et al. [
49
]
converted the time-series signals into the frequency domain using Fourier transform, while
some others [
79
81
] put a bandpass filter to extract the heartbeat signals and count the
heart rate.
For studies that rely on the head movement method, they use almost identical process-
ing steps. Both Li et al. [
80
] and Balakrishnan et al. [
81
] applied temporal filtering for the
signal obtained from the head movement trajectories. Then they use principal component
Sensors 2021,21, 7777 15 of 21
analysis (PCA) to obtain the periodic signal caused by the heartbeat. Finally, peak detection
is used to help determine the heart rate.
Each study shows promising results by obtaining identical value compared to the
reference ground truth, i.e., Kim et al. [
49
] obtained an average accuracy of 95.48%,
Gault et al. [
78
] reached 90% accuracy, Li et al. [
80
] had a mean error of 2.7%, and fi-
nally, Balakrishnan et al. [
81
] achieve mean error of 1.5%. These results were achieved by
comparing them to a contact-based device such as a pulse sensor and ECG. Unfortunately,
there is no detailed information regarding the reference equipment model or specification.
3.3. Body Temperature
In most cases, the body temperature is measured using the LWIR type thermal camera.
For example, Pan et al. [
36
]. used the P384-20 thermal camera, Rao et al. [
37
] used the
Mobotix 16TR camera consisting of RGB and LWIR thermal camera, while Lewicki et al. [
41
]
used the FLIR Lepton 3.5.
There are some steps to process the image and calculate the body temperature value.
The process starts from obtaining the image from the thermal camera, followed by de-
termining the ROI. In this case, some studies [
37
,
41
] used a combination of RGB and
thermal cameras and needed a calibration process between two images. This RGB camera
objective is to obtain enough information about the facial landmark and help select the
ROI. Rao et al. [
37
] also implemented some advanced algorithms in their system, such as
a prioritization algorithm to decide which person should be measured and background
removal to separate person object and background environment. In terms of tracking,
Rao et al. [
37
] used a neural network-based head tracker while another study [
36
] used an
elliptical head tracking method.
Matching the two images from RGB cameras and thermal cameras also requires a
series of processes. Instead of using Affine Transformation, Lewicki et al. [
41
] use coarse-
grained field of view method, while Rao et al. [
37
] use manual offset approach and dynamic
frame alignment.
Some authors also mentioned challenges related to accuracy. Rao et al. [
37
] imple-
mented the temperature correction algorithm to compensate the temperature with the
distance using a regression and multi-layer perceptron. The camera also needs to be
calibrated in order to achieve a minimum error [82].
Each study has its procedure to get the body temperature value. For example,
Pan et al. [
36
] determine the body temperature by measuring the highest temperature
value from every point on the face. In comparison, Rao et al. [
37
] set some algorithms
to prioritize the region of eyes and forehead followed by face and head. In contrast,
Lewicki et al. [
41
] use the average value of each point on the face as the body temperature.
The eye’s inner canthus or medial canthus is known as the most accurate region of the
face for measuring body temperature [
83
,
84
]. This region has a temperature that is almost
identical to the eardrum temperature [
85
,
86
]. This region also has a similar temperature
compared with the rectum, which has been the most identified as a reference for inner
core body temperature [
87
]. However, because the selection of the inner eye canthus ROI
in thermal images is quite challenging and requires an advanced method (for example,
with machine learning or cross correlation [
88
]), several studies [
87
,
89
] have chosen the
highest temperature value on the face or forehead as the body temperature which is also
quite accurate.
Several methods were used to validate the system result by comparing it with the
reference. Rao et al. [
37
], in their studies, used the black body measured temperature as the
ground truth and obtained 100% sensitivity and 96.9% specificity from 105 people as the
subject. On the other hand, Pan et al. [
36
] used an infrared ear thermometer as a reference
and achieved a CAND value above 0.9.
Sensors 2021,21, 7777 16 of 21
4. Discussion
This systematic review provides an overview of studies that use thermal cameras to
measure and monitor aspects of vital signs in the human body. This discussion section will
present the advantages, disadvantages, challenges, and future trends and works.
4.1. Advantages of Thermal Camera-Based Physiological Measurement
A thermal camera can be used to measure several parameters such as RR, HR, and
body temperature. Thus, it has the potential to be used in non-invasive, continuous mea-
surements such as in neonatal ICU monitoring, long-term monitoring, fitness applications,
and health screening. Moreover, during the COVID-19 pandemic, non-contact body tem-
perature measurement methods promise hygiene. Mainly, it could be implemented as
non-contact physiological measurement. Moreover, the thermal camera is able to operate
in low light conditions. Therefore, using the thermal camera-based non-contact method
offers more flexibility and convenience for the patient.
Since thermal camera overcomes the drawbacks of contact-based sensors, some clinical
applications are using this method. For instance, it can provide continuous monitoring for
neonates [
4
], classifying affective states [
90
], and monitoring exercise [
91
]. The thermal
camera method is also effective for sleep monitoring [
54
] and estimating the movement
during sleeping [
5
]. In addition, there are also some other implementations such as human
thermal comfort modeling [
92
], lie detector [
93
,
94
], mood and stress-related disorders [
95
],
sober and drunk classification [96], and many more.
4.2. Challenges of Thermal Camera-Based Physiological Measurement
A significant challenge of using a non-contact method that depends on the camera
is that it is very susceptible to movement—both the motion of objects in the frame and
the movement of the camera itself. Research in ROI tracking is needed to overcome
this shortcoming. Additionally, separating partially obscured individuals or objects of
the same temperature in thermal pictures can be challenging, as their pixels have the
same intensity [
97
]. In these cases, including depth information or color edges can aid
in disambiguation.
Related to the applicability, there is a concern for non-contact measurement using a
thermal camera when faced with the current standard medical method in terms of accuracy
and reliability. Although the accuracy of the prototype in this systematic review shows
good results, it is not enough to exceed the performance of existing medical standard
equipment. This happens because the systems developed in these studies have not been
tested in actual health care conditions. Moreover, it has not been standardized or medically
certified. Therefore, measurement involving several possibilities in real-case scenarios
is an important direction to take. Several studies emphasize the importance of further
clinical trials to ensure the reliability of the developed systems. The tests carried out need
to combine several test scenarios to test the reliability of the system in measuring RR,
for example, the use of various kinds of bed covers or blankets [
54
], the involvement of
various patient demographics [
40
], and further clinical trials in the health institution such
as hospital or clinic.
4.3. Future Trends and Works
Several suggestions and prospects need to be considered to improve the future of
thermal cameras for physiological signal measurements.
4.3.1. Healthcare Applications
In healthcare, the telemedicine revolution shifts illness prediction, prevention, and
treatment from a hospital-centered reactive paradigm to a person-centered one. Driven
by the COVID-19 pandemic and a growing need for home healthcare, telemedicine trends
have the potential to transform and enhance healthcare delivery and accessibility. Non-
contact physiological signal monitoring is a significant development that supports current
Sensors 2021,21, 7777 17 of 21
telemedicine technology [
98
], including using thermal cameras as part of non-contact
monitoring. Thermal cameras also provide a hygienic aspect for users where there are no
components attached to the body to minimize contact between users.
For the popularization of health care, another critical challenge is related to low-
cost implementation. Meanwhile, most of the studies in this systematic review have
not explained development costs in detail. Research related to low-cost development
is essential because the cost aspect is considered in implementing medical devices in
developing countries [99,100].
4.3.2. Machine Learning
Machine learning can be used as an enhancement to thermal images. As discussed in
the previous section, a thermal camera generally has a small resolution. The resulting image
can be enhanced to have a higher resolution by using the Thermal Image Enhancement
method using convolutional neural network (TEN-CNN) [
101
]. Furthermore, the challenge
in using machine learning is increasing the variety of datasets used to improve inference
engine capabilities and accuracy. This is also expressed by Lyra et al. in their study [
28
]. The
YOLOv4 they use relies on large-scale datasets to enhance their inference engine abilities.
Improvements to this dataset also need to consider aspects of data variation for various
races, ages, genders, weight and height, and other aspects.
4.3.3. Multi-Parameter and Data Fusion
Multi-parameter measurement could be future work for the thermal camera-based
physiological measurement system. Simplifying the measurement of several parameters
will benefit patients using this system, especially multi-parameters on vital signs such
as body temperature, blood pressure, heart rate, and respiratory rate. One study [
102
]
that evaluated a multi-parameter wireless wearable sensor to monitor vital signs showed
excellent and effective results. However, not many similar systems are accommodated in a
non-invasive method.
Multi-parameter focuses on various parameters being measured. In contrast, data
fusion is an aspect that combines several input data into the same process to achieve
an output from a system. In this systematic review, there are several studies [
40
,
42
,
47
]
that use data fusion in the form of combining several types of cameras. The aim and
benefit of combining various sensors and data are to eliminate each sensor’ or method’s
disadvantages and take advantage of every sensor. Further observation related to the
combination of various sensors should be addressed to achieve better signal measurement.
5. Conclusions
In this paper, we have reviewed the existing literatures regarding thermal cameras
to measure the human body’s respiratory rate, heart rate, and body temperature. The
general process stages in processing thermal images for physiological signal measurements
are discussed, compared, and evaluated to find advantages and challenges. The advan-
tages of using a thermal camera in measuring physiological signals include comfort and
convenience for the patient due to its non-invasive aspects, hygiene, and the ability to
capture images in low light conditions. On the contrary, the challenges include how to
reduce motion artifact and increase the accuracy and reliability of the physiological signal
measurement system.
This systematic review contributes a comprehensive overview by providing high-
lights of current methodological concerns related to using the thermal camera to measure
physiological signal. Furthermore, this systematic review can be used as an initial reference
for researchers to identify the existing research gap. In addition, we also provide several
future development directions, including integrating multi-parameter systems to improve
functionality, using data fusion and machine learning technology to improve measure-
ment accuracy and reliability, and developing low-cost thermal imaging applications to
increase penetration.
Sensors 2021,21, 7777 18 of 21
Author Contributions:
Conceptualization: M.C.T.M. and Y.-H.L.; studies screening: M.C.T.M. and
S.-J.L.; original draft preparation: M.C.T.M., Y.-H.L. and S.-J.L.; contextual review: Y.-H.L. and
N.-K.C.
; proofreading: S.-J.L., Y.-H.L. and N.-K.C.; editing: M.C.T.M., Y.-H.L. and N.-K.C. All authors
have read and agreed to the published version of the manuscript.
Funding:
This work was supported in part by a research grant from the Ministry of Science and
Technology, Taiwan for academic research under Grant MOST 109-2637-E-011-002- and Grant MOST
110-2221-E-011-123-, and financially supported by the Taiwan Building Technology Center from The
Featured Areas Research Center Program within the framework of the Higher Education Sprout
Project by the Ministry of Education in Taiwan.
Data Availability Statement:
The data used in this review are from published primary studies
available in the public domain.
Conflicts of Interest: The authors declare no conflict of interest.
References
1.
Zadeh, H.G.; Masoumzadeh, S.; Nour, S.; Kianersi, S.; Eyvazi Zadeh, Z.; Joneidi Shariat Zadeh, F.; Haddadnia, J.; Khamseh, F.;
Ahmadinejad, N. Breast cancer diagnosis by thermal imaging in the fields of medical and artificial intelligence sciences: Review
article. Tehran Univ. Med. J. 2016,74, 377–385.
2.
Balaji, A.S.; Makaram, N.; Balasubramanian, S.; Swaminathan, R. Analysis of pre- and post- fatigue thermal profiles of the
dominant hand using infrared imaging. ACM Int. Conf. Proceeding Ser. 2017,1, 53–57.
3.
Cardone, D.; Perpetuini, D.; Filippini, C.; Spadolini, E.; Mancini, L.; Chiarelli, A.M.; Merla, A. Driver stress state evaluation by
means of thermal imaging: A supervised machine learning approach based on ECG signal. Appl. Sci. 2020,10, 5673. [CrossRef]
4.
Topalidou, A.; Ali, N.; Sekulic, S.; Downe, S. Thermal imaging applications in neonatal care: A scoping review. BMC Pregnancy
Childbirth 2019,19, 381. [CrossRef]
5.
Mohammadi, S.M.; Enshaeifar, S.; Hilton, A.; Dijk, D.-J.; Wells, K. Transfer Learning for Clinical Sleep Pose Detection Using a
Single 2D IR Camera. IEEE Trans. Neural Syst. Rehabil. Eng. 2021,29, 290–299. [CrossRef] [PubMed]
6.
Mann, D.M.; Chen, J.; Chunara, R.; Testa, P.A.; Nov, O. COVID-19 transforms health care through telemedicine: Evidence from
the field. J. Am. Med. Inform. Assoc. 2020,27, 1132–1135. [CrossRef]
7.
Lee, S.M.; Lee, D. Opportunities and challenges for contactless healthcare services in the post-COVID-19 Era. Technol. Forecast.
Soc. Chang. 2021,167, 337–339. [CrossRef] [PubMed]
8.
Kim, H.; Jeong, J. Non-Contact Measurement of Human Respiration and Heartbeat Using W-band Doppler Radar Sensor. Sensors
2020,20, 5209. [CrossRef]
9.
McDuff, D.; Nishidate, I.; Nakano, K.; Haneishi, H.; Aoki, Y.; Tanabe, C.; Niizeki, K.; Aizu, Y. Non-contact imaging of peripheral
hemodynamics during cognitive and psychological stressors. Sci. Rep. 2020,10, 10884. [CrossRef]
10.
Hall, T.; Lie, D.Y.C.; Nguyen, T.Q.; Mayeda, J.C.; Lie, P.E.; Lopez, J.; Banister, R.E. Non-Contact Sensor for Long-Term Continuous
Vital Signs Monitoring: A Review on Intelligent Phased-Array Doppler Sensor Design. Sensors 2017,17, 2632. [CrossRef]
11.
Cheshire, W.P. Thermoregulatory disorders and illness related to heat and cold stress. Auton. Neurosci.
2016
,196, 91–104.
[CrossRef]
12.
Fu, M.; Weng, W.; Chen, W.; Luo, N. Review on modeling heat transfer and thermoregulatory responses in human body. J. Therm.
Biol. 2016,62, 189–200. [CrossRef] [PubMed]
13. Ivanov, K. The development of the concepts of homeothermy and thermoregulation. J. Therm. Biol. 2006,31, 24–29. [CrossRef]
14. Barr, E.S. The Infrared Pioneers—II. Macedonio Melloni. Infrared Phys. 1962,2, 67–74. [CrossRef]
15.
Wikipedia Contributors. Thermographic Camera. Available online: https://en.wikipedia.org/w/index.php?title=
Thermographic_camera&oldid=1052657772 (accessed on 30 October 2021).
16.
Tattersall, G.J. Infrared thermography: A non-invasive window into thermal physiology. Comp. Biochem. Physiol. Part A Mol.
Integr. Physiol. 2016,202, 78–98. [CrossRef] [PubMed]
17.
Howell, K.J.; Smith, R.E. Guidelines for specifying and testing a thermal camera for medical applications. Thermol. Int.
2009
,19,
5–12.
18.
Bestsennyy, O.; Gilbert, G.; Harris, A.; Rost, J. Telehealth: A Quarter-Trillion-Dollar Post-COVID-19 Reality? McKinsey
2020
.
Available online: https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/telehealth- a-quarter-
trillion-dollar-post-covid-19-reality (accessed on 4 November 2021).
19. Mikulska, D. Contemporary applications of infrared imaging in medical diagnostics. Ann. Acad. Med. Stetin. 2006,52, 35–40.
20.
Lahiri, B.; Bagavathiappan, S.; Jayakumar, T.; Philip, J. Medical applications of infrared thermography: A review. Infrared Phys.
Technol. 2012,55, 221–235. [CrossRef]
21.
El, E.N.; Una, D. Applications of Infrared Thermography in Sports: A Review. Rev. Int. Med. Cienc. Act. Física Deporte
2015
,15,
805–824.
22.
Znamenskaya, V.V.S.I.A.; Koroteeva, E.Y.; Khakhalin, A.V. Thermographic visualization and remote control of dynamical
processes around a facial area. Sci. Vis. 2016,8, 122–131.
Sensors 2021,21, 7777 19 of 21
23.
Moreira, D.G.; Costello, J.T.; Brito, C.J.; Adamczyk, J.G.; Ammer, K.; Bach, A.J.; Costa, C.M.; Eglin, C.; Fernandes, A.A.; Fernández-
Cuevas, I.; et al. Thermographic imaging in sports and exercise medicine: A Delphi study and consensus statement on the
measurement of human skin temperature. J. Therm. Biol. 2017,69, 155–162. [CrossRef]
24. Pan, C.-T.; Francisco, M.D.; Yen, C.-K.; Wang, S.-Y.; Shiue, Y.-L. Vein Pattern Locating Technology for Cannulation: A Review of
the Low-Cost Vein Finder Prototypes. Sensors 2019,19, 3573. [CrossRef]
25.
NAggarwal, N.; Garg, M.; Dwarakanathan, V.; Gautam, N.; Kumar, S.S.; Jadon, R.S.; Gupta, M.; Ray, A. Diagnostic accuracy of
non-contact infrared thermometers and thermal scanners: A systematic review and meta-analysis. J. Travel Med.
2020
,27, taaa193.
[CrossRef] [PubMed]
26.
Foster, J.; Lloyd, A.B.; Havenith, G. Non-contact infrared assessment of human body temperature: The journal Temperature
toolbox. Temperature
2021
, 1–14. Available online: https://www.tandfonline.com/action/showAxaArticles?journalCode=ktmp20
(accessed on 4 November 2021). [CrossRef]
27.
He, Y.; Deng, B.; Wang, H.; Cheng, L.; Zhou, K.; Cai, S.; Ciampa, F. Infrared machine vision and infrared thermography with deep
learning: A review. Infrared Phys. Technol. 2021,116, 103754. [CrossRef]
28.
Lyra, S.; Mayer, L.; Ou, L.; Chen, D.; Timms, P.; Tay, A.; Chan, P.; Ganse, B.; Leonhardt, S.; Antink, C.H. A Deep Learning-Based
Camera Approach for Vital Sign Monitoring Using Thermography Images for ICU Patients. Sensors 2021,21, 1495. [CrossRef]
29.
Nowara, E.M.; Duff, D.M. Combating the Impact of Video Compression on Non-Contact Vital Sign Measurement Using
Supervised Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Korea,
27–28 October 2019.
30.
AKirimtat, A.; Krejcar, O.; Selamat, A.; Herrera-Viedma, E. FLIR vs SEEK thermal cameras in biomedicine: Comparative diagnosis
through infrared thermography. BMC Bioinform. 2020,21 (Suppl. S2), 22.
31.
Khare, S.; Singh, M.; Kaushik, B.K. Development and validation of a quantitative model for the subjective and objective minimum
resolvable temperature difference of thermal imaging systems. Opt. Eng. 2019,58, 104111. [CrossRef]
32.
Kim, D.-I.; Kim, G.; Kim, G.-H.; Chang, K.S. Responsivity and Noise Evaluation of Infrared Thermal Imaging Camera. J. Korean
Soc. Nondestruct. Test. 2013,33, 342–348. [CrossRef]
33.
Da-xing, S.M.-M.P. Evaluation of Performance of Infrared Systems Using Noise Equivalent Temperature Difference. Infrared
2010
,
31, 22–25.
34.
Li, Y.; Pan, D.; Yang, C.; Luo, Y. NETD test of high-sensitivity infrared camera. In Proceedings of the 3rd International Symposium
on Advanced Optical Manufacturing and Testing Technologies: Optical Test and Measurement Technology and Equipment,
Chengdu, China, 8–12 July 2007; Volume 6723, pp. 836–840.
35.
Villa, E.; Arteaga-Marrero, N.; Ruiz-Alzola, J. Performance Assessment of Low-Cost Thermal Cameras for Medical Applications.
Sensors 2020,20, 1321. [CrossRef]
36.
Pan, C.Y.; Huang, C.S.; Horng, G.J.; Peng, P.L.; Jong, G.J. Infrared Image Processing for a Physiological Information Telemetry
System. Wirel. Pers. Commun. 2015,83, 3181–3208. [CrossRef]
37.
Rao, K.; Coviello, G.; Feng, M.; Debnath, B.; Hsiung, W.-P.; Sankaradas, M.; Yang, Y.; Po, O.; Drolia, U.; Chakradhar, S. F3S: Free
Flow Fever Screening. In Proceeding of the 7th IEEE International Conference on Smart Computing, Irvine, CA, USA, 23–27
August 2021; pp. 276–285.
38. HBudzier, H.; Gerlach, G. Calibration of uncooled thermal infrared cameras. J. Sens. Sens. Syst. 2015,4, 187–197. [CrossRef]
39.
König, S.; Gutschwager, B.; Taubert, R.D.; Hollandt, J. Metrological characterization and calibration of thermographic cameras for
quantitative temperature measurement. J. Sens. Sens. Syst. 2020,9, 425–442. [CrossRef]
40.
Scebba, G.; Da Poian, G.; Karlen, W. Multispectral Video Fusion for Non-Contact Monitoring of Respiratory Rate and Apnea.
IEEE Trans. Biomed. Eng. 2021,68, 350–359. [CrossRef] [PubMed]
41.
Lewicki, T.; Liu, K. AI thermometer for temperature screening: Demo abstract. In Proceedings of the 18th Conference on
Embedded Networked Sensor Systems, Yokohama, Japan, 16–19 November 2020; pp. 597–598.
42.
Negishi, T.; Abe, S.; Matsui, T.; Liu, H.; Kurosawa, M.; Kirimoto, T.; Sun, G. Contactless Vital Signs Measurement System Using
RGB-Thermal Image Sensors and Its Clinical Screening Test on Patients with Seasonal Influenza. Sensors
2020
,20, 2171. [CrossRef]
43.
Jagadev, P.; Giri, L.I. Non-contact monitoring of human respiration using infrared thermography and machine learning. Infrared
Phys. Technol. 2020,104, 103117. [CrossRef]
44.
Mutlu, K.; Rabell, J.E.; del Olmo, P.M.; Haesler, S. IR thermography-based monitoring of respiration phase without image
segmentation. J. Neurosci. Methods 2018,301, 1–8. [CrossRef] [PubMed]
45.
Jagadev, P.; Giri, L.I. Human respiration monitoring using infrared thermography and artificial intelligence. Biomed. Phys. Eng.
Express 2020,6, 35007. [CrossRef]
46.
Goldman, L.J. Nasal airflow and thoracoabdominal motion in children using infrared thermographic video processing. Pediatr.
Pulmonol. 2012,47, 476–486. [CrossRef]
47.
Negishi, T.; Sun, G.; Liu, H.; Sato, S.; Matsui, T.; Kirimoto, T. Stable Contactless Sensing of Vital Signs Using RGB-Thermal Image
Fusion System with Facial Tracking for Infection Screening. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2018,2018, 4371–4374.
48.
Negishi, T.; Sun, G.; Sato, S.; Liu, H.; Matsui, T.; Abe, S.; Nishimura, H.; Kirimoto, T. Infection screening system using
thermography and CCD camera with good stability and swiftness for non-contact vital-signs measurement by feature matching
and MUSIC algorithm. In Proceedings of the 41st Annual International Conference of the IEEE Engineering in Medicine and
Biology Society (EMBC), Berlin, Germany, 23–27 July 2019.
Sensors 2021,21, 7777 20 of 21
49.
Kim, Y.; Park, Y.; Kim, J.; Lee, E.C. Remote heart rate monitoring method using infrared thermal camera. Int. J. Eng. Res. Technol.
2018,11, 493–500.
50.
Pereira, C.B.; Yu, X.; Goos, T.; Reiss, I.; Orlikowsky, T.; Heimann, K.; Venema, B.; Blazek, V.; Leonhardt, S.; Teichmann, D.
Noncontact Monitoring of Respiratory Rate in Newborn Infants Using Thermal Imaging. IEEE Trans. Biomed. Eng.
2018
,66,
1105–1114. [CrossRef]
51.
Hu, M.-H.; Zhai, G.-T.; Li, D.; Fan, Y.-Z.; Chen, X.-H.; Yang, X.-K. Synergetic use of thermal and visible imaging techniques for
contactless and unobtrusive breathing measurement. J. Biomed. Opt. 2017,22, 1. [CrossRef]
52.
Chen, L.; Hu, M.; Liu, N.; Zhai, G.; Yang, S.X. Collaborative use of RGB and thermal imaging for remote breathing rate
measurement under realistic conditions. Infrared Phys. Technol. 2020,111, 103504. [CrossRef]
53.
Hu, M.; Zhai, G.; Li, D.; Fan, Y.; Duan, H.; Zhu, W.; Yang, X. Combination of near-infrared and thermal imaging techniques for
the remote and simultaneous measurements of breathing and heart rates under sleep situation. PLoS ONE
2018
,13, e0190466.
[CrossRef] [PubMed]
54.
Jakkaew, P.; Onoye, T. Non-contact respiration monitoring and body movements detection for sleep using thermal imaging.
Sensors 2020,20, 6307. [CrossRef]
55.
Boyko, N.; Basystiuk, O.; Shakhovska, N. Performance Evaluation and Comparison of Software for Face Recognition, Based on
Dlib and Opencv Library. In Proceedings of the 2018 IEEE Second International Conference on Data Stream Mining & Processing
(DSMP), Lviv, Ukraine, 21–25 August 2018; pp. 478–482.
56. King, D.E. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 2009,10, 1755–1758.
57. Weisstein, E.W. Affine Transformation. Available online: https://mathworld.wolfram.com/ (accessed on 25 July 2021).
58.
Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011
International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 2564–2571.
59.
Abdulmajeed, M.; Seyfi, L. Object recognition system based on oriented FAST and rotated BRIEF. In Proceedings of the 2nd
International Symposium on Innovative Approaches in Scientific Studies, Konya, Turkey, 30 November–2 December 2018.
60.
Viola, P.; Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, Kauai, HI, USA, 8–14 December 2001.
61.
Bennett, S.L.; Goubran, R.; Knoefel, F. Comparison of motion-based analysis to thermal-based analysis of thermal video in the
extraction of respiration patterns. In Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society (EMBC), Jeju, Korea, 11–15 July 2017; pp. 3835–3839.
62.
Scholkmann, F.; Boss, J.; Wolf, M. An Efficient Algorithm for Automatic Peak Detection in Noisy Periodic and Quasi-Periodic
Signals. Algorithms 2012,5, 588–603. [CrossRef]
63.
Palshikar, G. Simple algorithms for peak detection in time-series. In Proceedings of the 1st Int. Conf. Advanced Data Analysis,
Business Analytics and Intelligence, Ahmedabad, India, 6–7 June 2009; Volume 122.
64.
Sandya, H.B.; Hemanth, K.P.; Himanshi, P. Fuzzy rule based feature extraction and classification of time series signal. Int. J. Soft
Comput. Eng. 2013,3, 2231–2307.
65.
Huang, S.; Tang, J.; Dai, J.; Wang, Y. Signal Status Recognition Based on 1DCNN and Its Feature Extraction Mechanism Analysis.
Sensors 2019,19, 2018. [CrossRef]
66.
Laguna, P.; Moody, G.B.; Mark, R.G. Power spectral density of unevenly sampled data by least-square analysis: Performance and
application to heart rate signals. IEEE Trans. Biomed. Eng. 1998,45, 698–715. [CrossRef]
67.
Barandas, M.; Folgado, D.; Fernandes, L.; Santos, S.; Abreu, M.; Bota, P.; Liu, H.; Schultz, T.; Gamboa, H. TSFEL: Time Series
Feature Extraction Library. SoftwareX 2020,11, 100456. [CrossRef]
68.
Liu, H.; Allen, J.; Zheng, D.; Chen, F. Recent development of respiratory rate measurement technologies. Physiol. Meas.
2019
,40,
07TR01. [CrossRef] [PubMed]
69. Carol, K. Respiratory rate 1: Why measurement and recording are crucial. Nurs. Times 2018,114, 23–24.
70.
Takamoto, H.; Nishine, H.; Sato, S.; Sun, G.; Watanabe, S.; Seokjin, K.; Asai, M.; Mineshita, M.; Matsui, T. Development and
Clinical Application of a Novel Non-contact Early Airflow Limitation Screening System Using an Infrared Time-of-Flight Depth
Image Sensor. Front. Physiol. 2020,11, 552942. [CrossRef]
71. Flenady, T.; Dwyer, T.; Applegarth, J. Accurate respiratory rates count: So should you! Australas. Emerg. Nurs. J. 2017,20, 45–47.
[CrossRef]
72.
Sartini, C.; Tresoldi, M.; Scarpellini, P.; Tettamanti, A.; Carcò, F.; Landoni, G.; Zangrillo, A. Respiratory Parameters in Patients
With COVID-19 After Using Noninvasive Ventilation in the Prone Position Outside the Intensive Care Unit. JAMA
2020
,323,
2338–2340. [CrossRef]
73.
Total Hospital Bed Occupancy (COVID-19)|SCDHEC. Available online: https://scdhec.gov/covid19/hospital-bed-capacity-
covid-19 (accessed on 25 July 2021).
74.
Storck, K.; Karlsson, M.; Ask, P.; Loyd, D. Heat transfer evaluation of the nasal thermistor technique. IEEE Trans. Biomed. Eng.
1996,43, 1187–1191. [CrossRef]
75.
Hunsaker, D.H.; Riffenburgh, R.H. Snoring Significance in Patients Undergoing Home Sleep Studies. Otolaryngol. Neck Surg.
2006,134, 756–760. [CrossRef]
76.
Akbarian, S.; Ghahjaverestan, N.M.; Yadollahi, A.; Taati, B. Distinguishing Obstructive Versus Central Apneas in Infrared Video
of Sleep Using Deep Learning: Validation Study. J. Med. Internet Res. 2020,22, e17252. [CrossRef] [PubMed]
Sensors 2021,21, 7777 21 of 21
77.
Wang, C.W.; Hunter, A.; Gravill, N.; Matusiewicz, S. Unconstrained video monitoring of breathing behavior and application to
diagnosis of sleep apnea. IEEE Trans. Biomed. Eng. 2014,61, 396–404. [CrossRef] [PubMed]
78.
Gault, T.; Farag, A. A fully automatic method to extract the heart rate from thermal video. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition Workshops, Louisville, KY, USA, 23–28 June 2013; pp. 336–341.
79.
Bennett, S.L.; Goubran, R.; Knoefel, F. Adaptive eulerian video magnification methods to extract heart rate from thermal video. In
Proceedings of the 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Benevento, Italy,
15–18 May 2016; pp. 1–5.
80.
Li, F.; Zhao, Y.; Kong, L.; Dong, L.; Liu, M.; Hui, M.; Liu, X. A camera-based ballistocardiogram heart rate measurement method.
Rev. Sci. Instrum. 2020,91, 054105. [CrossRef]
81.
Balakrishnan, G.; Durand, F.; Guttag, J. Detecting Pulse from Head Motions in Video. In Proceedings of the 2013 IEEE Conference
on Computer Vision and Pattern Recognition, Portland, OR, USA, 23–28 June 2013; pp. 3430–3437.
82.
Netinant, P.; Vasprasert, P.; Rukhiran, M. Evaluations of Effective on LWIR Micro Thermal Camera IoT and Digital Thermometer
for Human Body Temperatures. In Proceedings of the 2021 The 5th International Conference on E-Commerce, E-Business and
E-Government, New York, NY, USA, 28 April 2021; pp. 20–24.
83.
Ring, E.F.J.; McEvoy, H.; Jung, A.; Zuber, J.; Machin, G. New standards for devices used for the measurement of human body
temperature. J. Med. Eng. Technol. 2010,34, 249–253. [CrossRef]
84. Mercer, J.B.; Ring, E.F.J. Fever screening and infrared thermal imaging: Concerns and guidelines. Thermol. Int. 2009,19, 67–69.
85.
Kim, N.W.; Zhang, H.Y.; Yoo, J.-H.; Park, Y.S.; Song, H.J.; Yang, K.H. The Correlation Between Tympanic Membrane Temperature
and Specific Region of Face Temperature. Quant. InfraRed Thermogr. Asia 2017,16, 1–7.
86.
Yeoh, W.K.; Lee, J.K.W.; Lim, H.Y.; Gan, C.W.; Liang, W.; Tan, K.K. Re-visiting the tympanic membrane vicinity as core body
temperature measurement site. PLoS ONE 2017,12, e0174120. [CrossRef] [PubMed]
87.
Dell’Isola, G.B.; Cosentini, E.; Canale, L.; Ficco, G.; Dell’Isola, M. Noncontact Body Temperature Measurement: Uncertainty
Evaluation and Screening Decision Rule to Prevent the Spread of COVID-19. Sensors 2021,21, 346. [CrossRef]
88.
Str ˛akowska, M.; Str ˛akowski, R. Automatic eye corners detection and tracking algorithm in sequence of thermal medical images.
Meas. Autom. Monit. 2015,61, 199–202.
89.
Chen, H.-Y.; Chen, A.; Chen, C. Investigation of the Impact of Infrared Sensors on Core Body Temperature Monitoring by
Comparing Measurement Sites. Sensors 2020,20, 2885. [CrossRef]
90.
Nhan, B.R.; Chau, T. Classifying Affective States Using Thermal Infrared Imaging of the Human Face. IEEE Trans. Biomed. Eng.
2010,57, 979–987. [CrossRef] [PubMed]
91.
Merla, A.; Mattei, P.A.; Di Donato, L.; Romani, G.L. Thermal Imaging of Cutaneous Temperature Modifications in Runners
During Graded Exercise. Ann. Biomed. Eng. 2009,38, 158–163. [CrossRef] [PubMed]
92.
Tejedor, B.; Casals, M.; Gangolells, M.; Macarulla, M.; Forcada, N. Human comfort modelling for elderly people by infrared
thermography: Evaluating the thermoregulation system responses in an indoor environment during winter. Build. Environ.
2020
,
186, 107354. [CrossRef]
93. Pavlidis, I.T. Lie detection using thermal imaging. Def. Secur. 2004,XXVI, 270–279.
94.
Warmelink, L.; Vrij, A.; Mann, S.; Leal, S.; Forrester, D.; Fisher, R.P. Thermal imaging as a lie detection tool at airports. Law Hum.
Behav. 2011,35, 40–48. [CrossRef]
95.
VEngert, V.; Merla, A.; Grant, J.; Cardone, D.; Tusche, A.; Singer, T. Exploring the Use of Thermal Infrared Imaging in Human
Stress Research. PLoS ONE 2014,9, e90782.
96.
Koukiou, G.; Anastassopoulos, V. Neural networks for identifying drunk persons using thermal infrared imagery. Forensic Sci.
Int. 2015,252, 69–76. [CrossRef]
97. Gade, R.; Moeslund, T.B. Thermal cameras and applications: A survey. Mach. Vis. Appl. 2014,25, 245–262. [CrossRef]
98.
Zhao, F.; Li, M.; Tsien, J.Z. Technology platforms for remote monitoring of vital signs in the new era of telemedicine. Expert Rev.
Med. Devices 2015,12, 411–429. [CrossRef] [PubMed]
99.
Weber, M.; Hiete, M.; Lauer, L.; Rentz, O. Low cost country sourcing and its effects on the total cost of ownership structure for a
medical devices manufacturer. J. Purch. Supply Manag. 2010,16, 4–16. [CrossRef]
100.
Balsam, J.; Ossandon, M.; Bruck, H.A.; Lubensky, I.; Rasooly, A. Low-cost technologies for medical diagnostics in low-resource
settings. Expert Opin. Med. Diagn. 2013,7, 243–255. [CrossRef] [PubMed]
101.
YChoi, Y.; Kim, N.; Hwang, S.; Kweon, I.S. Thermal Image Enhancement using Convolutional Neural Network. In Proceedings of
the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, 9–14 October 2016; pp.
223–230.
102.
Welch, J.; Kanter, B.; Skora, B.; McCombie, S.; Henry, I.; McCombie, D.; Kennedy, R.; Soller, B. Multi-parameter vital sign database
to assist in alarm optimization for general care units. J. Clin. Monit. 2015,30, 895–900. [CrossRef] [PubMed]
... The thermal camera has a high impact in enhancing non-invasive data acquisition [147]. Yoshikawa et al. developed a non-invasive thermal data acquisition system for the human body. ...
... For FIR imaging (thermal imaging), authors developed algorithms to estimate the body temperature [147,148], but there are some limitations that have to be addressed. One of these limitations is the estimation of the BT without reference point in order to increase the usability. ...
Article
Full-text available
The rapid development of technology has brought about a revolution in healthcare stimulating a wide range of smart and autonomous applications in homes, clinics, surgeries and hospitals. Smart healthcare opens the opportunity for a qualitative advance in the relations between healthcare providers and end-users for the provision of healthcare such as enabling doctors to diagnose remotely while optimizing the accuracy of the diagnosis and maximizing the benefits of treatment by enabling close patient monitoring. This paper presents a comprehensive review of non-invasive vital data acquisition and the Internet of Things in healthcare informatics and thus reports the challenges in healthcare informatics and suggests future work that would lead to solutions to address the open challenges in IoT and non-invasive vital data acquisition. In particular, the conducted review has revealed that there has been a daunting challenge in the development of multi-frequency vital IoT systems, and addressing this issue will help enable the vital IoT node to be reachable by the broker in multiple area ranges. Furthermore, the utilization of multi-camera systems has proven its high potential to increase the accuracy of vital data acquisition, but the implementation of such systems has not been fully developed with unfilled gaps to be bridged. Moreover, the application of deep learning to the real-time analysis of vital data on the node/edge side will enable optimal, instant offline decision making. Finally, the synergistic integration of reliable power management and energy harvesting systems into non-invasive data acquisition has been omitted so far, and the successful implementation of such systems will lead to a smart, robust, sustainable and self-powered healthcare system.
Article
Full-text available
Lameness has a high economic cost to the U.K. dairy industry; accurate and early detection of lameness minimises this cost. Infrared thermal imaging (IRT) devices have shown promising results for use as a lameness detection aid in cattle when used in research settings; these devices are typically high-cost, limiting their adoption. This study analysed the effectiveness of low-cost IRT devices (LCDs) as lameness detection aids, by comparing both maximum environmentally adjusted temperature values and hindfeet temperature difference collected by an LCD to the mobility score of the cow; this test was repeated for data collected by a research-specification device. Data collection occurred during routine milking of 83 cattle; each cow’s mobility was scored afterwards. Significant differences were found between lame and sound cows with the LCD, upon analysis of both methods. There was no significant difference between the data captured by differing devices. The maximum sensitivity and specificity values for the LCD were calculated as 66.95 and 64.53, respectively, compared with 70.34 and 70.94, respectively, for the research-specification device; optimum threshold values for these were equivalent for both devices, suggesting IRT lameness identification is not device-dependent. It was concluded that a minimal difference in effectiveness between tested devices suggests that LCDs could be used as a lameness detection aid; consequently, there is potential for widespread adoption as on-farm detection aids.
Article
Full-text available
The assessment of human internal/core temperature (T core) is relevant in many scientific disciplines , but also for public health authorities when attempting to identify individuals with fever. Direct assessment of T core is often invasive, impractical on a large scale, and typically requires close contact between the observer and the target subject. Non-contact infrared thermometry (NCIT) represents a practical solution in which T core can potentially be assessed from a safe distance and in mass screening scenarios, by measuring skin temperature at specific anatomical locations. However, the COVID-19 pandemic has clearly demonstrated that these devices are not being used correctly, despite expert guided specifications available in International Standard Organization (ISO) documents. In this review, we provide an overview of the most pertinent factors that should be considered by users of NCIT. This includes the most pertinent methodolo-gical and physiological factors, as well as an overview on the ability of NCIT to track human T core. For practical use, we provide a checklist based on relevant ISO standards which are simple to follow and should be consulted prior to using NCIT for assessment of human T core. Our intention is for users of NCIT to adopt this checklist, which may improve the performance of NCIT for its ability to track T core. ARTICLE HISTORY
Article
Full-text available
Infrared imaging-based machine vision (IRMV) is the technology used to automatically inspect, detect, and analyse infrared images (or videos) obtained by recording the intensity of infrared light emitted or reflected by observed objects. Depending on whether controllable excitation is used during the imaging of infrared rays, thermal IRMV can be categorised into passive thermography and active thermography. Passive thermography is an important supplement to conventional machine vision based on visible light and is a valid imaging tool for self-heating objects such as the human body and electrical power devices. Active thermography is a non-destructive testing method for the quality evaluation and safety assurance of non-self-heating objects. In active thermography, the trend is to inspect rapidly, reliably, and intelligently by introducing multiple-mode excitation sources and artificial intelligence. The rapid development of deep learning makes IRMV more and more intelligent and highly automated, thus considerably increasing its range of applications. This paper reviews the principle, cameras, and thermal data of IRMV and discusses the applications of deep learning applied to IRMV. Case studies of IRMV and deep learning on various platforms such as unmanned vehicles, mobile phones and embedded systems are also reported.
Article
Full-text available
Infrared thermography for camera-based skin temperature measurement is increasingly used in medical practice, e.g., to detect fevers and infections, such as recently in the COVID-19 pandemic. This contactless method is a promising technology to continuously monitor the vital signs of patients in clinical environments. In this study, we investigated both skin temperature trend measurement and the extraction of respiration-related chest movements to determine the respiratory rate using low-cost hardware in combination with advanced algorithms. In addition, the frequency of medical examinations or visits to the patients was extracted. We implemented a deep learning-based algorithm for real-time vital sign extraction from thermography images. A clinical trial was conducted to record data from patients on an intensive care unit. The YOLOv4-Tiny object detector was applied to extract image regions containing vital signs (head and chest). The infrared frames were manually labeled for evaluation. Validation was performed on a hold-out test dataset of 6 patients and revealed good detector performance (0.75 intersection over union, 0.94 mean average precision). An optical flow algorithm was used to extract the respiratory rate from the chest region. The results show a mean absolute error of 2.69 bpm. We observed a computational performance of 47 fps on an NVIDIA Jetson Xavier NX module for YOLOv4-Tiny, which proves real-time capability on an embedded GPU system. In conclusion, the proposed method can perform real-time vital sign extraction on a low-cost system-on-module and may thus be a useful method for future contactless vital sign measurements.
Article
Full-text available
The need to measure body temperature contactless and quickly during the COVID-19 pandemic emergency has led to the widespread use of infrared thermometers, thermal imaging cameras and thermal scanners as an alternative to the traditional contact clinical thermometers. However, limits and issues of noncontact temperature measurement devices are not well known and technical–scientific literature itself sometimes provides conflicting reference values on the body and skin temperature of healthy subjects. To limit the risk of contagion, national authorities have set the obligation to measure body temperature of workers at the entrance to the workplace. In this paper, the authors analyze noncontact body temperature measurement issues from both clinical and metrological points of view with the aim to (i) improve body temperature measurements accuracy; (ii) estimate the uncertainty of body temperature measurement on the field; (iii) propose a screening decision rule for the prevention of the spread of COVID-19. The approach adopted in this paper takes into account both the traditional instrumental uncertainty sources and clinical–medical ones related to the subjectivity of the measurand. A proper screening protocol for body temperature measurement considering the role of uncertainty is essential to correctly choose the threshold temperature value and measurement method to access critical places during COVID-19 pandemic emergency.
Article
Full-text available
Sleep quality is an important determinant of human health and wellbeing. Novel technologies that can quantify sleep quality at scale are required to enable the diagnosis and epidemiology of poor sleep. One important indicator of sleep quality is body posture. In this paper, we present the design and implementation of a non-contact sleep monitoring system that analyses body posture and movement. Supervised machine learning strategies applied to noncontact vision-based infrared camera data using a transfer learning approach, successfully quantified sleep poses of participants covered by a blanket. This represents the first occasion that such a machine learning approach has been used to successfully detect four predefined poses and the empty bed state during 8-10 hour overnight sleep episodes representing a realistic domestic sleep situation. The methodology was evaluated against manually scored sleep poses and poses estimated using clinical polysomnography measurement technology. In a cohort of 12 healthy participants, we find that a ResNet-152 pre-trained network achieved the best performance compared with the standard de novo CNN network and other pre-trained networks. The performance of our approach was better than other video-based methods for sleep pose estimation and produced higher performance compared to the clinical standard for pose estimation using a polysomnography position sensor. It can be concluded that infrared video capture coupled with deep learning AI can be successfully used to quantify sleep poses as well as the transitions between poses in realistic nocturnal conditions, and that this non-contact approach provides superior pose estimation compared to currently accepted clinical methods.
Article
Full-text available
We present the metrological characterization and calibration of three different types of thermographic cameras for quantitative temperature measurement traceable to the International Temperature Scale (ITS-90). Relevant technical specifications – i.e., the non-uniformity of the pixel-to-pixel responsivity, the inhomogeneity equivalent temperature difference (IETD), the noise equivalent temperature difference (NETD), and the size-of-source effect (SSE) – are determined according to the requirements given in the series of Technical Directives VDI/VDE 5585. The measurements are performed with the camera calibration facility of the Physikalisch-Technische Bundesanstalt. The data reference method is applied for the determination and improvement of the non-uniformity, leading to an improved IETD for all three cameras. Finally, the cameras are calibrated according to the different procedures discussed in the VDI/VDE 5585 series. Results achieved with the different calibration procedures are compared for each type of camera and among the three cameras. An uncertainty budget for the calibration of each camera is given according to GUM (ISO, 1995) and VDI/VDE 5585.
Article
This study examines the opportunities and challenges involved with contactless healthcare services in the post-COVID-19 pandemic era. First, we reviewed the literature to analyze contactless or contact-free healthcare services that have been utilized in pre-and during the COVID-19 pandemic periods. Then, we interviewed medical experts and hospital administrators to gain knowledge about how healthcare providers are currently working to mitigate the spread of COVID and preparing for the post-pandemic period. Thus, we analyzed the evolution and utilization of contactless services during the three different time periods: pre-, during-, and post-COVID-19. The results indicated that in the post-COVID-19 era, a new normal of hybrid healthcare services would emerge. While some of the contactless services that have been practiced during the pandemic may revert to the traditional face-to-face services, those innovative contactless healthcare services that have been proven effective during the pandemic would be practiced or even advanced in the post-pandemic period due to the accelerating technological developments. This study suggests many potential opportunities and daunting challenges for healthcare institutions, policymakers, and consumers regarding the implementation of contactless services in the post-COVID-19 era.
Article
Recent studies in computer vision show that, due to inhalation and exhalation, pixel variations can be captured by RGB-thermal videos to estimate the breathing rate (BR). In the last few years, considerable progress has been made to BR contactless monitoring, however, still many issues remain open. The performances of those methods significantly degrade under challenging conditions, specifically when subjects’ spontaneous movements, facial expressions and illumination variations are involved. In this paper, we propose a dynamic time warping-based optimization framework to automatically and accurately select regions that are most useful for robust BR estimation irrespective of these interfering factors. A signal quality index called Respiratory Signals Quality Index based on dynamic time warping (RSQI_dtw) is empirically developed. In order to address the undetected problems of facial areas due to large movements, two different methods based on face tracking or motion detection by RGB-thermal images are used to acquire the respiratory signals, and BRs are measured by the re-concatenated signal. In our framework, a time-domain processing procedure is further proposed, which contains de-trend by recursive least squares (RLS) algorithm, normalization and de-noise by band-pass filter for extracting mainstream signals. The results of validation experiments conducted with 36 subjects suggest that the proposed approach outperforms state-of-the-art BR estimation methods significantly under real-world conditions. Our study can be utilized to assist medical staff in diagnosis and treatment by remote and accurate respiratory rate detection and reduce close contact between medical staff and patients to a certain extent.