A Multimodal Perception System for Detection of Human Operators in Robotic Work Cells

Abstract

Workspace monitoring is a critical hw/sw component of modern industrial work cells or in service robotics scenarios, where human operators share their workspace with robots. Reliability of human detection is a major requirement not only for safety purposes but also to avoid unnecessary robot stops or slowdowns in case of false positives. The present paper introduces a novel multimodal perception system for human tracking in shared workspaces based on the fusion of depth and thermal images. A machine learning approach is pursued to achieve reliable detection performance in multi-robot collaborative systems. Robust experimental results are finally demonstrated on a real robotic work cell.

Full text

A Multimodal Perception System for Detection of
Human Operators in Robotic Work Cells*
Marco Costanzo, Giuseppe De Maria, Gaetano Lettera, Ciro Natale and Dario Perrone
Abstract— Workspace monitoring is a critical hw/sw com-
ponent of modern industrial work cells or in service robotics
scenarios, where human operators share their workspace with
robots. Reliability of human detection is a major requirement
not only for safety purposes but also to avoid unnecessary
robot stops or slowdowns in case of false positives. The present
paper introduces a novel multimodal perception system for
human tracking in shared workspaces based on the fusion of
depth and thermal images. A machine learning approach is
pursued to achieve reliable detection performance in multi-
robot collaborative systems. Robust experimental results are
finally demonstrated on a real robotic work cell.
I. INTRODUCTION
The paper proposes a sensor fusion strategy which com-
bines depth and thermal images to robustly detect human
operators, e.g., in industrial work cells or in professional
service robotics scenarios, and realize safe human-robot
collaboration (HRC) tasks. The main focus of the current
safety regulations is operators safety during industrial robotic
operations. The safety standards for these applications are
laid out by the International Organization for Standardization
(ISO) 10218-1 [1], 10218-2 [2] and by the upcoming ISO
Proposed Draft Technical Specification (TS) 15066 [3]. Four
types of collaborative scenarios are identified, which are
addressed in post-collision and pre-collision scenarios [4].
Industrial safety requirements do not permit to have the
use of post-collision systems because the physical impact
between the robot and the human operator occurs before the
complete stop of the machinery. Otherwise, a pre-collision
scheme makes use of appropriate exteroceptive sensors to
detect humans and prevent collisions. A Speed and Sepa-
ration Monitoring (SSM) scenario requires that the robot
speed should be monitored according to the robot separation
distance from the human operator. In this paper the SSM
scenario has been selected with the aim to maximize the
production time in industrial work cells or in any professional
service task, and preserve human safety at the same time.
However, it combines two uncertain worlds: the distance
monitoring, which requires an accurate and robust human
*This work was supported by the European Commission under the
H2020 Framework Cleansky 2 JTI – project LABOR, ID n. 785419
https://www.labor-project.eu.
M. Costanzo, G. De Maria, G. Lettera, C. Natale and
D. Perrone are with Dipartimento di Ingegneria, Univer-
sit`
a degli Studi della Campania Luigi Vanvitelli, 81031
Aversa, Italy marco.costanzo@unicampania.it,
giuseppe.demaria@unicampania.it,
gaetano.lettera@unicampania.it,
ciro.natale@unicampania.it,
dario.perrone1@studenti.unicampania.it
c
2019 IEEE. DOI: https://doi.org/10.1109/SMC.2019.8914519
detection algorithm, and the robot speed monitoring, which
should be reactive and efficient.
Distance monitoring can be solved through motion cap-
ture systems, range sensors or artificial vision systems [5].
Nevertheless, localizing human operators robustly is not an
easy task. It is often necessary to fuse several sensors with
different properties. Originally developed for military appli-
cations, thermal cameras provide relevant data to perform
breast cancer diagnostic [6], infrastructure and electrical
systems monitoring, gas or liquid detection [7], inspection
and control tasks in industrial applications [8]. Thermal
cameras are ideal for finding objects of a certain temper-
ature: human detection and tracking (HDT) well fits for this
case, as the body temperature is about 37◦C. Unfortunately,
thermal cameras do not support depth information, which
are necessary to correctly compute the separation distance
between the operator and the robot and apply the current
regulations. The main difficulty of fusing spatial and thermal
images is that a correspondence between corresponding
pixels needs to be found. Similar sensor fusion approaches
for indoor human detection combine RGB data with depth
information [9], using the Histogram of Oriented Gradients
(HOG) proposed in [10] together with depth feature that
describes the self-similarity of an image. Different strategies
are based on Convolutional Neural Networks (CNN), widely
used for object recognition [11] and human detection [12].
A CNN-based RGB-D human detector exploiting the depth
information to develop a region of interest selection method
(ROI) is proposed in [13]. However, the fusion of thermal
and spatial information has gained attention in the last few
years, especially in fields where the spatial data are used as
the main source of information [14], but nowadays there are
no standardized methods to robustly combine them.
The State-of-Art (SoA) publications propose some meth-
ods to deal with the HRC safety: a volumetric representation
of the areas occupied by operators and by the robot has been
studied in [15] to stop the robot when these areas overlap;
as well as, [16] proposes a potential field method to be
used to generate a collision free path. A further approach is
presented in [17] where a safety index is modeled to modify
the robot trajectories and preserve the cooperative task. Many
of these approaches rely on evasive actions to increase safety.
However, in industrial setting, it is generally recommended
to follow the robot predefined path without deviations.
This research paper tackles the HRC problem by intro-
ducing a novel approach to robustly detect human operators
in collaborative work cells through a multimodal perception
system aimed at minimizing false positives to avoid unnec-

essary robot stops. The paper guarantees human safety in
general multi-robot scenarios. In fact, the algorithm allows
computing the minimum separation distance between every
human operator and every robot within the collaborative
work cell, following the line of the current regulations. The
applicability of the approach in the manufacturing industry
has been obtained not by modifying the robot predefined
path but by scaling down the robot trajectory only when
indispensable, thus trying to maximize the production time
even in presence of humans.
II. HUMAN-ROBOT SEPARATION DISTANCE
This section proposes a novel point-cloud based method-
ology to compute the separation distance between a moving
robot and the closest human operator. Strong emphasis has
been devoted to the proposed HDT strategy which realizes
a robust algorithm to correctly detect human operators into
a HRC scenario in real-time. According to the current
ISOs, the robot speed must be modulated to eventually slow
down the robot pre-programmed trajectory when a dangerous
situation for the human operator occurs.
A. Experimental setup and camera calibration
Nowadays, thermal imaging provides relevant data to per-
form specific robotic applications that require thermal infor-
mation. When combining two or more sources of acquisition,
the resulting multi-sensor system has to be extrinsically
calibrated to find the relative pose between the adopted
sensors. This step can be performed by using a calibration
target. The section explains two developed methods for
camera calibration. The objective is to obtain the pose of
the two cameras as accurate as possible both for the success
of the next merging step and for the final separation distance
computation. Two steps have been necessary.
The first one reliably localizes the pose of the depth
camera with respect to the robot base frame. In literature,
this problem is solved by different calibration procedures,
especially for object recognition applications. Their typical
target is to recognize objects located at about 0.5m from the
camera frame. On the contrary, for the proposed experimental
application, the robot and the operators work about 2.5m far
from the camera. To obtain the desired accuracy, the novelty
of the depth camera extrinsic calibration procedure consists
in exploiting a sphere tracking as detailed below.
The experimental setup of this work is shown Fig. 1 and
consists of two cameras rigidly attached to each other. They
have been arranged in a way that their optical axes are
aligned. The adopted cameras have different field of views
(FOVs) and this implies that some depth pixels (Microsoft
Kinect v1, Focal length: 6.1 mm, FOV: 57x45, image size:
640x480) are outside the thermal image (Optris PI 450, Focal
length: 15 mm, FOV: 38x29, Spectrum: 7.5 to 13 m, image
size: 382x288) and they are not used in the merging step
(see Section II-C.1). The goal of the extrinsic calibration
is to obtain an accurate identification of the camera poses,
which guarantee the minimum accuracy error when the two
camera views are merged.
Fig. 1. The experimental perception system composed of a depth camera
(Microsoft Kinect v1) and a thermal camera (Optris PI 450).
A 3D tracking technique has been developed to calibrate
the depth sensor, by tracking a polystyrene sphere of 0.12m
diameter. The red sphere has been mounted at the robot
end effector, so as to match the center of the sphere with
the end effector frame origin. The developed procedure uses
the M-estimator SAmple Consensus (MSAC) algorithm [18]
(which is an extension of the best known RANdom SAmple
Consensus (RANSAC) algorithm [19]), to find a sphere
which satisfies a radius constraint and provides its geometric
model. The robot has been positioned at specific configura-
tions, which allow to correctly distinguish the target within
the camera view. From the robot joint states, the forward
kinematics immediately computes the position of the red
sphere. At the same time, the developed procedure acquires
the depth image, converts it into point-cloud data [20] (PCD)
and estimates the target model. The method is iterated to try
to cover the entire collaborative workspace and to minimize
the calibration error. Finally, the transformation matrix T
T
Tr
d,
between the robot frame Σrand the depth camera frame Σd,
has been evaluated through an optimization algorithm of a
cost function, which combines the corresponding data.
After the depth camera calibration, the extrinsic calibration
of the thermal camera with respect to the depth camera
has been solved. In [21] and [22] a thermal camera and
a depth camera are calibrated by using a perforated grid
placed in front of the sensors. The procedures assume that the
target is located close enough to the sensors lenses because
the objective is to practically solve the calibration but the
approach revealed unsuitable for the application scenario at
hand. The solution consists in using three spheres attached
to a flat cardboard support and heated to be distinguishable
by both the depth and the thermal cameras.
To obtain an estimation of the transformation matrix
T
T
Td
t, between the depth camera frame Σdand the thermal
camera frame Σt, the spheres have been moved inside the
collaborative workspace by placing the support in 10 config-
urations at distances from the camera in the range where the
human operator is expected to act during the collaborative
task. At every acquisition, the calibration target has been
suitably heated to be detectable from both cameras. The
coordinates p
p
pd
k=xd
kyd
kzd
kTof the kth center of the

target sphere have been directly calculated from the depth
image, while the corresponding thermal point coordinates
have been calculated from the thermal image, assuming the
distance from the lens equal to the depth value, i.e., zt
k=zd
k
and
xt
k=(ak−cxt)zt
k
fxt
(1)
yt
k=(bk−cyt)zt
k
fyt
,(2)
where akand bkare the pixel coordinates of the sphere center
in the thermal image, cxt,cytare the pixel coordinates of
the thermal image center and fxt,fytare the focal lengths
expressed in pixel-related units. Finally, the transformation
matrix T
T
Td
thas been estimated by minimizing a cost function
that combines the corresponding data.
Note that, the intrinsic calibrations have been performed
by using common patterns, i.e., a chessboard pattern for
the depth camera and an heated circular pattern grid for
the thermal camera. These procedures return the intrinsic
calibration matrices of the cameras, thus, the parameters cxt,
cxd,cyt,cyd,fxt,fxd,fyt,fyt(index drefers to the depth
camera), which are needed to compute the PCD and (1)-(2).
B. Segmentation pipeline
The basic assumption of the proposed segmentation al-
gorithm (blue pipeline in Fig. 2) is to process exclusively
the information related to the dynamic objects present into
the observed scene. This is because every point-cloud based
strategy always represents a computationally heavy opera-
tion, then a Background Segmentation step has been initially
adopted to subtract the static environment.
Section II-A describes the experimental setup in which the
cameras monitor the surroundings of the manipulator and the
robot kinematic chain is fully visible, as shown in Fig. 2A.
While the collaborative workspace is observed, the robot
executes its task, thus becoming a dynamic entity. Therefore,
the package Real-time URDF Filter [23] has been integrated
at the beginning of the pipeline to distinguish the depth pixels
belonging to the robot model from those belonging to other
dynamic entities and assign them a Not-a-Number (NaN)
value, see Fig. 2B.
The background filtering has been developed through
an efficient algorithm that performs the subtraction of a
stored background, at pixel level: 50 frames of the static
background are initially captured and the mean value of
each pixel is stored in a memory area. At every acquisition,
the current frame subtracts the static frame, as shown in
Fig. 2C. The depth image is then converted to PCD and
a uniform sampling filter is applied to make the algorithm
more reactive, by reducing the clouds density. Finally, the
detection of dynamic entities is executed through a PCD
Clustering step, which processes the point-cloud scene and
provides some clusters as many as single dynamic areas are
detected in the foreground. The Euclidean cluster extraction
method is performed to distinguish all the clusters into
the collaborative workspace. Figure 2D shows two detected
dynamic entities visualized in RViz together with the robot
model. To compensate the sensors measurement noise that
could sometimes provide false clusters, a first constraint is
enforced by defining a minimum cardinality that the areas in
the foreground should have, to be large enough to represent
a human entity. However, the correct discrimination about
validity of the cluster as a real human entity is done through
a novel developed HDT algorithm described in Section II-C.
The Human Validation step waits for the cluster check from
the HDT pipeline, which executes the human detection as
explained in Section II-C.
C. Human Detection and Tracking
The proposed human detection approach makes use of a
Convolutional Neural Network (CNN), whose innovative in-
put is obtained by combining depth and thermal images. The
two information have been processed though a simple and
intuitive pixel-by-pixel technique, never presented elsewhere.
This choice did not allow the authors to use pre-existing data-
sets and pre-trained CNNs but the CNN had to be trained
with a novel, multi-sensory data-set.
A depth camera is suitable for different kinds of environ-
ments because of its adaptability to different lighting con-
ditions: in this work it is adopted to compute the minimum
distance between the human operator and the robot to apply
regulations, but a depth image lets also to distinguish and
localize human surface shapes and their volume, so it is
appropriate to the HDT problem. On the other hand, the
thermal camera distinguishes temperatures and it is ideal for
finding objects of a specific temperature as the human body,
which is about 37◦C.
CNN solutions which process only thermal images can
be often not sufficient in those applications where large,
hot objects, e.g., tools used in the manual task or, more
generally, small temperature gradients could be present. On
the contrary, CNN trained to detect human shapes into pure
depth images can confuse human operators with objects of
similar shapes, e.g., a plastic mannequin, a coat rack, a
lamp. Therefore, merging depth and thermal images makes
the HDT approach more robust, avoiding false positives and
making correct decisions about human classification. More-
over, the proposed CNN strategy allows also to correctly
localize the human operators into the observed scene. This
information is then sent to the segmentation pipeline to select
who is the human cluster which is the closest to the robot
and continue the computation of the separation distance from
the manipulator. In these terms, thermal imaging is used as
complementary information to spatial ones and represents
a vision strategy that goes beyond SoA human detection
approaches based on background subtraction.
1) Depth-Thermal mapping: The extrinsic calibrations
explained in Section II-A are a first step towards a correct
mapping, that means finding matches between the depth
image and the thermal image. Since the adopted cameras
have different FOVs and resolutions, the resulting map size
must correspond to the smallest one. According to the
experimental setup shown in Fig. 1, the mapping step builds

Fig. 2. Implemented human detection and tracking pipeline: the background segmentation (first three blue labels) processes the depth image to subtract the
static environment and to highlight only dynamic shapes. At the same time, the depth image and the corresponding thermal image are merged into a single
RGB channel (green and red labels, respectively). The obtained images are then combined through a mapping matrix to reliably localize workers (orange
labels) and distinguish them from non-human clusters (human validation step); eventually, the minimum distance between the closest human operator and
the robot is computed.
Fig. 3. The algorithm for mapping depth and thermal images finds pixel-
by-pixel matches between the two images: the result is a 382×288 matrix.
a 382 ×288 matrix, the size of the thermal image shown in
Fig. 3.
The mapping step has been solved through a pixel-by-
pixel procedure: the pixel of the depth image, of indices
(m,n), contains the depth value, zd
m,n, which can be acquired
to compute the corresponding Cartesian point coordinates
p
p
pd
m,n=xd
m,nyd
m,nzd
m,nT, similarly to (1)-(2),
xd
m,n=(m−cxd)zd
m,n
fxd
(3)
yd
m,n=(n−cyd)zd
m,n
fyd
.(4)
The Cartesian point is then expressed with reference to the
thermal camera frame through the relation
p
p
pt
m,n
1=T
T
Tt
dp
p
pd
m,n
1.(5)
Using the intrinsic parameters of the thermal camera, the
corresponding pixel indices of the point p
p
pt
m,ninto the thermal
image (a,b)are finally computed by inverting (1)-(2). If
they are contained in the FOV of the thermal image, the
corresponding depth pixel indices (m,n)are written into
the mapping matrix at the indices (a,b); otherwise, they
are discarded because they are outside the mapping image
size. Note that, if the observed object is far enough, the
mapping matrix does not depend on the distance. Thus the
mapping matrix can be computed offline by using a fixed
value of zd
m,ncompatible with the working area, hence saving
computational load.
2) Sensor fusion: Multimodal sensor images can be com-
bined through different image fusion techniques, which
work at different merging levels: pixel-by-pixel, combining
signals, using relevant features or at symbol levels. This
paper provides an image fusion algorithm at pixel level but
represents a novel approach with respect to the most widely
used pixel-level image fusion algorithms [24] which never
merge depth and thermal information.
The first requirement of the proposed sensor fusion ap-
proach is to preserve all valid and useful information from
the two sources to be combined, while not introducing distor-
tions. For the purpose of this work, the depth image and the
corresponding thermal image have been merged to provide an
enhanced single view of a scene with extended information
content, through the mapping matrix of Section II-C.1. The
proposed approach, callable RGB Mapping Approach (RGB-
MA), consists in defining the intensities of empty RGB

channels. This is also because, to the best of the authors
knowledge, CNNs work better with RGB images. RGB-MA
strength is that it assigns the same priority to the input
sources. The result is no longer a grayscale, as a depth image
alone could be, or a weighted average image which assigns
different priorities to the sources, but it is an RGB image
where the depth data have been mapped on the green channel
(see Fig. 2H) and the temperature values have been mapped
on the red channel (see Fig. 2K). Specifically, the original
depth sensor value, sd, and the corresponding temperature
sensor value, st, have to be normalized into the interval [0,1]
(see Fig. 2G and J). To do this, a minimum and a maximum
variability ranges of the source values have been defined for
both thermal, mint,maxt, and depth, mind,maxd, cameras.
They do not actually correspond to the ranges of the sensors
technical specifications, but they have been chosen according
to the values detectable into the considered workspace. More
in detail, the detectable depth values are included between
0.30m and 4.0m, while the detectable temperature values
are within the range [0,50]◦C, which are suitable for any
type of human detection task.
The color information inserted into the specific channel of
the (i,j)-th pixel of the output image must be mapped to 8
bits. The R(red) value is computed, by acquiring st
i,jfrom
the thermal image, as
Ri,j=round 255 st
i,j−mint
maxt−mint!; (6)
the G(green) value is computed by acquiring sd
m,nfrom the
depth image, where mand nare contained into the (i,j)-th
value of the mapping matrix (Section II-C.1),
Gi,j=round 255 sd
m,n−mind
maxd−mind!; (7)
the B(blue) value of the resulting image is always zero.
The result is shown in Fig. 2L. Note that the proposed
image fusion technique leaves another channel that could be
used for a further input source. Section IV report the results
of the approach for a typical SSM scenario.
3) CNN for Human Detection: To assess the effectiveness
of the fused images of Section II-C.2, YOLOv3 [25] has been
used for real-time human detection. The selected framework
is an off-the-shelf SoA 2D object detector pre-trained on
ImageNet [26] and fine-tuned on the MS-Coco [27] data-set.
It is an extremely fast and accurate object detection system,
which is born to detect semantic objects of a certain class,
e.g., humans, buildings and cars, in RGB images. Nowadays,
there are no neural networks which have been trained on
combined images such as those proposed by this paper, so
the YOLOv3 CNN model has been re-trained to adapt the
detection system to the Depth-Thermal (D-T) images. The
following steps have been executed:
•definition of a Human class;
•exclusion of the YOLOv3 pre-trained classes from the
prediction;
•building of the training data-set acquiring frames from
D-T video stream;
•manual labelling of each frame;
•retrain of the YOLOv3 CNN weights.
After the training step, the CNN has been applied to the
real time D-T video stream to obtain the human prediction. A
bounding box is estimated around each detected human and
its coordinates are finally sent to the point-cloud pipeline,
as shown in Fig. 2M. Note that both the training and the
prediction process need high computational cost and they
have been executed on a proper GPU (NVIDIA Titan V).
D. Human-Robot separation distance
Once the predicted bounding boxes are sent to the seg-
mentation pipeline, each cluster is verified: each point of
the cluster is transformed into depth pixel coordinates (by
inverting (3)-(4)). Since the bounding box is expressed in
the thermal image plane, the selected pixel is converted
into depth image coordinates through the mapping matrix
(Section II-C.1). If at least 50% of the cluster points belong
to a bounding box, the cluster is labeled as human and passes
the check. Figure 2E shows two clusters: the red human
operator, which is correctly detected by the CNN, and the
yellow plastic mannequin, which is correctly not labeled as
human.
The Human Validation check is a fundamental step to
compute the correct separation distance between human
operators and the robot to apply the actual regulations of
industrial robotic applications. Therefore, the last step of the
segmentation pipeline (Fig. 2F) identifies the nearest pair of
points, one belonging to the robot (P
R) and the other one
belonging to the operator (P
H), that minimize the distance,
i.e.,
P
H∈H,P
R∈R|d(P
H,P
R)≤d(P0
H,P0
R)
∀P0
H∈H,P0
R∈R,(8)
where d(·,·)is the Euclidean distance between two points,
Hand Rrepresent the set of all points that belong to the
operator and the robot, respectively.
If P
His detected through the HDT strategy, a robot model-
ing method has been implemented to detect P
R. To take into
account the link volumes and not only specific points, the
proposed solution uses spheres as in [28] and [29] to model
robot links. The kinematic chain has been padded through 12
dummy frames to include the robot homogeneously, creating
a virtual 0.10m diameter safety sphere around each of them.
Therefore, the pair of closest points can be immediately
identified: the algorithm calculates the distance between all
points of the verified clusters point clouds and the origin of
every robot frame. The robot point P
Rwill be on the closest
virtual sphere along the line connecting the origin with P
H.
This step strongly justifies the choice of a point cloud
based approach. In fact, it provides satisfactory accuracy and
precision: it allows tracking humans also when they are not
completely visible from the camera view, unlike common
skeleton-based techniques; it is not necessary that human
operators are in front of the camera view because the point

Fig. 4. Identification of the minimum distance points between the whole
robot (yellow sphere) and the closest human operator (purple sphere).
cloud-based approach will recognize them anyway; more
detailed body parts can also be detected, e.g., a elbow, the
head, an hand, the chin or the chest. Figure 4 shows the
results: the developed CNN distinguishes human operators
belonging to the Human class (in red) from other clustered
objects (in yellow), i.e., a plastic mannequin and a chair,
which are not labeled as humans and they are not considered
for the safety separation distance computation, even if they
are possibly closer to the robot. Note that the closest human
cluster is selected in case of many human operators.
III. TRAJECTORY SCALING
Industrial SSM scenarios allow the robot system and the
human operator to move concurrently in the collaborative
workspace. Risk reduction is achieved by maintaining at least
the minimum protective separation distance, S, between the
operator and the robot [3], assumed constant here. During
robot motion, the robot system never gets closer to the
operator than S. When the separation distance, d, is less than
S, the robot system stops, before it can impact the operator.
When the operator moves away from the robot system, it can
resume the motion automatically while maintaining at least
the protective separation distance.
The proposed strategy ensures human-robot coexistence
according to the standard regulations methodology, introduc-
ing a low speed mode which slows down the robot nominal
velocity when the operator is inside a hazardous workspace.
The approach guarantees the robot task efficiency by using a
time-scaling method to change robot operating speed without
introducing acceleration discontinuities.
A. Single robot work cell
With reference to a single robot case, a typical industrial
pre-programmed task, T, is composed by Npositions, ˜
q
q
qi,
associated to velocities ˙
˜
q
q
qi, accelerations ¨
˜
q
q
qiand temporal
Fig. 5. Relation between the computed separation distance d(x axis) and
the scale factor k(y axis).
instants ˜
ti. Typically the pre-programmed joint positions have
to be interpolated according to the sampling time Tcrequired
by the robot. In this work a quintic interpolation is used, i.e.,
the planned interpolated trajectory is
˜
q
q
qh=p
p
p5(th;T)(9)
˙
˜
q
q
qh=p
p
p4(th;T)(10)
th+1=th+Tc(11)
where this the h-th discrete time instant, p
p
p4is the derivative
of the polynomial p
p
p5,˜
q
q
qhand ˙
˜
q
q
qhare the planned joint position
and velocity at time threspectively.
The proposed algorithm modulates the robot speed by
scaling the time with a safety scale factor k, which can
assume values inside the interval [0,1]. The scale factor is
related to d(Section II-D) as shown in Fig. 5. When dis
below the danger distance dd=S,kis 0 and the robot stops.
When the distance dis far from the warning distance dw>S,
i.e., d>dw, the robot can move at full speed to improve the
production time. Between ddand dwthe function in Fig. 5
smoothly varies to avoid acceleration discontinuities.
Practically, the trajectory is scaled by computing (9) with
a scaled time τh, i.e.,
q
q
qh=p
p
p5(τh;T)(12)
τh+1=τh+kTc(13)
where q
q
qhis the actual joint command at time th. Obviously,
the joint command q
q
qh, as well as the scaled time τh, are gen-
erated with sampling time Tc. The effect of this approach is
the actual scaling of the joints velocities. In fact, using (13),
˙
τ≈τh+1−τh
Ts
=k.(14)
By differentiation (12), the following equation demonstrates
that the velocity is scaled by the safety factor k
˙
q
q
qh=p
p
p4(τh;T)k.(15)
This approach guarantees that the task Tremains the same
in position, but, simultaneously, the resulting velocity is
scaled according to k. When the operator is going to be
into a dangerous situation, the robot operates at diminished
capacity with limited velocity according to human robot
collaboration norms, until restoration of safety conditions.
Experimental results are shown in Section IV.

Fig. 6. Two cases of human false positives: a dummy labeled as human
in the depth CNN approach (left); a hot moving robot labeled as human in
the temperature CNN approach (right).
B. Multi-robot work cell
The strategy discussed so far can be easily extended to
multi-robot work cells. It is necessary to pay close attention
to distinguish the independent robots case from the cooper-
ating robots case.
If the work cell is composed by robots which execute
independent taskseach robot must be slowed down according
to the separation distance with the proper closest human
operator. Thus, the whole pipeline proposed in this paper
is executed for each robot and each robot speed is scaled
independently from the others. This solution has a positive
impact on the production time because it reduces the speed
only of the robots involved in dangerous situations.
On the contrary, if the work cell is composed by coop-
erating robots, e.g., an assembly line or the transportation
of a commonly held object as in [17],the application of the
strategy to the single robots independently can compromise
the task execution. Whereas, the perception pipeline must
be executed for each robot worker until the computation of
the scaling factors, Then, the scale factor corresponding to
the most dangerous robot is applied to all the configuration
variables of the whole robotic system to preserve the coop-
erative task. Suitable danger metrics can be defined to take
this decision, e.g., the scale factor of the robot closest to the
human can be selected.
IV. EXPERIMENTAL RESULTS
The section shows a complete experiment that represents
a typical SSM collaborative scenario to describe the advan-
tages of the approach. Emphasis has been devoted to the
proposed DT-CNN to better highlight the performance.
A. Performances of DT-CNN
To test the performance of the Sensor Fusion approach,
two others CNNs have been trained based on the depth (D-
CNN) and temperature (T-CNN) information, respectively.
The networks have been trained and tested with the same
input data-set. Note that the training phase needed about
1000 training samples to reach the presented performance.
This observation represents another great advantage of using
the proposed approach if compared with many pre-trained
CNNs, which usually needed tens of thousands of images to
be trained.
The Mean Average Precision (maP) has been adopted
as a metric to measure the accuracy of each CNN [30].
TABLE I
CNNSTES TING RE SULTS
CNN mAP % False Positives % False Negatives %
D-CNN 65.09 36.87 17.53
T-CNN 62.76 64.35 4.37
DT-CNN 57.54 2.47 11.85
10 15 20 25 30 35 40 45 50
0
0.5
1
10 15 20 25 30 35 40 45 50
0
0.5
1
Fig. 7. Experiment: an operator enters the shared workspace while the
robot is moving. The top plot shows the estimated distance robot-operator
d, the dangerous distance ddand the warning distance dw. The bottom plot
shows the trajectory scaling factor kadopted to scale the robot velocity.
Considering the bounding boxes returned from the prediction
and the ground truth, the estimation of mAP is based on
the calculation of various metrics such as precision, recall
and Intersection over Union (IoU). Since the mAP does
not consider false positives and negatives, the percentage of
the erroneous detection has been estimated as an additional
metric.
Comparing the results of Table I, the DT-CNN provides
a mAP slightly lower than other two methods since it is
computed by considering only true positives. Whereas, the
percentage of false positives is considerably lower than the
others. In all cases the percentage of false negatives remains
low. The high percentage of false positives of single-source
approaches is to be found in cases where hot objects (T-
CNN) or objects with shapes comparable to human ones (D-
CNN) can be confused with a human (Fig. 6).
B. Complete experiment
To evaluate the combined approach (human detection and
trajectory scaling) a SSM scenario has been experimentally
tested. A video of the experiment is available at https:
//youtu.be/BrcvKmSiR9Q. A collaborative manufac-
turing industrial operation has been considered. The robot ex-
ecutes a pre-planned task at a given nominal speed. Suddenly,
an operator enters the robot workspace to perform some
manual operations (see the accompanying video). Results
are shown in Fig. 7. At about 15s the operator enters the
collaborative workspace and the system starts to measure
the separation distance d(blue line). When dgoes below the
warning distance dwthe trajectory scaling factor kbecomes
lower than 1 and the robot reduces its velocity without
changing its path. In some intervals of the experiment dgoes
below the dangerous distance ddthus kbecomes 0 and the

robot stops.
In the accompanying video the DT-CNN has been also
compared with the networks that use single sources (depth
or thermal). It is clear that the DT-CNN ensures a better
human detection minimizing the false positives.
This experiment demonstrates how the proposed approach
is able to automatically detect human operators in collabora-
tive workspaces and modulate the robot velocity according
to the current regulations.
V. CONCLUSIONS
This work shows a multimodal perception system based
on a thermal and a depth camera adopted to detect human
operators in general multi-robot work cells. The cameras
have been coupled in a fixed way and calibrated. Results
show that the calibration error is small enough to implement
a new sensor fusion technique to process the camera images
and robustly detect humans into the observed scenario. The
fusion technique consists in defining an RGB image by
combining depth and thermal images on two channels. The
fused image is then processed by a CNN specifically trained
to detect humans in the workspace. The approach based on
the fused images has been demonstrated to be more efficient
than single source perception data in a real collaborative
scenario. Future developments will be devoted to devise
an SSM strategy where, rather than assuming a constant
minimum protective distance, it is computed based on the
actual robot and human velocities estimated through the
same perception data used for distance computation. A risk
analysis based on the real velocity information can lead
to the definition of a less conservative minimum protective
distance, hence maximizing productivity.
REFERENCES
[1] “Robots and robotic devices - Safety requirements for industrial robots.
Part 1: Robots.” International Organization for Standardization” Tech-
nical report, 2011.
[2] “Robots and robotic devices - Safety requirements for industrial robots.
Part 2: Robot system and integration.” International Organization for
Standardization” Technical report, 2011.
[3] “Robots and robotic devices - collaborative robots.” International
Organization for Standardization” Technical report, 2016.
[4] J. Heinzmann and A. Zelinsky, “Quantitative safety guarantees
for physical human-robot interaction,” The International Journal of
Robotics Research, vol. 22, no. 7-8, pp. 479–504, jul 2003.
[5] F. Flacco, T. Kroger, A. D. Luca, and O. Khatib, “A depth space
approach to human-robot collision avoidance,” in 2012 IEEE Interna-
tional Conference on Robotics and Automation. IEEE, may 2012.
[6] N. Arora, D. Martins, D. Ruggerio, E. Tousimis, A. J. Swistel, M. P.
Osborne, and R. M. Simmons, “Effectiveness of a noninvasive digital
infrared thermal imaging system in the detection of breast cancer,”
The American Journal of Surgery, vol. 196, no. 4, pp. 523–526, oct
2008.
[7] M. Vollmer and K.-P. Mllmann, Infrared Thermal Imaging. Wiley-
VCH Verlag GmbH & Co. KGaA, dec 2017.
[8] H. Kaplan, Practical Applications of Infrared Thermal Sensing and
Imaging Equipment. SPIE Publications, 2007.
[9] B. Li, H. Jin, Q. Zhang, W. Xia, and H. Li, “Indoor human detection
using RGB-d images,” in 2016 IEEE International Conference on
Information and Automation (ICIA). IEEE, aug 2016.
[10] N. Dalal and B. Triggs, “Histograms of oriented gradients for human
detection,” in 2005 IEEE Computer Society Conference on Computer
Vision and Pattern Recognition (CVPR’05). IEEE, 2005, pp. pp.
886–893 vol. 1.
[11] M. Liang and X. Hu, “Recurrent convolutional neural network for
object recognition,” in 2015 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR). IEEE, jun 2015.
[12] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus,
and Y. LeCun, “Overfeat: Integrated recognition, localization and
detection using convolutional networks,” CoRR, vol. abs/1312.6229,
2013. [Online]. Available: http://arxiv.org/abs/1312.6229
[13] K. Zhou and A. Paiement, “Detecting humans in rgb-d data with
cnns,” in 15th IAPR International Conference on Machine Vision
Applications (MVA), Nagoya University, Nagoya, Japan. IEEE, 2017.
[14] S. Vidas, P. Moghadam, and M. Bosse, “3d thermal mapping of
building interiors using an RGB-d and thermal camera,” in 2013 IEEE
International Conference on Robotics and Automation. IEEE, may
2013.
[15] P. Rybski, P. Anderson-Sprecher, D. Huber, C. Niessl, and R. Sim-
mons, “Sensor fusion for human safety in industrial workcells,” in
2012 IEEE/RSJ International Conference on Intelligent Robots and
Systems. IEEE, oct 2012.
[16] P. Zhang, P. Jin, G. Du, and X. Liu, “Ensuring safety in human-robot
coexisting environment based on two-level protection,” Industrial
Robot: An International Journal, vol. 43, no. 3, pp. 264–273, may
2016.
[17] M. Lippi and A. Marino, “Safety in human-multi robot collaborative
scenarios: a trajectory scaling approach,” IFAC-PapersOnLine, vol. 51,
no. 22, pp. 190–196, 2018.
[18] S. Choi, T. Kim, and W. Yu, “Performance evaluation of ransac
family,” in Procedings of the British Machine Vision Conference 2009.
British Machine Vision Association, 2009.
[19] C. Papazov and D. Burschka, “An efficient ransac for 3d object
recognition in noisy and occluded scenes,” in Computer Vision – ACCV
2010. Springer Berlin Heidelberg, 2011, pp. 135–148.
[20] R. B. Rusu and S. Cousins, “3d is here: Point cloud library (PCL),” in
IEEE International Conference on Robotics and Automation. IEEE,
may 2011.
[21] T. Luhmann, J. Piechel, and T. Roelfs, “Geometric calibration of ther-
mographic cameras,” in Thermal Infrared Remote Sensing. Springer
Netherlands, 2013, pp. 27–42.
[22] J. Rangel and S. Soldan, “3d thermal imaging: Fusion of thermog-
raphy and depth cameras,” in Proceedings of the 2014 International
Conference on Quantitative InfraRed Thermography. QIRT Council,
2014.
[23] N. Blodow, “Realtime urdf filter,” 2012. [Online]. Available:
http://github.com/blodow/realtime urdf filter
[24] B. Yang, Z. liang Jing, and H. tao Zhao, “Review of pixel-level image
fusion,” Journal of Shanghai Jiaotong University (Science), vol. 15,
no. 1, pp. 6–12, feb 2010.
[25] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,”
CoRR, vol. abs/1804.02767, 2018. [Online]. Available: http://arxiv.
org/abs/1804.02767
[26] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei,
“ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09,
2009.
[27] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Doll´
ar, and C. L. Zitnick, “Microsoft COCO: Common objects in
context,” in Computer Vision – ECCV 2014. Springer International
Publishing, 2014, pp. 740–755.
[28] S. I. Choi and B. K. Kim, “Obstacle avoidance control for redun-
dant manipulators using collidability measure,” in Proceedings 1999
IEEE/RSJ International Conference on Intelligent Robots and Systems.
Human and Environment Friendly Robots with High Intelligence and
Emotional Quotients (Cat. No.99CH36289). IEEE, 1999.
[29] P. Bosscher and D. Hedman, “Real-time collision avoidance algorithm
for robotic manipulators,” in 2009 IEEE International Conference on
Technologies for Practical Robot Applications. IEEE, 2009, pp. 113–
121.
[30] W. Su, Y. Yuan, and M. Zhu, “A relationship between the average
precision and the area under the roc curve,” in Proceedings of the 2015
International Conference on The Theory of Information Retrieval,
ser. ICTIR ’15. New York, NY, USA: ACM, 2015, pp. 349–352.
[Online]. Available: http://doi.acm.org/10.1145/2808194.2809481