Human-Centric Digital Twins: Advancing Safety and Ergonomics in Human-Robot Collaboration

Abstract

Human-Robot Collaboration combines the reliability of robots with human adaptability. It is a prime candidate to respond to the trend of Mass Customization which requires frequent reconfiguration with variable lot sizes. But the close contact between humans and robots creates new safety risks, and ergonomic factors like robot-induced stress need to be considered. Therefore we propose a human-centric Digital Twin framework, where information about the human is stored and processed in a dedicated Digital Twin and can be transmitted to the robot’s Digital Twin for human-aware adaptations. We envision and briefly discuss three possible applications. Our framework has the potential to advance collaborative robotics but inherits technical challenges that come with Digital Twin based approaches and human modelling.

Full text

Human-Centric Digital Twins: Advancing Safety
and Ergonomics in Human-Robot Collaboration
Ben Gaffinet1,2, Jana Al Haj Ali2, Herv´e Panetto2, and Yannick
Naudet1
1Luxembourg Institute of Science and Technology, Belval, Luxembourg
{ben.gaffinet, yannick.naudet}@list.lu
2Universit´e of Lorraine, CNRS, CRAN, Nancy, France
{jana.al-haj-ali, herve.panetto}@univ-lorraine.fr
Abstract. Human-Robot Collaboration combines the reliability of robots
with human adaptability. It is a prime candidate to respond to the trend
of Mass Customization which requires frequent reconfiguration with vari-
able lot sizes. But the close contact between humans and robots creates
new safety risks, and ergonomic factors like robot-induced stress need
to be considered. Therefore we propose a human-centric Digital Twin
framework, where information about the human is stored and processed
in a dedicated Digital Twin and can be transmitted to the robot’s Digi-
tal Twin for human-aware adaptations. We envision and briefly discuss
three possible applications. Our framework has the potential to advance
collaborative robotics but inherits technical challenges that come with
Digital Twin based approaches and human modelling.
Keywords: Human-Robot Collaboration ·Human Digital Twin ·Cog-
nitive Digital Twin ·Human-Centric System
1 Introduction
Human-centric systems are designed following the human centered design paradigm
which focuses on people’s needs first [12]. Important pillars of the design philos-
ophy are, among others, usability, accessibility, ethical considerations and user
empowerment. The paradigm has been applied successfully to develop, for ex-
ample, healthcare information systems, interfaces for user electronics or Smart
Homes, especially for elderly care. Human-centric systems are not only imple-
mented to support consumers or patients, but also workers in an industrial con-
text. In manufacturing, the collaboration between humans and robots can benefit
from human-centric design approaches.
Where appropriate, industry has adopted automation to enabled efficient and
cost-effective production of large lot sizes, keeping unit cost down. But the high
upfront cost of fully automated systems are prohibitive for small production
runs. Additionally it is costly to change the configuration of a fully automated
production line, leading to limited flexibility when it comes to customizing a
product for a customer segment or an individual’s specific demands. As a polar

2 B. Gaffinet et al.
opposite manual labor is highly flexible and, provided the right training and
instructions, can customize products quickly to respond to demand, but at a
high unit cost. Therefore a compromise between cost-effectiveness and flexibility
is desirable for products that should be highly customizable with variable lot sizes
[41]. A prime candidate to achieve this is Human-Robot-Collaboration (HRC)
(see Figure 1) which combines the repeatability, accuracy and strength of robots
with the flexibility and versatility of humans.
Fig. 1. Economic viability of HRC as presented in [45]. The blue zone represents the
production volumes for which HRC provides the best unit cost and thus is economically
viable.
Interactions between humans and robots have evolved over time, starting
with mere coexistence in the same physical space with completely independent
operations. Eventually the interactions between humans and machines deepened,
Wang et al. [64] distinguished between Coexistence, Interaction, Cooperation
and Collaboration between Humans and Robots. For the remainder of this paper
we exclusively focus on Human-Robot-Collaboration, which means that both
actors are sharing the same workspace, resources and tasks while direct physical
contact is allowed.
With the close contact between robots and humans comes a non-negligible
risk of injury which motivates extensive research into robot safety methods and
collision avoidance in particular. Real-time situational awareness is a common
challenge and required to estimate distances or distinguish voluntary from acci-
dental contact. Robots need to be more flexible and adaptable in their program-
ming to work with the uncertainty that human behaviour introduces into the
system. Nonetheless successful HRC implementations can improve ergonomics

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 3
[39] by minimizing mental and physical fatigue by assigning repetitive and heavy
lifting tasks to the robot. But poorly implemented systems can induce stress in
the human worker and erode trust in the machine thus there is a need to track
the human state and ergonomic metrics for good collaboration.
In the field of system control, safety has always been a major concern for
industrial setups. Recognizing this, the Digital Twin (DT) has been identified as
a practical and efficient solution. Given its potential, we are inclined to propose a
framework that integrates DT. By emphasizing human interaction and involve-
ment in the process, the DT is positioned not only to bolster safety measures
but also to refine and optimize ergonomics. Through behavioural models addi-
tional information about human intent can be exploited to ensure safety through
adaptation of the robot’s movement. While tracking the human state allows the
computation of ergonomic indices for improved well-being or adapt operation
speed for comfort. A DT is a continuously updated digital representation of the
twinned entity. It centralizes data of the given entity, can run models to simu-
late situations before they occur, or emulate the system to possibly mitigate or
prevent dangerous situations and failures. DT and twinned entity are connected
via a bi-directional data link; captured data from the entity is fed into the DT
to update models; and the DT returns feedback, which can range from sugges-
tions to commands, to change the state of the twinned entity. Our framework
is composed of two separate DTs; (i) A dedicated Human Digital Twin (HDT)
[49] which gathers data from the human worker and models its intent, behaviour
and well being; (ii) A Cognitive Digital Twin (CDT) [50] that utilizes cognitive
functions to interpret the robot’s interactions, enhancing its ability to align its
operations with the human’s anticipated actions and intents.
The remainder of the paper is structured as follows; Section 2presents the
state of the art for assembly tasks in HRC work cells with a special focus on the
limitations and challenges in safety and ergonomics; in Section 3we present a
DT based approach to address perceived gaps in the state of the art; Finally in
Section 4we conclude and provide research directions.
2 State of the Art
Multiple core challenges of HRC are succinctly expressed by Wang et al. [64]:
“An essential aspect of HRC is how to cope with human ergonomics, process
time, emotions and reaction during collaboration, and safety aspects.”
While a wide array of challenges exist in HRC we choose to focus on, and
explore, two dimensions: safety & ergonomics. Both lead back to problems
with real-time situational awareness and integrating information about the hu-
man state and behaviour. Therefore human-centered approaches are particularly
applicable to address these challenges. It was in this context that the notion of
DT was introduced. DTs offer a precise digital replica of a real system or process,
enabling detailed simulations and analysis. These tools are proving essential in
tackling and solving the challenges of HRC. In what follows, we explore in depth
the applications and implications of DTs in this context.

4 B. Gaffinet et al.
2.1 Safety
The merits of HRC are discussed in [33] where Kruger et al. make the case
for how robot reliability and human adaptability can enable a new generation
of assembly processes. Safety becomes a prime concern as soon as the physical
separation between human workers and industrial robots is no longer guaranteed.
HRC, by definition, allows physical contact between human workers and robots
making physical separation inherently impossible.
In the industrial sector, many robots operate as Cyber-Physical Systems
(CPS), optimized for performance and the completion of specific tasks. Faced
with the rapid evolution of HRC, Yilma et al. [67] introduced an innovative
approach to Industry 4.0, based on the Cyber-Physical-Social Systems (CPSS)
paradigm. While human aspects and subtleties are frequently omitted from tradi-
tional CPS-centric models, the CPSS methodology aims to resolve this omission.
It aims to strengthen collaboration while guaranteeing security. Thanks to arti-
ficial intelligence, CPSS can assimilate and interpret human behaviors, making
robots, especially those based on CPS, more effective collaborative allies.
In [64,63] the causes for HRC accidents are organized in three categories; En-
gineering failures, human errors, and poor environmental conditions. For robotics
and machinery in general, engineering failures are addressed in the design phase
and guidance is provided in ISO 13855 [28]. The unavoidable physical contact
between robots and humans in HRC setups required an adaptation of standards,
which lead to ISO/PDTS 15066 [29]. It provides valuable guidance on risk as-
sessment, safety features, workspace design as well as task and process design.
Overall safety in robotics is a well researched subject, nonetheless the challeng-
ing context of HRC requires further investigation on how to enable setups that
can operate at relatively high efficiency and speed without compromising safety.
Research in HRC safety can be categorized into three categories; (i) under-
standing injury mechanisms and related standards; (ii) limiting the severity of
an impact; (iii) active collision avoidance. In the first category, Haddadin et al.
[22,23,24] made major contributions with their work on injury mechanisms of
collisions that might occur during HRC task execution. Further research from
Haddadin et al. [21] establishes that safety is guaranteed for robot velocities
below 2.7m/s and in [25] the link between mass, velocity, impact geometry and
resulting injury is described in detail. Instead of limiting operational velocity,
Laffranchi et al. [36] proposed to regulate the energy of the system instead. Ear-
lier research in [66] analyses pain tolerance of humans and establishes an early
method to reduce robot velocity upon impact, which allows the human operator
to reflexively withdraw and avoid serious injuries and pain. More recent research
introduces additional constraints to the control algorithms based on newly in-
troduced safety indicators [46]. Depending on the distance between robot and
human operator the kinetic and potential energy in the system is reduced to
limit the energy that could be dissipated in the event of a collision.
Detecting collisions and reacting appropriately represents a second major
category of research in HRC safety. Unmitigated movement of a robot after an
unwanted collision can lead to severe pain and injury. As contact between robot

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 5
and human is expected in HRC it is not appropriate to stop operations as soon
as any physical contact occurs, instead more subtle methods are required. On
the hardware level the addition of joint torque sensors enabled a new set of
safety features. Tonietti et al. [61] adjusts the stiffness of actuation to inherently
guarantee safety. Park et al. [55] developed a passive safety mechanism through
non-linear stiffness. During normal operations high stiffness is maintained, but
upon impact, that exceeds a set threshold force, the stiffness drops quickly. In
contrast Geravand et al. [20] base their control architecture on the motor currents
and joint velocity without the need of additional torque sensors. Kokkalis et al.
[30] compare provided motor currents with the expected current needed for the
planned trajectory to limit forces during contact. In a similar approach [48]
uses reference torque and actual torque to the same end, while [40] use Neural
Networks to detect torque disturbances. A last set of approaches uses additional
external sensors, such as optical cameras [15,34,19] or depth sensors [10,17] to
detect collisions.
Avoiding collisions altogether is approached from different angles by re-
searchers. A large swath of research uses on or the combination of detection
systems such as, time of flight cameras (or other depth sensors) [18,3,1] or opti-
cal [60] cameras. Creating 3D models of the robot, environment and/or human
worker is sometimes used in addition to the chosen sensing systems [47,65] to en-
able better distance estimations. Some approaches put the human worker at the
center of their investigation by tracking their movement, either with wearables
or external cameras to ultimately include human behaviour [69] as a parame-
ter for robot adaptation [37]. Augustsson et al. establish safety zones [6] based
on which the robot can adapt when a human enters the zone. In this case the
data about the human is communicated to the robot for adaptations. Efficient
real-time communication is generally a challenge and includes communicating
the human position to the robot as well as changes in assembly order or settings
to the human [7].
Decades of Robotics safety research have lead to hardware adaptions, and
methods that substantially reduce the risk of severe injuries. Force and speed
limits, alongside collision detection and appropriate reactions upon impact miti-
gate both pain and injury severity. But, we believe that further research is desir-
able in the field of collision avoidance. Increased situational awareness, including
the internal state of the human operator (e.g. stress, behavioural patterns), could
enable less restrictive and more dynamic speed adaptations and enable increased
efficiency of HRC work cells. In Section 3we introduce a DT based approach to
address the perceived limitations of traditional setups.
2.2 Ergonomics
In HRC setups robots should support the human worker to fulfill the defined
task as a human-robot team and adapt their own behaviour to make the human
comfortable, improve ergonomics performance and keep task completion time
low [64]. Ergonomics encompasses not only physical stress, which can be induced
by e.g. lifting heavy loads, but also mental stress. Both mental underload, from

6 B. Gaffinet et al.
monotonous tasks, and overload, typically from overstraining or high pace, need
to be addressed in HRC research.
A necessary component of HRC is that the human team member accepts to
work with the robot. Poorly adapted robot movement can induce stress, erode
trust and ultimately lead the the loss of propensity to work with the robot.
Trust is a major parameter for successful HRC setups and has been studied
extensively by Hancock et al. [26]. The dominant factor that can erode trust
is the performance of the human-robot team, while environmental and human
factors have a more limited impact. The distance between human and robot
alongside the robot’s movement speed has a direct impact on induced stress [4].
The more general impact of robot movement on the human affective state has
been investigated in [35]. Adapting the robot’s movement by designing human-
aware systems is a promising approach that has found success in the field.
Research addressed the problems through better motion planning based on
human-aware systems. The movement needs to be predictable, and perceived
as safe. Multiple paper seek to improve motion planning and task allocation by
including information about the human worker; such as the human’s goals [56],
mental model [52], perception of robot movement [54,13], next subtask [27] or
intended motion [62].
Many promising approaches to improve fluency of HRC systems and er-
gonomics rely on understanding the human worker’s behaviour and state better.
In Section 3we propose the use of HDT to gather human worker data and
relevant models for better adaptive systems.
2.3 Digital Twins
Safety, a long-underestimated major concern in robotics research, has become
a central preoccupation as intelligent collaborative robots, commonly known
as cobots, are increasingly integrated into various industries. These cobots are
equipped with sensors that enable them to detect and react to their environ-
ment, guaranteeing safe physical interactions with humans. These sensors not
only detect the presence of humans and potential obstacles, but also gather
complex data on cobot movements and forces, such as joint position and com-
pliance monitoring. Faced with this safety challenge, it is imperative to develop
advanced simulation models known as digital twins (DT). These are defined as,
“A set of adaptive models that emulate the behaviour of a physical system in a
virtual system getting real time data to update itself along its life cycle. The dig-
ital twin replicates the physical system to predict failures and opportunities for
changing, to prescribe real time actions for optimizing and/or mitigating unex-
pected events observing and evaluating the operating profile system” [58]. These
DT enable scenarios to be recreated [57] and tested virtually [32]. This data-
driven approach ensures that cobots can interact safely with humans, marking
a significant advance in automation within the manufacturing sector. Safety in
HRC setups is further delved into with a study [11], where a mixed-reality ap-
proach is proposed to enhance safety. This method employs sensors to measure
the real-time safety distance between humans and cobots. Further enhancing the

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 7
capabilities of DT, Droder [14] introduces a machine learning method for indus-
trial cobots to avoid obstacles and people, using the nearest neighbor approach
for trajectory planning and neural networks for detection.
With the aim of allowing DTs in virtual reality to guide the safe implemen-
tation of human-robot collaboration strategies in future factories, Oyekan [53]
established a DT workshop to study human responses to both predictable and
unpredictable cobot motions. The findings indicated that real-world human reac-
tions to robotic behaviors were consistent with those observed in the DT virtual
environment. It is essential to create an accurate and consistent DT model that
captures the physical components and their interactions. With this in mind, a
quadruple DT model has been suggested [42], encompassing separate models for
the cobot, the individual, the collaborative framework, and their interrelation-
ships.
In the field of ergonomics, the introduction of Digital Human (DH) technol-
ogy to the DT-controlled HRC represents a significant step forward, emphasizing
human centricity in a production context. This system [44] captures and sim-
ulates operator movements and physical constraints in real time, providing a
precise ergonomic assessment platform. It is structured around three integrated
modules: the first is a virtualized robotics module offering direct, real-time con-
trol of the PLC; the second is a DH module specifically dedicated to the analysis
and simulation of operator actions; and finally, a production management mod-
ule which guarantees adapted planning taking ergonomic criteria into account.
The DT has been employed in the design and reconfiguration of assembly
systems. Kousi et al. [31] has modeled production parameters at different lev-
els and updates them dynamically using data from 2D-3D sensors, combining
geometry and semantics to obtain an overview of the production process.
DT functions, like monitoring, prediction and optimization, offer effective
strategies for HRC assembly. Malik’s [43] study introduces a DT framework
aimed at optimizing the design, construction, and control of human-machine
assembly collaborations. Through computer simulations, a constantly updated
digital model of the collaborative environment is maintained, ensuring timely
improvements and mirroring the physical setup. Similarly, Bilberg’s [8] research
focuses on an object-oriented simulation of a flexible assembly cell, integrated
with a cobot to work alongside humans. Beyond traditional virtual models, this
DT emphasizes real-time control, dynamic task allocation between the cobot
and human, and adaptive cobot programming based on the task sequencing.
DT, while useful in a variety of fields, have certain limitations when applied
to HRC. One major limitation is their limited ability to anticipate and adapt
to unpredictable human behavior in dynamic work environments. In addition,
DT may struggle to model safety accurately enough to ensure safe interactions
between cobots and human workers. This highlights the need for a better un-
derstanding of the human processes at the core of these interactions. The notion
of cognition focuses primarily on knowledge and understanding [50]. It encom-
passes the processes by which sensory inputs are transformed, stored, and used

8 B. Gaffinet et al.
[51], including aspects such as attention, reasoning, learning, memory, percep-
tion, problem solving, and knowledge representation, among others [2].
In an HRC context, where safety and ergonomics are major concerns, it be-
comes clear that advanced cognitive capabilities are required. This is where the
Cognitive Digital Twin (CDT) comes in, which is “a digital representation of
a physical system that is augmented with certain cognitive capabilities and sup-
port to execute autonomous activities; comprises a set of semantically interlinked
digital models related to different lifecycle phases of the physical system includ-
ing its subsystems and components; and evolves continuously with the physical
system across the entire lifecycle” [70]. The CDT possesses knowledge manipula-
tion and problem-solving abilities. These cognitive capabilities include detection,
reasoning, and self-learning, enabling continuous and proactive adaptation.
Research such as that conducted by Shi et al. [59] on a CDT framework for
manufacturing systems with HRC based on 5G communication networks illus-
trates the increasing importance of CDT in the manufacturing field. The concept
of CDT in [68], thus provides the ability to dynamically handle more complex
and unpredictable situations through enhanced computational capabilities. As
an emerging concept, CDT has not yet been widely implemented and verified in
the industry. Most published studies explore the theoretical perspectives of CDT
or focus on its vision. Nonetheless, there are ongoing studies and projects target-
ing its practical feasibility in diverse industry scenarios. Notably, the active EU
project COGNITWIN [70] aims to enhance the cognitive capabilities of exist-
ing process control systems to enable self-organization and address unpredicted
behaviors.
3 Proposed Research
In this section we propose a DT based approach to respond to challenges in HRC
safety and ergonomics. We propose an adapted architecture for DTs of agents
and discuss their application in the context of HRC. Two functionalities of DTs
are of interest, namely emulation and simulation.
3.1 Single Agent Digital Twin
For each physical entity, or more precisely agent, we propose to instantiate a
dedicated DT that gathers all the data coming from the twinned entity, and
holds all the models describing and acting upon the twinned entity. Additionally
the ability to receive data from the environment or other physical entities is
required to provide the relevant context for some of the models. Figure 2depicts
the essential elements and their interactions required to build a DT of an agent.
Data from the Twinned Agent is fed to the Agent Digital Twin and stored in
the Data Storage alongside data from the Environment that is needed to give
required context. Data is used by the Models to track and predict parameters
of the Twinned Agent. The result of running a model can be fed back into the
Data Storage to update relevant parameters or trigger a feedback to the Twinned

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 9
Agent. For cobots the feedback could be a change in operational mode while a
human agent can not be directly controlled but only influenced via suggestions.
The last element is the Agent Digital Twin ’s interaction with other agents in the
system. Through and Access Interface data can be shared between agents and
fed to the relevant models to adapt behaviour accordingly. The Co-Simulation
Interface provides the ability to run simulations of what-if scenarios involving
multiple agents. In this case simulated data needs to be provided as both the
physical Environment and Twinned Agent are decoupled for simulation runs.
Fig. 2. Digital Twin Architecture for an Agent. Adapted from the Human Digital Twin
reference architecture proposed by L¨ocklin et al. [38]. In the context of HRC the agent
is either a human or a robot.
3.2 Digital Twins for Human-Cobot Collaboration Systems
Considering the specific case of an HRC scenario, there will be two DTs; a
Human Digital Twin and a Cognitive Digital Twin, see Figure 3for the proposed
architecture. The physical layer is composed of a Human alongside a Cobot. Both
agents have physical interactions with each other and the Environment itself.
Data is gather from both agents to feed into their respective DTs, which follow
the previously described architecture from Figure 2. A Communication Module
enables the exchange of data between both DTs. Each DT can run models to
track the state of the system and send feedback to its twinned entity. Feedback
to the Physical Layer is managed by a Control System, which can reallocate and
reschedule tasks dynamically, or change the operational mode of the Cobot.

10 B. Gaffinet et al.
Fig. 3. Digital Twin based approach for HRC systems with one human worker and
one Cobot. Dashed lines represent physical interactions while full lines represent data
flows.
The Human Digital Twin gathers data from the human worker, which can
include static data (e.g. age, skills, personality) and live data (e.g. biometrics,
video, sound). Through models meaningful information, such as ergonomic in-
dices, can be computed, stored and shared with the Cognitive Digital Twin and
Control System. Receiving live information about the state of the human worker
is the backbone of human-aware cobot adaptation.
The Cognitive Digital Twin, on the other hand, utilizes data received from
the Human Digital Twin, along with its own data and knowledge models, to
perform more advanced analyses. The Cognitive Digital Twin harnesses its cog-
nitive functions to perform various tasks. Through perception, it interprets sen-
sory data like facial expressions and voice tone. It employs reasoning to detect
patterns and anomalies, drawing from its memory of past experiences. This in-
forms its decision-making, allowing it to suggest adjustments like altering the
cobot’s parameters or task assignments based on real-time observations.
We see two possible adaptation that are possible in the DT based HRC
system; (i) reallocating or rescheduling tasks through the Control System,(ii)
changing the operational mode or settings of the Cobot. Example changes for the
Cobot include arm velocity and trajectory selection. For illustration purposes we
describe three cases where our framework could be applied in the next Section
3.3.

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 11
3.3 Use Cases and Limitations
Given the context of safety and ergonomics we provide three use cases below
and discuss the general challenges for prospective implementations. The cases
are; (i) Collision avoidance, (ii) Ergonomic improvements and (iii) Emulating
cobot reaction under extreme human behaviour.
I - Collision avoidance: The Cognitive Digital Twin leverages real-time emu-
lation capabilities to coordinate collision avoidance strategies. Data is gathered
from wearable devices or static cameras to feed the Human Digital Twin, which
calculates estimated positions of the human operator’s head and arms. Simul-
taneously, data concerning the cobot’s joint and end effector positions is con-
tinuously monitored and shared with the Cognitive Digital Twin. The Commu-
nication Module facilitates data exchange between human and cobot positions,
which is then analyzed by a collision avoidance model stored within the Cogni-
tive Digital Twin. The advantage is the possibility to connect models from the
Human Digital Twin, for example a intention model, with the collision avoidance
model of the Cognitive Digital Twin. The adaptive behaviour can incrementally
be enhanced by considering increasingly complex aspects of human behaviour.
II - Ergonomic improvements: In an analogous fashion information from the
human worker can be fed to a dedicated model of the Cognitive Digital Twin to
adapt the cobot’s movement in order to decrease induced stress. To improve the
human’s comfort not only the movement of the cobot can be adapted but also
the allocation of tasks. The Control System is kept up to date about the task
progress of both agents but can also receive information about the state of the
human worker. A threshold based approach could be used to trigger dynamic
reallocation of tasks when stress or fatigue increases too much.
III - Emulating cobot reactions under extreme human behaviour: The
DT framework can be used to run emulations by decoupling the physical entities
from the system and feeding in emulated data. In the context of HRC this
opens up the possibility to test the cobot’s adaptive response under extreme
human behaviours without putting any human at risk of injury. The ability
to emulate edge cases becomes especially important if the adaptive behaviour
relies on Machine Learning, knowledge-based systems (symbolic AI) or hybrid
approaches. Such integrated approaches can show unintended behaviour when
encountering edge cases, making emulation vital.
Related Works: Attempts to better account for the human factor in industrial
contexts exist. The novelty of the present proposal are the dedicated DTs for
each entity with a particular focus on HRC. Ascone et al. [5] consider Human-
Machine Systems and propose a holistic framework for DTs of such systems.
They argue that humans are an inseparable part and propose capturing hu-
man data and building human models. But they consider one single DT for the

12 B. Gaffinet et al.
whole system without having a dedicated DT for each agent. Bousdekis et al. [9]
consider a Manufacturing System which includes operators alongside machines
and processes. DTs for both the machines and humans are proposed to detect
abnormal situations and support the operator’s activities. The solution is not
focused on HRC but rather on joint decision making between the operator and
an AI system. Endo et al. [16] implement a Human Digital Twin to minimize the
physical load of a worker that is working alongside a robot. In the use case both
the robot and human pick items from boxes and store them in a cart without
entering in direct contact. Two DTs a instantiated with a physical load model
for the human and progress tracking of the overall process. The solution enables
the dynamic reallocation of tasks and successfully decreases the work performed
by the human and maximizes the usage of the robot.
Proposed Experimental Validation: In the context of Industry 4.0, where
cyber-physical systems play a key role, we propose to apply the framework to
a case of human-robot collaboration (HRC). More specifically an assembly task
performed jointly by a human and a cobot. We envision a shared workstation
where the cobot and human share resources to assemble a specific product, with-
out the need to move away from the workstation. An expected challenge is to
reduce the complexity of the human model to a minimal amount of parameters
(e.g: stress or fatigue) while maintaining the ability to meaningfully adapt the
cobot’s speed and trajectory selection. A second challenge, which is common
in DT based approaches, is the requirement for real time operations, thus im-
plementations that rely on computationally expensive methods are not viable.
Through a questionnaire it is possible to assess the amount of stress experienced
by participants for an adaptive and non-adaptive system. We would judge an
implementation as successful if stress is reduced without significantly increasing
the overall cycle time.
4 Conclusion and Discussion
We discussed how a human-centric, DT based, approach can be applied to re-
spond to challenges in Human-Robot Collaboration (HRC). We explored, in
detail, the state of the art two particular problems; safety & ergonomics. Ex-
tensive safety research has established limits to velocity and energy for human
safety. Furthermore methods to limit the severity of an unwanted collision are
well established. The most challenging approach to safety is avoiding unwanted
collisions altogether, which requires excellent situational awareness. Ergonomics
research established a link between robot movement and the human’s feeling of
safety, trust and stress. The human experience and well-being can be improved
through adaptive motion planning and dynamic task allocation but requires a
good understanding and tracking of the human state.
From the challenges in safety and ergonomics we identified situational aware-
ness to be particularly important for successful HRC implementations. To achieve

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 13
excellent situational awareness it is necessary to track the human state and ac-
count for the behaviour and intent of humans. As an additional benefit ergonomic
adjustments could be made using the tracked data about the human worker.
In our discussions on HRC, we have highlighted the growing importance of
safety, particularly with the emergence of collaborative robots. These cobots,
equipped with advanced sensors, guarantee safe interactions with humans. Hav-
ing explored the state of the art in the application of DT in HRC, it is clear
that DTs play an essential role in simulating and predicting interactions to en-
hance safety. Digital Human (DH) technology reinforces this vision by faithfully
reproducing human movements to provide a complete ergonomic assessment.
However, despite the considerable advantages of DHs, they face the challenge
of anticipating unpredictable human behavior. This is where the CDT comes
in, adding advanced cognitive capabilities to traditional DT. Although CDTs
represent a promising advance, their practical implementation remains largely
theoretical.
We proposed a DT based framework where each agent has its own dedi-
cated Agent Digital Twin (see Figure 2). Applied to a HRC system it translates
to a human and a cobot, each with their DT (see Figure 3). Through a com-
munication module, data can pass between the DTs to run models that require
awareness of each other’s state. The Human Digital Twin is designed with the ex-
plicit role of gathering the data about the human, and running models, required
for human-aware approaches to safety and ergonomics. Likewise, the Cognitive
Digital Twin is designed to analyze this data in greater depth, thus enabling
anticipatory adaptations during human-robot interactions. We briefly presented
three use cases for our framework; (i) Collision avoidance models can be sup-
ported with human intent models; (ii) Stress can be reduced by adapting the
cobot’s movement or dynamically reassigning tasks, (iii) Cobot reactions under
extreme human behaviour can be emulated without putting humans at risk.
Human-robot collaboration, particularly through the use of a DT, presents
multiple challenges. The intrinsic complexity of human behavior poses modeling
difficulties, often resulting in discrepancies between the DT’s predictions and
real-world outcomes. Additionally, these models might lack the adaptability to
cope with rapid behavioral changes, necessitating substantial data for training.
The real-time demands of tools like Computer Vision can introduce performance
bottlenecks, making the system’s speed a paramount concern. Task design for
such collaborations is crucial; poorly planned interactions can lead to ineffi-
ciencies or even failures. Navigating the regulatory landscape while employing
these technologies is another consideration. Moreover, when tapping into per-
sonal data, such as biometrics, there’s an added emphasis on securing that data,
ensuring its safety and confidentiality.
With a view to future research directions in human-robot collaboration, the
distinction between HDT and CDT models is crucial. For HDT, a thorough
exploration of human models is necessary to ensure a faithful, nuanced digital
representation of the individual. At the same time, for CDT, it’s crucial to
identify and integrate the cognitive functions representative of robots. Once these

14 B. Gaffinet et al.
elements have been established and differentiated, they can be applied to specific
use cases, laying the foundations for more harmonious and productive human-
robot collaboration.
Acknowledgement
This work has been partially supported by the ANR French National Research
agency and the FNR Luxemburgish National Research funds project AI4C2PS
(INTER/ANR/22/17164924/AI4C2PS), 2023-2025.
References
1. Ahmad, R., Plapp er, P.: Safe and automated assembly process using vision assisted robot ma-
nipulator. Procedia Cirp 41, 771–776 (2016)
2. Al Faruque, M.A., Muthirayan, D., Yu, S.Y., Khargonekar, P.P.: Cognitive digital twin for
manufacturing systems. In: 2021 Design, Automation & Test in Europe Conference & Exhibition
(DATE). pp. 440–445 (2021). https://doi.org/10.23919/DATE51398.2021.9474166, ISSN: 1558-
1101
3. Anton, F.D., Anton, S., Borangiu, T.: Human-robot natural interaction and collision avoidance
in flexible manufacturing cells. IFAC Proceedings Volumes 45(6), 835–840 (2012)
4. Arai, T., Kato, R., Fujita, M.: Assessment of operator stress induced by robot collaboration in
assembly. CIRP annals 59(1), 5–8 (2010)
5. Ascone, C., Vanderhaegen, F.: Towards a holistic framework for digital twins of human-machine
systems. IFAC-PapersOnLine 55(29), 67–72 (2022)
6. Augustsson, S., Christiernin, L.G., Bolmsj¨o, G.: Human and robot interaction based on safety
zones in a shared work environment. In: Proceedings of the 2014 ACM/IEEE international
conference on Human-robot interaction. pp. 118–119 (2014)
7. Augustsson, S., Olsson, J., Christiernin, L.G., Bolmsj¨o, G.: How to transfer information between
collaborating human operators and industrial robots in an assembly. In: Proceedings of the 8th
nordic conference on human-computer interaction: fun, fast, foundational. pp. 286–294 (2014)
8. Bilb erg, A., Malik, A.A.: Digital twin driven human–robot collaborative assembly. CIRP Annals
68(1), 499–502 (2019). https://doi.org/10.1016/j.cirp.2019.04.011,https://www.sciencedirect.
com/science/article/pii/S000785061930037X
9. Bousdekis, A., Ap ostolou, D., Mentzas, G.: A human cyber physical system framework for
operator 4.0–artificial intelligence symbiosis. Manufacturing letters 25, 10–15 (2020)
10. Casalino, A., Guzman, S., Zanchettin, A.M., Rocco, P.: Human pose estimation in presence of
occlusion using depth camera sensors, in human-robot coexistence scenarios. In: 2018 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS). pp. 1–7. IEEE (2018)
11. Choi, S.H., Park, K.B., Roh, D.H., Lee, J.Y., Mohammed, M., Ghasemi, Y., Jeong,
H.: An integrated mixed reality system for safety-aware human-robot collaboration using
deep learning and digital twin generation. Robotics and Computer-Integrated Manufactur-
ing 73, 102258 (2022). https://doi.org/10.1016/j.rcim.2021.102258,https://www.sciencedirect.
com/science/article/pii/S0736584521001381
12. Co oley, M.: On human-machine symbiosis. In: Human machine symbiosis: The foundations of
human-centred systems design, pp. 69–100. Springer (1996)
13. Dragan, A., Srinivasa, S.: Integrating human observer inferences into robot motion planning.
Autonomous Robots 37, 351–368 (2014)
14. Dr¨oder, K., Bobka, P., Germann, T., Gabriel, F., Dietrich, F.: A machine learning-enhanced
digital twin approach for human-robot-collaboration. Procedia CIRP 76, 187–192 (2018).
https://doi.org/10.1016/j.procir.2018.02.010,https://www.sciencedirect.com/science/article/
pii/S2212827118300295
15. Eb ert, D.M., Henrich, D.D.: Safe human-robot-cooperation: Image-based collision detection for
industrial robots. In: IEEE/RSJ international conference on intelligent robots and systems.
vol. 2, pp. 1826–1831. IEEE (2002)
16. Endo, Y., Maruyama, T., Tada, M.: Dhaibaworks: A software platform for human-centered
cyber-physical systems. International Journal of Automation Technology 17(3), 292–304 (2023)
17. Fischer, M., Henrich, D.: 3d collision detection for industrial robots and unknown obstacles
using multiple depth images. In: Advances in Robotics Research: Theory, Implementation, Ap-
plication, pp. 111–122. Springer (2009)
18. Flacco, F., Kro eger, T., De Luca, A., Khatib, O.: A depth space approach for evaluating distance
to objects: with application to human-robot collision avoidance. Journal of Intelligent & Robotic
Systems 80, 7–22 (2015)

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 15
19. Gecks, T., Henrich, D.: Human-robot cooperation: Safe pick-and-place operations. In: ROMAN
2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005.
pp. 549–554. IEEE (2005)
20. Geravand, M., Flacco, F., De Luca, A.: Human-robot physical interaction and collaboration
using an industrial robot with a closed control architecture. In: 2013 IEEE International Con-
ference on Robotics and Automation. pp. 4000–4007. IEEE (2013)
21. Haddadin, S., Albu-Schaffer, A., De Luca, A., Hirzinger, G.: Collision detection and reaction: A
contribution to safe physical human-robot interaction. In: 2008 IEEE/RSJ International Con-
ference on Intelligent Robots and Systems. pp. 3356–3363. IEEE (2008)
22. Haddadin, S., Albu-Sch¨affer, A., De Luca, A., Hirzinger, G.: Evaluation of collision detection
and reaction for a human-friendly robot on biological tissues. Proceedings IARP 2008 (2008)
23. Haddadin, S., Albu-Sch¨affer, A., Hirzinger, G.: Safety evaluation of physical human-robot inter-
action via crash-testing. In: Robotics: Science and systems. vol. 3, pp. 217–224. Citeseer (2007)
24. Haddadin, S., Albu-Sch¨affer, A., Hirzinger, G.: Requirements for safe robots: Measurements,
analysis and new insights. The International Journal of Robotics Research 28(11-12), 1507–
1527 (2009)
25. Haddadin, S., Haddadin, S., Khoury, A., Rokahr, T., Parusel, S., Burgkart, R., Bicchi, A., Albu-
Sch¨affer, A.: On making robots understand safety: Embedding injury knowledge into control.
The International Journal of Robotics Research 31(13), 1578–1602 (2012)
26. Hanco ck, P.A., Billings, D.R., Schaefer, K.E., Chen, J.Y., De Visser, E.J., Parasuraman, R.:
A meta-analysis of factors affecting trust in human-robot interaction. Human factors 53(5),
517–527 (2011)
27. Hawkins, K.P., Vo, N., Bansal, S., Bobick, A.F.: Probabilistic human action prediction and
wait-sensitive planning for responsive human-robot collaboration. In: 2013 13th IEEE-RAS In-
ternational Conference on Humanoid Robots (Humanoids). pp. 499–506. IEEE (2013)
28. ISO: Iso 13855 safety of machinery — positioning of safeguards with respect to the approach
speeds of parts of the human body (2010)
29. ISO: Iso/ts 15066 rob ots and robotic devices — collaborative robots (2016)
30. Kokkalis, K., Michalos, G., Aivaliotis, P., Makris, S.: An approach for implementing power and
force limiting in sensorless industrial robots. Procedia CIRP 76, 138–143 (2018)
31. Kousi, N., Gkournelos, C., Aivaliotis, S., Lotsaris, K., Bavelos, A., Baris, P., Michalos, G.,
Makris, S.: Digital twin for designing and reconfiguring human–robot collaborative assembly
lines. Applied Sciences 11, 4620 (2021). https://doi.org/10.3390/app11104620
32. Kritzinger, W., Karner, M., Traar, G., Henjes, J., Sihn, W.: Digital twin in manufactur-
ing: A categorical literature review and classification. IFAC-PapersOnLine 51(11), 1016–1022
(2018). https://doi.org/10.1016/j.ifacol.2018.08.474,https://www.sciencedirect.com/science/
article/pii/S2405896318316021
33. Kr ¨uger, J., Lien, T.K., Verl, A.: Cooperation of human and machines in assembly lines. CIRP
annals 58(2), 628–646 (2009)
34. Kr ¨uger, J., Nickolay, B., Heyer, P., Seliger, G.: Image based 3d surveillance for flexible man-
robot-cooperation. CIRP annals 54(1), 19–22 (2005)
35. Kulic, D., Croft, E.A.: Affective state estimation for human–robot interaction. IEEE transactions
on robotics 23(5), 991–1000 (2007)
36. Laffranchi, M., Tsagarakis, N.G., Caldwell, D.G.: Safe human robot interaction via energy regu-
lation control. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.
pp. 35–41. IEEE (2009)
37. Liu, H., Wang, Y., Ji, W., Wang, L.: A context-aware safety system for human-robot collabora-
tion. Procedia Manufacturing 17, 238–245 (2018)
38. L¨ocklin, A., Jung, T., Jazdi, N., Ruppert, T., Weyrich, M.: Architecture of a human-digital twin
as common interface for operator 4.0 applications. Procedia CIRP 104, 458–463 (2021)
39. Lorenzini, M., Lagomarsino, M., Fortini, L., Gholami, S., Ajoudani, A.: Ergonomic human-robot
collaboration in industry: A review. Frontiers in Robotics and AI 9, 262 (2023)
40. Lu, S., Chung, J.H., Velinsky, S.A.: Human-robot collision detection and identification based on
wrist and base force/torque sensors. In: Proceedings of the 2005 IEEE international Conference
on Robotics and Automation. pp. 3796–3801. IEEE (2005)
41. Lu, Y., Adrados, J.S., Chand, S.S., Wang, L.: Humans are not machines—anthropocentric
human–machine symbiosis for ultra-flexible smart manufacturing. Engineering 7(6), 734–737
(2021)
42. Ma, X., Qi, Q., Cheng, J., Tao, F.: A consistency method for digital twin model
of human-robot collaboration. Journal of Manufacturing Systems 65, 550–563 (2022).
https://doi.org/10.1016/j.jmsy.2022.10.012,https://www.sciencedirect.com/science/article/
pii/S0278612522001832
43. Malik, A.A., Bilb erg, A.: Digital twins of human robot collaboration in a production setting. Pro-
cedia Manufacturing 17, 278–285 (2018). https://doi.org/10.1016/j.promfg.2018.10.047,https:
//www.sciencedirect.com/science/article/pii/S2351978918311636
44. Maruyama, T., Ueshiba, T., Tada, M., Toda, H., Endo, Y., Domae, Y., Nakabo, Y., Mori, T.,
Suita, K.: Digital twin-driven human robot collaboration using a digital human. Sensors (Basel,
Switzerland) 21(24), 8266 (2021). https://doi.org/10.3390/s21248266

16 B. Gaffinet et al.
45. Matthias, B.: Industrial safety requirements for collaborative robots and applications. In: ERF
2014 Workshop: Workspace Safety in Industrial Robotics: trends, integration, and standards
(2014)
46. Meguenani, A., Padois, V., Da Silva, J., Hoarau, A., Bidaud, P.: Energy based control for safe
human-robot physical interaction. In: 2016 International Symposium on Experimental Robotics.
pp. 809–818. Springer (2017)
47. Mohammed, A., Schmidt, B., Wang, L.: Active collision avoidance for human–robot collabo-
ration driven by vision sensors. International Journal of Computer Integrated Manufacturing
30(9), 970–980 (2017)
48. Morinaga, S., Kosuge, K.: Collision detection system for manipulator based on adaptive
impedance control law. In: 2003 IEEE International Conference on Robotics and Automation
(Cat. No. 03CH37422). vol. 1, pp. 1080–1085. IEEE (2003)
49. Naudet, Y., Baudet, A., Risse, M.: Human digital twin in industry 4.0: Concept and preliminary
model. In: IN4PL. pp. 137–144 (2021)
50. Naudet, Y., Panetto, H., Yilma, B.A.: Towards cognitive interoperability in cyber-physical en-
terprises. In: 22nd IFAC World Congress, IFAC 2023. vol. Preprints of the 22nd IFAC World
Congress Yokohama, Japan, July 9-14, 2023 \copyright2023 the authors. Accepted by IFA
(2023), https://hal.science/hal-04189335, issue: 755-766
51. Neisser, U.: Cognitive psychology. New York, Appleton-Century-Crofts (1967), http://archive.
org/details/cognitivepsychol00neis
52. Nikolaidis, S., Shah, J.: Human-robot interactive planning using cross-training: A human team
training approach. In: Infotech@ Aerospace 2012, p. 2536. AIAA (2012)
53. Oyekan, J.O., Hutabarat, W., Tiwari, A., Grech, R., Aung, M.H., Mariani, M.P., L´opez-D´avalos,
L., Ricaud, T., Singh, S., Dupuis, C.: The effectiveness of virtual environments in develop-
ing collaborative strategies between industrial robots and humans. Robotics and Computer-
Integrated Manufacturing 55, 41–54 (2019). https://doi.org/10.1016/j.rcim.2018.07.006,https:
//www.sciencedirect.com/science/article/pii/S0736584517303150
54. Pandey, A.K., Alami, R.: Mightability maps: A perceptual level decisional framework for co-
operative and competitive human-robot interaction. In: 2010 IEEE/RSJ International Confer-
ence on Intelligent Robots and Systems. pp. 5842–5848. IEEE (2010)
55. Park, J.J., Lee, Y.J., Song, J.B., Kim, H.S.: Safe joint mechanism based on nonlinear stiff-
ness for safe human-robot collision. In: 2008 IEEE International Conference on Robotics and
Automation. pp. 2177–2182. IEEE (2008)
56. Pellegrinelli, S., Admoni, H., Javdani, S., Srinivasa, S.: Human-robot shared workspace collab-
oration via hindsight optimization. In: 2016 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS). pp. 831–838. IEEE (2016)
57. Ramasubramanian, A.K., Mathew, R., Kelly, M., Hargaden, V., Papakostas, N.: Digital twin for
human–robot collaboration in manufacturing: Review and outlook. Applied Sciences 12, 4811
(2022). https://doi.org/10.3390/app12104811
58. Semeraro, C., Lezo che, M., Panetto, H., Dassisti, M.: Digital twin paradigm:
A systematic literature review. Computers in Industry 130, 103469 (2021).
https://doi.org/10.1016/j.compind.2021.103469,https://www.sciencedirect.com/science/
article/pii/S0166361521000762
59. Shi, Y., Shen, W., Wang, L., Longo, F., Nicoletti, L., Padovano, A.: A cognitive digital
twins framework for human-robot collaboration. Procedia Computer Science 200, 1867–1874
(2022). https://doi.org/10.1016/j.procs.2022.01.387,https://www.sciencedirect.com/science/
article/pii/S1877050922003969
60. Tan, J.T.C., Arai, T.: Triple stereo vision system for safety monitoring of human-robot collab-
oration in cellular manufacturing. In: 2011 IEEE International Symposium on Assembly and
Manufacturing (ISAM). pp. 1–6. IEEE (2011)
61. Tonietti, G., Schiavi, R., Bicchi, A.: Design and control of a variable stiffness actuator for
safe and fast physical human/robot interaction. In: Proceedings of the 2005 IEEE international
conference on robotics and automation. pp. 526–531. IEEE (2005)
62. Unhelkar, V.V., Lasota, P.A., Tyroller, Q., Buhai, R.D., Marceau, L., Deml, B., Shah, J.A.:
Human-aware robotic assistant for collaborative assembly: Integrating human motion prediction
with planning in time. IEEE Robotics and Automation Letters 3(3), 2394–2401 (2018)
63. Vasic, M., Billard, A.: Safety issues in human-robot interactions. In: 2013 ieee international
conference on robotics and automation. pp. 197–204. IEEE (2013)
64. Wang, L., Gao, R., V´ancza, J., Kr¨uger, J., Wang, X.V., Makris, S., Chryssolouris, G.: Symbiotic
human-robot collaborative assembly. CIRP annals 68(2), 701–726 (2019)
65. Wang, L., Schmidt, B., Nee, A.Y.: Vision-guided active collision avoidance for human-robot
collaborations. Manufacturing Letters 1(1), 5–8 (2013)
66. Yamada, Y., Hirasawa, Y., Huang, S., Umetani, Y.: Fail-safe human/robot contact in the safety
space. In: Proceedings 5th IEEE International Workshop on Robot and Human Communication.
RO-MAN’96 TSUKUBA. pp. 59–64. IEEE (1996)
67. Yilma, B.A., Panetto, H., Naudet, Y.: A meta-model of cyb er-physical-social system: The CPSS
paradigm to support human-machine collaboration in industry 4.0. In: Camarinha-Matos, L.M.,
Afsarmanesh, H., Antonelli, D. (eds.) Collaborative Networks and Digital Transformation. pp.
11–20. IFIP Advances in Information and Communication Technology, Springer International
Publishing (2019). https://doi.org/10.1007/978-3-030-28464-02

Human-Centric Digital Twins: Advancing Safety and Ergonomics in HRC 17
68. Yin, Y., Zheng, P., Li, C., Wang, L.: A state-of-the-art survey on augmented reality-assisted
digital twin for futuristic human-centric industry transformation. Robotics and Computer-
Integrated Manufacturing 81, 102515 (2023). https://doi.org/10.1016/j.rcim.2022.102515,
https://www.sciencedirect.com/science/article/pii/S0736584522001971
69. Zhao, X., Pan, J.: Considering human behavior in motion planning for smooth human-robot
collaboration in close proximity. In: 2018 27th IEEE international symposium on robot and
human interactive communication (RO-MAN). pp. 985–990. IEEE (2018)
70. Zheng, X., Lu, J., Kiritsis, D.: The emergence of cognitive digital twin: vision, chal-
lenges and opportunities. International Journal of Production Research 60(24), 7610–
7632 (2022). https://doi.org/10.1080/00207543.2021.2014591,https://www.tandfonline.com/
doi/full/10.1080/00207543.2021.2014591