Human-machine collaboration in virtual reality for adaptive production engineering (presentation)

Abstract

Presentation of conference paper at FAIM 2017: Human-machine collaboration in virtual reality for adaptive production engineering

Full text

FAIM2017
27-30 June 2017 –University of Modena and RE
Human-machine collaboration in virtual reality
for adaptive production engineering
Andrea de Giorgio, KTH Royal Institute of Technology
Mario Romero, Mauro Onori, Lihui Wang

FAIM2017
27-30 June 2017 –University of Modena and RE Andrea de Giorgio
The laboratory is directed by prof. Lihui Wang, chair of
FAIM 2004 in Canada. More recently at FAIM 2015 with a
Panel Talk on “Cyber-Physical Systems and Cloud Robotics”.
Human-robot collaboratory at KTH, Sweden

FAIM2017
27-30 June 2017 –University of Modena and RE
Overview
•Human-machine collaboration
•Virtual reality (VR): Production vs gaming industry
•Key elements for immersive VR
•Issues on integration of VR in production software
•Human-machine collaboration in VR
•Cyber-physical systems and game engines
•Role of Unity in VR research
•Use case: Human-robot collaboration in VR
•Conclusions and future work
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Human-machine collaboration (HMC)
HMC is defined as “the coordinated interaction of human
and machine with a common goal.”
(Source: CIRP Encyclopedia of Production Engineering)
The ultimate goal of HMC is to increase the flexibility of
production, other than reducing the overall costs.
Flexibility is the quality of having options. We need a
production system that is adaptable to different scenarios.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Virtual reality: Production vs gaming industry
Virtual, mixed or augmented realities (VR/MR/AR) are not
new ideas.
Consumer-accessible VR/MR/AR devices are just born and
being produced at exponential rate.
Gaming industry is leading the way, with their game
engines and compatible devices.
Production industry is still exploring opportunities.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Key elements for immersive VR
•A virtual world
•Immersion
•Feedback
•Interactivity
•Participants
They can be associated to corresponding HMC challenges,
as a direct contribution from the gaming industry research.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Key elements for immersive VR: A virtual world
A virtual world is a collection of objects in a space
governed by rules and relationships, where the objects
are CAD models of industrial machines and other industrial
equipment that completes a factory scene.
In a game engine such as Unity, the rules are defined using
classes, which are called GameObjects, and the
relationships are defined by components and scripts that
can be attached to any GameObjects. All together these
objects form a virtual world.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Key elements for immersive VR: Immersion
Immersion refers to the status of the industrial operator
being involved in an activity within a virtual space, to an
extension that their mind is separated from the physical
space they are living in.
A game engine such as Unity provides full support for
HMD devices making the full immersive modality a default
setting for the simulation.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Key elements for immersive VR: Feedback
Feedback gives the operator the ability to observe the
results (outputs) of their activities (input) in the VR. The
standard feedback with HMD devices is visual/auditory,
but it has also become common the use of haptic devices
that can provide a sense of touch. Taste and smell remain
difficult to explore.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Key elements for immersive VR: Interactivity
The ability to interact with the virtual world is
fundamental. Sensors and devices allow to capture the
operator’s body actions and transform them into virtual
actions. Navigation and direct manipulation are the two
main aspects of an immersive virtual reality.
HMDs, e.g. the HTC Vive, are often sold in bundle with
some controllers that operators can hold in their hands.
Voice and gesture commands can also be captured by
microphones and cameras mounted in the environment.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Key elements for immersive VR: Participants
Human operators are an essential element of the VR
experience. They can be grouped by experience and
offered different VR representations based on their
capacity to interact with the virtual objects. A good
advantage of the VR is that it allows an operator to be
trained simply by using it. This is achieved by structuring
the virtual world in different levels that are called, in turn,
each time that the operator acquires enough knowledge to
perform more complex operations. Exactly as a player
advances level by level in a videogame.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Issues on integration of VR in production software
•Production simulations are based on proprietary
software for which support from the VR devices
producers is not offered.
•CAD models that are mainly used are not VR friendly. In
fact, graphic engines for VR make use of scene-graphs
for visualization (e.g. Openscenegraph, OpenSG or
OpenGL Performer), which are hierarchical data
structures (triangulated mesh geometry, spatial
transforms, lighting, material properties, etc.).
Renderers provide methods to visualize these data
structures at interactive frame rates.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Issues on integration of VR in production software
•Production simulations are often not meant to be
executed remotely, while games need to exchange the
common virtual world details in real time with the
players connected from different locations, therefore
they are optimized to share only the strictly-needed
information. Cloud-based software can store on servers
all the simulation variables, or even entire models, but
this doesn’t mean that they can be effectively retrieved
and exploited for local interactive visualizations.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Human-machine collaboration in VR
The interaction between operators and machines can now
happen in two different cases:
•When the machine is directly operated (eventually with
support of AR information);
•When the digital twin (DT) of a machine is operated by
a projection of the operator in the virtual environment.
A challenge lies in addressing conflicts when the real and
virtual operations are not aligned, e.g. two different
operators act on the same machine (a real one and its DT).
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Cyber-physical systems (CPS) and game engines
The concept of digital twins (DTs) that is becoming more
and more popular for CPS can be modelled around the
already existing game objects, by adding scripts that
transmit bidirectional information with the real devices.
The challenge is not reinventing digital models that can be
exploited in VR, but either integrating the existing ones
into production software, or production software into
game engines.
Ultimately the question is: do we need accurate
simulations when interacting with CPS in a virtual
environment?
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Unity leads research on VR/MR/AR
As an example, at FAIM 2017 on 270 accepted papers:
•9 mention Unity (3 out of 5 papers just in this session)
•none mention Unreal
•none mention cryENGINE
•none mention GameMaker
•…
Unity is easy to learn and use, free for non-commercial use
(R&D) and has small fees for commercial use. It also has
partnerships with all major VR/MR/AR device producers.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Use case: Human-robot collaboration in VR (1)
We have implemented a digital twin of an ABB IRB 120
industrial robot manipulator using the game engine Unity.
The DT in Unity is controlled by using an HTC Vive head
mounted display and two hand controllers that are
connected to the computer running Unity.
The collaborative task is a typical pick and place, where the
operator is directly in charge of the planning and the robot
performs it, with the advantage of having the super-
human strength to deal with very heavy objects.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Use case: Human-robot collaboration in VR (2)
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Use case: Human-robot collaboration in VR (3)
The operator has two tasks to execute:
•Define a good (non necessarily optimal) trajectory for
the end effector;
•Replace the typical image analysis and/or machine
learning algorithms to choose successful pick and place
directions and forces.
Whatever an operator does can be fine-tuned using simple
algorithms and/or machine learning methods. For
example, the trajectory can be recomputed to become
energy or time efficient and feasible w.r.t . the inverse
kinematics of the robot.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Use case: Human-robot collaboration in VR (4)
Since the digital twin of the robot manipulator is
controlled by a human arm, the degrees of freedom (read
number of joints) are different. The two different vectors
of joint positions are mapped with a machine learning
function trained with genetic algorithms.
The end effector (EE) position is defined by the operator’s
hand position and the inverse kinematics for the robot
obtained from the machine-learned function.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Inverse kinematics issues (1)
The kinematics algorithm solves the following issues:
1. Evaluate the relative EE position corresponding to the
operator’s arm extension from the headset;
2. Adapt the robot space to the reachability of the
operator’s arm extension;
3. Let the robot move to configurations which the
operator’s arm cannot possibly mimic with their
degrees of freedom by readapting and smoothing the
robot movements, given the operator’s arm position.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Inverse kinematics issues (2)
The kinematics algorithm solves the following issues:
1. Evaluate the relative EE position corresponding to the
operator’s arm extension from the headset.
Solution: Tactile feedback in the form of vibration is
provided to the operator whenever the Inverse Kinematic
(IK) algorithm is unable to compute the target
configuration. The operator can easily learn from that what
is the available space.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Inverse kinematics issues (3)
The kinematics algorithm solves the following issues:
2. Adapt the robot space to the reachability of the
operator’s arm extension.
Solution: Valid trajectories are calculated between the
solutions provided by the IK algorithm. If an operator’s
movement reaches a singularity, the robot switches to
another configuration to continue the movement. The
perceived discontinuity doesn’t matter because the final
trajectory will be optimized after the full movement.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Inverse kinematics issues (4)
The kinematics algorithm solves the following issues:
3. Let the robot move to configurations which the
operator’s arm cannot possibly mimic with their degrees of
freedom by readapting and smoothing the robot
movements, given the operator’s arm position.
Solution: The problem is solved by allowing the user,
rather than the robot, to reposition themselves. The robot
follows the relative movement of the controller, but only
when the operator is pressing a trigger button.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Other improvements (1)
•The speed of the robot can be slowed down w.r.t. the
operator movements in order to improve accuracy. This
is controlled by the operator who can press a button
with pressure sensor on the hand controller:
fully pressed = same speed
lower pressure = robot speed reduced w.r.t. operator speed
•Degrees of freedom reduction. The wrist of the robot
can be locked in a specific position in order to improve
the accuracy of the pick and pose approach trajectory
from the operator’s arm input.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Other improvements (2)
•1st person perspective. A teach
pendant can be typically used to
manually move the robot, but it
only allow for the robot
coordinate system perspective to
be taken care of. In VR, the
Andrea de Giorgio
operator’s relative coordinate system can be exploited
to allow their arm movements to correspond exactly to
the robot movements from the operator’s perspective,
as our human brain would logically assume. A hard task
becomes “intuitive”.

FAIM2017
27-30 June 2017 –University of Modena and RE
Conclusions and future work (1)
•The introduction of an immersive VR in HMC can shift
the operator’s actions toward more intuitive actions.
Learning is easier, production is faster. The hybrid
approach between real and virtual environments needs
to be explored further.
•Integration of VR equipment in production industry
cannot simply ignore the advancements in gaming
industry, as silly as the mixture can sound. The use of
cyber-physical systems is the perfect scenario where a
converging approach between digital twins and game
objects is tested.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Conclusions and future work (2)
•HRC is just one special case of HMC that can be tested
with advancements coming from the gaming industry
and the arrival of the IoT allows us to consider every
device an independent machine that can be modeled
in gaming software, other than common simulation
software. Also, machine to machine collaboration, or
simply coordination, can be explored from such
perspective. Future scenarios can depict remote
production, remote learning, immersive production
design and real-time adaptive production, where both
human and machines are virtually interacting.
Andrea de Giorgio

FAIM2017
27-30 June 2017 –University of Modena and RE
Thank you!
Andrea de Giorgio
KTH Royal Institute of Technology, Sweden
Connect with me on andreadegiorgio.com
andreadg@kth.se