Human - Robot Complementarity
Learning each other and collaborating
MARIA DAGIOGLOU∗,National Center for Scientic Research ‘Demokritos’, Greece
STASINOS KONSTANTOPOULOS, National Center for Scientic Research ‘Demokritos’, Greece
As robot capabilities increase, the complexity of controlling and manipulating them becomes complex and cumbersome making
intuitive Human-Robot Interaction all the more necessary for seamless human-robot collaboration. In this paper, we look into the
ability of collaborators to understand each other’s intentions and act accordingly in order to promote the collaboration. We focus
on scenarios that human intentions are communicated through movement. In order to endow robots with understanding of human
intentions, as well as with robot behaviours that humans interpret correctly, we need to look at mechanisms humans recruit to perceive
and communicate intentions. We then need to distil the essence of these mechanisms so that they can be applied to completely
non-anthropomorphic robot collaborators.
Additional Key Words and Phrases: Human-Robot Collaboration, Human intentions, Robot intentions, Assistive teleoperation
Intuitive Human-Robot Interaction is necessary to allow human-robot collaboration in a seamless way. While the
capabilities of robots increase, the complexity of controlling and manipulating them becomes complex and cumbersome.
is is further stressed out in cases where the user is impaired and must be relieved from extreme physical or cognitive
Essential for achieving intuitive and seamless human-robot collaboration is the ability of both human and robot to
be able to understand each other’s intentions and act accordingly in order to promote the collaboration. In this paper
we present our research plan for developing methods that will endow robots with this understanding, allow robots to
behave in ways that humans interpret correctly, and enable shared control between a robot and a user to achieve a goal.
We are interested in applications where the medium of interaction is human movement, which guides the robot’s
actions either explicitly (e.g., assistive teleoperation) or implicitly (e.g., shared-workspace collaboration). To guarantee
seamless human-robot collaboration three high-level requirements must be satised:
(1) Communicate to humans the intentions of robots
(2) Communicate to robots the intentions of humans
Shared control that weighs intention of both sides, decides about the appropriate action and does not disrupt
human or robot behaviour.
To that end, it is of great interest to explore how a human and a robot can collaborate to learn a task in the rst
place. In other words, how do we prompt human motor learning (and thus performance) by involving the robot in the
process and how do we manipulate robot learning to take advantage of human expertise? Such a co-training gives the
∗e corresponding author: firstname.lastname@example.org
e research described in this paper has been carried out in the context of Roboskel, the robotics activity of Soware and Knowledge Engineering Lab,
Institute of Informatics and Telecommunications, NCSR “Demokritos”. Please see for more details: hp://roboskel.iit.demokritos.gr.
2 Maria Dagioglou and Stasinos Konstantopoulos
human and the robot an understanding of each other’s intentions and behaviour that would be extremely dicult to
2.1 Communicating robot intentions to humans
Humans could understand a robot’s intention by predicting its behaviour or even by using their own motor system to
understand intention in case of anthropomorphic robots. While interacting with a robot, a human is expected to build
an internal representation of the robot movement behaviour in a similar way the brain retains internal models of own
body (Wolpert and Ghahramani 2000), external objects (Cerminara et al
2009) and the world in general (Berkes et al
2011). ese internal representations allow prediction of own movements, movements of external objects, etc.
An interesting observation is that the control of anthropomorphic and non-anthropomorphic robots dier, possibly
because (in case of anthropomorphic robots) the user controls the robot as if it was their own body (Oztop et al
2015). It seems that in this scenario humans could recruit processes similar to those used in social interactions. In
social interactions humans can predict other people’s actions and intentions by observing among other things their
movements (Frith and Frith 2006). Wolpert et al. (2003) have suggested that we use our motor system to understand
actions and that this is an ecient mechanism for making inferences in social interactions. Internal modelling
processes have been proposed to be linked with mirror neurons during cognitive processes, like human interactions
and understanding intention of other (Miall 2003).
2.2 Communicating human intentions to robots
Translating human intention to appropriate signals for the robot could be perhaps a more complicated process. Research
relevant to communicating intention in physical interactions (Strabala et al
2012) could be used as a starting point for
reading human intention in human-robot interactions. In certain applications, like robotic assistive therapy, movement
intention could be provided by a brain-computer interface (Wairagkar et al. 2016).
We are mostly interested in exploring how human intention can be modelled in scenarios where the human is
controlling the robot via a medium like a joystick. To the best of our knowledge there are no recent papers discussing
this issue. In an older study (Horiguchi et al
2000), the authors explored naturalistic human-robot collaboration based
upon mixed-initiative interactions in a teleoperating environment.
2.3 Shared control and collaboration
roughout literature there are several interesting directions and challenges that one can take to build shared-control
systems that promote collaboration.
A rst question is how can we manipulate behaviour to facilitate intention inference? While humans can use motor
prediction and understand action intention to interact with robots, robots can be built to further facilitate that the
inference of intention unambiguously resolves to the correct robot goals. Movements are usually planned based on cost
functions related to shortest distance, minimum energy consumption etc. An interesting idea is to modify robot motion
planning so that it explicitly prompts human inferences about robot’s expected action (Dragan et al
to Dragan et al. (2013) motion planning in collaborative agents must be based in eligibility rather than predictability to
allow ‘intention reading’ at an early stage of robot’s motion execution. e authors argue that robots must be eligible
in all collaboration scenarios, including shared-workspace collaboration, robot learning and assistive teleoparation. In a
Human - Robot Complementarity 3
Fig. 1. A DrRobot Jaguar 4x4 robot can be teleoperated to climb a step.
human-robot collaborative study it was actually demonstrated that eligible robotic movements (rather than predictable
or functional ones) mediated more uent collaborations with humans (Dragan et al. 2015).
Another interesting idea is that of understanding each other’s capabilities interactively. Awais and Hernich (2013)
give a very interesting example of interaction based on human intentions. ey suggest a framework where the robot
starts reacting by copying human movements. With each repetition of the task the interaction improves by using history,
action randomness, and heuristic-based action predictions. Specically in the case of assistive robotic arms, a major
issue is how to collaborate with a dexterous and complicated robotic arm by only using a simple joystick. Mapping a
high-dimensional system to low dimensional inputs can denitely be a barrier for collaboration. Time-optimal mode
switching could create a level of shared control and allow user to express their intentions (Herlant et al. 2016).
3 RESEARCH PLAN
Taking into account the body of literature discussed above, we propose a research plan that assumes these ndings as
a starting point to develop the technical capabilities for communicating intentions and shared control. To make this
more concrete, we consider a specic collaborative task, and, namely, the task of teleoperating a rover robot to climb a
step (Figure 1). e robot’s wheels are big enough to clear the step, but some caveats are there: the operator needs to
approach the step at an appropriate angle and speed, or the platform will slam against the step rather than raise its
wheels over it; a sudden acceleration or sharp turn during the manoeuvre can overturn the robot; but the manoeuvre
cannot be performed slowly, as there comes a point when the wheels do not touch the ground and the platform needs
to have picked up enough speed to get on the step.
is simple experiment becomes more interesting when setup in a way that control is shared between the operator
and the robot: the robot uses its accelerometer to predict that some teleoperation has the potential to overturn it, and
should refuse to carry out the command. e operator, on the other hand, has the understanding of the dynamics
behind climbing a step that is tall enough to be possible but challenging.
e actual demonstration task that will be used to collect data will be to challenge novice users to teleoperate the
robotic platform up a step. Initial experiments have shown that managing to teleoperate the robot up the step in
Figure 1requires skilled manoeuvres, and that novice users can either operate the robot into dangerous inclinations or
4 Maria Dagioglou and Stasinos Konstantopoulos
fail to accomplish the task. On the other hand, it is not straightforward how to implement such an autonomous skill for
arbitrary, previously unseen steps and environments. We believe that this paradigm provides plenty of opportunities to
study human-robot collaboration in a series of studies:
A non-anthropomorphic robot uses movement behaviour to communicate intentions to the user. One such experi-
ment is to compare how well dierent movement behaviours communicate robot ‘objections’ to executing a
teleoperation command. For example, the robot might need to communicate a deviation from its plan (i.e., the
operator is moving away from the goal set) or a more urgent safety concern (i.e., the operator requested an
A robot understands user intention and correctly interprets teleoperation commands. One such experiment is to
develop and compare dierent strategies for interpreting the operator’s intentions rather than blindly executing
the detailed teleoperation commands. For example, the robot should use context and previous interactions with
the user to decide if the operator’s directive is to move against a step should be interpreted as a command to
climb the step, to park by it, or to slam against it. Experiments will include understanding the trade o between
making early and making safe predictions of the operators’ intention, and adapting to dierent operators.
Integrating the above into a shared control interface. ese, more ambitious, experiment will synthesize commu-
nicating robot objections and human operation into a collaboration that achieves the goal.
At the current state of development, we have identied a hardware, an environment, and a task that is impossible
for standard autonomous navigation, possible but challenging for experienced operators, and impossible for novice
operators. We are currently developing the safety controls that will allow novice users to try the task without risking
overturning robot, and will soon proceed to initial experiments along the lines described above.
M. Awais and D. Henrich. 2013. Human-Robot Interaction in an Unknown Human Intention Scenario. In 11th International Conference on Frontiers of
Information Technology. 89–94. DOI:hp://dx.doi.org/10.1109/FIT.2013.24
P. Berkes, G. Orb
an, M. Lengyel, and J. Fiser. 2011. Spontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment.
Science 331, 6013 (2011), 83–87. DOI:hp://dx.doi.org/10.1126/science.1195870
N. L. Cerminara, R. Apps, and D. E Marple-Horvat. 2009. An Internal Model of a Moving Visual Target in the Lateral Cerebellum. e Journal of physiology
587, Pt 2 (Jan. 2009), 429–42. DOI:hp://dx.doi.org/10.1113/jphysiol.2008.163337
A. Dragan, S. Bauman, J. Forlizzi, and S. Srinivasa. 2015. Eects of Robot Motion on Human-Robot Collaboration. In Human-Robot Interaction.
A. Dragan, K. Lee, and S. Srinivasa. 2013. Legibility and Predictability of Robot Motion. In Human-Robot Interaction.
C. D. Frith and U. Frith. 2006. How we Predict what Other People Are Going to Do. Brain Research 1079, 1 (2006), 36 – 46.
brainres.2005.12.126 Multiple Perspectives on the Psychological and Neural Bases of Understanding Other People’s Behavior.
L. V. Herlant, R. M. Holladay, and S. S. Srinivasa. 2016. Assistive Teleoperation of Robot Arms via Automatic Time-Optimal Mode Switching. In Proc. 11th
ACM/IEEE Intl Conf. Human Robot Interaction (HRI 2016). IEEE Press, 35–42. hp://dl.acm.org/citation.cfm?id=2906831.2906839
Y. Horiguchi, T. Sawaragi, and G. Akashi. 2000. Naturalistic human-robot collaboration based upon mixed-initiative interactions in teleoperating
environment. In Proceedings Intl Conf. Systems, Man, and Cybernetics, Vol. 2. 876–881. DOI:hp://dx.doi.org/10.1109/ICSMC.2000.885960
R. C. Miall. 2003. Connecting mirror neurons and forward models. NEUROREPORT 14 (2003), 2135–2137.
E. Oztop, E. Ugur, Y. Shimizu, and H. Imamizu. 2015. Humanoid Brain Science. CRC Press, Chapter 2.
K. Strabala, M. K. Lee, A. Dragan, J. Forlizzi, and S. Srinivasa. 2012. Learning the Communication of Intent Prior to Physical Collaboration. In Proceedings
of the 21st IEEE International Symposium on Robot and Human Interactive Communication.
M. Wairagkar, I. Zoulias, V. Oguntosin, Y. Hayashi, and S. Nasuto. 2016. Movement Intention Based Brain Computer Interface for Virtual Reality and So
Robotics Rehabilitation Using Novel Autocorrelation Analysis of EEG. In Proceedings of the 6th IEEE International Conference on Biomedical Robotics
and Biomechatronics (BioRob 2016). 685–685. DOI:hp://dx.doi.org/10.1109/BIOROB.2016.7523705
D. M. Wolpert, K. Doya, and M. Kawato. 2003. A unifying computational framework for motor control and social interaction. Philosophical Transactions of
the Royal Society of London 358 (2003), 593–602.
D. M. Wolpert and Z. Ghahramani. 2000. Computational Principles of Movement Neuroscience. Nature neuroscience 3 Suppl, november (Nov. 2000),