Content uploaded by Chuang Yu
Author content
All content in this area was uploaded by Chuang Yu on Jun 16, 2023
Content may be subject to copyright.
Robot Theory of Mind with Reverse Psychology
Chuang Yu
Cognitive Robotics Lab,
The University of Manchester, UK
chuang.yu@manchester.ac.uk
Baris Serhan
Cognitive Robotics Lab,
The University of Manchester, UK
baris.serhan@manchester.ac.uk
Marta Romeo
Cognitive Robotics Lab,
The University of Manchester, UK
marta.romeo@manchester.ac.uk
Angelo Cangelosi
Cognitive Robotics Lab,
The University of Manchester, UK
angelo.cangelosi@manchester.ac.uk
Figure 1: The pipeline of robot theory of mind with psychology. Player
𝑃1
and the robot work together as a team while player
𝑃2
plays by itself. Based on reinforcement learning, the robot learns how to decide which suggestions to give for a better team
performance. We assume that the robot knows the actions of
𝑃2
. However, player
𝑃1
is unaware that the robot has this extra
knowledge. When player
𝑃1
possesses a false belief in the robot performance, corresponding to a low trust level, the optimal
robot policy uses reverse psychology to give opposite advice to encourage
𝑃1
to do what the robot desires for a better team
performance.
ABSTRACT
Theory of mind (ToM) corresponds to the human ability to infer
other people’s desires, beliefs, and intentions. Acquisition of ToM
skills is crucial to obtain a natural interaction between robots and
humans. A core component of ToM is the ability to attribute false
beliefs. In this paper, a collaborative robot tries to assist a human
partner who plays a trust-based card game against another human.
The robot infers its partner’s trust in the robot’s decision system via
reinforcement learning. Robot ToM refers to the ability to implicitly
anticipate the human collaborator’s strategy and inject the predic-
tion into its optimal decision model for a better team performance.
In our experiments, the robot learns when its human partner does
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specic permission
and/or a fee. Request permissions from permissions@acm.org.
HRI ’23 Companion, March 13–16, 2023, Stockholm, Sweden
©2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9970-8/23/03. . . $15.00
https://doi.org/10.1145/3568294.3580144
not trust the robot and consequently gives recommendations in its
optimal policy to ensure the eectiveness of team performance. The
interesting nding is that the optimal robotic policy attempts to use
reverse psychology on its human collaborator when trust is low.
This nding will provide guidance for the study of a trustworthy
robot decision model with a human partner in the loop.
CCS CONCEPTS
•Human-centered computing
→
Interaction design;•Com-
puter systems organization →Robotics;
KEYWORDS
robot theory of mind, reverse psychology, human-robot trust
ACM Reference Format:
Chuang Yu, Baris Serhan, Marta Romeo, and Angelo Cangelosi. 2023. Ro-
bot Theory of Mind with Reverse Psychology. In Companion of the 2023
ACM/IEEE International Conference on Human-Robot Interaction (HRI ’23
Companion), March 13–16, 2023, Stockholm, Sweden. ACM, 3 pages. https:
//doi.org/10.1145/3568294.3580144
HRI ’23 Companion, March 13–16, 2023, Stockholm, Sweden Chuang Yu, Baris Serhan, Marta Romeo, and Angelo Cangelosi
1 INTRODUCTION
Theory of mind (ToM) relates to the ability to attribute mental
states to ourselves and others. Human ToM plays a crucial role
in cognitive development and natural social interaction [
3
]. To
build advanced intelligent robot partners, researchers are trying to
empower robots with this cognitive skill [
1
,
8
,
9
]. With the human in
the loop, Romeo et al. [
8
] explored how robot ToM inuences human
decision-making in a challenging maze game. Without the human
in the loop, Chen et al. [
1
] built an AI observer to study the visual
behavior modeling for robot ToM. Their AI observer could predict a
robot actor’s future trajectory only given one image frame showing
the actor robot’s initial scene. ToM can be formalized as inverse
reinforcement learning [
5
]. Reinforcement learning (RL) tries to
nd the optimal policy with the guidance of a reward function. In
contrast, based on the observed behavior history of the agent or its
policy, inverse reinforcement learning (IRL) recovers the reward
function of the agent. In [
5
], the reward function, policy, and world
model in RL are mapped respectively to desires, intentions, and
beliefs in the ToM model. Rabinowitz et al. [
7
] came up with an
articial ToM model, namely ToMnet, which uses meta learning
to learn the policy of other agents through their behavior history.
The ToMnet model implicitly revealed the agent’s false beliefs (a
vital component of human ToM) about the world.
Reverse psychology refers to a manipulative behavior that af-
fects another individual’s behavior as it desires. By using reverse
psychology, an agent tries to make the interactor behave as wanted
by advocating an opposite behavior to the desired one. Guo et al.
[
4
] show the phenomenon in which the optimal robot policy tries
to exploit reverse psychology in a reconnaissance mission within
a human-robot team. The paper also proposes two trust-behavior
models and investigates how the dierent models aect a robot’s
optimal policy and HRI team performance.
In our paper, we explore robot ToM through reverse psychology
in a multiagent “Cheat game” scenario. From our experiment re-
sults, we found that when player
𝑃1
has a false belief in the robot
performance, corresponding to a low trust level, the optimal robot
policy always uses reverse psychology to give the opposite advice
to encourage player
𝑃1
to do what the robot desires for a better team
performance. This implicitly certies that the robot successfully
learned the false belief state of its human partner, a core component
of ToM.
2 METHODS
The “Cheat game” experimental setting is shown in Fig. 1. The
“Cheat game” is a card game with multiple players who try to get
rid of all their cards. In each round, one player discards one or more
cards face down and declares the number and the rank of the cards.
They can either tell the truth or lie about the rank and number of
cards they are playing. At this point, the opponents can decide to
call “cheat” on the hand that was just played. If “cheat” is called,
the hand is revealed and, if the player who had declared it lied, they
collect all the cards on the table. If they did not lie, whoever called
“cheat” collects the cards on the table. The game ends when one
player wins the game by managing to get rid of all of their cards.
We present a modied version with one robot and two human
players,
𝑃1
(co-playing with the robot) and
𝑃2
. At the start, each
player has 12 cards randomly chosen from a whole deck of cards.
Player
𝑃1
collaborates with the robot as a team during the multia-
gent game. The robot will give its advice (whether to call cheat or
not on
𝑃2
) to the human player
𝑃1
, guided by the maximal collabora-
tion performance. We assume that the robot knows the actions of
𝑃2
but
𝑃1
does not know that the robot has this additional knowledge.
The robot policy can be modeled as a POMDP (Partially Observ-
able Markov Decision Process). The states are: the card situation
of
𝑃1
hand (for example, how many cards that
𝑃2
claimes and
𝑃1
has), the number of the claimed card played by
𝑃2
, the action of
𝑃2
(cheat or not cheat), and the belief (belief is modeled with shape
parameters of beta distribution used in [
4
]) on
𝑃1
trust on the robot
performance.
𝑃1
trust in the robot is continuous in the range of
0 to 1. As
𝑃1
trust cannot be directly observed by the robot, the
robot models a belief state of trust based on its performance. We
dene the world state
𝑠∈𝑆
, the robot action
𝑎𝑅∈𝐴𝑅
and player
𝑃1
action
𝑎𝑃1∈𝐴𝑃1
. The state changes with a transition probability
𝑃𝑠′|𝑠, 𝑎𝑅, 𝑎𝑃1
. After the state changes, a reward
𝑟𝑠, 𝑎𝑅, 𝑎𝑃1, 𝑠 ′
is received. The robot reward function is based on the change of
the number of cards in 𝑃1and 𝑃2’s hands in each round of game.
We developed a simulation for the multiagent cheat game to
record data in order for the robot to learn the optimal policy to
give recommendations to maximise the team performance. In the
simulation,
𝑃1
and
𝑃2
are simulated human players. The behavior
of
𝑃1
is modeled with a trust dynamics model and a trust-based
policy model [
2
,
4
]. During the simulation for the data recording,
the robot player always performed random actions (gave random
recommendations). In total, 8000 games were recorded. The dataset
was divided in 4800 for training, 1600 for validation, and 1600
for testing. Oine RL algorithms can eectively learn a policy
from previously-collected static datasets without further interac-
tion, which is convenient for RL model in human-robot interaction.
Hence, this paper used the oine reinforcement learning model,
namely the Conservative Q-Learning (CQL) [
6
] to learn the optimal
robot policy. The CQL model minimizes the action-values under
the current policy and maximizes values under data distribution to
overcome the underestimation issue.
3 RESULTS
Our oine RL model is completed with the
𝑑
3
𝑟𝑙 𝑝𝑦
library [
10
].
After training the oine reinforcement learning model, the robot
learns how to decide which suggestion to give to maximize the
team performance. We tested our trained RL model (trained for
3963 epoches) in the simulation.
𝑃2
plays the same actions as in
the random policy case, in order to compare the dierence of team
performance between a random policy and optimal policy in the
same setting. The results are shown in Figure 2. The results show
that the optimal robot decision model (robot CQL policy) indeed
uses reverse psychology for a better team performance. For team
performance, the robot CQL policy gets an accuracy of 77.3% while
the random model 63.3%. We dene as accuracy how many times
𝑃1
guesses correctly the actions of
𝑃2
, after hearing the robot rec-
ommendation. When
𝑃2
cheats and
𝑃1
has low trust in the robot
(trust < 0.5), the optimal robot model mostly suggests to
𝑃1
not to
call “cheat” and
𝑃1
does not follow the robot and nally get the
right guess on
𝑃2
action (nearly 400 times). However, the robot
Robot Theory of Mind with Reverse Psychology HRI ’23 Companion, March 13–16, 2023, Stockholm, Sweden
Figure 2: The testing results on trained RL policy and random policy.
𝐴𝑐𝑡𝑖𝑜𝑛_𝑃
2”cheat” or ”not cheat” refers to whether
𝑃2
cheats on
𝑃1
.
𝐴𝑐𝑡𝑖𝑜𝑛_𝑃
1”cheat” or ”not cheat” refers to whether
𝑃1
calls ”cheat” or not. The reverse psychology phenomenon in
RL model testing happens when
𝑃1
trust is low. For example, in the top left subgure, the robot mostly advises not to cheat, and
𝑃1chooses to cheat.
random policy (not trained on the aquired dataset) does not use
reverse psychology in the same situation. When
𝑃2
cheats and
𝑃1
has high trust in the robot (trust > 0.5), the robot does not use
reverse psychology and advises to call “cheat”,
𝑃1
mostly follows
the advice in this case (more than 400 times). Similar results are
reached when 𝑃2does not lie when playing their hand.
4 DISCUSSION AND FUTURE WORKS
In this paper, we certied the existence of reverse psychology in
an optimal robot policy during a multiagent trust-based card game.
When the human trust in the robot is low, the optimal robot policy
using ToM uses reverse psychology for a better team performance.
However, this might be hazardous to long-term human-robot inter-
action. If the human interactor realizes that reverse psychology is
being used trust will collapse. We should take trust into account in
the robot reward function to avoid this collapse, which in turn can
damage the team’s performance. Hence, the robot model should
aim to strike a balance between trust maintenance and team per-
formance. How to reach this balance will be explored in future
work. Finally, this paper’s limitation is that all results are based on
simulated data. We will explore human-joined experiments in the
future.
ACKNOWLEDGMENTS
This work was funded and supported by the UKRI TAS Node on
Trust (EP/V026682/1) and the project THRIVE/THRIVE++ (FA9550-
19-1-7002).
REFERENCES
[1]
Boyuan Chen, Carl Vondrick, and Hod Lipson. 2021. Visual behavior modelling
for robotic theory of mind. Scientic Reports 11, 1 (2021), 1–14.
[2]
Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha Srini-
vasa. 2020. Trust-aware decision making for human-robot collaboration: Model
learning and planning. ACM Transactions on Human-Robot Interaction (THRI) 9,
2 (2020), 1–23.
[3] Jerry A Fodor. 1992. A theory of the child’s theory of mind. Cognition (1992).
[4]
Yaohui Guo, Cong Shi, and Xi Jessie Yang. 2021. Reverse psychology in trust-
aware human-robot interaction. IEEE Robotics and Automation Letters 6, 3 (2021),
4851–4858.
[5]
Julian Jara-Ettinger. 2019. Theory of mind as inverse reinforcement learning.
Current Opinion in Behavioral Sciences 29 (2019), 105–110.
[6]
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva-
tive q-learning for oine reinforcement learning. Advances in Neural Information
Processing Systems 33 (2020), 1179–1191.
[7]
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and
Matthew Botvinick. 2018. Machine theory of mind. In International conference on
machine learning. PMLR, 4218–4227.
[8]
Marta Romeo, Peter E McKenna, David A Robb, Gnanathusharan Rajendran,
Birthe Nesset, Angelo Cangelosi, and Helen Hastie. 2022. Exploring Theory of
Mind for Human-Robot Collaboration. In 2022 31st IEEE International Conference
on Robot and Human Interactive Communication (RO-MAN). IEEE, 461–468.
[9]
Brian Scassellati. 2002. Theory of mind for a humanoid robot. Autonomous Robots
12, 1 (2002), 13–24.
[10]
Takuma Seno and Michita Imai. 2022. d3rlpy: An Oine Deep Reinforcement
Learning Library. Journal of Machine Learning Research 23, 315 (2022), 1–20.
http://jmlr.org/papers/v23/22-0017.html