Detecting Deception in Natural Environments Using Incremental Transfer Learning

Full text

Detecting Deception in Natural Environments Using Incremental
Transfer Learning
Muneeb Imtiaz Ahmad Abdullah Alzahrani Sunbul M. Ahmad
Swansea University Swansea University Cardi University
Swansea, UK Swansea, UK Cardi, UK
m.i.ahmad@swansea.ac.uk 2043528@swansea.ac.uk ahmads31@cardi.ac.uk
ABSTRACT
Existing work on detecting deception has mainly relied on collect-
ing datasets evolving from contrived user interactions. We argue
that naturally occurring deception behaviours can inform more
reliable datasets and improve detection rates. Therefore, in this
paper, we discuss the ndings of two experiments which enabled
participants to freely and naturally engage in deceptive and truthful
behaviours in a game environment. We collected physiological and
oculomotor behaviour (PB, & OB) data including electrodermal
activity, blood volume pulse, heart rate, skin temperature, blinking
rate, and blinking duration during the deceptive and truthful states.
We investigate the changes in both PB and OB across repeated inter-
actions and explore the potential of incremental transfer learning
in detecting deception. We found signicant dierences in elec-
trodermal activity, and skin temperature between deception and
non-deception groups in both studies. The incremental transfer
learning method with a logistic regression classier detected decep-
tion with 80% accuracy, outperforming previous research. These
results highlight the importance of collecting data from multiple
sources and promote the use of incremental transfer learning to
accurately detect deception in real time.
CCS CONCEPTS
• Human-centered computing
→
Human Robot interaction ;
User studies; • Computer systems organization → Robotics.
KEYWORDS
Deception, Measurement, Dataset, Physiological and oculomotor
behaviours, Blu Game, Human-Robot Game Interaction
ACM Reference Format:
Muneeb Imtiaz Ahmad, Abdullah Alzahrani, and Sunbul M. Ahmad. 2024.
Detecting Deception in Natural Environments Using Incremental Transfer
Learning. In INTERNATIONAL CONFERENCE ON MULTIMODAL INTER-
ACTION (ICMI ’24), November 04–08, 2024, San Jose, Costa Rica. ACM, New
York, NY, USA, 10 pages. https://doi.org/10.1145/3678957.3685702
1 INTRODUCTION
Deception can be dened as an agent acting or speaking to induce a
false belief in a target or victim [28]. Deception detection has been
This work is licensed under a Creative Commons Attribution International
4.0 License.
ICMI ’24, November 04–08, 2024, San Jose, Costa Rica
© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0462-8/24/11
https://doi.org/10.1145/3678957.3685702
widely studied across many elds such as psychology, social science,
criminology and neuroscience due to its pervasive nature in human
interactions and implication in many social contexts [21].
The traditional approach to detecting deception has mainly con-
sidered polygraph tests that extract physiological measurements
such as heart rate, respiration rate, skin conductance, and skin
temperature [
19
]. Researchers have raised concern about the use
of these measures to reliably and non-invasively detect deception.
Further, ndings suggest that trained individuals can trick the sys-
tem resulting in bias and error [
27
]. However, we argue that due
to advancements in wearable technology design (see [
17
,
55
]), im-
proved machine learning classication methods for deception detec-
tion [
18
], and combining multiple psychophysiological indicators
or using a hybrid approach [
6
], current devices can reliably and
non-invasively collect various psychophysiological indicators in
real-time to support deception detection. For instance, together
with physiological measurements, studies have shown that blink
duration and count can be useful for the detection of deception
[
22
,
40
,
43
]. Recent work by George et al
. [22]
has shown that par-
ticipants’ blink duration and count were signicantly higher in the
deception condition. Similarly, Marchak
[40]
has found that train-
ing machine learning classiers on blink rate and response time
can help identify deceptive and non-deceptive behaviours. Conse-
quently, we consider both oculomotor and physiological behaviours
(OB, & PB) for detecting deception.
While current work has considered hybrid approaches, the work
on the collection of data to detect deception has mostly considered
rather unnatural or articially created tasks such as truthfully and
quickly answering general questions, interview questions of dif-
ferent categories, or analysing video interactions [
6
,
8
,
24
,
47
,
49
].
Consequently, the existing dataset resulting from these interactions
lacks collecting data on deception behaviour in a natural way. Sim-
ilarly, recent ndings have highlighted the limitations of relying
on laboratory-created lies to study human lie detection and have
called for researching natural means to study human deception
detection [
53
]. Alaskar et al
. [6]
conducted a comprehensive review
on machine-intelligence techniques for detecting deception and
concluded that available datasets are not diverse as they are col-
lected in simulated environments which are not realistic to train
the deceptive and truthful behaviour, have been based on limited
participants, and, have used static questions as a task.
Another aspect lies in investigating the changes in deception
behaviour during repeated interaction and understanding how OBs
and PBs of deception change over time. Further, how investigat-
ing changes in deception behaviour over time can enrich datasets.
Existing research on collecting datasets has also used one-o inter-
actions, thus, the change in deception behaviour during repeated

ICMI ’24, November 04–08, 2024, San Jose, Costa Rica Ahmad et al.
interaction has also not been studied [
6
]. We see evidence from
deception research on repeated interviews that liars are not less
consistent than truth tellers [
41
]. Thus, it becomes an important
question to understand how deception behaviour naturally evolves.
Furthermore, most existing datasets have been collected under
a single context. Recent research has shown that the context in
which deception occurs can inuence the likelihood and type of
deception [
59
]. Therefore, applying incremental transfer learning
to detect deception across multiple contexts is important. However,
due to most datasets being collected within a single context and to
the best of our knowledge, the role of incremental transfer learning
to detect deception has not been explored yet.
Considering these aspects, this paper uniquely employs the
GAME (Game As a Method of eliciting Emotions) paradigm pro-
posed by Ahmad et al
. [5]
, Shahid
[48]
to collect data on naturally
occurring deception behaviour. We consider collaborative and com-
petitive interaction game contexts and investigate both PBs and OBs
during naturally occurring deception behaviour. Lastly, as social
robots have begun to take on dierent social yet professional roles
such as an interviewer [
4
,
30
], or a teacher [
36
], or a therapist [
11
]
or a detective [
24
], we consider the Human-robot game interac-
tion context and foresee a future where robots detect deception in
real-time. The paper investigates the following research questions
(RQs):
•
RQ1: How do naturally occuring human OBs and PBs dier be-
tween deception and truthful states during interactions with a
robotic agent?
•
RQ2: How do naturally occuring human OBs and PBs evolve as
individuals gain experience during repeated interactions?
•
RQ3: Do collecting data on OBs and PBs during truthful and de-
ceptive states naturally improve the deception detection accuracy?
•
RQ4: Which naturally occuring human OBs and PBs are predic-
tive of deception and truthful behaviours?
To investigate these RQs, we conducted two experiments that
tasked participants to play a game involving showing natural in-
stances of both deception and truthful behaviour to and with a
NAO robot. We recorded both OBs and PBs, including electroder-
mal activity (EDA), blood volume pulse (BVP), heart rate (HR), skin
temperature (SKT), blinking rate (BR), and blinking duration (BD)
to detect humans’ deception in real-time. The novel contributions
of this paper are as follows:
•
Using an incremental transfer learning technique, we have shown
that data gathered on (OB) and (PB) in natural interactions can ef-
fectively dierentiate between deceptive and truthful states with
an accuracy rate of 80%. This method surpasses most existing tech-
niques and highlights the potential for reliable deception detection
and development of adaptive robotic systems.
•
We show that repeated exposure to the same deceptive scenarios
can lead to habituation and emotional desensitisation, resulting in
fewer physiological changes and consistent behaviours.
•
We share the study materials and evolving datasets with the
community to advance knowledge on deception which can be found
here.
2 BACKGROUND
Theoretical Knowledge on Deception - Numerous theories
have been proposed to explain why individuals engage in deceptive
behaviours [
9
,
62
]. We focus on the four-factor theory of deception
and the interpersonal theory of deception, both suggesting that
liars exhibit increased general arousal, emotional load, cognitive
load, and attempts at control and impression management to appear
honest. These conditions can lead to changes in verbal and non-
verbal behaviours, such as increased blinking and pupil dilation,
heightened voice pitch, speech errors, pausing, and other speech
hesitations, ultimately aecting physiological behaviours like SKT,
EDA, and HR. Additionally, we consider the truth default [
38
] and
interpersonal theory, which justify lying for reasons such as goal
attainment, where honesty is seen as counterproductive. Thus,
using games that naturally create situations requiring deception to
win is considered.
Methods for Detecting Deception - Broadly, three methods have
been applied to detect deception: 1) psychological, 2) professional,
and 3) computational [
6
]. Psychological approaches examine the
relationship between nonverbal and verbal behaviours and the act
of lying, including physiological changes such as increased pupil
size and higher-pitched voices [
14
]. Studies have also identied
unique hand movements and speech-related gestures as indicators
of deception, using measures of facial expression smoothness and
asymmetry to connect them to deceptive acts [1].
Physiological techniques, including polygraph tests and fMRI,
are historically considered limited due to their need for complex
setups and skilled operators [
50
]. However, recent advancements
in hardware design and technology may challenge these limita-
tions [
17
,
55
]. Lastly, signicant advancements in data mining and
machine learning algorithms have led to the rise of computational
methods [
6
]. These methods analyse micro-expressions, voice stress,
heart rate, skin activity, and breathing patterns to detect deception.
Interdisciplinary research continues to rene these methods, aiming
to develop reliable tools for various settings. Despite advancements,
existing work is often limited to datasets collected under a single
context and does not use increment transfer learning to detect de-
ception [
6
]. This paper addresses this gap by collecting datasets
from two experiments and applying transfer learning to investigate
the accuracy of deception detection.
Datasets on Detecting Deception - Various methods have been
used to create datasets for detecting deception, encompassing three
primary categories: verbal, non-verbal, and hybrid approaches. In
verbal methods, researchers have leveraged features such as text
sequences [
31
], linguistic attributes [
13
], sparse elements [
60
], and
acoustic characteristics [
57
,
61
]. Non-verbal methods involve fea-
tures such as EEG signals [
7
], facial expressions [
33
,
54
], and micro
eye movements [
34
]. Hybrid methods, combine dierent feature
categories to create comprehensive datasets. These hybrid datasets
encompass a range of features, including visual and vocal charac-
teristics [
32
], MFCCs (Mel-Frequency Cepstral Coecients), and
statistical measures of speed, pitch, and loudness [
23
], as well as
facial expressions and body motions extracted from videos [
15
],
and integrated data from videos, audios, EEG, and gaze tracking
[
25
,
35
]. Some hybrid datasets combine EEG, video, audio, and gaze
data, while others integrate gaze and speech features [20, 42].

Detecting Deception in Natural Environments Using Incremental Transfer Learning ICMI ’24, November 04–08, 2024, San Jose, Costa Rica
In summary, many studies resulting in datasets to detect decep-
tion demonstrate low accuracy rates, often below 70% [
6
]. Deception
detection models are frequently trained on datasets lacking diver-
sity and realism, collected in simulated environments that may not
accurately reect real-world deceptive behaviours. These datasets
typically involve a limited number of subjects, and participants
are often asked static questions, restricting response authenticity.
Additionally, there are conicting ndings on whether verbal or vi-
sual cues are more important in detecting deception. Psychological
studies suggest verbal cues are primary, while professional inves-
tigators and deep learning models emphasize visual cues. These
inconsistencies highlight the challenges in developing eective de-
ception detection methods. Therefore, we create a dataset based on
naturally occurring deceptive behaviour and use a hybrid approach
combining physiological and oculomotor behaviour from a large
number of participants. We apply an incremental transfer learning
algorithm to detect deception [12].
Game as a Method to Detect Deception -Researchers have
used various games to study deception behaviour, including Cheap-
Talk Games signalling game [
45
], iterated prisoner’s dilemma [
52
],
and the Maa party game [13]. Signalling games involve strategic
communication where one party sends a message, and the other
decides if it is truthful or deceptive [
45
]. The iterated prisoner’s
dilemma allows players to repeatedly choose cooperation or defec-
tion, with opportunities for deception to gain higher payos [
52
].
The Maa party game involves players with secret roles lying to
conceal their identity and objectives [
13
]. The Maa party game
has been widely used to create video-based datasets for detecting
deceptive behaviours [
13
,
29
]. Additionally, Bag of Lies is a game-
based approach where participants describe images honestly or
deceptively [25].
However, researchers have identied several shortcomings in the
Maa party game, including being context-specic [
29
], imposing
a high cognitive load on players that may impact their ability to
deceive naturally [
29
,
50
], and oering low classication accuracy
in identifying deceptive players in the real world [
25
]. In this paper,
we introduce a simple card game known as the Blu game [
56
],
inspired by Ahmad and Alzahrani
[2]
, to study the truthfulness of
the robot. The game allows players to depict truthful and decep-
tive behaviours without a high cognitive load, making it ideal for
studying deception behaviours. The cognitive load in a game can
vary depending on factors such as the number of players, game
complexity, and player familiarity [
44
]. To minimise impact, the
game is played by two players with minimal complexity. Cards are
managed and distributed to each player, with multiple sessions con-
ducted to ensure familiarity. In summary, this paper employs the
Blu game to create a dataset to detect deception, testing players’
abilities to lie naturally and convincingly.
3 METHODOLOGY
We conducted two studies that involved participants to naturally
engage in deceptive and truthful behaviours in a fun and entertain-
ing manner while playing the blu game. The two studies diered
in context. In study 1, we enabled participants to play against the
robot thus presenting a competitive context while in study 2, partic-
ipants played the game where the robot acted as an advisor, hence
presenting a cooperative context. In both studies, we only focused
on the instances where participants were engaged in displaying
deceptive and truthful behaviours. Such interactions were not in-
formed or mediated by the the role of robot in both contexts. We
investigate the following hypotheses:
•
[H1]: Human PBs and OBs, including EDA, BVP, HR, SKT, BR,
and BD, will show signicant dierences when participants are en-
gaging in deceptive versus truthful behaviours during interactions
with a robotic agent in both competitive (H1a) and cooperative
(H1b) settings.
•
[H2]: Signicant interaction eects between the session num-
ber (1, 2, 3, and 4) and the chosen PBs and OBs will be observed,
indicating dierences in PBs and OBs responses to deceptive and
truthful behaviours in both competitive (H1a) and cooperative
(H1b) settings.
•
[H3]: Classication algorithms will be able to classify instances
of deception with potentially high accuracy, demonstrating the
feasibility of using PBs and OBs for real-time detection of deception
in dierent settings.
Study 1 investigates H1a, H2a, and H3 while study 2 investigates
H1b, H2b and H3 respectively. In essence, H1 is based on the exist-
ing ndings that both PBs and OBs tend to dier during deceptive
and truthful acts of humans [
50
]. H2 is based on the existing re-
search that familiarity with the situation can signicantly inuence
humans’ deceptive behaviour [
39
]. Lastly, H3 is based on the nding
suggesting that data collected through humans naturally occurring
deceptive and truthful behaviour can improve the reliability and
detection rates [6].
Ethics - We submitted an application to the university’s ethics
committee to ensure the ethical integrity of our research involving
human participants. After review, the application was approved
[160322/5031].
3.1 The Game
We have created a card game called the "Blu Game" using the
Python programming language. The game can be played in two
ways: a human player can compete against a robot (study 1), or
a human and a robot can team up against an adversary (study
2). The game involves a deck of 52 cards with four sets of each
number from 1 to 10 and the face cards (jack, queen, and king).
The game interface has play and decision buttons, which make it
easy for players and the game to interact seamlessly. At the start of
the game, each player receives 15 cards, and to win the game, the
goal is to eliminate all the cards before the opponent. The game is
turn-based, and at each turn, a player must choose a set of 2-4 cards
to discard. This requires the player to decide whether to deceive or
be truthful about the cards in their hand. The opponent then has
to decide whether to believe the player is telling the truth or lying.
If the opponent believes that the player is truthful, the cards are
discarded and remain unseen. The opponent then takes the next
turn, and the game continues. If the opponent does not believe
that the player is truthful, the discarded cards are revealed. If the
player is found to be truthful, the opponent loses the round, and
the opponent receives the player cards. If the player is found to
be deceptive, the player must take back the cards, and the game
continues. The game ends when one player has discarded all their

ICMI ’24, November 04–08, 2024, San Jose, Costa Rica Ahmad et al.
cards. The list of each player’s cards is updated dynamically after
every turn.
3.2 Study 1
3.2.1 Interaction Scenario. The Nao robot was designed to interact
verbally with participants across the game events. We used the
Wizard of Oz method (WOz) to manage the game’s control with-
out revealing this to the participants, ensuring unbiased responses.
The interaction had three phases: welcoming and introducing the
game, playing the game, and concluding the game. At the begin-
ning, the robot greeted the participant warmly and oered a brief
introduction:“Hello. I am a Nao robot. Today, we will be playing a
card game against each other. Are you ready?”. Participants played
the game four times with a 5-minute break between each session.
In the second, third, and fourth sessions, the robot thanked the
participants and reintroduced the game by saying: “Hello again.
Thank you for playing. We are going to play another game. Are
you ready?” and “Let us start” respectively. At the start of the game,
the Nao robot informed the participant by saying “the game starts
now”. The robot initiated the rst turn and followed the game rules
by interacting with the participant during various game events in
the following manner:
(1)
When the robot selected its set of cards and declared them, e.g.,
“I selected two Kings”..
(2)
When the participant believed the robot’s claim, it responded
with: “It is your turn”.
(3)
When the participant did not believe the robot and the robot’s
card declaration was accurate, the robot stated: “I was telling the
truth”.
(4)
When the participant did not believe the robot and the robot’s
card declaration was incorrect, the robot stated: “You got me, and
it is your turn”.
(5)
When the robot believed the participant, it said: “I trust you,
and it is my turn”.
(6)
When the robot did not believe the participant, it said: “I think
you are blung”. If the participant told the truth, the robot said:
“Oh, I was wrong, and it is your turn now”.
(7)
If the robot did not believe the participant and the participant
was incorrect, the robot stated: “Yes, I got you, and it is my turn
now”.
After each game, the robot congratulated or wished the partic-
ipant luck for the next round. Upon a victory, the robot cheered:
“Congratulations! You’ve won. Thank you, and see you in the next
round”. In case of defeat, the robot encouraged by saying: “You’ve
just lost the game. Good luck in the following rounds”. In the nal
session, the robot bid farewell as it concluded the experiment.
3.2.2 Participants. The study initially aimed to involve 45 people
aged between 18 and 60 years old. However, data collection issues
were encountered with two of the participants, and the eective
number was adjusted to 43, with a mean age of 29.53 and a stan-
dard deviation of 6.71. The group consisted of 16 females, 26 males,
and one individual who preferred not to specify their gender. We
recruited participants by sending out invitations through univer-
sity email lists and posting yers around the campus. Interested
individuals signed up via the online platform, Calendly.
Figure 1: Experimental setup of study 1 and study 2
To determine the participants’ level of familiarity with robotics,
we categorised them based on their experience as high, medium,
low, or none. Participants who had controlled or constructed a ro-
bot were categorised as having high experience. Those who had
repeated use of robots were categorised as having medium expe-
rience, and those with occasional interactions were considered to
have low experience. The remaining participants, who had never
interacted with robots, were noted as having no experience. The
nal tally showed that 2 participants had high experience, 2 had
medium experience, 24 had low experience, and 15 had no prior
interactions with robots.
3.2.3 Experimental Setup and Equipment. The experimental setup
was splitted across two rooms, depicted in Figure 1. The rst room
hosted the interactive game setup, where participants were seated at
a table facing the Nao robot, with a laptop facilitating the card game.
To capture physiological responses, participants were equipped
with Pupil Invisible Eye Tracking Glasses and the Empatica E4
Wristband. A tablet was also provided to enter demographic details.
This room was specially arranged to ensure uniform environmental
conditions, such as controlled lighting and temperature, to prevent
any external inuence on physiological data such as BD, BR, and
SKT. In the second room, an experimenter oversaw the experiment
and manipulated the robot’s actions through a laptop, ensuring
seamless interaction between the participant and the robot. PBs
and OBs data collection was carried out using two sophisticated de-
vices: the Empatica E4 Wristband and Pupil Invisible Eye Tracking
Glasses. The E4 Wristband is known for its accuracy in measuring
heart rate, electrodermal activity, and body temperature, making it
an invaluable tool for this study. Similarly, the Pupil Invisible Eye
Tracking Glasses, with their high-resolution cameras and sensors,
were pivotal in tracking eye movements and providing insights into
the participants’ focus and cognitive engagement during interac-
tions with the robot.
3.2.4 Experimental Procedures. The study has the following steps:
(1)
Initially, each participant was briefed about the study through
an informational sheet and provided with detailed game instruc-
tions, followed by the signing of a consent form.
(2)
Next, they lled out a demographic questionnaire detailing their
prior experiences with robotics.
(3)
The participants then equipped themselves with eye-tracking
glasses and a physiological data recording wristband. Upon starting
the data recording, the experimenter vacated the room.
(4)
The actual game play commenced with participants engaging
in the card game against the Nao robot, which was manipulated
remotely by the experimenter from a separate room.

Detecting Deception in Natural Environments Using Incremental Transfer Learning ICMI ’24, November 04–08, 2024, San Jose, Costa Rica
(5)
Following the completion of each game round, the experimenter
re-entered the room to pause the data recording and requested the
participants to ll out a questionnaire assessing their interaction
with the robot. It’s important to note that the responses to this ques-
tionnaire were not analyzed in this paper due to their irrelevance
to the study’s primary objectives.
(6)
The steps 3, 4, and 5 were repeated in four dierent game ses-
sions.
(7)
To conclude, participants were informed of their £10 Amazon
voucher as appreciation for their contribution to the study.
3.3 Study 2
3.3.1 Interaction Scenario. The interaction was consistent during
the welcoming and introducing and concluding the game phases.
Due to the dierent role of robot, the role of robot interaction was
limited to occurrences where participants need to take advise on
whether to accept or reject the opponent move during the game
phase. Following the game rules, the robot interacted with the
participants during various game events. The game’s ow involved
the robot interacting with the participants during decisions and
other situations in the game as follows:
(1)
During the experiment, the robot consistently followed a pre-
dened protocol and strategy when participants asked about the
decision-making process in the accept condition. The robot pro-
vided feedback as follows: “Given the game has just started, I think
we could accept the claim for now; what do you think?”, “I think
we could accept, what do you think?”, “I suggest accept, what do
you think?”, or “I think it seems reasonable to accept the claim,
what do you think?”.
(2)
In the reject claims condition, the robot said, “I think they might
want to discard non-similar cards rst, what do you think?”, “I think
they are blung, what do you think?”, “I suggest rejecting the claim;
what do you think?”
(3)
If the participants agreed with the robot’s suggestion to accept,
the robot said “Okay, let’s continue”, “Okay, let’s proceed”, or “Okay,
let’s see how to conclude”.
(4)
If the participants agreed with the robot’s suggestion of reject-
ing the claims, the robot said “Okay, let’s see”.
(5)
If the participants disagreed with the robot’s suggestion, the
robot said “Okay, it is up to you”.
(6)
If the participants asked the robot to repeat the suggestion, the
robot repeated the suggestion for them.
(7)
If the robot did not hear the participants, the robot said “Sorry,
I did not hear that, could you please repeat it”.
(8)
If the participants seem to have been occupied with something
else, the robot said “You seem occupied with something else, could
you please focus on the game”.
(9)
If the participants asked the robot for anything else during the
game, the robot said “I can only advise you when you are deciding
to accept or reject”.
3.3.2 Participants. The recruitment process for participants in
Study 2 was similar to that of Study 1, with the aim of maintain-
ing consistency in the sample population by targeting a similar
demographic prole. Due to data collection issues, only 41 partici-
pants were included in the study, instead of the intended 45. The
participants, with an average age of 30.45 years and a standard
deviation of 4.14, were a balanced mix of genders. This diversity of
demographics provides varied perspectives on robotic interaction,
which enriches the research. To classify the participants based on
their experience with robotics, we analysed their frequency of in-
teraction with robots, ranging from daily usage to no interaction at
all. We categorised them into four groups: high, medium, low, or
none. 3 participants were classied as having high experience due
to their daily engagement with robots, 6 as having medium expe-
rience reecting weekly interaction, 20 as having low experience,
and 12 as having no experience with robots.
3.3.3 Setup and Materials. The setup and materials for Study 2
closely mirror those outlined in Study 1, with adjustments made
to accommodate the robot’s position to be next to the participant
(see Figure 1). Key components include the interactive card game
environment, physiological sensors for capturing participants’ re-
sponses, and the data collection system. The detailed descriptions
of these elements are provided in the Setup and Materials section
of Study 1.
3.3.4 Experiment Procedure. The experiment procedure for Study
2 follows the structure established in Study 1. This includes partici-
pant orientation, sensor attachment and calibration, introduction to
the game’s mechanics and the robot’s function, gameplay sessions,
and post-experiment questionnaire. For a detailed explanation of
these procedures, please refer to the experiment Procedure section
of Study 1.
3.4 Measurements for Deception Detection
3.4.1 Physiological Measures. We collected participants’ PBs and
OBs responses in real-time during deceptive or truthful decisions
using eye tracking technology for recording BR and BD, and a
wristband for monitoring EDA, BVP, HR, and SKT. These responses
were chosen for their relevance to deception detection and non-
intrusive nature of collection.
3.4.2 Behavioural Measures. The game interactions captured the
decision-making process of individuals choosing to display a de-
ceptive or truthful behaviour. The outcomes of each participant’s
decision were documented and coded as binary values, with 0 rep-
resenting truthfulness and 1 representing deception. Additionally,
the researchers recorded the timestamps for the beginning and
end of each decision phase to better understand the physiological
responses associated with specic moments of decision-making
related to deception or honesty.
3.5 Data Preparation for Deception Analysis
3.5.1 Behavioral Data Processing. Behavioural data from the game
sessions were processed to extract meaningful analytics:
(1)
Outcomes processing: The choices of whether to deceive or
to be honest were counted and sorted into dierent categories for
statistical analysis.
(2)
Timing analysis: We carefully matched the decision-making
periods with physiological data collection, ensuring precise analysis
of participants’ responses during critical moments.

ICMI ’24, November 04–08, 2024, San Jose, Costa Rica Ahmad et al.
3.5.2 Physiological Data Preprocessing. To begin with, we per-
formed the following steps prior to analysing the physiological
data:
(1)
Noise reduction: A low-pass lter was applied to remove noise
and artifacts, ensuring signal integrity for accurate analysis.
(2)
Data segmentation: The physiological data stream was seg-
mented according to the game rounds, aligning with the decision-
making phases for each participant.
(3)
Feature extraction: For each decision-making interval, we
calculated average values for EDA, BVP, HR, SKT, BR, and BD. This
step transformed raw data into a structured format conducive to
detecting deception-related physiological changes.
(4)
Dataset compilation: The nal dataset was compiled by aver-
aging physiological measures across sessions and decisions, creat-
ing a comprehensive prole for each participant’s deceptive and
honest behaviours.
By following these steps, we successfully generated two datasets
suitable for analysing deception and honest behaviours using PBs
and OBs. The datasets alongside codes can be accessed here. In the
given link, the les named as “Dataset1_sessions” and “Dataset2_sessions”
represent the datasets for session 1, 2, 3 and 4 respectively, while,
the les named as “Dataset1_all” and “Dataset2_all” represent all
session data.
4 RESULTS
In this section, we present the ndings of both studies. We inves-
tigate whether there are signicant dierences in OBs and PBs
during deceptive or truthful states in both studies. In addition, we
investigated whether familiarity with the task enabled a signicant
change in OBs and PBs during deceptive or truthful states during
repeated interactions. Lastly, we investigated how accurately we
could classify deception in the two datasets and also applied incre-
mental transfer learning to nd whether we could detect deception
with high accuracy.
To test H1a, & H1b and H2a, & H2b, we conducted a repeated-
measures ANOVA with deception and truthful states as a between-
subject variable and the interactive session (session 1, session 2,
session 3, and session 4) as a within-subject variable on the physio-
logical measures (EDA, BVP, HR, SKT, BR, and BD) as dependent
variables (DVs). We found that there was a signicant eect of
deception in study 1 on EDA ((
𝐹 (
1
,
69
) =
4
.
270,
𝑝 = .
04,
𝜂
2
𝑝= .
058))
and SKT (
𝐹 (
1
,
69
) =
20
.
124,
𝑝 < .
001,
𝜂
2
𝑝= .
226) scores and
in study 2 on EDA (
𝐹 (
1
,
76
) =
8
.
730,
𝑝 = .
004,
𝜂
2
𝑝= .
103), HR
(
𝐹 (
1
,
76
) =
4
.
141,
𝑝 = .
045,
𝜂
2
𝑝= .
052), and SKT (
𝐹 (
1
,
76
) =
23
.
570,
𝑝 < .
001,
𝜂
2
𝑝= .
237). However, we did not see a signicant eect
of deception on BVP (
𝐹 (
1
,
69
) = .
591,
𝑝 = .
445,
𝜂
2
𝑝= .
008), HR
(
𝐹 (
1
,
69
) =
2
.
925,
𝑝 = .
092,
𝜂
2
𝑝= .
041), BR (
𝐹 (
1
,
69
) = .
00,
𝑝 > .
983,
𝜂
2
𝑝= .
00) and BD (
𝐹 (
1
,
69
) = .
066,
𝑝 > .
798,
𝜂
2
𝑝= .
001) respec-
tively in study 1 and BVP (
𝐹 (
1
,
76
) = .
834,
𝑝 = .
364,
𝜂
2
𝑝= .
011),
BR (
𝐹 (
1
,
76
) = .
428,
𝑝 = .
515,
𝜂
2
𝑝= .
006), and BD (
𝐹 (
1
,
76
) = .
796,
𝑝 = .
375,
𝜂
2
𝑝= .
010) in study 2. The mean and standard deviation
for all the DVs in both studies can be seen in Table 4.
A signicant interaction eect of session and deception was
observed for BVP measures in study 2 (
𝐹 (
3
,
74
) =
2
.
664,
𝑝 = .
054,
𝜂
2 = .
097). However, we did not observe a signicant interac-
tion eect of session and deception (session * deception) on EDA
(
𝐹 (
3
,
67
) =
1
.
073,
𝑝 = .
366,
𝜂
2 = .
046), BVP (
𝐹 (
3
,
67
) = .
606,
𝑝 = .
613,
𝜂
2 = .
026), HR (
𝐹 (
3
,
67
) = .
727,
𝑝 = .
539,
𝜂
2 = .
032), SKT
(
𝐹 (
3
,
67
) = .
447,
𝑝 = .
720,
𝜂
2 = .
020), BR [
𝐹 (
3
,
67
) =
1
.
036,
𝑝 > .
382,
𝜂
2 = .
044], and BD (
𝐹 (
3
,
67
) = .
392,
𝑝 > .
759,
𝜂
2 = .
017) in study 1
and EDA (
𝐹 (
3
,
74
) = .
374,
𝑝 = .
772,
𝜂
2 = .
015), HR (
𝐹 (
3
,
74
) = .
241,
𝑝 = .
867,
𝜂
2 = .
010), SKT (
𝐹 (
3
,
74
) = .
061,
𝑝 = .
980,
𝜂
2 = .
002),
BR (
𝐹 (
3
,
74
) = .
170,
𝑝 = .
916,
𝜂
2 = .
007), and BD (
𝐹 (
3
,
74
) = .
570,
𝑝 = .
637,
𝜂
2 = .
023) in study 2. Lastly, we observed that only SKT
varies across sessions in both study 1 and 2 (
𝐹 (
3
,
67
) =
18
.
957,
𝑝 >< .
001,
𝜂
2 = .
020), (
𝐹 (
3
,
74
) =
22
.
834,
𝑝 < .
001,
𝜂
2 = .
481),
respectively.
We conducted post-hoc Bonferroni tests to assess whether SKT
diered signicantly between sessions 1, 2, 3, and 4 in two studies.
In Study 1, the analysis conrmed that SKT was signicantly higher
in session 1 compared to session 2 (
𝑝 <
0
.
001), session 3 (
𝑝 <
0
.
001),
and session 4 (
𝑝 <
0
.
001). No signicant dierences were found
when comparing session 2 to sessions 3 and 4, nor between session
3 and session 4, suggesting that SKT levels were more stable across
these later sessions. In Study 2, similar trends were observed, with
SKT signicantly higher in session 1 compared to session 2 (
𝑝 <
0
.
001), session 3 (
𝑝 <
0
.
001), and session 4 (
𝑝 <
0
.
001). These
ndings highlight the variability in SKT responses across dierent
sessions, underscoring the impact of session-specic factors on skin
temperature measurement.
To test H3, which aimed to investigate whether PBs and OBs
can be utilised to classify truthful and deception behaviour, we fol-
lowed the structured approach proposed by Ahmad et al
. [3]
. Seven
classiers were implemented: Random Forest (RF), Logistic Regres-
sion (LR), Support Vector Machines (SVM), Decision Tree (DT),
AdaBoost (AB), Neural Network (NN), and Naive Bayes (NB). The
performance of these classiers was evaluated using 5-fold cross-
validation. The ndings revealed that RF, LR, and SVM achieved
the highest accuracies at 75%, 71%, and 71%, respectively, while the
other classiers also performed well (refer to Table 1).
To provide a more detailed analysis of the accuracy ndings,
we have presented the results in the form of a classication report
in Table 1. This report shows the F1 score for each class, which
evaluates the performance of each classier. The results indicate
that for the RF, LR, and SVM classier, both deception and non-
deception were predicted correctly, with a 71% both, 70% and 71%
and 71%, 74% accuracy rate on the test data, suggesting that LR
and SVM have relatively higher accuracy compared to the other
classiers.
4.1 Feature importance for Deception and
Non-deception
We examined each PB in the datasets and evaluated their ability
to determine whether the subject was being truthful or deceptive.
We calculated the F1 score for each class separately to gauge the
eectiveness of each feature in accurately classifying the subjects.
The RF, LR, and SVM classiers exhibited the best performance
in predicting deception or non-deception. Thus, we only present
the feature importance for these classiers. In Study 1, the feature
importance for the deception and truthful states were: EDA (0.39,
0.62), BVP (0.58, 0.39), HR (0.58, 0.58), SKT (0.67, 0.66), BR (0.52,

Detecting Deception in Natural Environments Using Incremental Transfer Learning ICMI ’24, November 04–08, 2024, San Jose, Costa Rica
Classier Accuracy (%) F1-Scores
Study 1 Study 2 Study 1 Study 2
S1 S2 S3 S4 All S1 S2 S3 S4 All D T D T
RF 60% 60% 67% 64% 69% 65% 77% 63% 63% 75% 0.70 0.66 0.76 0.74
LR 67% 70% 62% 67% 71% 71% 69% 66% 64% 69% 0.71 0.71 0.75 0.72
SVM 65% 66% 59% 69% 71% 64% 69% 70% 62% 69% 0.70 0.71 0.71 0.66
DT 55% 51% 55% 59% 60% 47% 60% 53% 67% 67% 0.61 0.59 0.68 0.65
AB 60% 51% 56% 60% 61% 54% 59% 57% 63% 69% 0.63 0.58 0.69 0.66
NN 72% 59% 56% 62% 70% 64% 77% 67% 67% 68% 0.70 0.69 0.71 0.67
NB 66% 63% 66% 65% 67% 60% 64% 71% 58% 65% 0.69 0.64 0.71 0.50
Table 1: Performance Metrics of Classiers across Dierent Sessions (S) and Studies for Deception (D) and truthful (T) behaviours.
Classier Source (Study 1) Target (Study 2)
RF 79% 77%
LR 77% 80%
SVM 76% 77%
DT 73% 58%
AB 76% 75%
NN 82% 77%
NB 71% 67%
Table 2: Performance of various classiers using incremental
transfer learning on two datasets.
0.49), and BD (0.43, 0.57) for LR respectively. For RF in Study 2,
the values were: EDA (0.56, 0.52), BVP (0.53, 0.53), HR (0.53, 0.52),
SKT (0.56, 0.56), BR (0.65, 0.66), and BD (0.55, 0.55). SVM in Study 1
showed: EDA (0.28, 0.63), BVP (0.69, 0.18), HR (0.50, 0.56), SKT (0.66,
0.65), BR (0.66, 0.06), and BD (0.36, 0.51). The features HR and SKT
were consistent and eective in predicting deception and truthful
classes.
4.2 Incremental Transfer Learning Results
We conducted two studies that produced two datasets. To handle
this, we adopted an incremental transfer learning approach utilis-
ing seven classiers as proposed by Chui et al
. [12]
. Our process
involved selecting two datasets, one as the source (Dataset 1) and
the other as the target (Dataset 2). We divided each dataset into
equally sized subsets and trained an initial model (Model 1.1) on
the rst subset of Dataset 1. Then, we transferred the knowledge
from Model 1.1 to train Model 2.1 on the rst subset of Dataset 2.
We continued updating the models with subsequent subsets until
the last subsets were used. We utilised seven models, including RF,
LR, SVM, DT, AB, NN and NB, with parameters illustrated in Table
3. Our method achieved signicant accuracies: RF achieved 79%
(source) and 77% (target), LR scored 77% and 80%, SVM showed 76%
on both, DT reported 73% and 58%, AB recorded 76% and 75%, NN
reached 82% and 77%, and NB showed 71% and 67% (see table 2 for
more information).
5 DISCUSSION
This study investigated whether PBs & OBs can be collectively
used to detect deception. In this section, we discuss whether the
hypotheses were accepted or rejected in the light of the ndings.
Model Parameters
RF n_estimators=150, max_depth=10, criterion=’entropy’
LR penalty=’l2’, tol=0.0001, C=1.0, t_intercept=True, solver=’lbfgs’, max_iter=100
SVM probability=True
AB base_estimator=DecisionTreeClassier(max_depth=1), n_estimators=50
DT max_depth=3
NN hidden_layer_sizes=(100,), max_iter=500
NB GaussianNB()
Table 3: Models Parameters
Feature Non-deception Deception
Study 1 Study 2 Study 1 Study 2
M SD M SD M SD M SD
EDA 0.99 2.59 0.37 0.44 0.38 0.46 0.99 0.84
BVP 0.02 0.18 0.03 0.30 0.03 0.32 0.01 0.17
HR 105.1 18.1 101.5 14.7 101.4 18.01 105.9 20.64
SKT 28.1 1.4 26.87 1.5 26.9 1.5 28.02 1.53
BR 2.5 2.04 2.4 1.63 2.5 1.9 2.29 2.03
BD 310.4 125.52 317.02 120.5 319.08 132.6 302.1 144.94
Table 4: Mean (M) and Standard Deviation (SD) for the physi-
ological features under truthful and deceptive states across
two sets of data.
H1 suggested a signicant dierence in human PBs & OBs re-
sponses, such as EDA, BVP, HR, SKT, BR, and BD, between deceptive
and truthful states during HRI. We found that both EDA and SKT
diered signicantly during deceptive and truthful states in both
experiments. EDA and SKT are physiological measures linked with
galvanic skin responses that can detect deception [
51
]. EDA mea-
sures skin conductivity, which increases during stress or arousal
states related to deception. SKT reects changes in blood ow to
the skin, which can vary due to the complex interplay between the
sympathetic and parasympathetic nervous systems. Studies have
shown that lying often induces nervousness or stress, as well as
cognitive load, both of which are related to increased (sympathetic
nervous system activity [51].
We observed a signicant dierence in HR between deceptive
and truthful states in study 2, which represented a cooperative
context. However, no such eects were seen in Study 1, which pre-
sented a competitive context. Previous research has shown that the
variability in HR response to deception can be inuenced by sev-
eral situational and individual factors, which can explain why HR
may dier in one situation and not in another [
39
]. The emotional

ICMI ’24, November 04–08, 2024, San Jose, Costa Rica Ahmad et al.
response to lying can vary depending on the stakes involved, the
potential consequences of being caught, and the individual’s moral
compass. Although cognitive load and stress were consistent in
both studies, we believe that participants may have felt more pres-
sure in the presence of the robot and attempted to maintain a moral
compass [
58
]. This nding can be due to interpersonal dynamics,
as the relationship between the deceiver and the observer can inu-
ence HR. For instance, lying to a stranger may not elicit the same
physiological response as lying to a loved one [
39
]. Furthermore,
studies have shown inconsistencies in the correlation between HR
and deception. Some studies have found an increased HR in guilty
individuals, while others have indicated that lying could decrease
HR [
26
]. Lastly, a recent study suggests that the presence of robots
can have a similar impact on HR as working with other humans,
potentially due to the development of trust and the integration of
robots as team members [16].
On the other hand, the other PBs and OBs (BVP, BR, and BD)
did not show signicant dierences between truthful and deceptive
states. We understand that BVP and blinking rates can be inuenced
by stress and cognitive load. The regulation of BVP is complex and
can be maintained across dierent emotional states, and blinking
rates are subject to voluntary control and are inuenced by various
contextual factors. Therefore, these measures could not dier sig-
nicantly in truthful and deceptive states. In addition, the context
of the interaction can inuence blinking rates. For example, if an
individual is in a relaxed and informal setting, they may blink less
frequently, regardless of whether they are being truthful or decep-
tive [
10
]. Moreover, changes in BVP and other PBs correlate with
anxiety, but such conditions may not have been present during the
game-based context. The absence of pressure elements, such as time
constraints and individual dierences in stress response and cogni-
tive load, may have contributed to the variability in physiological
responses.
In summary, the hypothesis H1a and H1b were partially accepted
as we did not nd signicant dierences for all the PBs.
H2 hypothesised an interaction eect (session and decision to
be truthful and deceptive) on PBs. Our results did not conrm this
hypothesis, as we did not nd a signicant interaction eect of ses-
sion and decision (session * decision) on all PB features except BVP
in study 2. We understand that deception can cause consistent phys-
iological behaviours during repeated interactions due to several
psychological and physiological factors [
46
]. Individuals can adapt
to the act of deception over time, leading to a decrease in physio-
logical responses. With repeated exposure to the same deceptive
scenarios, individuals can become habituated to the stress associ-
ated with lying leading to a reduction in physiological responses.
Frequent deception can make individuals more skilled at lying,
resulting in less pronounced physiological changes and more con-
sistent behaviours. Repeatedly engaging in deceptive behaviour can
lead to emotional desensitisation, where the emotional impact of ly-
ing diminishes over time, resulting in fewer physiological changes
and consistent behaviours. Increased eciency at lying reduces
physiological indicators of deception. Individuals have dierent
baseline physiological responses, and familiarity with the situation
reduces the physiological response to deception. If the context of
the repeated interactions remains consistent, it makes it harder to
discern dierences between truthful and deceptive behaviours.
H3 suggested that classication algorithms will classify instances
of deception with potentially high accuracy. The results of the two
studies were promising, with LR, SVM, NN, and RF classiers de-
tecting deception with accuracy rates of over 70% and 75%. In both
studies, HR, and SKT features were crucial for detecting decep-
tion in the best-performing classiers. The importance of these
features is linked to their association with emotional arousal, cog-
nitive eort, and rapid physiological changes that typically occur
in response to deception in game contexts [37].
Our study utilised a new incremental transfer learning algorithm
and achieved an accuracy rate of 80%, surpassing the current decep-
tion detection rates based on PBs and OBs [
6
]. This indicates that
our hypothesis (H3) was accepted. The high accuracy was possible
due to the negative transfer avoidance algorithm included in in-
cremental transfer learning, which reduces the risk of transferring
irrelevant information and facilitates the transfer of knowledge
[
12
]. Additionally, the use of multiple PBs and OBs is crucial since
PBs are often dependent on the task or environment [
3
]. To sum-
marise, our dataset consisted of natural and repeated interactions,
including both deceptive and truthful states, resulting in a large
and diverse set of data that helped us achieve good results.
6 CONCLUSION & FUTURE WORK
This paper highlights the limitations of current datasets used for
deception detection, as they lack diversity and realism and have
a limited number of subjects in simulated environments that do
not accurately reect real-world deceptive behaviours. To over-
come this, we used the GaME (game as a method to elicit emotions
naturally) paradigm and created a dataset based on organically de-
picted deceptive and truthful interactions. The dataset was based on
repeated interactions, which is a further signicant improvement
from existing work that only oers a dataset on one-o interactions.
We conducted two experiments involving 83 participants to inves-
tigate whether dierent physiological and oculomotor behaviours
(PBs & OBs) collected naturally during deceptive and truthful states
dier signicantly. Additionally, we explored whether combining
PBs and OBs can accurately predict humans in deceptive and truth-
ful states during HRI. Our ndings conrmed that PBs such as EDA
and SKT diered in deceptive and truthful states. It indicated that
multiple PBs collectively detect deception in real time during HRI.
For the rst time, we used the novel incremental transfer learning
to detect deception and achieved an 80% accuracy, surpassing most
of the existing work. We encourage the research community to
use the GaME paradigm in dierent contexts to improve the rate
of deception detection. We promote incremental transfer learning
techniques to yield optimal results in the target (new) models.
While this study shows promise, it is important to note its limi-
tations. Findings are specic to game-based robot interactions and
may not apply to other contexts or human interactions. The limited
demographic characteristics, mainly consisting of students, may
restrict the generalisability of the results. Future research will in-
volve testing in various environments and including participants
from diverse backgrounds to enhance the ndings. We also plan
to explore how combining facial and speech features along with
PBs and OBs in dierent contexts can improve detecting deception
rates. We aim to use such detection mechanisms to develop adaptive
robotic systems that can have wider applications.

Detecting Deception in Natural Environments Using Incremental Transfer Learning ICMI ’24, November 04–08, 2024, San Jose, Costa Rica
REFERENCES
[1]
Rachel Adelson. 2004. Detecting deception. APA Monitor on Psychology 35, 7
(2004), 70–73.
[2]
Muneeb Ahmad and Abdullah Alzahrani. 2023. Crucial Clues: Investigating
Psychophysiological Behaviors for Measuring Trust in Human-Robot Interaction.
In Proceedings of the 25th International Conference on Multimodal Interaction.
135–143.
[3]
Muneeb Imtiaz Ahmad, Ingo Keller, David A Robb, and Katrin S Lohan. 2020.
A framework to estimate cognitive load using physiological data. Personal and
Ubiquitous Computing (2020), 1–15.
[4]
Muneeb Imtiaz Ahmad, Omar Mubin, and Hiren Patel. 2018. Exploring the
potential of NAO robot as an interviewer. In Proceedings of the 6th International
Conference on Human-Agent Interaction. 324–326.
[5]
Muneeb I Ahmad, Hassan Tariq, Mehreen Saeed, Suleman Shahid, and Emiel
Krahmer. 2011. Guess who? An interactive and entertaining game-like platform
for investigating human emotions. In Human-Computer Interaction. Towards
Mobile and Intelligent Interaction Environments: 14th International Conference,
HCI International 2011, Orlando, FL, USA, July 9-14, 2011, Proceedings, Part III 14.
Springer, 543–551.
[6]
Haya Alaskar, Zohra Sbaï, Wasiq Khan, Abir Hussain, and Arwa Alrawais. 2023.
Intelligent techniques for deception detection: a survey and critical study. Soft
Computing 27, 7 (2023), 3581–3600.
[7]
Faryal Amber, Adeel Yousaf, Muhammad Imran, and Khurram Khurshid. 2019.
P300 based deception detection using convolutional neural network. In 2019
2nd International Conference on Communication, Computing and Digital systems
(C-CODE). IEEE, 201–204.
[8]
Alexander Mois Aroyo, Jonas Gonzalez-Billandon, Alessia Tonelli, Alessandra
Sciutti, Monica Gori, Giulio Sandini, and Francesco Rea. 2018. Can a humanoid
robot spot a liar?. In 2018 IEEE-RAS 18th International Conference on Humanoid
Robots (Humanoids). IEEE, 1045–1052.
[9]
David B Buller and Judee K Burgoon. 1996. Interpersonal deception theory.
Communication theory 6, 3 (1996), 203–242.
[10]
Judee K Burgoon and Norah E Dunbar. 2016. Accuracy of distinguishing truth
from lie. The social psychology of perceiving others accurately (2016), 71–97.
[11]
Luis Vicente Calderita, Luis J Manso, Pablo Bustos, Cristina Suárez-Mejías, Fer-
nando Fernández, and Antonio Bandera. 2014. THERAPIST: towards an au-
tonomous socially interactive robot for motor and neurorehabilitation therapies
for children. JMIR rehabilitation and assistive technologies 1, 1 (2014), e3151.
[12]
Kwok Tai Chui, Varsha Arya, Shahab S Band, Mobeen Alhalabi, Ryan Wen Liu,
and Hao Ran Chi. 2023. Facilitating innovation and knowledge transfer between
homogeneous and heterogeneous datasets: Generic incremental transfer learning
approach and multidisciplinary studies. Journal of Innovation & Knowledge 8, 2
(2023), 100313.
[13]
Sergey Demyanov, James Bailey, Kotagiri Ramamohanarao, and Christopher
Leckie. 2015. Detection of Deception in the Maa Party Game. In Proceedings
of the 2015 ACM on International Conference on Multimodal Interaction (Seattle,
Washington, USA) (ICMI ’15). Association for Computing Machinery, New York,
NY, USA, 335–342. https://doi.org/10.1145/2818346.2820745
[14]
Bella M DePaulo and Wendy L Morris. 2004. Discerning lies from truths: Behav-
ioral cues to deception and the indirect pathway of intuition. BM DePaulo, WL
Morris, ad. by PA Granhag. The detection of deception in forensic contexts//New
York: Cambridge (2004), 15–40.
[15]
Mingyu Ding, An Zhao, Zhiwu Lu, Tao Xiang, and Ji-Rong Wen. 2019. Face-
focused cross-stream network for deception detection in videos. In Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7802–7811.
[16]
Gwendolyn Edgar, Ayca Aygun, Matthew McWilliams, and Matthias Scheutz.
[n. d.]. Towards Genuine Robot Teammates: Improving Human-Robot Team
Performance Beyond Shared Mental Models with Proactivity. ([n. d.]).
[17]
Empatica Inc. Year of access. Empatica E4 Research Device. https://www.empatica.
com/en-eu/research/e4/
[18]
Lynette Gabonthone. 2019. Critical Evaluation of Machine Learning Research
Aimed at Improving Lie Detection Accuracy. Selected Computing Research Papers
(2019), 1.
[19]
Andrea Gaggioli et al
.
2018. Beyond the truth machine: emerging technologies
for lie detection. Cyberpsychology, Behavior and Social Networking 21, 2 (2018),
144–144.
[20]
Ascensión Gallardo-Antolín and Juan M Montero. 2021. Detecting deception
from gaze and speech using a multimodal attention LSTM-based framework.
Applied Sciences 11, 14 (2021), 6393.
[21]
Matthias Gamer and Wolfgang Ambach. 2014. Deception research today. Frontiers
in psychology 5 (2014), 88426.
[22]
Sabu George, MM Manohara Pai, Radhika M Pai, and Samir Kumar Praharaj.
2017. Eye blink count and eye blink duration analysis for deception detection.
In 2017 International conference on advances in computing, communications and
informatics (ICACCI). IEEE, 223–229.
[23]
Mandar Gogate, Ahsan Adeel, and Amir Hussain. 2017. Deep learning driven
multimodal fusion for automated deception detection. In 2017 IEEE symposium
series on computational intelligence (SSCI). IEEE, 1–6.
[24]
Jonas Gonzalez-Billandon, Alexander M Aroyo, Alessia Tonelli, Dario Pasquali,
Alessandra Sciutti, Monica Gori, Giulio Sandini, and Francesco Rea. 2019. Can a
robot catch you lying? a machine learning system to detect lies during interac-
tions. Frontiers in Robotics and AI 6 (2019), 64.
[25]
Viresh Gupta, Mohit Agarwal, Manik Arora, Tanmoy Chakraborty, Richa Singh,
and Mayank Vatsa. 2019. Bag-of-lies: A multimodal dataset for deception detec-
tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition Workshops. 0–0.
[26]
Danielle Hipsky, Elizabeth Parimanath, Joseph Mueller, Mike McGinnis, and
Stephanie Wright. 2014. Analyzing Fluctuations in Heart Rate, Respiration, and
Perspiration as Physiological Indicators of Deception. (2014).
[27]
Charles R Honts and John C Kircher. 1994. Mental and physical countermeasures
reduce the accuracy of polygraph tests. Journal of Applied Psychology 79, 2 (1994),
252.
[28]
Ray Hyman. 1989. The psychology of deception. Annual review of psychology 40,
1 (1989), 133–154.
[29]
Samee Ibraheem, Gaoyue Zhou, and John DeNero. 2022. Putting the con in
context: Identifying deceptive actors in the game of maa. arXiv preprint
arXiv:2207.02253 (2022).
[30]
Koji Inoue, Kohei Hara, Divesh Lala, Kenta Yamamoto, Shizuka Nakamura, Kat-
suya Takanashi, and Tatsuya Kawahara. 2020. Job interviewer android with
elaborate follow-up question generation. In Proceedings of the 2020 International
Conference on Multimodal Interaction. 324–332.
[31]
Nishant Jain, Abhay Kumar, Shekhar Singh, Chirag Singh, and Suraj Tripathi.
2019. Deceptive reviews detection using deep learning techniques. In Natural
Language Processing and Information Systems: 24th International Conference on
Applications of Natural Language to Information Systems, NLDB 2019, Salford, UK,
June 26–28, 2019, Proceedings 24. Springer, 79–91.
[32]
Mimansa Jaiswal, Sairam Tabibu, and Rajiv Bajpai. 2016. The truth and nothing
but the truth: Multimodal analysis for deception detection. In 2016 IEEE 16th
International Conference on Data Mining Workshops (ICDMW). IEEE, 938–943.
[33]
Hamid Karimi. 2018. Interpretable multimodal deception detection in videos. In
Proceedings of the 20th ACM international conference on multimodal interaction.
511–515.
[34]
Wasiq Khan, Keeley Crockett, James O’Shea, Abir Hussain, and Bilal M Khan.
2021. Deception in the eyes of deceiver: A computer vision and machine learning
based automated deception detection. Expert Systems with Applications 169 (2021),
114341.
[35]
Gangeshwar Krishnamurthy, Navonil Majumder, Soujanya Poria, and Erik Cam-
bria. 2018. A deep learning approach for multimodal deception detection. In
International Conference on Computational Linguistics and Intelligent Text Process-
ing. Springer, 87–96.
[36] David W Kupferman. 2022. I, robot teacher. , 1513–1522 pages.
[37]
Jennifer S Lerner, Ye Li, Piercarlo Valdesolo, and Karim S Kassam. 2015. Emotion
and decision making. Annual review of psychology 66 (2015), 799–823.
[38]
Timothy R Levine. 2014. Truth-default theory (TDT) a theory of human deception
and deception detection. Journal of Language and Social Psychology 33, 4 (2014),
378–392.
[39]
Timothy R Levine. 2019. An overview of detecting deceptive communication.
The palgrave handbook of deceptive communication (2019), 289–301.
[40]
Frank M Marchak. 2013. Detecting false intent using eye blink measures. Frontiers
in psychology 4 (2013), 36551.
[41]
Jaume Masip, Iris Blandón-Gitlin, Carmen Martínez, Carmen Herrero, and Iza-
skun Ibabe. 2016. Strategic interviewing to detect deception: Cues to deception
across repeated interviews. Frontiers in psychology 7 (2016), 1702.
[42]
Karnati Mohan and Ayan Seal. 2021. Deception detection on “Bag-of-Lies”:
integration of multi-modal data using machine learning algorithms. In Proceedings
of International Conference on Machine Intelligence and Data Science Applications:
MIDAS 2020. Springer, 445–456.
[43]
Merylin Monaro, Pasquale Capuozzo, Federica Ragucci, Antonio Maei, Antoni-
etta Curci, Cristina Scarpazza, Alessandro Angrilli, and Giuseppe Sartori. 2020.
Using blink rate to detect deception: A study to validate an automatic blink detec-
tor and a new dataset of videos from liars and truth-tellers. In Human-Computer
Interaction. Human Values and Quality of Life: Thematic Area, HCI 2020, Held as
Part of the 22nd International Conference, HCII 2020, Copenhagen, Denmark, July
19–24, 2020, Proceedings, Part III 22. Springer, 494–509.
[44]
Fred Paas and John Sweller. 2014. Implications of cognitive load theory for
multimedia learning. The Cambridge handbook of multimedia learning 27 (2014),
27–42.
[45]
Jerey Pawlick and Quanyan Zhu. 2015. Deception by design: evidence-based
signaling games for network defense. arXiv preprint arXiv:1503.05458 (2015).
[46]
Richard Reardon, Annette L Folwell, Jode Keehr, and Trevor Kauer. 2019. Eects
of deception on the deceiver: An interdisciplinary view. The Palgrave handbook
of deceptive communication (2019), 107–125.
[47]
M Umut Şen, Veronica Perez-Rosas, Berrin Yanikoglu, Mohamed Abouelenien,
Mihai Burzo, and Rada Mihalcea. 2020. Multimodal deception detection using
real-life trial data. IEEE Transactions on Aective Computing 13, 1 (2020), 306–319.

ICMI ’24, November 04–08, 2024, San Jose, Costa Rica Ahmad et al.
[48]
Suleman Shahid. 2012. Fun & Face: Exploring non-verbal expressions of emotion
during playful interactions. (2012).
[49]
Jeremy Speth, Nathan Vance, Adam Czajka, Kevin W Bowyer, Diane Wright, and
Patrick Flynn. 2021. Deception detection and remote physiological monitoring:
A dataset and baseline experimental results. In 2021 IEEE International Joint
Conference on Biometrics (IJCB). IEEE, 1–8.
[50]
R Weylin Sternglanz, Wendy L Morris, Marley Morrow, and Joshua Braverman.
2019. A review of meta-analyses about deception detection. The Palgrave hand-
book of deceptive communication (2019), 303–326.
[51]
Sabine Ströfer, Matthijs L Noordzij, Elze G Ufkes, and Ellen Giebels. 2015. Decep-
tive intentions: Can cues to deception be measured before a lie is even stated?
PloS one 10, 5 (2015), e0125237.
[52]
Michele K Surbey and Jerey J McNally. 1997. Self-deception as a mediator of
cooperation and defection in varying social contexts described in the iterated
prisoner’s dilemma. Evolution and Human Behavior 18, 6 (1997), 417–435.
[53]
Leanne Ten Brinke, Julia J Lee, and Dana R Carney. 2019. Dierent physiological
reactions when observing lies versus truths: Initial evidence and an intervention
to enhance accuracy. Journal of personality and social psychology 117, 3 (2019),
560.
[54]
Su-Jing Wang, Bing-Jun Li, Yong-Jin Liu, Wen-Jing Yan, Xinyu Ou, Xiaohua
Huang, Feng Xu, and Xiaolan Fu. 2018. Micro-expression recognition with small
sample size by transferring long-term convolutional neural network. Neurocom-
puting 312 (2018), 251–262.
[55]
Whoop. [n. d.]. WHOOP. https://www.whoop.com/us/en/ Accessed on January
29, 2024.
[56]
Wikipedia contributors. 2024. Cheat (game) — Wikipedia, The Free Encyclopedia.
https://en.wikipedia.org/wiki/Cheat_(game) [Online; accessed 3-May-2024].
[57]
Yue Xie, Ruiyu Liang, Huawei Tao, Yue Zhu, and Li Zhao. 2018. Convolutional
bidirectional long short-term memory for deception detection with acoustic
features. IEEE Access 6 (2018), 76527–76534.
[58]
Aakash Yadav, Patralika Ghosh, Malik Rawashdeh, Diane Lee, Thomas Bolf, and
Ranjana K Mehta. 2023. All Human Versus Human-Robot Teaming: Measuring
Neurophysiological Synchrony, Team Performance, and Trust during Search
and Rescue. In Proceedings of the Human Factors and Ergonomics Society Annual
Meeting, Vol. 67. SAGE Publications Sage CA: Los Angeles, CA, 432–433.
[59]
Ke Zhang, Lara A Frumkin, Alex Stedmon, and Glyn Lawson. 2013. Deception
in context: Coding nonverbal cues, situational variables and risk of detection.
Journal of Police and Criminal Psychology 28 (2013), 150–161.
[60]
Yan Zhou, Heming Zhao, Xinyu Pan, and Li Shang. 2015. Deception detecting
from speech signal using relevance vector machine and non-linear dynamics
features. Neurocomputing 151 (2015), 1042–1052.
[61]
Yan Zhou, Heming Zhao, and Li Shang. 2017. Lying speech characteristic extrac-
tion based on SSAE deep learning model. In Intelligent Computing Methodologies:
13th International Conference, ICIC 2017, Liverpool, UK, August 7-10, 2017, Proceed-
ings, Part III 13. Springer, 672–681.
[62]
Miron Zuckerman, Bella M DePaulo, and Robert Rosenthal. 1981. Verbal and
nonverbal communication of deception. In Advances in experimental social
psychology. Vol. 14. Elsevier, 1–59.