![Conceptual framework illustrating the relationship between workload, attention and performance (adopted from [55]).](https://www.researchgate.net/profile/Ehsan-Tarkesh-Esfahani/publication/322324510/figure/fig1/AS:580706210648064@1515462557443/Conceptual-framework-illustrating-the-relationship-between-workload-attention-and_Q320.jpg)
Content uploaded by Ehsan Tarkesh Esfahani
Author content
All content in this area was uploaded by Ehsan Tarkesh Esfahani on Jan 10, 2018
Content may be subject to copyright.
Available via license: CC BY-NC-ND 4.0
Content may be subject to copyright.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier xx.xxxx/ACCESS.201x.DOI
Physiological Measures for Human
Performance Analysis in Human-Robot
Teamwork: Case of Tele-Exploration
AMIRHOSSEIN H. MEMAR1, and EHSAN T. ESFAHANI1, (Member, IEEE)
1Department of Mechanical and Aerospace Engineering, University at Buffalo, Buffalo, NY 14260 USA
Corresponding author: Ehsan T. Esfahani (e-mail: ehsanesf@buffalo.edu).
ABSTRACT Continuous monitoring of mental workload and situation awareness in operational environ-
ments are useful for understanding and prediction of human performance. Such information can be used
to develop real-time adaptive systems to enhance human performance. In this paper, we investigate the use
of workload- and attention-related physiological measures to predict operator performance and situation
awareness in the context of tele-exploration with a small team of robots. A user study is conducted based
on a simulated scenario involving visual scanning and manual control tasks with varying levels of task-
load. Brain activity and eye movements of the participants are monitored across the experimental tasks
using electroencephalogram (EEG) and eye tracker sensors. The performances of the subjects are evaluated
in terms of target detection and situation awareness (primary metrics) as well as reaction time and false
detection (secondary metrics). Moreover, individual differences in two specific visual skills, visual search
(VS) and multi-object tracking (MOT), are considered as between-subject factors in the experimental design.
The main effects of task type and individual differences reveal that VS and MOT skill have significant effects
on target detection and situation awareness, respectively. The correlations of physiological measures with
the task performance and situation awareness are analyzed. The results suggest that brain-based features
(mental workload and distraction) which represent the covert aspect of attention are better suited to predict
the secondary performance metrics. On the other hand, glance based features which represent the overt
aspect of attention, are shown to be the best predictors of the primary performance metrics.
INDEX TERMS Human-Robot Interaction, Human Performance, Individual Differences, Mental Work-
load, Physiological Measures, Situation Awareness.
I. INTRODUCTION
MOBILE robots are becoming more involved in the
variety of tele-exploration tasks such as surveillance
of large areas [1], search and rescue [2] and military missions
[3]. However, fully autonomous systems are still vulnerable
in unconstrained environments where unanticipated events
and uncertainties are unavoidable. As a result, human-robot
teamwork is often required in time-critical and risk-sensitive
missions. Depending on the level of autonomy and the nature
of the task, one human operator may even be able to coop-
erate with multiple robots simultaneously [4]. For instance,
in supervisory control where team coordination is the main
task of the operator, it would be feasible for a single operator
to cooperate with up to 12 unmanned aerial vehicles [5], [6].
However, for low-level control tasks such as visual scanning
or manual maneuvering, which is the focus of this study,
the size of the human-robot team is significantly smaller to
attenuate the cognitive burden on the operator [7].
To consistently sustain vigilance, an operator needs to
maintain a high-level of Situation Awareness (SA) while
dividing his/her attention among team members. The most
important factors contributing to the loss of SA are mental
workload, level of expertise and multitasking (management
of attention) [8]. For instance, performing multiple tasks with
a limited level of autonomy may increase the mental work-
load to the extent that it exceeds operator mental capabilities
and ultimately leads to the loss of SA and performance
degradation. To predict and avoid such situations, a coherent
human-aware framework is required to enhance human per-
formance by adapting the system behaviors, modifying the
task allocations or changing robots’ initiative level, in order
to maintain the operator’s performance within an acceptable
VOLUME X, 2018 1
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
range.
The main challenge in the design of such adaptive frame-
works is the lack of direct and real-time evaluations of
performance factors. For instance, although reaction time and
the accuracy of the target detection are the most popular
performance metrics in visual search tasks [9], their assess-
ments require prior knowledge about the exact locations of
the targets which are not available in real tele-operation
scenarios. In other words, the performance metrics are hidden
variables that need to be estimated from some observable
measurements.
To address this issue, this paper proposes a cognitive
model to estimate the operator’s performance and situation
awareness from their physiological measurements and also
taking into account their individual differences in cognitive
abilities. For this purpose, we have considered cognitive
features extracted from two physiological modalities: brain
activity and eye movements. By studying the relationship
between these features and the operator’s performance and
situation awareness, we develop a model to estimate his/her
performance in real-time. In the next section, we provide
a background on physiological monitoring in human-robot
interaction and summarize the main contributions of this
paper.
II. LITERATURE REVIEW
Recent advances in the non-invasive sensing technologies
have provided portable and wearable solutions to record and
process physiological data in out-of-lab settings. As a result,
these systems have received much attention in recent years to
estimate different cognitive states such as mental workload,
engagement [10], satisfaction [11], distraction and fatigue
[12], or task difficulty [13].
For instance, Bekele and Sarkar [14] developed an adap-
tive framework to modify the behavior of a social robot
through monitoring physiological responses. Esfahani and
Sundararajan [11] proposed an approach to detect human sat-
isfaction level in human-robot interaction. Largest Lyapunov
exponents and alpha-asymmetry in the frontal cortex were
the main features of that study which were extracted from
brain activity of the subjects. Wright et al. [15] studied the
relationship between eye movement features and subjective
measures of workload and SA in a collaborative task with a
team of mobile robots. They observed positive correlations
between eye fixation count and self-reported workload and
SA in manual control of robots. Novak et al. [16] utilized
physiological signals including EEG, autonomic nervous sys-
tem responses and eye tracking to estimate human workload
and effort in physical interactions with a robotic arm. Similar
modalities are also used to classify task difficulty in software
development to avoid programmers from introducing bugs in
their codes [13].
Physiological data has also been studied to estimate human
performance in variety of domains such as driving [17],
aviation [18], rehabilitation [19] and surgical training [20]. In
most of these applications, mental workload has been exten-
sively investigated as the main cognitive factor that influences
human performance. In these cases, adaptive automation
techniques are used to keep operator workload within an
acceptable level [21], [22]. Although mental workload-based
adaptive frameworks have been found to enhance the per-
formance of human-machine interaction, they face several
challenges that limit their successful implementation [23].
Some of these limitations are summarized as follow:
First, individual differences in cognitive abilities signif-
icantly influence the human performance in multi-tasking
environments. Depending on the nature of a task, different
cognitive skills such as attentional control, spatial ability,
working memory capacity and gaming experience can affect
the operator’s performance in supervisory control of multiple
robots [24], [25]. In complex visual demanding tasks, action-
video-gamers have been found to outperform infrequent/non-
gamers. They have demonstrated enhanced spatial resolu-
tion in visual processing [26], faster reaction time [27]
and improved attention allocation [28]. Therefore, a reliable
performance estimation model should properly include the
individual differences most relevant to each performance
metric.
Second, mental workload estimated from the brain activ-
ity, particularly EEG, is task-specific [23], [29]. An EEG-
based classifier can determine the mental workload very well
when used for the task on which the classifier was trained,
but may fail if used for similar tasks not included in the
training data. This is particularly important in the case of
tele-operation, where the physical separation between the
operator and the physical environment increases the variabil-
ity and uncertainty in the task type and make it impractical
to train a classifier on all possible scenarios. Yet, there is
no study to demonstrate to what extent the standard EEG-
based workload classifiers can be applied for performance
prediction in complex human-machine systems.
Third, increasing the level of mental workload typically
leads to performance degradation, however based on the
nature of a task, performance and workload might be dissoci-
ated [30]. Therefore, workload-based adaptation methods are
more effective in high-risk or long duration tasks [31] where
operators are subjected to high levels of workload and safety
is the main concern. It has been suggested that measures
of both mental workload and SA are vital for elevating the
performance of complex human-machine systems as they
complement each other [32], [33].
The purpose of the present study is to examine the relation-
ship between physiological measures, operator performance
and SA in a tele-exploration task with a small group of
robotic drones. Compared to prior works which are mostly
limited to the workload analysis [16], [17], [20], [31], in
this study we aim to address the above mentioned problems
through: (1) Developing a more comprehensive performance
prediction model by including cognitive features extracted
from two different modalities (eye movement and brain ac-
tivity) as well as the effect of individual differences in our
assessment analysis. The cognitive features are selected to
2VOLUME X, 2018
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
Eye tracker
Simulation
Environment
Graphical User
Interface
Brain Computer
Interface
Joystick
FIGURE 1: The recording setup of the experiment.
capture information about the mental workload and both the
overt and covert aspect of attention. (2) We also examine
the effects of individual differences on the physiological
measures and show how the identified factors can be used
to individualize the performance estimation models. (3) Fur-
thermore, we investigate the generalizability of EEG indices
of workload and distraction obtained from standard neuro-
cognitive assessment tests, to be used in operational environ-
ments.
In general, the outcome of this study is expected to ad-
vance knowledge on the real-time prediction of the operator’s
performance and his/her level of situation awareness in tele-
operations. The developed performance prediction model in
this paper can be utilized in future studies to define and
develop an intelligent layer (e.g., [34]) between human op-
erators and robotic agents, where the estimated performance
metrics play a critical role in the adaption of the system
behavior and enhancing task performance.
III. MATERIALS AND METHODS
An interactive physics-based simulation environment is de-
veloped to study the cognition of human operators during in-
teraction with a small team of robots (Fig. 1). Eye movements
and brain activity, recorded in terms of EEG signals, are
the two physiological modalities used to determine human
cognitive states. An eye tracker provides physiological data
in terms of gaze location that is usually an indicator of
the overt aspect of attention, while EEG signals reflect the
neural activity of the brain which can be considered as the
covert aspect of attention. Therefore, by combining these two
modalities we took the advantage of both overt and covert
aspects of attention in our cognitive analysis. The details of
the experimental setup and conducted user study are provided
in this section.
A. PARTICIPANTS
22 subjects (16 males and 6 females) were recruited from
University at Buffalo School of Engineering. Participants’
ages ranged from 23 to 37 years (M = 26.8, SD = 3.7)
and they had normal or corrected to normal vision. None of
the participants had prior experience with manual control of
robotic drones, however, they were frequent computer users.
B. APPARATUS
In order to conduct experiments in a controlled environment,
a simulator is developed based on the V-REP framework
[35] which enables users to create and visualize detailed and
realistic 3D virtual environments. A customized version of
the built-in drone model in V-REP is used as the robotic
agent. Two vision sensors are attached to the drone’s main
body to provide users with RGB video streams from the top-
down and front field of views. The angle of view for vision
sensors is set to 45◦. Drone dynamics and corresponding
low-level controllers are also implemented in the simulation
environment such that each drone can fly in either stabilized
manual or autonomous mode by tracking point-to-point tra-
jectories. The details of the developed interactive simulation
environment can be found in [36].
Furthermore, a graphical user interface (GUI) is developed
to enable users to interact with the robotic drones for the
tele-exploration task of this study. For each drone, this GUI
provides users with a message box, rough map, and video
stream switchable between the front and top-down cameras.
A commercial 4-axes joystick is also used as the command
interface and navigation tool for the manual control.
C. DATA COLLECTION
1) Brain Activity
The B-Alert X10 wireless headset (Advanced Brain Mon-
itoring ©, Carlsberg, CA, USA) is utilized to capture the
brain activity of the subjects, non-invasively. EEG signals
are recorded with a sampling rate of 256 Hz at 9 sensor
locations on the scalp and referenced with respect to linked
electrodes placed on the mastoids. The sensor locations are
Fz, Cz, POz, F3, F4, C3, C4, P3, and P4 channels based
on the 10/20 international EEG system. Real-time cognitive
states including the level of mental workload and distraction
are then extracted from artifact-free EEG data for each 1-
second epoch [10].
2) Eye Movement
The Eye Tribe eye tracker (The EyeTribe ©, Copenhagen,
Denmark) is used to record eye movements at a sampling
rate of 30 Hz. This is a non-contact tracking system that
captures user’s binocular gaze information via an infrared
camera. The Eye Tribe is placed below the users’ monitor
and calibrated for each individual participant prior to the
experimental study. Eye movement features are extracted
from estimated gaze location on the computer display as well
as pupil dilation.
3) Individual Differences
Visual Search (VS) and Multiple Object Tracking (MOT)
tests are used to capture individual differences in visual skills.
These two skills are shown to be related to the performance of
video gamers in terms of target detection, situation awareness
and reaction time [28]. In our specific application, VS is
used as an indication of how fast and accurate targets can be
VOLUME X, 2018 3
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
+
+
L
L
L
L
L
T
L
Fixation
Stimuli
Response
500 (ms)
1000 (ms)
RT (ms)
FIGURE 2: Trial sequence of the Visual Search paradigm.
detected, whereas MOT represents the number of objects that
an individual can track simultaneously. The results of these
quantitative assessments are used to categorize subjects’ skill
levels into low and high groups. This information is then
used in the statistical analysis and also personalizing the
performance prediction models.
D. INDIVIDUAL DIFFERENCES ASSESSMENT
After briefing the participants about the experimental proce-
dure, individual differences in visual skills are assessed. EEG
baseline of cognitive states is also recorded by conducting
a neuro-cognitive test. Finally, participants are instructed to
perform a tele-exploration task in which their brain activity,
eye movements, and performance are measured. Followings
are the description of the experiments for recording VS, MOT
and EEG baselines.
1) Visual Search
The VS test conducted in this study is adopted from the easy
display VS experiment designed by Castel et al. [37]. Fig. 2
illustrates the sequence of a VS trial.
Participants are asked to determine the presence or absence
of a target letter (“T”) by pressing keyboard buttons “p” for
presence and “a” for absence. At the beginning of each trial,
a cross shape is illustrated in the center of the display as
the fixation point for 500 ms. In each trial, the target letter
and a set of distractors are randomly located on the screen
with 50% probability for the presence of the target letter. A
total number of 200 trials are recorded from each participant.
These trials are randomly selected from four different sizes
of 8, 12, 16 and 20 letters, each of which repeated 50 times.
Upon receiving a response key from the participant, the
trial is ended and both the reaction time and response key
are recorded. Participants are asked to react as quickly and
accurately as possible. A publicly-available MATLAB code
[38] based on PsychToolbox [39] is modified and used to
measure participants’ VS skill.
2) Multiple Object Tracking
MOT test offers an experimental approach to evaluate the
number of moving objects that a person can track simul-
taneously [40], [41]. Contrary to the VS task that requires
FIGURE 3: Trial sequence of Multiple Object Tracking paradigm. Red
circles in the third screen represent the target objects which the participants
were asked to keep track of them.
discrete shifts of attention over time, MOT task depends upon
concurrent allocation of attention to several moving objects.
For example, driving requires allocation of attention to mul-
tiple moving objects, such as other vehicles and pedestrians.
Likewise, in the case of interaction with multiple drones, the
level of operator’s MOT skill can potentially affect his/her
performance and situration awareness.
Fig. 3 demonstrates the sequences of a sample MOT trial
that is conducted in this study. At the beginning of each
trial, a set of 12 identical circles appear on the screen and
move in random directions. After 4 seconds, a random subset
of these circles is temporally highlighted to identify the
target objects. After 4 seconds of temporary highlights, the
target objects turn back to the normal color and become
visually indistinguishable from the others. Participants are
asked to track the moving targets even after the highlights
are removed. After 6 seconds of tracking, circles stop and the
participants are asked to identify all the target circles. This
experiment is conducted with 5 sets of 2 to 6 targets. Ten
trials are repeated per each set resulting in a total of 50 trials.
The number of correctly identified targets is averaged over
all the trials and is used as the MOT score.
3) EEG Baselines
Upon the completion of VS and MOT tests, a neuro-cognitive
assessment (Auditory Psycho-Vigilance [10], [42]) is con-
ducted to individualize the EEG classifiers. It is a standard
eyes closed vigilance task used as a baseline for estimating
the level of distraction/relaxed wakefulness of subjects.
E. EXPERIMENTAL PROCEDURE
Participants receive training and then practice on the simu-
lator’s elements and the required tasks. They are also asked
to perform a short practice trial to gain adequate experience
in controlling the drones by joystick and accomplish all the
task objectives independently. In the main experiment, partic-
ipants perform an aerial exploration using two robotic drones
to detect and identify specific geometrical objects among
distractors (Fig. 4d), which are randomly distributed over the
search area (Fig. 4b). The experiment is approximately 20
minutes long during which participants monitor two drones;
4VOLUME X, 2018
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
(a) (b)
Top View
Side View
MGMC
FMC
(d)
Build 1
Build 2
Build 3
(c)
FIGURE 4: Simulated tele-exploration task: (a) top view of search area, (b)
Random distributions of targets and distractors over the search area, (c) side
view of the search area, (d) Target shapes (cube and torus) and 6 distractors.
Tasks
VS-VS
LVS-VS HVS-VS
Low Task-Load High Task-Load
Task-Type 1
1st Drone : Autonomous
2nd Drone : Autonomous
MC-VS
GMC-VS FMC-VS
Low Task-Load High Task-Load
Task-Type 2
1st Drone : Manual
2nd Drone : Autonomous
FIGURE 5: Factorial design of the experiment.
one always in autonomous mode, and the other one requires
manual control after a while.
As shown in Fig. 5, the experimental scenario consists of
two levels of Task-Type, each of them includes two levels
of Task-Load resulting in a factorial design. The experiment
begins with the simultaneous visual scanning of the camera
feeds of the two drones (VS-VS). In this level of Task-
Type, both drones autonomously track predefined trajectories
and the participant is asked to visually scan the captured
videos from top-down cameras and record the location of
observed targets by pressing a button on the joystick. Two
levels of Task-Load including low (LVS-VS) and high (HVS-
VS) visual load are included in the experiment. The higher
visual task load is achieved by increasing the speed of the
drones and accordingly faster video streams to process by
the operator.
The second level of Task-Type is composed of simulta-
neous manual control and visual scanning tasks (MC-VS).
Instantly after VS-VS task, the first drone sends a message
and asks to be switched to the manual mode while the second
drone continues the exploration in the autonomous mode.
In the manual mode, users have access to the drone’s front
camera to navigate easier. There are also two levels of Task-
Loads in manual maneuvering, that are gross and fine manual
control. In the Gross Manual Control (GMC), subjects are
asked to navigate the first drone along a provided path on
the map (see the GMC path in Fig. 4a) and simultaneously
detect targets on the second drone’s camera (GMC-VS). This
maneuvering is conducted in the presence of no obstacle, thus
it is considered as low level difficulty. Contrary to GMC-VS,
the second level of Task-Load requires fine manual control
of the first drone along with visual scanning of the second
drone (FMC-VS). This task is called fine manual control
since obstacle avoidance and fine motion control are needed
to enter and explore the inside of two abandoned buildings
(see Fig. 4a and c).
IV. DATA ANALYSIS
A. EYE TRACKING METRICS
Eye tracking metrics are mainly defined in an overall or AOI-
based (Area Of Interest) fashions. AOIs are regions of the
display or visual stimuli that contain objects of interest and
are usually defined based on the semantic information of the
display. In this study, the GUI is divided into 7 AOIs as shown
in Fig.6. These AOIs include the message boxes, maps and
camera views of the drones.
To identify periods of gaze fixations from noisy data of
eye movements, a publicly available software [43] relying on
a dispersion-threshold identification algorithm [44] is used.
This method identifies each fixation as a group of consecutive
points within a particular dispersion, or maximum separation
in a sliding window of 11 samples (corresponding to about
360 ms). For each trial, a threshold of 7.5% of the maximum
variance is used to identify saccades corresponding to peaks
in signal variance. Then, the fixation detection criterion is de-
fined as the period of time between two successive saccades
with a minimum duration of 120 ms as suggested in [44].
The optimal choices of thresholds for variance and minimum
fixation duration are obtained through visual inspection on
a random subset of the recorded data. Each fixation point is
then compared with the predefined AOIs to identify glances
FIGURE 6: Operator GUI and defined AOIs along with a heatmap of gaze
locations corresponding to a sample practice session.
VOLUME X, 2018 5
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
made at each of the AOIs.
Holmqvist et al. [45] have gathered a comprehensive re-
view of different eye tracking measures among which three
metrics are adopted in this study that best represent underly-
ing cognitive activities of the tele-exploration. These metrics
are Fixation Rate (FR), Glance Ratio (GR) and Pupil Size
(PS) which are defined as follow:
Fixation Rate: FR is the number of fixations occurred
within one second time windows. The interpretation of this
metric varies based on the task type and associated cognitive
functions. Usability studies have shown that in general, FR
is related to semantic informativeness and search difficulty
[46].
Glance Ratio: GR represents the percent of time glances
are within an area of interest. In our study, it is defined as the
percentage of the glances within AOIs associated with the
first drone (1, 3 and 5) to the ones associated with the second
drone (2, 4 and 6). Typically, glance duration correlates with
the level of attention to an AOI [45].
Pupil Size: PS or dilation is one of the relatively continu-
ous measurements that is provided by eye tracker systems.
The pupil size is idiosyncratic and varies across subjects.
To address this issue, we convert the raw PS values to z-
scores for each individual. Moreover, to minimize the effect
of luminance variation on PS, the scene light is maintained
relatively at the same level throughout the experiment.
B. EEG METRICS
Two cognitive measures, Mental Workload (MW) and Dis-
traction level (DS), are extracted from EEG signals. MW
reflects working memory load and cognitive processing. It
corresponds to the level of task difficulty and reasoning
process. DS represents the inability of a subject to maintain
passive attention and consequently involuntary switching of
attention. The B-Alert© software is used to acquire EEG
signals and quantify the aforementioned EEG metrics. Fol-
lowing is a brief description of the procedure and main
features used by the B-Alert software to extract these metrics.
Additional details on signal processing and classification
methods can be found in [10], [47].
All the EEG signals are band-passed filtered (0.1-70 Hz
and 20 dB/decade roll-off) and then converted to digital
format with 256 Hz sampling rate and 16 bits/sample res-
olution. Filtered signals are then transmitted via Bluetooth
to a PC. Wavelet transformation is then used to identify and
decontaminate artifacts caused by muscle movements and
eye blinks [10]. Finally, EEG absolute and relative power
spectral density (PSD) for bipolar electrodes listed in Table
1 are computed using Fast-Fourier transform applied to 1-sec
epoch data with 50% overlapping Kaiser windowing.
Mental workload is quantified based on a linear Discrim-
inant Function Analysis (DFA) trained on a dataset of EEG
signals recorded from 67 participants who performed three
working memory benchmark tasks (mental arithmetic, grid
location and digit-span tasks) with different levels of diffi-
culty. Regarding distraction, a quadratic DFA is used by the
B-Alert© software. The coefficients of this model are indi-
vidualized for each participant according to the baseline tasks
described in the experimental procedure. The corresponding
PSD features used by the workload and distraction classifiers
are listed in Table 1.
TABLE 1: EEG variables used for classification of mental workload and
distraction level.
1-4 Hz
5-7 Hz
8-13 Hz
14-24 Hz
25-40 Hz
Absolute and relative PSD variables for MW classifier
C3C4
×
×
×
×
CzPOz
×
×
F3Cz
×
×
×
F3C4
×
×
×
FzC3
×
×
×
FzPOz
×
×
×
×
Absolute and relative PSD variables for DS classifier
FzPOz
×
×
CzPOz
×
×
×
×
×
C. INDIVIDUAL DIFFERENCES
Participants are divided into two groups of low and high
skill with respect to their VS scores. For each participant,
the average score of recorded reaction times is used to assign
group membership. Participants with reaction times less than
one standard deviation below the mean (M=1.68,SD =.23)
are categorized as high and the rest as low skill level. Sim-
ilarly, participants are categorized into high and low MOT
skills. The average percentage of correctly identified moving
targets is used to define the skill level of each participant.
One standard deviation above the mean (M=.84,SD =.07)
is considered as the threshold to group participants into two
levels of MOT skill. Fig. 7 illustrates the group distribution
of VS and MOT skills.
Visual Search score (sec)
1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2
Multiple Object Tracking score (%)
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
High MOT
Low MOT
High
VS Low
VS
FIGURE 7: The group distribution of VS and MOT skills. Dashed lines
indicate the threshold values.
D. TASK PERFORMANCE METRICS
Steinfeld et al. [48] have studied common operator perfor-
mance metrics for task-oriented human-robot interaction and
suggested that performance metrics should provide informa-
tion about the operator’s attention/awareness and workload.
6VOLUME X, 2018
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
TABLE 2: List of task performance metrics.
Metrics
Definition
Target Detection (TD)
Percentage of correctly identified targets
False Detection (FD)
Number of distractor misidentified as target
Reaction time (RT)
Time between the appearance of a target in a
camera view to its detection by operator
Level 1- Situation
Awareness (SA)
Cumulative time period that new information
is displayed but no glance has been made on
relevant AOI, divided by the total task time
Based on this criteria, we define four different metrics to as-
sess tele-exploration performance of each individual operator
as listed in Table 2.
Reaction times and hit rates (measured in terms of target
detection) are indirect indicators of cognitive workload and
have been widely used in human factor studies [49], [50] and
therefore, are adopted in this study as the “workload-related
performance metrics”. The second group of performance
metrics are false detection and situational awareness that are
considered as “attention-related metrics”.
Situation awareness is defined by Endsley as “the percep-
tion of the elements in the environment within a volume of
time and space, the comprehension of their meaning, and the
projection of their status in the near future” [8]. Endsley’s
model defines three level of SA including the perception of
the elements in the environment (level-1), comprehension of
the current situation (level-2) and projection of the future
status (level-3). In our target detection task, the perception
and comprehension elements of SA correspond to spatiotem-
poral visual attention and identification of the targets from
distractor objects, respectively.
Traditionally, SA is measured at the end of the experiment
using subjective methods such as SAGAT (Situation Aware-
ness Global Assessment Technique) and SART (Situation
Awareness Rating Technique) questionnaires [51]. Another
approach is to employ explicit techniques throughout the ex-
periment like SPAM (Situation Present Assessment Method)
[52]. Neither of these techniques can be used for real-time
SA assessment. For this reason, eye movement data are used
as the best alternative to estimate SA in real-time.
Eye tracker systems can provide the fixation points to
which a user pays attention without moving his/her eyes
(overt attention). In other words, it can capture the percep-
tual aspects of SA (level-1) in visual demanding tasks [53].
These features are often combined with the environmental
information to provide a more reliable estimate of SA. For
instance, percentage of time fixating on relevant AOIs is used
as prediction of overall SA [54].
A similar approach is utilized in our study to measure
level-1 SA. Information of eye tracking data and virtual scene
are combined to mathematically define this measure in a time
window with a length Twas (1),
SA = 1 −Tloss
Tw
= 1 −Pn
i=1 ∆ti
Tw
i=1:n(1)
where Tloss is the cumulative duration of time periods (∆ti),
in which new visual information is presented in the GUI but
the operator fails to capture (by not making glances on the
relevant AOIs). To identify whether a participant responded
to new visual information by making eye fixations on it, a
time-threshold is defined for each video stream by (2),
χ=2htan α
s(2)
where sand hare the drone speed and altitude, respectively,
and αdenotes the angle of view of the top-down camera. In
fact, χrepresents the amount of time that a target is observ-
able in the camera’s view. It should be noted that the proposed
SA metric contains information about the drones motions and
hence differs from the eye movement features described in
section IV.A. To better understand the underlying mechanism
for measuring the SA, a sample case is presented in Fig. 8.
Suppose that χ1and χ2are the time-threshold related to
the first and second drones, respectively. The bottom graph
illustrates the glance state that can be directed toward the first
(D1) or second drone (D2) or neither of them (none). Based
on the glance state, a timer measures the elapsed time form
the instant that the glance is switched and if the elapsed time
reaches its threshold the associated ∆tiwill be computed.
Finally, SA metric can be computed using (1).
Time
Elapsed 1 Elapsed 2
none
Glance
Δ𝑡1
Δ𝑡2
𝑇
𝑤
𝜒2
𝜒1
D2
D1
FIGURE 8: A sample case to illustrate SA measurement.
V. RESULTS OF STATISTICAL ANALYSIS
The purpose of the statistical analysis provided in this section
is threefold:
(i) To investigate the main effects of changing Task-Type
and increasing Task-Load on the cognitive features and
performance metrics (test of within-subject factors).
These results are used to identify significant effects on
the cognitive features and also validate the difficulty
manipulation incorporated in the experimental design.
(ii) To study the influence of individual differences on the
performance metrics and their interactions with cogni-
tive features.
VOLUME X, 2018 7
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
(iii) To construct a linear model for predicting the perfor-
mance and situation awareness (hidden states) from
cognitive features (observable states) by taking into
account characterized individual differences.
For this purpose, task performance, eye movement, and
EEG metrics are separately subjected to a series of 2×2×
2×2(VS ×MOT ×Task-Type ×Task-Load) multivariate
analysis of variance (MANOVA) with VS and MOT groups
as between-subject factors, whereas Task-Type and Task-
Load are assigned as within-subject factors. When significant
results are identified, multiple univariate ANOVAs are con-
ducted for each dependent variable as a follow-up procedure
to interpret the multivariate effect.
To protect against Type-I error inflation due to conducting
multiple univariate tests, Bonferroni adjustment of the alpha
value (α/ number of dependent variables) is imposed. This
adjustment leads to α=.0125,.0167 and .025 for perfor-
mance, eye features and EEG features, respectively. In case
of a significant two-way interaction, test of simple main
effects is conducted to examine the differences between each
levels of independent factors.
A. MAIN EFFECTS OF WITHIN-SUBJECT FACTORS
MANOVA reveals statistically significant main effects of
Task-Type and Task-Load on the performance metrics, eye-
tracker features as well as EEG features. A significant Task-
Type ×Task-Load interaction is also observed in the per-
formance and eye-tracker metrics but not EEG features.
MOT ×Task-Load interaction is also found significant for
eye-tracker metrics. The detailed statistical analysis of the
significant main effects performed by the MANOVA and the
follow-up ANOVA analysis are listed in Table.3.
TABLE 3: Significant Main Effect of MANOVA and follow-up ANOVA
analysis.
Main
MANOVA
Follow up ANOVA
Effect
Group
F*
P-value
2
Factor
F**
P-value
2
Task
Type
Performance
10.3
<0.001
0.73
TD
14.7
0.001
0.45
RT
34.3
<0.001
0.65
SA
24.2
<0.001
0.57
Eye
42.2
<0.001
0.89
GR
16.5
0.001
0.48
FR
11.1
0.004
0.38
PS
130.0
<0.001
0.88
Brain
19
<0.001
0.69
MW
37.8
<0.001
0.68
DS
21.5
<0.001
0.55
Task
Load
Performance
7.9
0.001
0.68
TD
9.6
0.006
0.35
SA
36.8
<0.001
0.67
Eye
5.3
0.01
0.5
GR
10.9
0.001
0.47
Brain
5.18
0.017
0.38
DS
10.4
0.005
0.37
Type
x
Load
Performance
6.2
0.004
0.62
RT
19.8
<0.001
0.53
Eye
16.2
<0.001
0.75
GR
12.2
0.001
0.5
PS
23.9
<0.001
0.57
MOTx
Load
Eye
3.3
0.049
0.38
PS
8.3
0.01
0.32
* MANOVA Analysis: Performance F(4,15), Eye F(3,16), EEG F(2,17)
**ANOVA Analysis: F(1,18)
0
0.3
0.6
0.9
1.2
1.5
-1
1
3
5
7
9
-0.1
0.1
0.3
0.5
0.7
0.9
*
* *
Target
Detection (%)
(a)
(b)
(c)
*
* * *
* *
LVS-VS HVS-VS GMC-VS FMC-VS
False
Detection (#) Reaction
Time (sec) Situation
Awareness (%)
*
*
**
**
Pupil Size (Z-Score)Fixation Rate Glance Ratio
DistractionMental Workload
*
*
FIGURE 9: Mean and standard deviations of (a) performance, (b) eye
movement, and (c) EEG metrics. * indicates a significant simple main effect.
Fig. 9a shows the mean and standard deviation of perfor-
mance metrics in each of the the two Task-Types and Task-
Loads. Statistical analysis indicates that changing the Task-
Type from visual search to manual control and increasing
Task-Load degrade the performance in terms of target de-
tection (TD) and situation awareness (SA) consistently. For
reaction time (RT) a two-way interaction of Task-Type ×
Task-Load is found significant. Follow-up tests of simple
main effects for this interaction demonstrate that increasing
Task-Load in the visual scanning (VS-VS) results in a faster
RT, however, an opposite behavior is observed for manual
control (MC-VS) task. No significant observation is obtained
for false detection (FD).
Fig. 9b illustrates the results regarding the eye-tracking
features. Univariate tests of within-subject factors reveal
significant main effects of Task-Type on fixation rate (FR),
glance ratio (GR) and pupil size (PS) and a significant main
effect of Task-Load on GR. These results indicate that by
switching from visual scanning to manual control FR de-
creases, whereas GR and PS increase. Task-Type ×Task-
Load interactions are revealed for GR and PS. Tests of simple
8VOLUME X, 2018
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
(a) (b)
FIGURE 10: The effect of individual differences in VS and MOT skill on
(a) target detection and (b) situation awareness.
main effects for these interactions indicate that increasing
Task-Load in MC-VS task led to higher PS and GR, whereas
increasing Task load in the VS-VS task results in a smaller
pupil size and no significant change in GR.
Finally, the follow-up ANOVAs for EEG features reveal a
significant main effect of Task-Type on the mental workload
(MW) and distraction (DS) as well as a significant main
effect of Task-Load on the DS. Tests of simple main effects
reveal that switching from visual scanning to manual control
causes an increase in MW and a decrease in DS. However,
increasing the Task-Load within the Task-Types leads to
no significant change in the EEG features. Fig. 9c provide
a comparison between the EEG indices of workload and
distraction recorded during the two Task-Types and Task-
Loads.
B. MAIN EFFECTS OF BETWEEN-SUBJECTS FACTORS
Univariate tests of between-subject factors reveal signifi-
cant main effects of Visual Search (VS) skill on target
detection (F(1,18) =7.4, p =.012, η2
p=.3) and MOT skill on
situation awareness (F(1,18) =8.12, p =.011, η2
p=.31). Par-
ticipants with high VS skills outperform others in target
detection. As shown in Fig.10a, increasing the task load has
no effect on their target detection performance, however, it
affects the performance of the remaining subjects in low
VS group significantly. Interestingly, for the situation aware-
ness (SA), MOT score is the dominant individual difference
among the subjects. As shown in Fig.10b, subjects with high
MOT scores demonstrate a higher situation awareness.
Moreover, an MOT ×Task-Load interaction is revealed
(F(1,18) =8.3, p =.01, η2
p=.32) that significantly affects
PS. Tests of simple main effects show that increasing Task-
Load only increase the pupil sizes of participants with low
MOT and has no effect on the ones with high MOT scores.
C. RESULTS OF REGRESSION ANALYSIS
In addition to the MANOVAs, separate stepwise multiple
linear regressions are conducted to identify significant cor-
relations between observable cognitive features and hidden
performance metrics. The results are summarized in Ta-
ble 4. The statistical significance of each of the regression
models can be realized through the adjusted coefficient of
determination R2. By comparing this value for each of the
regression models, it can be seen that situation awareness and
target detection can be described very well with only three
regressors out of which one is an individual difference metric.
The results of stepwise regression analysis shows that PS
has a negative relationship with TD and SA; FR is negatively
related to FD; MW has a negative correlation with RT. More-
over, GR is found to be the dominant predictor of operator
performance, as it is strongly related to variations in TD, RT
and SA. DS is also found as a significant predictor for FD
such that an increased number of wrongly detected targets is
associated with high DS. Furthermore, the individual differ-
ences in terms of VS and MOT skills are positively correlated
with TD and SA, respectively.
VI. DISCUSSION
Let’s recall that our performance metrics are categorized into
two groups: workload-related (Target Detection and Reaction
Time), and attention/awareness-based (Situation Awareness
and False Detection). Among these factors, RT and FD
are considered as the secondary measures since they can’t
capture the inability of subjects in processing the incoming
information. For instance, increasing the load in the visual
scanning task improves the reaction time (see Fig. 9c). This is
due to higher visual load which is imposed by increasing the
drones’ speed and consequently providing participants with
a shorter time window to process video streams and make
decisions. Thus, participants miss more targets (lower TD)
while the identified ones are detected faster (lower RT). A
similar trend is observed in FD. The number of incorrectly
detected targets is reduced in tasks with higher workload but
it doesn’t imply a better performance as SA is significantly
decreased.
With respect to the primary metrics, significant perfor-
mance degradation in target detection and situation aware-
ness is observed as the task becomes cognitively more chal-
lenging (switching to manual control or increasing the task
load). This is well-documented in the human factor literature
[25] that higher mental workload will cause performance
degradation and loss of situation awareness. Although these
two observations are often accompanied, they are two distinct
concepts that are intricately related such that one affects
and is affected by the other [55]. They both are influenced
TABLE 4: Results of the regression analysis.
Performance
Significance
Variable
Weight
P-value
TD
F*= 44.3, R2 =0.75
P-value <0.001
GR
-0.53
<0.001
PS
-0.54
0.001
Task Type x VS
0.38
0.001
SA
F*= 77.5, R2 = 0.85
P-value <0.001
GR
-0.59
<0.001
PS
-0.29
0.004
MOT
0.27
<0.001
RT
F**= 44.3, R2 =0.59
P-value <0.001
GR
0.69
<0.001
MW
0.29
0.006
FD
F**= 44.3, R2 = 0.2
P-value <0.001
FR
0.34
0.019
DS
0.33
0.02
*F(3,40), **F(2,41)
VOLUME X, 2018 9
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
by many of the same human factors (e.g., limited working
memory, individual differences) and system variables (task
type and difficulty). A conceptual relationship between men-
tal workload and situation awareness (shown in Fig. 11.)
is provided by Vidulich and Tsang [55] which is used to
discuss the outcome of the statistical analysis. It considers
two main factors, the ‘Mental Workload’ and the ‘Strategic
Management of Attentional Resources’, as the sources of
the performance degradation at higher task loads. It should
be noted that the supply of attentional resources is mod-
ulated by ‘individual differences’. These three factors are
highlighted in Fig. 11 and the physiological monitoring are
used to extract overt and covert cognitive features from the
working memory (blue lines) and strategic management of
the attentional resources (red lines) of the framework. In the
rest of this section, we will discuss the results of statistical
analysis observed on these factors.
W
O
R
L
D
Task Type
& Load
Performance
Perception +
Mental
Workload
Situational
Awareness
Working
Memory
Long term
Memory
+
Expertise
Attentional
Resources Strategic
Management
EEG and Eye
Features
EEG and Eye
Features
Individual
Differences
VS + MOT
FIGURE 11: Conceptual framework illustrating the relationship between
workload, attention and performance (adopted from [55]).
A. WORKLOAD-RELATED FEATURES
Transitioning from visual scanning to manual control,
workload-related features (EEG workload, fixation rate and
pupil size) are significantly changed because of higher atten-
tion demand required for performing manual control com-
pared to visual scanning. However, increasing Task-Load
within each level of these Task-Types doesn’t affect EEG
workload (MW) and fixation rate (FR).
Mental workload (MW), extracted from EEG activities, is
an indication of overall working memory load and includes
different aspects (e.g., visual, auditory, cognitive, motor,
etc.). Therefore, it may not be sensitive enough to detect
Task-Load variations within the same Task-Type. As a re-
sult, relying on EEG as a stand-alone modality to measure
operator states may need identification of task-specific EEG
patterns. Task dependency of EEG features to estimate men-
tal workload has been observed and pointed out by the other
researchers as well [23], [29].
Pupil size (PS) is generally an indication of visual work-
load [56] which is significantly changed at different levels
of task load and type. The average pupil size is increased by
transitioning from visual to motor task as well as by increas-
ing task load in the motor task from gross to fine manual
control which indicates increased workload. Surprisingly, a
higher visual load in the visual scanning task (VS-VS Task-
Type) causes a significant decrease in PS. This discrepancy
can be an effect of the frequency of targets and the duration
of which they are presented as discussed by Privitera et al.
[57]. In our experiment, higher visual task load is associated
with a faster video streams and accordingly shorter presence
of targets and distractors in the associated AOIs which has a
negative effect on the PS size.
B. ATTENTIONAL FEATURES
In this study, there are two cognitive features, glance ratio
(GR) and distraction (DS), that mostly capture the ability of
a subject in using his/her atentional and cognitive resources.
DS is significantly decreased in the manual task type but
there is no significant effect when the load is increased within
each Task-Type. However, GR is significantly increased in
the presence of higher motor control load (switching from
VS-VS to GMC-VS and then to FMC-VS).
Glance based metrics, such as GR on specific AOIs pro-
vide information about the overt aspect of users attention.
Typically, GR increases when there is difficulty in extracting
visual information, or some elements of the scene require
higher attention [53]. In other words, high GR values can
be interpreted as attentional biases in information gathering
from specific AOIs and subsequently the loss of SA [46].
In our experiment, the motion dynamics of drones causes
manual control task (MC-VS) to be more time consuming
than visual scanning (VS-VS), therefore, subjects’ attention
is more biased toward the first drone and accordingly higher
GR values. This trend is also observed by increasing motor
control load in MC-VS task (from GMC-VS to FMC-VS).
The EEG index of distraction (DS) measures the involun-
tary switching of attention as a basis of attentional distrac-
tion. Statistical analysis reveal a significant decrease of DS
from visual scanning to manual control task. This is in agree-
ment with the previous studies indicating that distraction is
mediated by working memory load [58].
C. PERFORMANCE PREDICTION
We observe that our measures of individual differences, ob-
tained from Visual Search (VS) and Multiple Object Tracking
(MOT) test, have distinct effects on the performance mea-
sures. Participants with high VS scores outperform in target
detection. On the other hand, the situation awareness feature
is not affected by VS but MOT score such that individuals
with higher MOT scores demonstrate higher SA throughout
the experiment. Although the tele-exploration task of this
study is more analogous to VS paradigm than MOT, partici-
pants’ performance in terms of maintaining SA is affected by
their skills in MOT. In fact, the VS score can properly reflect
10 VOLUME X, 2018
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
the speed of visual processing and discrete shift of attention,
whereas MOT can measure the ability of dividing attention
between different tasks in multi-tasking environments.
Individual differences in VS and MOT are also found to
be significant factors in the performance prediction. Task-
Type ×VS is revealed as a predictor of target detection and
MOT as a predictor of situation awareness indicating how
individual differences can be effectively used to individualize
regression models to predict performance based on physio-
logical data.
The overall results of performance prediction based on
regression analysis indicate that metrics measuring covert
aspects of attention, such as EEG indices of mental workload
(MW) and distraction (DS), mainly contribute to the predic-
tion of secondary performance metrics that are reaction time
and false detection.
VII. CONCLUSION
In this paper, we present a multi-modal physiological mon-
itoring approach to estimate the performance and situation
awareness of human operator in tele-exploration with a small
team of robotic drones. Unlike previous works which are
mostly limited to the workload analysis, our proposed ap-
proach consider individual differences as well as attention-
based features in addition to workload features.
EEG features are task and subject specific and therefor
EEG workload classifiers need to be trained on data of each
specific tasks which may not be practical in multi-tasking
domains. Using a standard working memory test, we demon-
strate that even without including the task specific informa-
tion in the EEG analysis, EEG features measuring the covert
aspect of attentions are suitable for estimating the secondary
performance metrics (reaction time and false detection). We
also demonstrate that the primary performance metrics are
well predicted by eye tracking features measuring the overt
aspect of attention. Our final contribution is to identify that
the individual differences in multiple object tracking and
visual search have significant effect in prediction of situation
awareness and target detection, respectively. In fact, by re-
moving the individual differences from the regression analy-
sis, we observe that the adjusted coefficient of determination,
R2, decreases from 0.75 to 0.67 in the case of target detection
and from 0.85 to 0.78 in the case of situation awareness. This
result can also be extended to other domains such as driving
that include both visual scanning and motor control tasks.
The future works should further investigate the individ-
ualization of the performance prediction model based on
the identified individual factors. Moreover, the prediction of
secondary performance metrics can benefit from including
EEG features that provides the modulations of attentions and
visual workload. Nonlinear and probabilistic models can also
be investigated as an alternative for multiple linear regression
analysis.
It is also crucial to note that we only conducted our ex-
periment with the minimum possible team of robotic agents
to reduce the complexity of the interaction. Future work
should also take into consideration the effect of the number
of robots. Moreover, this study was only focused on the sta-
tistical analysis of the main effects and correlations between
hidden performance factors and observable physiological
features. Future work should use the presented road-map, to
develop a probabilistic approach to estimate the operator’s
performance in real-time using the suggested physiological
features.
REFERENCES
[1] D. Perez, I. Maza, F. Caballero, D. Scarlatti, E. Casado, and A. Ollero, “A
Ground Control Station for a Multi-UAV Surveillance System,” Journal of
Intelligent & Robotic Systems, vol. 69, no. 1-4, pp. 119–130, jan 2013.
[2] M. Bernard, K. Kondak, I. Maza, and A. Ollero, “Autonomous transporta-
tion and deployment with aerial robots for search and rescue missions,”
Journal of Field Robotics, vol. 28, no. 6, pp. 914–931, nov 2011.
[3] K. P. Valavanis and G. J. Vachtsevanos, Handbook of Unmanned Aerial
Vehicles. Springer Publishing Company, Incorporated, 2014.
[4] J. W. Crandall and M. L. Cummings, “Developing performance metrics for
the supervisory control of multiple robots,” Proceeding of the ACM/IEEE
international conference on Human-robot interaction - HRI ’07, p. 33,
2007.
[5] M. L. Cummings and P. J. Mitchell, “Predicting controller capacity in
supervisory control of multiple uavs,” IEEE Transactions on Systems,
Man, and Cybernetics-Part A: Systems and Humans, vol. 38, no. 2, pp.
451–460, 2008.
[6] R. Parasuraman, S. Galster, P. Squire, H. Furukawa, and C. Miller, “A
flexible delegation-type interface enhances system performance in human
supervision of multiple robots: Empirical studies with roboflag,” IEEE
Transactions on systems, man, and cybernetics-part A: Systems and Hu-
mans, vol. 35, no. 4, pp. 481–493, 2005.
[7] J. Y. C. Chen and M. J. Barnes, “Supervisory Control of Multiple Robots:
Effects of Imperfect Automation and Individual Differences,” Human
Factors, vol. 54, no. 2, pp. 157–174, apr 2012.
[8] M. R. Endsley, “Toward a Theory of Situation Awareness in Dynamic
Systems,” Human Factors, vol. 37, no. 1, pp. 32–64, mar 1995.
[9] V. J. Gawron, Human Performance, Workload, and Situational Awareness
Measures Handbook, 2nd Edition. CRC Press, 2008.
[10] C. Berka, D. J. Levendowski, M. N. Lumicao, A. Yau, G. Davis, V. T.
Zivkovic, R. E. Olmstead, P. D. Tremoulet, and P. L. Craven, “EEG
correlates of task engagement and mental workload in vigilance, learning,
and memory tasks.” Aviation, space, and environmental medicine, vol. 78,
no. 5 Suppl, pp. B231–44, may 2007.
[11] E. Esfahani and V. Sundararajan, “Using brain-computer interfaces to de-
tect human satisfaction in human-robot interaction,” International Journal
of Humanoid Robotics, vol. 8, no. 1, pp. 87–101, 2011.
[12] G. Borghini, L. Astolfi, G. Vecchiato, D. Mattia, and F. Babiloni, “Mea-
suring neurophysiological signals in aircraft pilots and car drivers for the
assessment of mental workload, fatigue and drowsiness,” Neuroscience &
Biobehavioral Reviews, vol. 44, pp. 58–75, 2014.
[13] T. Fritz, A. Begel, S. C. Müller, S. Yigit-Elliott, and M. Züger, “Using
psycho-physiological measures to assess task difficulty in software devel-
opment,” in Proceedings of the 36th International Conference on Software
Engineering. ACM, 2014, pp. 402–413.
[14] E. Bekele and N. Sarkar, “Psychophysiological feedback for adaptive
human-robot interaction (hri),” in Advances in Physiological Computing.
Springer, 2014, pp. 141–167.
[15] J. L. Wright, S. A. Quinn, J. Y. Chen, and M. J. Barnes, “Individual
differences in human-agent teaming an analysis of workload and situation
awareness through eye movements,” in Proceedings of the Human Factors
and Ergonomics Society Annual Meeting, vol. 58, no. 1. SAGE Publica-
tions, 2014, pp. 1410–1414.
[16] D. Novak, B. Beyeler, X. Omlin, and R. Riener, “Workload estimation
in physical human-robot interaction using physiological measurements,”
Interacting with Computers, vol. 27, no. 6, pp. 616–629, 2015.
[17] K. A. Brookhuis and D. de Waard, “Monitoring drivers’ mental workload
in driving simulators using physiological measures,” Accident Analysis &
Prevention, vol. 42, no. 3, pp. 898–903, 2010.
[18] C. Collet, P. Averty, and A. Dittmar, “Autonomic nervous system and
subjective ratings of strain in air-traffic control,” Applied ergonomics,
vol. 40, no. 1, pp. 23–32, 2009.
VOLUME X, 2018 11
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
[19] A. Koenig, X. Omlin, L. Zimmerli, M. Sapa, C. Krewer, M. Bolliger,
F. Muller, and R. Riener, “Psychological state estimation from physi-
ological recordings during robot-assisted gait rehabilitation,” Journal of
rehabilitation research and development, vol. 48, no. 4, p. 367, 2011.
[20] K. A. Guru, E. T. Esfahani, S. J. Raza, R. Bhat, K. Wang, Y. Hammond,
G. Wilding, J. O. Peabody, and A. J. Chowriappa, “Cognitive skills
assessment during robot-assisted surgery: separating the wheat from the
chaff,” BJU International, vol. 115, no. 1, pp. 166–174, jan 2015.
[21] C.-H. Ting, M. Mahfouf, A. Nassef, D. A. Linkens, G. Panoutsos,
P. Nickel, A. C. Roberts, and G. R. J. Hockey, “Real-Time Adaptive
Automation System Based on Identification of Operator Functional State
in Simulated Process Control Operations,” IEEE Transactions on Systems,
Man, and Cybernetics - Part A: Systems and Humans, vol. 40, no. 2, pp.
251–262, mar 2010.
[22] G. F. Wilson and C. A. Russell, “Performance enhancement in an unin-
habited air vehicle task using psychophysiologically determined adaptive
aiding.” Human factors, vol. 49, no. 6, pp. 1005–18, dec 2007.
[23] Y. Ke, H. Qi, F. He, S. Liu, X. Zhao, P. Zhou, L. Zhang, and D. Ming,
“An eeg-basedmental workload estimator trained on working memory task
can work well under simulated multi-attribute task,” Frontiers in human
neuroscience, vol. 8, p. 703, 2014.
[24] R. McKendrick, T. Shaw, E. de Visser, H. Saqer, B. Kidwell, and
R. Parasuraman, “Team Performance in Networked Supervisory Control
of Unmanned Air Vehicles: Effects of Automation, Working Memory, and
Communication Content,” Human Factors, vol. 56, no. 3, pp. 463–475,
may 2014.
[25] J. Y. C. Chen and M. J. Barnes, “Human-Agent Teaming for Multirobot
Control: A Review of Human Factors Issues,” IEEE Transactions on
Human-Machine Systems, vol. 44, no. 1, pp. 13–29, feb 2014.
[26] C. Green and D. Bavelier, “Action-Video-Game Experience Alters the
Spatial Resolution of Vision,” Psychological Science, vol. 18, no. 1, pp.
88–94, jan 2007.
[27] M. W. Dye, C. S. Green, and D. Bavelier, “Increasing Speed of Processing
With Action Video Games,” Current Directions in Psychological Science,
vol. 18, no. 6, pp. 321–326, dec 2009.
[28] M. Dye, C. Green, and D. Bavelier, “The development of attention skills in
action video game players,” Neuropsychologia, vol. 47, no. 8-9, pp. 1780–
1789, jul 2009.
[29] D. Popovic, M. Stikic, T. Rosenthal, D. Klyde, and T. Schnell, “Sensitive,
diagnostic and multifaceted mental workload classifier (physioprint),” in
International Conference on Augmented Cognition. Springer, 2015, pp.
101–111.
[30] W. J. Horrey, M. F. Lesch, and A. Garabet, “Dissociation between driving
performance and drivers’ subjective estimates of performance and work-
load in dual-task conditions,” Journal of safety research, vol. 40, no. 1, pp.
7–12, 2009.
[31] M. Boyer, M. Cummings, L. B. Spence, and E. T. Solovey, “Investigating
mental workload changes in a long duration supervisory control task,”
Interacting with Computers, vol. 27, no. 5, pp. 512–520, 2015.
[32] R. Parasuraman, T. B. Sheridan, and C. D. Wickens, “Situation awareness,
mental workload, and trust in automation: Viable, empirically supported
cognitive engineering constructs,” Journal of Cognitive Engineering and
Decision Making, vol. 2, no. 2, pp. 140–160, 2008.
[33] R. Ma and D. B. Kaber, “Situation awareness and workload in driving
while using adaptive cruise control and a cell phone,” International Journal
of Industrial Ergonomics, vol. 35, no. 10, pp. 939 – 953, 2005.
[34] D. A. Sanders, H. M. Sanders, A. Gegov, and D. Ndzi, “Rule-based system
to assist a tele-operator with driving a mobile robot,” in Proceedings of SAI
Intelligent Systems Conference. Springer, 2016, pp. 599–615.
[35] E. Rohmer, S. P. N. Singh, and M. Freese, “V-REP: A versatile and scalable
robot simulation framework,” in 2013 IEEE/RSJ International Conference
on Intelligent Robots and Systems. IEEE, nov 2013, pp. 1321–1326.
[36] A. Memar and E. T. Esfahani, “Human Performance in a Mixed Human-
Robot Team: Design of a Collaborative Framework,” in ASME 2016
International Design Engineering Technical Conferences and Computers
and Information in Engineering Conference. Charlote, NC: ASME, 2016,
pp. V01BT02A056–13.
[37] A. D. Castel, J. Pratt, and E. Drummond, “The effects of action video game
experience on the time course of inhibition of return and the efficiency of
visual search,” Acta Psychologica, vol. 119, no. 2, pp. 217–230, jun 2005.
[38] K. A. Ehinger, “PsychToolbox examples: Visual cognition
experiments [Computer software],” 2015. [Online]. Available:
http://kehinger.com/PTBexamples.html
[39] D. H. Brainard, “The Psychophysics Toolbox,” Spatial Vision, vol. 10,
no. 4, pp. 433–436, jan 1997.
[40] Z. Pylyshyn, “Multiple object tracking,” Scholarpedia, vol. 2, no. 10, p.
3326, 2007.
[41] Z. W. Pylyshyn and R. W. Storm, “Tracking multiple independent targets:
Evidence for a parallel tracking mechanism,” Spatial vision, vol. 3, no. 3,
pp. 179–197, 1988.
[42] R. R. Johnson, D. P. Popovic, R. E. Olmstead, M. Stikic, D. J. Lev-
endowski, and C. Berka, “Drowsiness/alertness algorithm development
and validation using synchronized EEG and cognitive performance to
individualize a generalized model,” Biological Psychology, vol. 87, no. 2,
pp. 241–250, may 2011.
[43] V. C. Laboratory, “Fixations identification algo-
rithm [Computer software],” 2015. [Online]. Available:
http://www.unifr.ch/inph/vclab/home/internal/eye-tracking-software
[44] D. D. Salvucci and J. H. Goldberg, “Identifying fixations and saccades in
eye-tracking protocols,” in Proceedings of the 2000 symposium on Eye
tracking research & applications. ACM, 2000, pp. 71–78.
[45] J. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka,
H., van de Weijer, Eye Tracking: A Comprehensive Guide to Methods and
Measures. Oxford, UK.: Oxford University Press, 2011.
[46] R. J. K. Jacob and K. S. Karn, “Eye tracking in human-computer interac-
tion and usability research: Ready to deliver the promises,” in The Mind’s
Eye: Cognitive and Applied Aspects of Eye Movement Research, eds. J.
Hyona, R. Radach, and H. Deubel, Ed. Oxford: Elsevier Science, 2003,
pp. 573–605.
[47] C. Berka, D. J. Levendowski, M. M. Cvetinovic, M. M. Petrovic, G. Davis,
M. N. Lumicao, V. T. Zivkovic, M. V. Popovic, and R. Olmstead, “Real-
time analysis of eeg indexes of alertness, cognition, and memory acquired
with a wireless eeg headset,” International Journal of Human-Computer
Interaction, vol. 17, no. 2, pp. 151–170, 2004.
[48] A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, and
M. Goodrich, “Common metrics for human-robot interaction,” in Pro-
ceeding of the 1st ACM SIGCHI/SIGART conference on Human-robot
interaction - HRI ’06. New York, New York, USA: ACM Press, 2006,
p. 33.
[49] D. Crundall, G. Underwood, and P. Chapman, “Driving experience and the
functional field of view,” Perception, vol. 28, no. 9, pp. 1075–1087, 1999.
[50] C. J. Patten, A. Kircher, J. Östlund, and L. Nilsson, “Using mobile tele-
phones: cognitive workload and attention resource allocation,” Accident
analysis & prevention, vol. 36, no. 3, pp. 341–350, 2004.
[51] M. R. Endsley, S. J. Selcon, T. D. Hardiman, and D. G. Croft, “A
Comparative Analysis of Sagat and Sart for Evaluations of Situation
Awareness,” Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, vol. 42, no. 1, pp. 82–86, oct 1998.
[52] C. Durso, F.T., Truitt, T.R., Hackworth, C.A., Crutchfield, J.M., Nikolic,
D., Moertl, P.M., Ohrt, D. & Manning, “Expertise and chess: A pilot study
comparing situation awareness methodologies,” in In D. J. Garland &
M. Endsley (Eds.) Experimental Analysis and Measurement of Situation
Awareness, Daytona Beach, FL, 1995.
[53] G. Hauland, “Measuring Individual and Team Situation Awareness During
Planning Tasks in Training of En Route Air Traffic Control,” The Interna-
tional Journal of Aviation Psychology, vol. 18, no. 3, pp. 290–304, may
2008.
[54] K. Moore and L. Gugerty, “Development of a Novel Measure of Situation
Awareness: The Case for Eye Movement Analysis,” Proceedings of the
Human Factors and Ergonomics Society Annual Meeting, vol. 54, no. 19,
pp. 1650–1654, sep 2010.
[55] M. A. Vidulich and P. S. Tsang, “Mental workload and situation aware-
ness,” in Handbook of Human Factors and Ergonomics. John Wiley &
Sons, Inc., 2012, pp. 243–273.
[56] G. Marquart, C. Cabrall, and J. de Winter, “Review of eye-related measures
of drivers’ mental workload,” Procedia Manufacturing, vol. 3, pp. 2854–
2861, 2015.
[57] C. M. Privitera, L. W. Renninger, T. Carney, S. Klein, and M. Aguilar,
“Pupil dilation during visual target detection,” Journal of Vision, vol. 10,
no. 10, pp. 3–3, 2010.
[58] F. B. Parmentier, “The cognitive determinants of behavioral distraction by
deviant auditory stimuli: A review,” Psychological Research, vol. 78, no. 3,
pp. 321–338, 2014.
12 VOLUME X, 2018
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2018.2790838, IEEE Access
AMIRHOSSEIN H. MEMAR (SM’15) received
the B.Sc. and M.Sc. degrees in mechanical en-
gineering from Isfahan University of Technology
and University of Kashan, Iran, respectively. In
2014, he was a researcher at PRISMA Lab, Uni-
versity of Naples Federico II, Italy. He is cur-
rently a Ph.D. candidate in the mechanical and
aerospace engineering department of Univeristy
at Buffalo - SUNY. His research interests include
Robotics, Human-Robot Interaction, Haptics and
Brain-Computer Interfaces.
EHSAN T. ESFAHANI (M’06) received the M.S.
degree in electrical engineering and the Ph.D. de-
gree in mechanical engineering from the Univer-
sity of California Riverside, Riverside, CA, USA,
in 2012. He is currently an Assistant Professor in
the Department of Mechanical and Aerospace En-
gineering, University at Buffalo, SUNY, Buffalo,
NY, USA. His main research interests include hu-
man in the loop systems, human robot interactions,
human activity monitoring and biorobotics.
VOLUME X, 2018 13
