Content uploaded by James Bliss
Author content
All content in this area was uploaded by James Bliss on Oct 21, 2015
Content may be subject to copyright.
Task Shedding and Control Performance as a Function of
Perceived Automation Reliability and Time Pressure
James P. Bliss
John W. Harden
Old Dominion University
Norfolk, Virginia
H. Charles Dischinger, Jr.
NASA-Marshall Space Flight Center
Huntsville, Alabama
Research has demonstrated that workload and past machine performance influences operator allocation of
task responsibilities to machines. We extended past investigations by offering task operators the
opportunity to relinquish task control to a robotic entity. Forty-three participants navigated a remotely
controlled vehicle around a prescribed course under conditions of low or high time pressure. While
navigating, they could allocate camera monitoring to a low- or high-reliability automated agent. Results
showed most participants retained control of the camera; others relinquished control immediately. Time
pressure and reliability did not interact to influence task performance. Course navigation time was faster
under high time pressure but errors were unaffected. Bivariate correlations revealed a positive relation
between self-ratings of robotic expertise and pressure to perform, and between pressure to perform and
errors committed during navigation. These results demonstrate low levels of trust in the robotic camera and
comparative sensitivity of navigation time to time pressure.
INTRODUCTION
For several decades, technology growth has
influenced the way humans perform complex
tasks. The change is particularly evident in the
air and ground transportation, industrial,
medical, and military task domains. Though
technology increases have stimulated greater
productivity, better quality control, and
increased production, they have at times reduced
operator situation awareness, exacerbated
cognitive workload, and produced variability in
operator attitudes toward the automation itself.
Parasuraman, Sheridan, and Wickens (2000)
presented a model of automation functionality
that drew connections to four stages of an
operator’s information processing: information
acquisition, information analysis, decision and
action selection, and action implementation.
They proposed that any particular automated
system could act within any number of those
stages, and that automating within these stages
fundamentally changes the actions required by
the operator to successfully achieve target goals.
Endsley and Kaber (1999) proposed a model
of levels of automation comprised of 10 levels at
which automation could function within a
system. In this research, the authors argued that
there were several problems with implementing
automation at levels which resulted in removing
the operator from “the loop.” In evaluating these
various levels of automation, the authors found
that lower levels of automation tended to result
in superior operator performance and that, in the
event of a failure, operator intervention was
quicker (Endsley & Kaber, 1999).
One particularly interesting finding is that
human users vary with regard to their reliance
and compliance rates and behaviors, even when
the reliability of the automation is advertised or
well known. This may be due to trust
development, complacency, workload, or a host
of other mediating or moderating factors
(Parasuraman, Molloy, & Singh, 1993; Dixon &
Wickens, 2006; Merritt & Ilgen, 2008).
Researchers have recorded reactions to
automated aids themselves as an index of trust
with some success (Ross, Szalma, Hancock,
Barnett, & Taylor, 2008). However, equally
revealing is the propensity for operators to shed
tasks to automated agents. Task shedding, or
what Parasuraman and Hancock (2001) refer to
as adaptive task allocation to the machine (ATA-
M), has been investigated by many researchers
interested in automation and its impact on
workload (c.f., Byrne & Parasuraman, 1996;
Scerbo, 1996). Recent interest in human-robot
interaction has increased the relevance of such
behaviors.
ATA-M has been shown to increase during
conditions of high primary task workload and
low certainty (Parasuraman & Hancock, 2001).
The purpose of the current experiment is to
replicate those findings in a paradigm that pairs
human operators with automated (robotic)
agents. The unique contribution is the presumed
interaction between automation reliability (and
by association, operator trust) and workload
(influenced by time pressure).
Given prior research (Bliss, Dunn, & Fuller,
1995), we hypothesized that participants would
be more liable to relinquish control of a camera
to a robot if the robot were advertised as
reliable. We also hypothesized that time
pressure would result in quicker and more
frequent task allocation to the robot (Kirlik,
1993).
METHOD
Design
For the current experiment we manipulated
variables according to a split-plot, 2 X 2
experimental design. The between-groups
independent variable was advertised automated
camera controller reliability rate. Participants in
the low-reliability group were told that
automating the camera control function had been
successful at making the overall task completion
time quicker either 75% or 95% of the time.
The within-groups variable was time pressure,
manipulated by the amount of time participants
had to complete the maneuvering task. During
the low-pressure condition, participants were
told that they were to complete the task in 10
minutes. For the high-pressure condition,
participants were told that they were to complete
the task in 5 minutes. In both cases, participants
were encouraged to complete the task as quickly
and accurately as possible.
Performance dependent measures included
the overall time taken to maneuver a remote-
control truck around a predefined course (in
secs), the number of errors made (driving
outside demarcated lines) while doing so,
whether or not participants chose to automate
the camera control task, and the time (in secs)
taken to so do.
We also collected questionnaire data,
including demographic information, trust
assessments, and information about the strategy
participants used during the task.
Participants
The 43 participants tested (20 male, 23
female) included 23 undergraduate students
enrolled in a general psychology course at The
University of Alabama in Huntsville and 20
employees at NASA’s Marshall Space Flight
Center in Huntsville, Alabama. The average age
of the participants was 23.05 years (SD=7.72).
Participants indicated that they had corrected-to-
normal visual acuity and hearing. Students at
UAH earned credit toward their psychology
class, whereas employees of NASA earned a
$10 Starbucks gift card for their participation.
Participants from each location were equally
distributed in the two reliability groups.
Materials
Initial questionnaires included informed
consent forms for UAH and NASA and a
background questionnaire that included
demographic items (age, sex, robot familiarity,
robotic control skill level, and general computer
use frequency). Following the experiment,
participants completed Jian, Bisantz, & Drury’s
(1999) trust scale twice; once to indicate trust of
the remotely controlled truck and once to
indicate their trust of the remotely controlled
camera. All participants also completed an
opinion questionnaire that allowed them to
discuss their perceived motivation for the
experiment, the strategy(ies) they used to
complete the task, and the level of effort they
expended during the experiment.
Remotely Controlled Truck – Participants
used a control device to maneuver a toy
remotely controlled truck around a demarcated
course. The course was 12 inches wide, U-
shaped, and bounded by walls on two sides (see
Figure 1). The starting position for the truck
was intentionally slanted 45 degrees to the track,
forcing participants to attain proper vehicle
alignment before proceeding. They then
maneuvered the truck through the course to the
end, whereupon the experimenter would reverse
the truck’s direction so participants could drive
the truck back to the starting point. Thus, the
number of left and right turns was equal.
Participants were not allowed to view the
truck and course directly. Instead, they were
required to complete the task by referring to a
laptop computer screen that showed a view of
the course recorded by a remotely mounted
camera. The perspective of the camera was
adjustable by the participant, so that he or she
could focus on particular parts of the course
while maneuvering the truck. The remote
controller for the truck had two small joysticks.
The right joystick controlled forward and
backward movement; the left joystick controlled
orientation of the front wheels to allow steering.
Remotely Controlled Camera – The video
camera was mounted on a military robot
(MarcBot unmanned ground vehicle) that was
positioned at a height of approximately 31
inches (see Figure 1). Participants used an X-
Box controller (left direction pad) to yaw the
camera. In this way, they could keep the track
and truck in view at all times.
Figure 1. Experimental Setup
Procedure
Following their arrival at the experimental
laboratory, participants completed the informed
consent forms and the background information
form. Participants were then trained to
concurrently manipulate the remote-controlled
truck and the remote camera. The experimenter
demonstrated proper use of an X-box controller
for the camera and the remote control unit for
the truck, then let the participant become
comfortable with each. Participants completed
one practice trial, during which they drove the
truck around the course while viewing it
directly. When participants indicated that they
understood the controls and the task, the first
experimental session began.
Participants were seated with their back to
the course so that they were required to rely on
the video feed from the remote camera to
maneuver the truck. Participants were told that
they had a limited time to complete the course (5
minutes or 10 minutes, depending on
counterbalanced condition). They were also told
that they could elect to automate the camera
control at any time during the session. At this
time, the experimenter emphasized that other
participants who had chosen to automate had
achieved performance improvement 75% or
95% of the time (depending on reliability group
assignment). Once participants indicated that
they understood the instructions, they began to
maneuver the truck through the course.
After completing the first experimental
session, participants were given a five-minute
break, and then began the second session. The
second session was identical to the first, except
that the time limit was counterbalanced to
ensure completion of low and high time pressure
sessions. After finishing the course the second
time, participants completed the trust
questionnaire for both the truck and the camera.
They then completed the opinion questionnaire,
and were debriefed and dismissed. In all,
participation took approximately 45 minutes.
RESULTS
We began our analyses by ensuring that the
data were coded correctly. We then calculated
descriptive statistics to ensure that the data were
normally distributed with no outliers. We noted
missing data for one participant’s maneuvering
error score. We also coded Time to Automate as
missing if participants chose not to automate the
camera task. For the following analyses, we
adopted a p = .10 significance level to account
for the exploratory nature of our work and
associated benign implications of committing a
type I error.
For Decision to Automate, a Chi-Square test
revealed that participants were more likely to
want to maintain control of the camera task than
to delegate the task, χ2(1) = 8.395, p < .01. Of
the 86 control decisions made, participants
wanted to delegate control of the camera 24
times. Thirteen of the 43 participants in the high
pressure condition decided to automate the
camera task; 11 of the 43 participants did so in
the low pressure condition. In each case, eight
of those deciding to automate did so
immediately (at the start of the task).
To predict the binary outcome of whether
participants decided to automate camera control
or not, we computed a binary logistic regression.
We first computed the regression analyses using
a prediction model that included reliability
group (low or high), sex (male or female), age,
robotic experience, skill level, perceived
pressure to perform, perceived problems with
completing the maneuvering task, and perceived
demand.
Results from the standard logistic regression
indicated that the combination of the predictors
significantly predicted the outcome, χ2(8) =
14.888, p = .061, Negelkerke R2 = .478.
However, results from each individual Wald
statistic indicated that only reliability group and
perceived problems with completing the
maneuvering task were significant predictors of
their deviation decision. Therefore, we
conducted a follow-up standard logistic
regression including just these two predictors.
Results from this analysis indicated that the
combination of the two predictors significantly
predicted the outcome, χ2(2) = 5.648, p=.059, R2
= .205.
A total of 75.0% of all participants’
decisions were correctly predicted with this
model. Type I error was 6.0%, indicating that
94% of participants’ decisions to want to
automate the task were correctly classified. Type
II error was 19.5%, indicating that 80.5% of
teams’ decisions to want to retain camera
control were correctly classified. Participants in
the low reliability group were .267 times more
likely to want to automate the camera task.
Participants were 1.471 times more likely to
want to automate the camera task if they
perceived a problem completing the
maneuvering task (see Table 1).
Table 1
Standard Logistic Regression
_______________________________________
Variable B SE Wald Odds Ratio
_______________________________________
Rel. Group -1.321 .815 2.625 .267
Problem .386 .901 2.160 1.471
_______________________________________
To determine whether truck maneuvering
speed or errors varied as a function of advertised
automation reliability or time pressure, we
computed 2 X 2 ANOVAs for those variables.
For time taken to traverse the course, there was
no significant interaction and no main effect of
reliability. However, there was a main effect of
time pressure, F(1,47) = 3.45, p = .07, partial n2
= .078, showing that participants traversed the
course more quickly in the high pressure
condition (233.98 secs) than in the low pressure
condition (271.86 secs). For errors made, there
was no significant interaction, no effect of
reliability, and no main effect of time pressure
(p > .10).
Our final step was to calculate correlations
among demographic, experience, strategy and
performance. Because of the numerous
correlations computed, we adopted a p = .01
significance level. Participant self-ratings of
robotic expertise were positively related to level
of perceived pressure to perform, r = .458, p = .
005. In turn, greater pressure to perform
associated with more numerous errors
committed during the low-pressure task session,
r = .496, p = .002.
DISCUSSION
Kirlik (1993) demonstrated that in some
cases operators may weigh the comparative
benefits of automating by considering workload
and manual control skills. Ultimately, they may
decide to not automate a task because it is
inconvenient or perceived as costly toward
overall performance or workload. In the current
experiment automation delegation was the
exception rather than the norm. Most
participants elected to retain control of the
camera, even when the advertised reliability and
time pressure were relatively high. This may
suggest that operators are cognizant of the task
changes that may follow automation
(Parasuraman et al., 2000).
One especially intriguing finding here was
that the participants who did choose to automate
the camera task did so immediately (in some
cases before the task even began), rather than in
the middle of a session. This suggests that
participants may make task strategy decisions
prior to engaging in the task to optimize
workload or situation awareness, thereby
minimizing the problems Endsley and Kaber
(1999) proposed that accompany automation
implementation.
Varying the amount of time given to
participants to complete the course seemed to
influence operator performance speed,
suggesting that the manipulation we chose
effectively changed perceived workload. Yet,
contrary to Byrne and Parasuraman’s (1996)
findings, the number of participants electing to
automate the camera was comparable across
time pressure conditions. The fact that most
people electing to automate the camera did so
immediately may have masked any differences
attributable to perceived workload.
Questions exist concerning the malleability
of automation decisions made prior to task
operation, as well as identification of factors
influencing the timing of such decisions. Future
research concerning these aspects could help
predict the potential for reliance on robotic
agents.
ACKNOWLEDGMENTS
The experimenters acknowledge Mr. Nick
Harris and Mr. Marshall Bliss, who assisted with
data collection.
REFERENCES
Bliss, J.P., Dunn, M., & Fuller, B.S. (1995).
Reversal of the cry-wolf effect: An investigation of
two methods to increase alarm response rates.
Perceptual and Motor Skills, 80, 1231-1242.
Byrne, E.A., & Parasuraman, R. (1996).
Psychophysiology and adaptive automation.
Biological Psychology, 42, 249-268.
Dixon, S., & Wickens, C.D. (2006). Automation
reliability in unmanned aerial vehicle control: A
reliance-compliance model of automation
dependence in high workload. Human Factors,
48(3), 474-486.
Endsley, M. R., & Kaber, D. B. (1999). Level of
automation effects on performance, situation
awareness and workload in a dynamic control task.
Ergonomics, 42(3), 462-492.
Kirlik, A. (1993). Modeling strategic behavior in
human-automation interaction: Why an “aid” can
(and should) go unused. Human Factors, 35(2), 221–
242.
Merritt, S. M., & Ilgen, D. R. (2008). Not all
trust is created equal: Dispositional and history-based
trust in human-automation interactions. Human
Factors, 50(2), 194–210.
Parasuraman, R., & Hancock, P. (2001).
Adaptive control of mental workload. In P. Hancock
and P. Desmond (eds.), Stress, Workload, and
Fatigue. New York: CRC Press.
Parasuraman, R., Molloy, R., & Singh, I.L.
Performance consequences of automation-induced
“complacency.” The International Journal of
Aviation Psychology, 3(1), 1-23.
Parasuraman, R., Sheridan, T. B., & Wickens, C.
D. (2000). A model for types and levels of human
interaction with automation. Transactions on
Systems, Man and Cybernetics, Part A: Systems and
Humans, IEEE, 30(3), 286-297.
Ross, J.M., Szalma, J.L., Hancock, P.A., Barnett,
J.S., & Taylor, G. (2008). The effect of automation
reliability on user automation trust and reliance in a
search-and-rescue scenario. Proceedings of the
Human Factors and Ergonomics Society 52nd
Annual Meeting. Santa Monica, CA: Human Factors
and Ergonomics Society.
Scerbo, M.S. (1996). Theoretical perspectives
on adaptive automation. In R. Parasuraman and M.
Mouloua (Eds.), Automation and Human
Performance: Theory and Applications. Mahwah,
NJ: Lawrence Erlbaum Associates.
