Content uploaded by Umberto Saetti
Author content
All content in this area was uploaded by Umberto Saetti on May 24, 2023
Content may be subject to copyright.
Full-Body Haptic Cueing Algorithms for Augmented Pilot Perception in
Degraded/Denied Visual Environments
Michael T. Morcos
Graduate Research Assistant
Spencer M. Fishman
Graduate Research Assistant
Alessandro Cocco
Postdoctoral Researcher
Umberto Saetti
Assistant Professor
Department of Aerospace Engineering
University of Maryland
College Park, MD 20740
Tom Berger
Flight Controls Group Lead
U.S. Army Combat Capabilities Development
Command Aviation & Missile
Moffett Field, CA 94035
Martine Godfroy-Cooper
Senior Research Psychologist
Edward N. Bachelder
Senior Research Engineer
San Jose State University
Moffett Field, CA 94035
ABSTRACT
This paper demonstrates the development, implementation, and testing of full-body haptic cueing algorithms for aug-
mented pilot perception. Full-body haptics is in the form of localized electrical muscle stimulation (EMS) is achieved
via a commercial, off-the-shelf product called TESLASUIT. Cueing algorithms are developed for roll-axis compen-
satory tracking tasks where the pilot acts on the displayed error between a desired input and the comparable vehicle
output motion to produce a control action. The error is displayed to the pilot using three different cueing modalities:
visual, haptic, and combined visual and haptic. For the visual and combined visual and haptic modalities, visual cues
are also considered in degraded visual environments (DVE). Full-body haptic cueing algorithms that are based on a
proportional-derivative (PD) compensation strategy on the tracking error are found to provide satisfactory pilot vehicle
system (PVS) performance for the task in consideration when using haptic feedback only (no visual cues) and to im-
prove PVS performance in DVE when using combined visual and haptic feedback. These results indicate that the use
of secondary sensory cues such as full-body haptics to augment the pilot perception can lead to improved/partially-
restored PVS performance when primary sensory cues like vision are impaired or denied.
INTRODUCTION
In symbiotic piloting, a human pilot shares perception and
control authority with an artificial intelligence (AI) agent to
control the motion of a vehicle, acting as a symbiotic organ-
ism (Fig. 1). Here, a vehicle is intended as any generic n-DOF
machine (e.g., airplanes and helicopters, drones, etc.) that
can move across regions of physical space. The vehicle may
physically host or be teleoperated by a human pilot, and its
dynamics are augmented by an AI agent with human-like per-
Presented at the VFS International 79th Annual Forum &
Technology Display, West Palm Beach, FL, May 16–18, 2023.
Copyright © 2023 by the Vertical Flight Society. All rights reserved.
Distribution Statement A. Approved for public release; distribution
is unlimited.
ception and actuation dynamics. Current perception models
for symbiotic human-machine piloting of vehicles are based
on the dominant visual (i.e., sight) and vestibular (i.e., equi-
librium) cues, but neglect less dominant perception cues, such
as somatosensory cues (e.g., haptics) or auditory cues (e.g.,
3-D audio), and interactions between primary and secondary
sensory channels. As such, there is limited understanding on
shared human-machine perception solutions for vehicle pilot-
ing when the symbiotic system operates with denied/impaired
sensory channels (e.g., degraded visual environment), de-
nied/impaired interactions between sensory channels, or with
primary sensory channels augmented with traditionally sec-
ondary perception cues.
Denial or impairment of perceptual channels may arise envi-
1
Fig. 1: Symbiotic pilot vehicle system.
ronment conditions (e.g., weather), factors limiting machine
abilities (e.g., sensor failure), or human pilot abilities (e.g.,
temporary or permanent disabilities, e.g., fatigue). Within the
context of manned piloting of aircraft, modeling of secondary
sensory cues could help develop cueing strategies to enable
emergent pilot vehicle system (PVS) performance, or to sup-
plement for pilot spatial disorientation which may be caused
by temporary or permanent loss/malfunction of their vestibu-
lar and/or visual systems.
Within the context of somatosensory cueing, previous stud-
ies have shown that cueing the relative motion of a hovering
helicopter with respect to a target fixed in space through full-
body tactile cueing is an effective strategy to increase spatial
awareness and handling qualities, and decrease pilot workload
in and out of degraded visual environment (DVE). In these
studies, the relative linear and angular positions, velocities,
and accelerations were cued to the pilot through an array of
tactors, and using tactor pulse patterns with varying ampli-
tude, frequency, and waveform (Refs. 1–4). Similar strategies
to those of Refs. 1, 2 could be used in the proposed investiga-
tion and adapted through localized electric muscle stimulation
(EMS) provided by the full-body haptic suits.
As such, the objectives of the present investigation are three-
fold. The first objective is to extend the crossover model
(Refs. 5) to secondary sensory cueing paths like full-body
haptics. The second objective is to develop efficient haptic
cueing strategies that can be used in compensatory tracking
experiments (Refs. 5, 6) in place of, or in combination with,
visual cues. The third objective is to test haptic cueing strate-
gies and identify the corresponding PVS dynamics. Compar-
isons between visual, haptic, and combined visual and haptic
cueing will help understand the potential differences in pilot
equalization when leveraging this secondary sensory path to
control the motion of an aircraft. Moreover, these experiments
will shed light on potential benefits of haptic cueing.
The paper begins with a discussion of the overall methodol-
ogy adopted in this investigation, including an overview of
pilot modeling. This is followed by a description of the equip-
ment adopted for the experimental studies. Next, the develop-
ment and implementation of full-body haptic cueing strategies
that make use of proportional-derivative compensation are de-
scribed in detail. Discussions on experiment design and para-
metric identification from input-output experimental data fol-
lows. Results feature both time- and frequency-domain anal-
yses of the experimental data. Quantitative data from the ex-
periments is compared to qualitative data from pilot feedback.
Final remarks summarize the overall findings of the study and
future developments are identified.
METHODOLOGY
Overview
The present effort focuses on closed-loop compensatory
tracking tasks in which the pilot acts on the displayed error
ebetween a desired command input iand the comparable ve-
hicle output motion mto produce a control action c. Histori-
cally, information about the error ewas presented to the pilot
through visual displays, examples of which is shown in Fig.
2. In Fig. 2a, the green bar with the inner and outer reticles
is aircraft pitch attitude indicator m, whereas the orange bar
with the center dot is the commend i. The difference between
these two is error that the pilot attempts to minimize within the
defined desired and adequate performance constraints. These
constraints are given by the inner and outer reticles, respec-
tively. Figure 2b shows another kind of compensatory display
referred to as bow tie display. This particular display is used
to cue roll and pitch attitude errors simultaneously. For pitch
evaluations with this display, the objective is to capture and
hold the green dot within the magenta circles for each com-
manded pitch attitude. For roll evaluations with the same dis-
2
play, the objective is to capture and hold the green line within
the diagonal bow tie bounds for each commanded bank angle.
The idea behind this research is to replace and/or augment
these visual displays with full-body haptic (tactile) displays.
More specifically, the tactile display will make use of local-
ized EMS where the electrodes distribution is anatomically
informed.
(a) Pitch axis compensatory tracking display.
(b) Bow tie display for roll and pitch compensatory tasks.
Fig. 2: Visual displays.
Pilot Vehicle System Modeling
As the base for the prediction of PVS performance and iden-
tification of the PVS dynamics, a pilot model structure needs
to be specified. Due to the preliminary nature of the investi-
gation, a crossover model (Ref. 7) is assumed as it constitutes
a simple yet powerful way to represent the combined PVS dy-
namics. The PVS dynamics in a compensatory tracking task
is shown qualitatively in Fig. 3.
In this study, the vehicle and display dynamics are all com-
bined into the controlled element with a transfer function
Yc(s). The portion of the pilot’s control action linearly cor-
related with the input is represented by the quasi-linear de-
scribing function Yp(s), which also includes the effects of the
manipulator characteristics. In a compensatory control task,
it was shown through extensive research (see, e.g., Ref. 5)
that the human pilot adapts his or her dynamics so that near
crossover frequency ωc(i.e., where |YpYc|=1, or 0 dB) the
open-loop dynamics are given by:
YOL(s) = YpYc=ωc
se−sτe(1)
where τeis the effective time delay, including transport delays
and neuromuscular lags. It is worth noting that the crossover
frequency is equivalent to the loop gain and accounts for the
pilot’s adaptive compensation for the controlled element gain.
The aircraft dynamics is assumed to have the following gen-
eral form:
Yc(s) = Ke
s(τs+1)(2)
and is representative of the roll attitude response to the pi-
lot lateral stick. The simplest pilot describing function form
for this particular aircraft dynamics, which corresponds to the
open-loop crossover model, is:
Yp(s) = Kpe−sτe(3)
where Kpis the pilot static gain (Ref. 5). The pilot adjusts
his or her gain Kpto place the crossover frequency where re-
quired to complete the task.. Thus, given known aircraft dy-
namics, the human pilot parameters to be identified for each
cueing modality are Kpand τe. Moreover, particular focus is
be placed on estimating and comparing ωcfor each cueing
modality.
If a sensory modality and/or vehicle configuration yields a
high crossover frequency, i.e.,ωc>4 rad/s (high perfor-
mance) the pilot’s neuromuscular mode will start to influence
the open-loop response such that the latter can become much
flatter than what the crossover model predicts. In this case, the
pilot model proposed in Ref. 8 will be used, which accounts
for the neuromuscular dynamics.
EQUIPMENT
Full-body haptic feedback is achieved through full-body hap-
tic suits (TESLASUITs), shown in Fig. 4a. These suits pro-
vide number of functions: (i) full-body haptics via local-
ized electrical muscle stimulation (EMS) achieved through
80 EMS channels and 114 electrodes that can operate at fre-
quencies of 1-150 Hz and current of 1-60 mA, (ii) biometry
through heart rate (HR), arterial oxygen saturation (SpO2),
and pulse rate variability (PRV) monitoring, and (iii) full-
body motion tracking through a 6/9 axes Inertial Measure-
ment Units (IMU), and 10 internal motion capture sensors
operating at a sampling rate of 100 Hz. The distribution of
the electrodes on the suit is anatomically informed such that
each major muscle has an electrode (Fig. 4b). EMS is con-
trolled through frequency, amplitude, and pulse width inputs.
Note that the electrode pulse width is defined as the sequence
of turning the electrode on and off, which is different from the
carrier frequency (i.e., the frequency of the electrode when the
electrode is on).
The controller used for tracking task is a Logitech X52 Pro
joystick, shown in Fig. 5. The joystick has three degrees of
freedom: left/right, fore/aft, and twist. The joystick adopts a
base spring with constant stiffness.
3
Fig. 3: Pilot vehicle system in a compensatory tracking task.
(a) TESLASUIT. (b) Electrodes map.
Fig. 4: Full-body haptic suit.
(a) Front. (b) Side.
Fig. 5: Logitech X52 Pro joystick.
HAPTIC CUEING STRATEGY
Continuous cueing of the roll attitude tracking error and/or
its time derivative was found to be an effective haptic cueing
strategy. The general form of this strategy is a proportional-
derivative (PD) compensation on the roll attitude tracking er-
ror:
POW =|KPeφ+KD˙eφ|+POWmin (4)
where POW is the TESLASUIT power, POWmin is an arbi-
trary power offset, and eφis the roll attitude tracking error.
The attidude tracking error is defined as the difference be-
tween the commanded and actual roll angles eφ=φcmd −φ.
Within the context of Fig. 3, the displayed error ecorresponds
to eφ, the desired command input iis φcmd and the vehicle out-
put motion mis represented by φ. Additionally, Kpand KDare
the proportional and derivative gains, respectively. It is worth
noting that the power is proportional to the EMS channel am-
plitude and can be adjusted for each subject based on their
perception of EMS. Minimum and maximum power are cali-
brated for each subject such that minimum power corresponds
to a barely perceivable stimulus, and maximum power corre-
sponds to the maximum stimulus that is perceived by the pilot
as not annoying and does activate muscle contraction. The
proportional gain is tuned based on the error bounds for Ade-
quate performance (discussed later in the paper), such that:
KP=(POWmax −POWmin)
eφAde
(5)
where POWmax =100% and POWmin =30%. This way, the
TESLASUIT power will reach 100% if for eφ=eφAde. The
derivative gain is tuned by trial and error. An integral gain
was considered in the early stages of development of the al-
gorithm, but was eventually discarded as it did not appear
to yield PVS performance improvements, nor seemed to be
liked by the test subjects. In any case, integral gain will be
studied more thoroughly in future investigations. The loca-
tion and number of the electrodes to be activated depends on
the sign of KPeφ+KD˙eφand on the magnitude of POW. If
KPeφ+KD˙eφ>0 and |KPeφ+KD˙eφ|+POWmin ≤100%, then
a single electrode on the right shoulder is activated (6a). In the
case where |KPeφ+KD˙eφ|+POWmin >100% then, a second
tactor on the upper right arm is activated in addition to that on
the right shoulder (6b). Vice versa, if KPeφ+KD˙eφ<0 and
|KPeφ+KD˙eφ|+POWmin ≤100%, then a single electrode on
the left shoulder is activated (6c). A second electrode on the
upper left arm is activated in case |KPeφ+KD˙eφ|+POWmin >
100% (6d). Based on this strategy, the pilot will perceive a
haptic feedback in the direction of the error or, equivalently,
in the verse of the control action to be taken to reduce the er-
ror. For instance, assume the actual roll attitude to be φ>0
(right wing down) the commanded roll attitude to be wings
level, i.e.,φcmd =0. Additionally, assume that ˙eφ=0. Then,
eφ<0, which activates the electrodes on the left side of the
upper body. As such, the pilot feels they should make a cor-
rective action to the left to bring the aircraft back to level. This
cueing strategy is summarized in Table 1.
It is worth noting that this cueing strategy differs from that in
Refs.1–4 in that these algorithms: (i) were based on cueing of
translational position, velocity, and acceleration; (ii) cued po-
sition, velocity, and acceleration separately; and (iii) adopted
a binning, non-continuous strategy to cue the magnitude of
position, velocity, and acceleration.
4
Table 1: Haptic cueing strategy.
Location Power [%] Condition 1 Condition 2
Right shoulder |KPeφ+KD˙eφ|+POWmin POW ≤100% KPeφ+KD˙eφ>0
Right shoulder + upper right arm 100% POW >100 KPeφ+KD˙eφ>0
Left shoulder |KPeφ+KD˙eφ|+POWmin POW ≤100% KPeφ+KD˙eφ<0
Left shoulder + upper left arm 100% POW >100 KPeφ+KD˙eφ<0
(a) Right shoulder electrode. (b) Left shoulder + upper right
arm electrodes.
(c) Left shoulder electrode. (d) Left shoulder + upper left arm
electrodes.
Fig. 6: Haptic cueing algorithm using a TESLASUIT.
EXPERIMENT DESIGN
The set of experiments object of this study involve a precision,
non-aggressive, closed-loop tracking task in the roll axis only
where the forcing function (command input) is chosen to be a
sum of signs (SOS), as per Ref. 6. The SOS forcing function
is used to drive the compensatory tracking task through which
the pilot attempts to minimize the displayed error within de-
sired performance constraints. A Fibonacci series-based SOS
input is designed to emphasize the frequency range that en-
compasses key vehicle dynamics and typical human pilot con-
trol action, ranging from 0.3 to approximately 6 rad/s. The
length for scoring time is 60 seconds, with a 10 second ramp
in/ramp out period on each end of the 60 second “for score”
window. Five trials are conducted for each cueing modality.
Each of these five trials used a different sum of sines, where
the sum of sines are based on randomly-generated phasing
angles and stored offline. The five sum of sines are the same
across each cueing modality and are presented to the test sub-
ject in the same order. Experiments make use of four subjects,
one of which is a test pilot. Desired performance is defined
such that the tracking error on the roll attitude be less than
5 deg for at least 50% of the scoring time. Adequate perfor-
mance requires the roll attitude tracking error to be less than
10 deg for at least 75% of the scoring time. These require-
ments are summarized in Table 2.
The joystick is positioned between the legs of the test subject,
who is seated with their eyes at approximately 1 m from the
computer screen. The central positioning of the joystick is
justified by the effort in replicating the use of a cyclic stick
in a conventional utility helicopter. By the same rationale, the
vertical position of the joystick is adjusted to have the test
subjects rest their elbow on their thigh when operating the
joystick. This setup is shown in Fig. 7.
Fig. 7: Experimental setup.
The following five cueing modalities are tested:
1. Visual: The pilot performs the compensatory tracking
task by looking at the visual display only, where the vi-
sual display is the bow tie display shown in Fig. 2b.
2. Visual DVE: Here, visual cues are disturbed by the use
5
Table 2: Desired and Adequate performance metrics.
Performance Tracking Error Bounds [deg] Scoring Time [%]
Desired |eφ|<5≥50
Adequate |eφ|<10 ≥75
of foggles to simulate a DVE.
3. Visual + Haptic: Information about the tracking error is
provided to the pilot via the visual (bow tie) display and
via full-body haptics. Three different haptic cueing algo-
rithms are studied: (i) proportional-only compensation,
(ii) derivative-only compensation, and (iii) proportional-
derivative compensation.
4. Visual DVE + Haptic: This is similar to the cueing
modality above, but visual cues are disturbed by the use
of foggles.
5. Haptic: Information about the tracking error is provided
to the pilot via full-body haptics only while the pilot
is blindfolded. Two different haptic cueing algorithms
are studied: (i) proportional-only compensation and (ii)
proportional-derivative compensation. Derivative-only
compensation was discarded a priori as pilots as the test
subjects incurred in pilot-induced oscillations (PIOs).
This test matrix is summarized in Table 3. The aircraft dy-
namics is chosen to be representative of the roll dynamics of a
conventional utility helicopter similar to a UH-60 with Level
1 handling qualities. The specific transfer function used in
this study is:
Yc(s) = φ
δlat
(s) = Lδlat
s(s−Lp)(6)
where Lp=−3.5 1/s is the roll acceleration due to the roll
rate and Lδlat =0.147 1/(s2-%) is the roll acceleration due to
a lateral stick displacement. The SOS code and visual cueing
interface of Ref. 6 are developed in C/C++, along with in-
terface between these two, the TESLASUIT application pro-
gramming interface (API), and the aircraft dynamics. The air-
craft dynamics is implemented in MATLAB®/Simulink and
subsequently compiled to C/C++ code.
PARAMETRIC IDENTIFICATION
Parametric identification studies make use of the CIFER®
(Ref. 9) software tool. The identification procedure is based
on a two-step process. First, frequency responses of the air-
craft output to the tracking error are extracted from piloted
simulation data. Next, state-space models are identified from
the frequency response data.
Pilot Vehicle System Dynamics
Consider the PVS dynamics of Eq. (1):
YOL(s) = φ
eφ
(s) = ωc
se−sτe(7)
To be suitable for state-space parametric identification, these
dynamics are transformed into state-space form:
˙
φ=0φ+ωceφ(t−τe)(8)
As such, the parameters to be identified from input-output
data are ωcand τe.
Pilot Dynamics
Consider now expressing the PVS dynamics of Eq. (1) in
terms of the pilot and aircraft dynamics parameters:
YOL(s) = φ
eφ
(s) = YpYc=Kpe−sτeLδlat
s(s−Lp)(9)
For the sake of state-space parametric identification, these dy-
namics are transformed into state-space form as well:
˙p
˙
φ=Lp0
1 0p
φ+KpLδlat
0eφ(t−τe)(10)
RESULTS
Pilot Vehicle System Performance
The PD gains used for the set of experiments object of this
analysis is Kp=7 %/deg and KD=5 %/(deg-s). The pro-
portional gain stems from Eq. (5) whereas the derivative gain
was found via trial and error. Figure 8 shows an example time
history for a non-aggressive compensatory tracking task with
a haptic-only cueing modality and a PD compensation.
The Desired and Adequate performance success rate of each
test subject and for each cueing modality is shown in Fig. 9.
Desired performance success rates, shown in Fig. 9a, indi-
cate that haptic-only cueing adopting proportional compensa-
tion (light blue boxes) is not sufficient to achieve satisfactory
performance. On the other hand, haptic-only cueing adopt-
ing PD compensation (blue boxes) provides satisfactory per-
formance for three out of four test subjects, where the mean
performance of test subjects not meeting the requirements is
still close to the satisfactory threshold. This constitutes a sig-
nificant result in that it suggests that the task in considera-
tion can be flown without the primary sense of vision, but
rather with a secondary perceptual channel like haptics. Fog-
gles appear to be an effective strategy to simulate DVE in that
desired performance in DVE when using visual-only cueing
is significantly degraded (light green and green boxes). The
adoption of proportional-only (pink boxes) and derivative-
only (red boxes) compensation strategies in combined visual
6
Table 3: Test matrix.
Experiment # Cueing Modality KP[%/deg] KD[%-s/deg]
1 Haptic P 7 0
2 Haptic PD 7 5
3 Visual - -
4 Visual DVE - -
5 Visual + Haptic P 7 0
6 Visual + Haptic D 0 5
7 Visual + Haptic PD 7 5
8 Visual DVE + Haptic P 7 0
9 Visual DVE + Haptic D 0 5
10 Visual DVE + Haptic PD 7 5
0 10 20 30 40 50 60
-20
-10
0
10
20
φ [deg]
Desired
Actual
0 10 20 30 40 50 60
-40
-20
0
20
˙eφ[d e g / s ]
0 10 20 30 40 50 60
Time [s]
-20
-10
0
10
20
δlat [%]
Fig. 8: Example time history for a non-aggressive
compensatory tracking task with a haptic-only cueing
modality and a PD compensation.
and haptic cueing does not appear to particularly enhance or
degrade performance. On the other hand, combined visual and
haptic cueing making use of PD compensation (light orange
boxes) shows a slight increase in performance across all pi-
lots. However, the most accentuated increase in performance
is when haptic cueing is paired with visual cueing in DVE
(orange, liliac, and purple boxes). For some test subjects, the
performance lost from degraded vision is almost entirely re-
covered through the use of haptic cueing, particularly when
using PD compensation. Similar observations can be made
for Adequate performance success rates in Fig. 9b.
Figure 10 shows the root-mean-square (RMS) of the roll at-
titude tracking error. These results further substantiate what
is observed in the Desired and Adequate success rate plots.
These results suggest that the adoption of haptic cueing might
be especially useful when visual perception is degraded or de-
nied. On the other hand, pairing haptic cueing with vision
when the latter is not degraded appears to yield only mod-
est improvements. Additionally, the PD haptic compensation
strategy provides better performance than proportional-only
(a) Desired performance.
(b) Adequate performance.
Fig. 9: Desired and Adequate performance success rates.
and derivative-only compensation.
7
Fig. 10: Root mean square (RMS) of tracking error.
Identified Pilot Vehicle System Dynamics
While time-domain Desired and Adequate metrics for task
performance might be representative of the performance of
the haptic cueing strategies developed herein, it is important
to also perform a frequency-domain analysis to get a better un-
derstanding of these haptic cueing laws. As such, frequency-
domain identification is performed both on the open-loop pi-
lot vehicle and on the pilot dynamics. The frequency range
used for the identification process is 0.5≤ωc≤4 rad/s for
non-DVE visual-only cueing tasks, 0.5≤ωc≤3 rad/s for
DVE visual and combined visual and haptic cueing tasks,
and 0.5≤ωc≤2 rad/s for haptic-only cueing tasks. The
average cost functions associated with the identification of
each cueing modality for each subject were always less than
J=100 and typically less than J=50. An average cost func-
tion across all cost functions for each frequency response of
J≤100 generally reflects an acceptable level of accuracy for
flight dynamics modeling, whereas a cost function of J≤50
can be expected to produce a model that is nearly indistin-
guishable from the original both in the frequency and time
domains. However, some of the individual cost functions can
reach J≤200 without resulting in a noticeable loss of overall
predictive accuracy (Ref. 9).
Figure 11 shows the identified crossover frequency and trans-
port delay of the open-loop PVS dynamics. This figure shows
that haptic-only cueing adopting a proportional compensation
strategy (light blue circles) is associated with high time delays
and low crossover frequencies. On the other hand, haptic-
only cueing making use of a PD compensatory strategy (blue
pluse signs) is characterized by significantly lower time de-
lays and somewhat higher crossover frequencies. Visual-only
feedback (light green asterisks) shows low time delays and
high crossover frequencies whereas visual-only feedback in
DVE (green crosses) shows comparatively higher time de-
lays and lower crossover frequencies. Combined haptic and
visual cueing in non-DVE conditions (pink squares, red dia-
monds, and light orange downward-pointing triangles) shows
slightly higher crossover frequencies than visual-only cueing,
especially for PD compensation. The most significant gains
are observed for combined visual and haptic cueing in DVE
conditions (orange right-pointing triangles, liliac left-pointing
triangles, and purple pentagrams) when compared to visual-
only cueing in DVE, where time delays are significantly lower
whereas crossover frequencies are significantly higher.
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3
Crossover frequency, ω
c [rad/s]
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Transport delay, τ [s]
Haptic P
Haptic PD
Visual
Visual DVE
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Visual DVE + Haptic P
Visual DVE + Haptic D
Visual DVE + Haptic PD
Fig. 11: Crossover frequency and transport delay of the PVS.
Figure 12 shows the identified pilot gain and transport delay
of the pilot dynamics. These results are in line with those in
Fig. 11 as the pilot gains for the pilot and aircraft models
in consideration is Kp=ωcLp/Lδlat. This result is obtained by
equating Eq. (1) with the product of Eqs. (2) and (3).
20 30 40 50 60 70 80
Pilot gain, k p [%/deg]
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Transport delay, τ [s]
Haptic P
Haptic PD
Visual
Visual DVE
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Visual DVE + Haptic P
Visual DVE + Haptic D
Visual DVE + Haptic PD
Fig. 12: Pilot gain and transport delay of the PVS.
These results are in agreement with the Desired and Adequate
success rates, in that (i) haptic-only cueing necessitates PD
compensation and (ii) haptic cueing is particularly useful in
8
restoring lost perception when paired with degraded visual
cueing.
Frequency-Domain Analysis
This section focuses on the frequency responses identified
from input-output data. The frequency responses shown are
those for the open-loop pilot vehicle dynamics of Eq. (1), i.e.,
for φ/eφ(s). These frequency responses are averaged across
all pilots to be representative of the mean behavior of all test
subjects.
Figure 13 shows the open-loop PVS response for haptic-only
cueing. Proportional-only compensation is associated with
a dynamics approximately described by a gain around the
crossover frequency whereas PD compensation is represen-
tative of a first-order dynamic system close to crossover fre-
quency and thus, it behaves like a crossover model (Ref. 5).
The phase lag of PD compensation appears to be lower than
that of proportional-only compensation, which is expected as
PD compensation provides information on the time derivative
of the tracking error.
10 0
0
2
4
6
8
10
Mag [dB]
Haptic P
Haptic PD
10 0
-180
-160
-140
-120
-100
Phase [deg]
10 0
Frequency, ω [rad/s]
0
0.2
0.4
0.6
0.8
1
Coh
Fig. 13: Averaged frequency responses of the open-loop PVS
dynamics for haptic-only cueing.
Figure 14 shows the open-loop PVS response for visual-only
cueing. Visual cueing in DVE is associated with a lower gain
and crossover frequency across the frequency range of inter-
est compared to non-DVE visual cueing, Additionally, visual
cueing in DVE exhibits a higher phase lag for frequencies ap-
proximately greater than 1 rad/s. This is indicative of the fact
that foggles are an effective way to decrease PVS performance
due to the degraded visual perception of the pilot.
Figure 15 shows the open-loop PVS response for haptic-only
cueing (PD compensation), non-DVE visual-only cueing, and
10 0
-5
0
5
10
15
Mag [dB]
Visual
Visual DVE
10 0
-200
-150
-100
-50
Phase [deg]
10 0
Frequency, ω [rad/s]
0
0.2
0.4
0.6
0.8
1
Coh
Fig. 14: Averaged frequency responses of the open-loop PVS
dynamics for visual-only cueing.
for combined visual and haptic cueing. Haptic-only cueing
is associated with a low gain across all frequencies of inter-
est when compared to the other strategies, along with a higher
phase lag at low frequency. This suggests that PD haptic cue-
ing is perceived by the pilot slower than visual cueing. Com-
bined visual and haptic cueing do not seem to significantly
affect the response gain or phase lag. These results suggest
that augmenting visual cueing with haptics yields minor im-
provements when compared to visual-only feedback.
Figure 16 shows the open-loop PVS response for haptic-only
cueing (PD compensation), non-DVE and DVE visual-only
cueing, and for combined visual and haptic cueing in DVE.
Non-DVE visual-only cueing is associated with the highest
gain among all cueing modalities shown whereas visual-only
cueing in DVE exhibits the lowest gain. Notably, the gain loss
stemming from DVE is partially recovered through the used of
haptic cueing, and especially PD haptic compensation. More-
over, combined visual cueing in DVE and PD haptic compen-
sation shows a lower phase delay than visual-only cueing in
DVE, with the phase lag overlapping that of visual-only cue-
ing in DVE up to about the crossover frequency. This suggests
that PVS performance lost because of DVE can be partially
restored by leveraging secondary sensory cues. In summary,
combined visual and full-body haptics in DVE yields higher
gain and lower phase lag than with visual feedback alone.
This result further substantiates the results obtained through-
out the paper, which indicate that haptic feedback may be par-
ticularly helpful when visual cues are degraded or denied.
Figure 17 shows the same responses as Fig. 16 but for test
subject 4 only. According to the results in Figs. 9a and 9b,
test subject 4 is the best interpreter of haptic-only feedback
among all test subjects. Their frequency response is shown to
9
10 0
-5
0
5
10
15
Mag [dB]
10 0
-200
-150
-100
-50
Phase [deg]
10 0
Frequency, ω [rad/s]
0
0.2
0.4
0.6
0.8
1
Coh
Haptic PD
Visual
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Fig. 15: Averaged frequency responses of the open-loop PVS
dynamics for haptic-only cueing (PD compensation),
non-DVE visual-only cueing, and for combined visual and
haptic cueing.
10 0
-5
0
5
10
15
Mag [dB]
10 0
-200
-150
-100
-50
Phase [deg]
10 0
Frequency, ω [rad/s]
0
0.2
0.4
0.6
0.8
1
Coh
Haptic PD
Visual
Visual DVE
Visual DVE + Haptic P
Visual DVE + Haptic D
Visual DVE + Haptic PD
Fig. 16: Averaged frequency responses of the open-loop PVS
dynamics for haptic-only cueing (PD compensation),
non-DVE and DVE visual-only cueing, and for combined
visual and haptic cueing in DVE.
demonstrate that haptic-only cueing can provide a compara-
ble response to visual-cueing-only feedback, although with a
higher phase lag. Moreover, for this particular test subject, the
performance lost for visual-only cueing in DVE is almost en-
tirely recuperated via PD haptic cueing. It is also worth noting
that haptic-only cueing and combined visual and haptic cue-
ing show very similar performance, indicating that this test
subject probably predilects haptics over vision when there is
no clear relative dominance of these two perceptual channels.
10 0
-10
-5
0
5
10
15
Mag [dB]
10 0
-200
-150
-100
-50
Phase [deg]
10 0
Frequency, ω [rad/s]
0
0.2
0.4
0.6
0.8
1
Coh
Haptic PD
Visual
Visual DVE
Visual DVE + Haptic P
Visual DVE + Haptic D
Visual DVE + Haptic PD
Fig. 17: Test subject 4 frequency responses of the open-loop
PVS dynamics for haptic-only cueing (PD compensation),
non-DVE and DVE visual-only cueing, and for combined
visual and haptic cueing in DVE.
Identified Crossover Model at Discrete Frequencies
The parameters of the open-loop PVS crossover model of
Eq. (1) were identified from the open-loop PVS frequency
responses (e.g., Figs. 13 through 17) using only the sum-of-
sines frequency where the forcing energy is. Figures 18 and
19 show the crossover frequency and equivalent delay, respec-
tively. As expected based on the performance for the differ-
ent cueing methods, haptic-only cueing (P compensation) has
the lowest overall crossover frequency and highest equivalent
delay (close to τe=1.0 sec). Haptic-only with PD compen-
sation reduces the equivalent delay by nearly 50% to around
τe=0.4−0.6 sec, due to teh additional lead provided by the
derivative path. The highest crossover frequency and lowest
delay are seen for combined visual cueing non-DVE and PD
haptic, which also showed the best overall task performance.
The PVS stability margins were calculated from the identi-
fied open-loop PVS crossover model. Figures 20 and 21 show
the gain and phase margins of the open-loop PVS system for
the different cueing methods, respectively. The mean stability
margins are similar for all cases (Gm ≈5 dB and Pm≈35),
which are typical numbers for PVS (Ref. 10), and similar
to the assumed values used in the ADS-33 (Ref. 11) piloted
bandwidth requirements (6 dB and 45 deg). It is clear that the
10
Haptic P
Haptic PD
Visual
Visual DVE
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Visual + Haptic DVE P
Visual + Haptic D
Visual + Haptic DVE PD
0.5
1
1.5
2
2.5
3
c [rad/sec]
Pilot 1
Pilot 2
Pilot 3
Pilot 4
Fig. 18: Identified open-loop PVS crossover frequency.
Haptic P
Haptic PD
Visual
Visual DVE
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Visual + Haptic DVE P
Visual + Haptic D
Visual + Haptic DVE PD
0
0.2
0.4
0.6
0.8
1
1.2
e [sec]
Pilot 1
Pilot 2
Pilot 3
Pilot 4
Fig. 19: Identified open-loop PVS equivalent time delay.
subjects adjust their gain (PVS crossover freuqency) based on
the equivalent loop delay such that they operate with similar
stability margins for all cases. Increasing their gain further
to try to achieve better performance will result in a reduction
in stability margins, a more oscillatory closed-loop responses,
and ultimately worse performance.
Test Pilot Qualitative Feedback
This section reports the feedback from the test subject who is
a test pilot. The test pilot comments are articulated below:
1. The vision restricting devices (”foggles”) were mission-
representative in that they gave the perception of flying
in and out of the clouds and contributed to spatial disori-
entation. It was significantly harder to track commanded
roll attitude with visual feedback only. Often it was nec-
essary for the attitude to exceed the bow tie boundary
Haptic P
Haptic PD
Visual
Visual DVE
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Visual + Haptic DVE P
Visual + Haptic D
Visual + Haptic DVE PD
0
5
10
15
GM [dB]
Pilot 1
Pilot 2
Pilot 3
Pilot 4
Mean
Fig. 20: Identified open-loop PVS gain margin
Haptic P
Haptic PD
Visual
Visual DVE
Visual + Haptic P
Visual + Haptic D
Visual + Haptic PD
Visual + Haptic DVE P
Visual + Haptic D
Visual + Haptic DVE PD
0
20
40
60
80
PM [deg]
Pilot 1
Pilot 2
Pilot 3
Pilot 4
Mean
Fig. 21: Identified open-loop PVS phase margin.
before a deviation was visually apparent, requiring large
roll corrections in the correct direction. However, large
and rapid control inputs are not a recommended control
strategy in degraded visual environments.
2. Proportional-only haptic compensation coupled with
vision was straightforward to comprehend in that it
matched with visual cues. When the actual attitude indi-
cator crossed the commanded attitude indicator, the hap-
tic signal changed sign and switched from the right/left
to the right/left portion of the upper body. However, the
perceived workload was high because corrections needed
to be very prompt: any time delay in lateral cyclic input
could result in getting out of phase as the roll rate re-
versed.
3. Derivative-only haptic compensation coupled with vi-
sion restriction led to spatial disorientation due to the
absence of roll attitude feedback. As such, it was dif-
11
ficult to determine the efficacy of the corrective inputs.
The pilot response was characterized by extremely rapid
discrete lateral stick displacements in the direction of the
feedback, waiting until the next signal, and then applying
the appropriate impulsive input again. This was proba-
bly a result of the high workload: only the direction of
the input could be determined quickly, the time needed to
properly assess and command the appropriate magnitude
for the required correction was excessive.
4. PD haptic compensation coupled with visual feedback
was initially confusing because the haptic signal might
not switch signs when the visual attitude indicator
crossed the commanded attitude line, depending on the
error of the tracking error derivative. This contributed
to initial mistrust and hesitation in response in the haptic
cueing algorithm. However, with some practice, it was
favored over proportional- and derivative-only compen-
sation, and performance improved.
5. Regarding the perceived workload, the pilot reported that
combined visual and haptic cueing required more “men-
tal reserve” in order to decide which one to trust. This
could be attributed to several factors. The first possibil-
ity is mistrust in the novel cueing method presented to the
pilot, possibly curable with practice. The second factor
could be non-congruent haptic and visual cueing, per-
haps due to different delays between visual and haptic
systems. A third explanation is that accuracy and per-
ception of visual perception in DVE may be reduced to
a point where they are comparable with those of haptic
cueing, so there is no longer a dominant cue between vi-
sion and haptics. This would force the pilot to choose
which one to trust, adding to cognitive workload, espe-
cially if the cues were conflicting or incongruent.
6. The addition of haptic cueing was received favorably by
the pilot, particularly in DVE. Haptic cueing helped re-
store the perception of both the tracking error and its rate
of change.
7. The test pilot reported an increased sensitivity to auditory
noise when using haptic-only feedback. The amount of
mental reserve that could be applied to the haptic feed-
back probably increased, however.
8. When flying with haptic cueing only, good performance
(as evaluated with Adequate and Desired performance
metrics) came with a feeling of “having done terribly”
and of being in a “massive pilot-induced oscillation
(PIO)” while poor performance was accompanied by a
feeling of having done well. This will be investigated
further in subsequent experiments.
9. At first the electro-stimulation feedback felt awkward to
the pilot, but the system made significantly more sense
when flying completely blind. The activation of one
shoulder muscle group by the suit easily triggered the
brain to activate the right arm and to move laterally in
that direction. The association was logical and somewhat
easily executed with minimal training.
CONCLUSIONS
Full-body haptic cueing algorithms were developed, imple-
mented, and tested for augmented pilot perception. These
haptic cueing algorithms are based on localized electrical
muscle stimulation (EMS) obtained via a commercial, off-
the-shelf full-body haptic suit. Cueing algorithms were de-
veloped for roll-axis compensatory tracking tasks where the
pilot acts on the displayed error between a desired input and
the comparable vehicle output motion to produce a control ac-
tion. The tracking error was displayed to the pilot using three
different cueing modalities: visual, haptic, and combined vi-
sual and haptic. For the visual and combined visual and hap-
tic modalities, visual cues were also considered in degraded
visual environments (DVE). Experiments that involved four
test subjects, one of which was a test pilot, were conducted to
gather quantitative data and qualitative feedback for analyzing
the performance of the haptic cueing algorithms. Time- and
frequency-domain analyses on the test data were performed.
Based on this work, full-body haptic cueing algorithms that
are based on a proportional-derivative (PD) compensation
strategy on the tracking error were found to provide satis-
factory PVS performance for the task in consideration when
using haptic feedback only (no visual cues) and to improve
PVS performance, especially in degraded visual environment
(DVE) when using combined visual and haptic feedback.
These results indicate that the use of secondary sensory cues
such as full-body haptics to augment the vehicle percep-
tion can lead to improved/partially-restored PVS performance
when primary sensory cues like vision are impaired or denied.
Future work will focus on extending the haptic cueing algo-
rithms to aggressive and non-aggressive tasks with the varying
dynamic characteristics of the aircraft (e.g., response repre-
sentative of Level 2 and Level 3 handling qualities), to other
axes singularly (e.g., pitch), and to multiple axes tracking
tasks (e.g., roll and pitch) with and without cross-coupling
between the axes. Stabilization tasks for the roll axis only,
pitch axis only, and combined roll and pitch axes will also be
considered. Additionally, these algorithms will be extended
to 3-D (or spatial) audio cueing. Visua, full-body haptic, and
spatial audio cueing will be combined to assess the interac-
tional dynamics of these three sensory paths. The pilot work-
load will be estimated for each cueing modality using the ap-
proach in Ref. 8. All of these measures will be compared to
address potential differences in pilot equalization and work-
load as a function of cueing strategy.
ACKNOWLEDGMENTS
This research was partially funded by the U.S. Government
under agreement no. N000142312067. The views and con-
clusions contained in this document are those of the authors
and should not be interpreted as representing the official poli-
cies, either expressed or implied, of the DEVCOM Aviation
12
& Missile Center Technology Development Directorate or the
U.S. Government.
REFERENCES
1McGrath, B. J., “Tactile Instrument for Aviation,” Techni-
cal report, NAMRL Monograph 49, Naval Aerospace Medical
Research Laboratory, Pensacola, FL, 2000.
2McGrath, B. J., Estrada, A., Braithwaite, M. G., Raj, A. K.,
and Rupert, A. H., “Tactile Situation Awareness System Flight
Demonstration,” Technical report, Army Aeromedical Re-
search Laboratory, Fort Rucker, AL, 2004.
3Wolf, F. and Kuber, R., “Developing a head-mounted tac-
tile prototype to support situational awareness,” International
Journal of Human-Computer Studies, Vol. 109, 2018, pp. 54–
67.
doi: 10.1016/j.ijhcs.2017.08.002
4Jennings, S., Cheung, B., Rupert, A., Schultz, K., and
Craig, G., “Flight-Test of a Tactile Situational Awareness Sys-
tem in a Land-based Deck Landing Task,” Proceedings of the
Human Factors and Ergonomics Society 48th Annual Meet-
ing, September 20-24, 2004.
doi: 10.1177/154193120404800131
5McRuer, D. T., Clement, W. F., Thompson, P. M., and
Magdaleno, R. E., “Minimum Flying Qualities. Volume 2: Pi-
lot Modeling for Flying Qualities Applications,” Technical re-
port, Systems Technology, Inc Hawthorne, CA, 1990.
6Klyde, D. H., Ruckel, P., Pitoniak, S. P., Schulze, P. C.,
Rigsby, J., Xin, H., Brewer, R., Horn, J., Fegely, C. E., Con-
way, F., Ott, C. R., Fell, W. C., Mulato, R., and Blanken, C. L.,
“Piloted Simulation Evaluation of Tracking MTEs for the As-
sessment of High-Speed Handling Qualities,” Proceedings of
the 74th Annual Forum of the American Helicopter Society,
May 14-17, 2018.
7McRuer, D. T. and Jex, H. R., “A Review of Quasi-Linear
Pilot Models,” IEEE Transations on Human Factors in Elec-
tronics, Vol. 8, (3), 1967, pp. 231–249.
doi: 10.1109/THFE.1967.234304
8Bachelder, E. N. and Godfroy-Cooper, M., “Pilot Work-
load Estimation: Synthesis of Spectral Requirements Analysis
and Weber’s Law,” AIAA 2014-1467, AIAA SciTech Forum,
January 7-11, 2019.
doi: 10.2514/6.2019-1228
9Tischler, M. B. and Rample, R. K., Aircraft and Rotor-
craft System Identification: Engineering Methods and Flight
Test Exemples, Second Edition, Princeton University Press,
Princeton, NJ, 1972.
10McRuer, D., Graham, D., and Reisener, W., “Human Pi-
lot Dynamics in Compensatory Systems,” Technical report,
AFFDL-TR-65-15, Air Force Flight Dynamics Laboratory,
Air Force Systems Command, Wright-Patterson Air Force
Base, OH, Redstone Arsenal, AL, 1965.
11Anon, “Aeronautical design standard performance spec-
ification handling qualities requirements for military rotor-
craft,” Technical report, U.S. Army Aviation and Missile
Command Aviation Engineering Directorate, Redstone Arse-
nal, AL, 2000.
13
