Content uploaded by Umberto Saetti

Author content

All content in this area was uploaded by Umberto Saetti on Feb 18, 2020

Content may be subject to copyright.

Probabilistic Assessment of Pilot-Vehicle System Performance and Perceived

Vehicle Handling Qualities

Umberto Saetti

Postdoctoral Fellow

School of Aerospace Engineering

Georgia Institute of Technology

Atlanta, GA 30313

Jonathan Rogers

Associate Professor

School of Aerospace Engineering

Georgia Institute of Technology

Atlanta, GA 30313

ABSTRACT

This paper proposes a novel mathematical methodology for probabilistic handling qualities evaluation using the Koop-

man operator. The Koopman operator is used to quantify the impact of parametric uncertainty in a pilot model to

speciﬁc pilot-vehicle system performance speciﬁcations and to perceived vehicle handling qualities. The considered

speciﬁcations include the Hover Mission Task Element (MTE), broken loop gain and phase margins, and pilot opinion

ratings of workload. The application to each speciﬁcation is illustrated through an example involving the UH-60 heli-

copter. Results demonstrate the utility of the proposed method in mapping uncertainty in the pilot model parameters

to uncertainty in handling qualities evaluations.

INTRODUCTION

A key requirement in the certiﬁcation of a new or modiﬁed

rotorcraft is that the aircraft meet speciﬁc handling qualities

(HQ), performance, and stability requirements. In a typical

certiﬁcation process, evaluation of these criteria involves the

use of pilot opinion ratings of the aircraft according to the

Cooper-Harper rating scale (Ref. 1) for various Mission Task

Elements (MTEs) (Ref. 2). However, inter/intra pilot vari-

ability in opinion rating exists due to pilot physiological dif-

ferences and differing levels of aggressiveness. Mathematical

pilot models that account for such differences have been de-

veloped in the past (Refs. 3–5). Particularly, the parameter-

ized pilot model proposed in Ref. 5accounts for differences

in both level of aggressiveness and central nervous and neu-

romuscular systems. When combined with uncertainty quan-

tiﬁcation (UQ) techniques, these models may be exploited to

assess how the uncertainty in level of aggressiveness and pilot

physiological characteristics propagates to the pilot-vehicle

system performance and perceived vehicle handling qualities.

In this manner, assessments can be made as to the probability

that a vehicle conforms to speciﬁc handling qualities, given

the uncertainty in the pilot-vehicle closed-loop dynamics.

Pilot-vehicle system performance variability and pilot rating

variability have been the subject of a signiﬁcant body of prior

work (Refs. 3,6–8). Likewise, some prior research investi-

gating probabilistic handling qualities evaluation techniques

has been documented (Refs. 9–12). However, to the au-

thors’ knowledge there has been no prior attempt to bridge

the gap between variability in pilot models and variability

in pilot-vehicle system performance and/or perceived vehicle

Presented at the 2020 Rotorcraft Handling Qualities Technical Meet-

ing, Huntsville, AL, Feb 19–19, 2020. Copyright c

2020 by the

Vertical Flight Society. All rights reserved.

handling qualities. In light of this, there is a need for computa-

tionally efﬁcient methodologies that can map the uncertainty

in the pilot model parameters to the resulting uncertainty in

handling qualities evaluations. Further, all the cited research

efforts investigating the effects of parametric uncertainty on

handling qualities use uncertainty quantiﬁcation methods that

suffer from poor scalability. In the UQ domain, explicit uncer-

tainty quantiﬁcation approaches, particularly the Frobenius-

Perron (FP) Operator (Ref. 13), are used to map a probability

density explicitly through a nonlinear transformation with im-

proved scalability characteristics with respect to Monte Carlo.

Recently, Meyers et al. (Ref. 14) applied the FP opera-

tor and its adjoint, the Koopman operator, to various optimal

decision-making problems under uncertainty, demonstrating

the scalability beneﬁts of these explicit techniques compared

to Monte Carlo. For this reason, the Koopman operator is

chosen as the preferred method for this study.

This paper presents a novel methodology for computing prob-

abilistic handling qualities performance assessments using the

Koopman operator. The algorithm begins with a discretiza-

tion of the uncertainty space comprised of the uncertain pilot

model parameters. Each discretized point is mapped through

relevant transformations to the handling qualities evaluation

domain, resulting in a handling qualities performance assess-

ment for each discretized point (similar to Monte Carlo, ex-

cept usually with signiﬁcantly less points). Using this map-

ping, the handling qualities evaluation is pulled back to the

initial uncertainty domain using the Koopman operator. The

expected value of the handling qualities performance is then

computed through an expected value integral with the initial

joint PDF, yielding a probabilistic assessment with respect

to relevant speciﬁcations. Following a detailed discussion of

the methodology and its computational implementation, ex-

1

(a)

(b)

Figure 1: (a) Forward Mapping of Uncertainty Domain to

HQ/Performance Domain (b) Pull-Back of HQ Evaluation

Score to Uncertainty Domain Using Koopman Operator.

amples are presented for a modiﬁed UH-60 helicopter model.

The paper begins with an explanation of the mathematical

methodology behind probabilistic handling qualities evalua-

tion using the Koopman operator. In the second section, the

mathematical framework for a quasi-nonlinear rotorcraft sim-

ulation model is deﬁned. The third section describes the im-

plementation of a multi-axis structural pilot model for holding

position and heading in hover. The fourth and ﬁfth sections

applies the Koopman operator to the propagation of the un-

certain parameters in the pilot model to speciﬁc pilot-vehicle

system performance speciﬁcations and to perceived vehicle

handling qualities including the Hover MTE, broken loop gain

and phase margins, and pilot opinion ratings of workload. For

each type of speciﬁcation, the relevant state transformation

and cost function deﬁnition(s) is identiﬁed. Each is illustrated

through an example involving the UH-60 helicopter. These

results demonstrate the utility of the proposed method in map-

ping uncertainty in the pilot model parameters to uncertainty

in handling qualities evaluations. Final remarks summarize

the overall ﬁndings of the study and identify areas for future

work.

METHODOLOGY

Deﬁne a closed-loop pilot-rotorcraft system, with state vector

~xand parameterized by ~p∈Rpaccording to:

˙

~x=~

f(~x,~p,~u)(1)

where ~uis the vector of controls. A new augmented state

is deﬁned by combining the rigid body state with uncertain

model parameters, and any other components needed to eval-

uate HQ ratings. Deﬁne the set of uncertain model parameters

as ~pu∈Rdwhere ~pu⊂~pand dis the dimension of the uncer-

tainty space. Likewise, ~pk∈Rp−dis the vector of known pa-

rameters. The augmented state is then given as ¯x={~x,~pu}T.

Suppose that a map exists that such that, for each unique ini-

tial state vector ¯x0, a state vector at a ﬁnal time ¯xfis generated

that may be used to evaluate handling qualities:

¯xf=S(¯x0)(2)

In Eq. 2, the transformation Smay represent various types

of algebraic or dynamic transformations. For instance, in the

evaluation of ADS-33E performance standards, Mission Task

Elements (MTEs) are used to categorize ﬂight performance

into Desired, Adequate, or Inadequate performance (or equiv-

alently, into Level 1, Level 2, or Level 3 performance). In

this case, evaluation of this speciﬁcation requires executing

the MTE from the prescribed initial condition using a se-

lected value of ~p. Thus, Smay involve propagation of a ﬂight

dynamic model from the prescribed initial condition for the

MTE to the prescribed ﬁnal condition, with the ﬁnal state

being ¯xf. Other handling qualities, performance, or stabil-

ity metrics may involve other types of dynamic or algebraic

transformations Sto obtain a vector ¯xf. In subsequent discus-

sion, this ﬁnal state vector will be referred to as ¯xeval =¯xffor

clarity, to represent the state vector on which the HQ ratings

can be evaluated.

Given this mapping, a “score” function gis deﬁned that as-

signs a numerical score based on ¯xeval representing the han-

dling qualities or performance speciﬁcation associated with

the forward mapping of the augmented state ¯x0. This score

function value may be “pulled back” to the initial condition

domain of the augmented state through use of the Koopman

operator deﬁned as,

USg=USg(¯xeval) = g(S(¯x0)) (3)

The only restriction on gis that it must be ﬁnite, i.e., ||g∞|| <

∞. The score or cost function value is associated with a

certain handling qualities criterion – for instance, a value of

g(¯xeval ) = 1 may be assigned for ¯xeval in a particular region

associated with Desired handling qualities, g(¯xeval ) = 2 for

regions associated with Adequate handling qualities, and so

on. Note that the Koopman operator USin Eq. 3is deﬁned

speciﬁc to a particular transformation S.

This overall process is illustrated in Fig. 1. In Fig. 1(a),

each discretized point in the uncertainty domain is mapped to

a speciﬁc numerical value in the HQ or performance domain

using the forward mapping S. In Fig. 1(b), the score/cost

function associated with each forward-mapped point is pulled

back to the original uncertainty domain, yielding USg. With

this pulled-back score function, the expected value of the han-

dling qualities assessment function gcan be computed given

the joint probability density function on the uncertain param-

eters ~puaccording to:

E[g(~x,~p)] = ZΩ

~

fpu(~pu)USg(¯xeval )d~pu(4)

where Ωis the support of the joint probability density of the

uncertain parameters, ~

fpu(~pu). This expected value represents

2

the expected handling qualities evaluation score for this par-

ticular metric signiﬁed by the cost or score function g. For

instance, in the example shown in Fig. 1, calculation of Eq.

4would provide a value between 0 and 1, representing the

probability of the vehicle achieving the Desired level of per-

formance for the particular handling qualities or performance

metric. Note that by deﬁning several binary score functions

for each handling qualities ranking, probability values can be

generated that specify the probability that the vehicle will con-

form to each level. This represents a generalization of the

chance constraint techniques detailed in Refs. 14,15.

ROTORCRAFT MODEL

This study uses an 14-state model representative of the ﬂight

dynamics of a UH-60 in hover. The stability and control

derivatives and trim data are taken from Ref. 16. The deriva-

tives and trim data are combined with nonlinear equations of

motion to produce a quasi-nonlinear simulation model (Ref.

17). Consider the dynamics of the ﬂight-identiﬁed linear

model from Ref. 16:

˙

x6

˙

xH=A6A6H

AH6 AHH x6

xH+B6

BHu(5)

where:

x6are the states relative to the rigid-body dynamics,

xHare the nHstates relative to the higher-order dynamics,

A66 contains the rigid-body stability derivatives,

A6H contains the higher-order to rigid-body stability deriva-

tives,

AH6 contains the rigid-body to higher-order stability deriva-

tives,

AHH contains the higher-order stability derivatives,

B6contains the rigid-body control derivatives,

BHcontains the higher-order control derivatives, and

uis the control vector.

The two portions of the state vector are:

xT

6=uvwpqr(6a)

xT

H=β1cβ1s(6b)

where:

u,v,ware the body velocities,

p,q,rare the body angular rates, and

β1c,β1sare the longitudinal and lateral ﬂapping angles.

The control vector is:

u=δlat δlon δcol δped(7)

where:

δlat,δlon are the lateral and longitudinal cyclic inputs,

δcol is the collective input, and

δped is the pedal input.

In order to calculate the aerodynamic perturbation forces and

moments, it is necessary to obtain the state and control pertur-

bations:

∆x6=X6−[X6]0(8a)

∆xH=XH−[XH]0(8b)

∆u=U−U0(8c)

where:

X6are the nonlinear simulation rigid-body states,

[X6]0are the trim rigid-body states in hover,

XHare the nonlinear simulation higher-order states,

[XH]0are the trim higher-order states in hover,

Uis the nonlinear simulation control vector, and

U0is the trim control vector in hover.

It is worth noting that the rigid-body perturbation vector does

not include Euler angles or the position since the aerodynamic

forces and moments do not depend explicitly on the Euler an-

gles or the position (Ref. 17). The aerodynamic perturbation

forces and moments are:

∆Faero

∆Maero=M(A66 ∆x6+A6H∆xH+B6∆u)(9)

where:

M=

m

m

m

Ixx −Ixz

Iyy

−Ixz Izz

(10)

is the mass matrix containing the mass mand inertia terms

Ixx,Iyy,Izz,Ixz for the helicopter. The aerodynamic trim forces

and moments, based on the trim Euler angles Φ0,Θ0in hover,

are:

Faero0=

gsinΘ0

−gcosΘ0sinΦ0

−gcosΘ0cosΦ0

m(11)

The nonlinear simulation gravitational forces, which are non-

linear with respect to the Euler angles, are:

Fgrav =

−gsinΘ

gcosΘsinΦ

gcosΘ0cosΦ

m(12)

The aerodynamic perturbation forces and moments are

summed with the trim and gravitational forces and moments

to obtain the total aerodynamic forces and moments:

F

M=∆Faero +Faero0+Fgrav

∆Maero (13)

These forces and moments are incorporated in the nonlinear

equations of motion in the body-ﬁxed frame to provide the

fuselage 6-DOF dynamics:

˙

U=RV −QW +Fx

m(14a)

3

˙

V=PW −RU +Fy

m(14b)

˙

W=QU −PV +Fz

m(14c)

˙

P=Izx(Ixx −Iyy +Izz)PQ −(I2

zz −IyyIzz +I2

xz)QR +IzzL+IxzN

IxxIzz −I2

xz (14d)

˙

Q=(Izz −Ixx)PR −Ixz(P2−R2) + M

Iyy

(14e)

˙

R=(I2

xx −IxxIyy +I2

xz)PQ −Izx(Ixx −Iyy +Izz)QR +IxzL+Ixx N

IxxIzz −I2

xz (14f)

˙

Φ=P+QsinΦtanΘ+RcosΦtanΘ(14g)

˙

Θ=QcosΦ−RsinΦ(14h)

˙

Ψ=QsinΦsecΘ+RcosΦsecΘ(14i)

˙

X=UcosΘcosΨ+V(sinΦsinΘcosΨ−cosΦsinΨ)

+W(cosΦsinΘcosΨ+sinΦsinΨ)(14j)

˙

Y=UcosΘsinΨ+V(sinΦsinΘsinΨ+cosΦcosΨ)

+W(cosΦsinΘsinΨ−sinΦcosΨ)(14k)

˙

Z=−UsinΘ+Vsin Φcos Θ+Wcos Φcos Θ(14l)

The higher-order dynamics are given by:

˙

B1c

˙

B1s=˙

xH=AH6x6+AHH xH+BH∆u(15)

STRUCTURAL PILOT MODEL

The human pilot is modeled using a Structural Pilot Model

taken from Ref. 4. The closed-loop pilot-rotorcraft system for

a multi-loop tracking task is shown in Fig. 2. The individual

blocks in Fig. 2are given by:

YP

n=ω2

n

s2+2ζnωn+ω2

n

(16a)

Yf=K1s

s+1

T1

(16b)

Ym=K2s

s+1

T2k−1(16c)

where:

Table 1: Structural pilot model inner loop parameters.

Axis ωc[rad/s] T2[sec] Ke

Lateral 2.5 0.246 36.69 [in/rad]

Longitudinal 2 1.534 64.28 [in/rad]

Directional 1 3.238 17.02 [in/rad]

Vertical 1 3.572 -1.331 [in/ft]

YP

nis the open-loop neuromascular dynamics,

Yfand Ymemulate the dynamics associated with muscle

spindles, Golgi tendon organs, and higher-level signal pro-

cessing,

ωnand ζnare the natural frequency and damping ratio as-

sociated with the open-loop neuromuscular dynamics,

Ycis the aircraft dynamics,

YPis the pilot dynamics,

Kedeﬁnes the pilot level of aggressiveness,

τ0is the delay in visual process and motor nerves conduc-

tion times,

Kyand TLdescribe the outer-tracking loop pilot compensa-

tion, and

K1,K2,T1,T2,k, are other parameters deﬁned in Ref. 4.

It is worth noting that the pilot workload, for which a measure

is the root mean square (RMS) of the ratio between the output

of Ymand the pilot gain Ke(i.e. um/Ke), can be related through

the Cooper-Harper rating scale to pilot opinion ratings (Refs.

4,18).

A multi-axis pilot model is created around the unaugmented

(i.e. no automatic ﬂight control system) aircraft. Inner atti-

tude and outer position loops are implemented for the roll and

pitch axes to track lateral and longitudinal positions, respec-

tively. Single loop strategies are implemented for the direc-

tional and vertical axes to track heading and vertical position,

respectively. The structural pilot model parameters are found

using the procedure in Ref. 4. The inner loop crossover fre-

quencies ωcchosen for the open-loop pilot-vehicle system are

reported in Table 1. To achieve crossover model characteris-

tics in the region of crossover for all axes, lead compensation

is needed (i.e. k=2 according to Ref. 4). Again, according to

Ref. 4, if k=2, then K1=1, K2=10, T1=2.5 sec, τ0=0.15

sec, ζn=0.707, and ωn=10 rad/s. T2is chosen to ensure

K/s-like pilot-vehicle system characteristics around the inner

loop crossover frequency. For this reason, T2for each axis is

taken as the opposite of the inverse of the damping derivative

for such axis (e.g., for the roll axis T2=−1/Lp). Values for

T2are reported in Table 1. Next, the pilot gain for each axis

is computed to ensure that the desired crossover frequency is

obtained:

Ke=1

|YP(jωc)|

1

|Yc(jωc)|(17)

The pilot gain values for each axis are reported in Table 1. For

the lateral and longitudinal outer position loops, the crossover

frequencies are reduced by a factor of four with respect to the

inner loop crossover frequencies. Values of the lead time TL

and outer loop gains Kyare found in Table 2.

4

Figure 2: A pilot-rotorcraft system for a multi-loop tracking task.

Table 2: Structural pilot model outer loop parameters.

Axis ωc[rad/s] TL[sec] Ky[rad/ft]

Lateral 0.625 3 0.0074

Longitudinal 0.5 3 -0.0066

Figure 3shows the response of the closed-loop pilot-rotorcraft

system for a step in the longitudinal position. Speciﬁcally,

Fig. 3(a) illustrates that the closed-loop system is able to

track the command while holding lateral position and altitude.

Figure 3(b) shows that the heading is successfully regulated

throughout the maneuver.

PILOT-VEHICLE SYSTEM PERFORMANCE

Time-Domain Speciﬁcations

This section illustrates the mapping of parametric uncertainty

in the pilot model to uncertainty in quantitative handling qual-

ities. Consider the case in which a handling qualities metric is

deﬁned over a transformed augmented state vector. ADS-33E

(Ref. 2) deﬁnes the Hover MTE in which a vehicle must main-

tain hover with certain performance requirements on the ve-

hicle state throughout the maneuver. Suppose a pilot-vehicle

system simulation model with state ~xand uncertain parame-

ters ~puis used to evaluate performance against the assigned

MTE performance criteria. Since central nervous and neu-

romuscular systems properties differ from pilot to pilot, and

since each pilot may ﬂy the rotorcraft with different levels of

aggressiveness, let the pilot lateral and longitudinal gains Keφ,

Keθ, and the neuromuscular delay τ0be uncertain parameters

with joint probability fKeφ,Keθ,τ0(Keφ,Keθ,τ0). The MTE can

be simulated from an initial condition ~x0and speciﬁc set of

parameters to obtain a time history of ~xthroughout the ma-

neuver. Key values from this time history needed to evaluate

the MTE metrics, such as maximum position and heading ex-

cursions, may be extracted from this time history to produce

(a)

(b)

Figure 3: Closed-loop pilot-rotorcraft response to a step in

commanded longitudinal position: (a) position (b) attitude.

5

Table 3: Maximum position and heading excursions as

deﬁned in the ADS-33E Hover MTE (Ref. 2)

Performance X[ft] X[ft] Z[ft] Ψ[deg]

Desired ±3±3±2±5

Adequate ±6±6±4±10

an evaluation state ¯xeval. Maximum position and heading ex-

cursions for different performance levels of the Hover MTE

are shown Table 3. The mapping S, which maps ¯x0to ¯xeval is

thus the dynamic simulation model of the helicopter. Given

¯xeval, a cost or score function g(¯xeval )may be deﬁned based on

the MTE deﬁnition which categorizes performance into De-

sired, Adequate, and Inadequate based on the speciﬁc thresh-

olds listed in ADS-33E. Once the simulation model is used

to compute USg(¯xeval)over a discretized set of points in the

initial augmented state space, the expected score can be com-

puted as:

E[g(¯xeval)] =

ZΩ

fKeφ,Keθ,τ0(Keφ,Keθ,τ0)USg(¯xeval)d~xd~pu(18)

As an alternative, suppose three different score functions are

used to assess performance. Each of these is deﬁned as an

indicator function such that:

gDes(¯xeval) = (1 if ¯xeval ∈(¯xeval)Des

0 else (19)

where ¯xeval ∈(¯xeval)Des is the set of evaluation vectors that is

deﬁned to give Desired performance according to the MTE

deﬁnition. Analogous score function deﬁnitions can be cre-

ated for the Adequate and Inadequate categories. Then, the

probability that the system satisﬁes the Desired, Adequate,

and Inadequate speciﬁcations can be computed as:

E[gDes(¯xeval] =

ZΩ

fKeφ,Keθ,τ0(Keφ,Keθ,τ0)USgDes(¯xeval)d~xd~pu(20)

In practice, each uncertain parameter dimension is discretized

into np=8 evenly-spaced grid points spanning µ±4σ, where

µand σare the mean and standard deviation of each parame-

ter. Since the number of uncertain parameters is 3, this leads

to a total of n3

p=512 data points. The number of grid points

npis chosen to ensure that the numerical solution of the inte-

grals deﬁned on the uncertainty domain (Eqs. 18,20) is sufﬁ-

ciently accurate. Table 4shows mean and standard deviation

values of the uncertain parameters. While mean and standard

deviation for the neuromuscular time delay τ0are taken from

Ref. 5, the standard deviations for the lateral and longitudinal

pilot gains are selected somewhat arbitrarily for the purposes

of this initial study.

A Control Equivalent Turbulence Input (CETI) model is used

to simulate turbulence (Ref. 19). The CETI model is a

Table 4: Uncertain parameters statistics.

Parameter Mean, µStandard Deviation, σ

Keφ36.69 [in/rad] 5% ¯

Keφ[in/rad]

Keθ64.28 [in/rad] 5% ¯

Keθ[in/rad]

τ00.15 [sec] 0.025 [sec]

Table 5: CETI turbulence model parameters (Ref. 20).

Turbulence Level U0[ft/s] σv[ft/s] σw[ft/s]

Moderate 60.0 8.5 6.5

Light 30.0 4.3 3.3

Ambient 13.3 1.9 1.5

hover/low-speed turbulence model that simulates the effects

of atmospheric turbulence by injecting turbulence equivalent

inputs into the control system of an aircraft. The CETI model

used in this simulation study is:

δ0

lat,lon

ξ(s) = 0.2783σ0.9912

wrU0

πL1

s+αw(21a)

δ0

col

ξ(s) = 0.0682σ0.5490

wr3U0

πL1+10.2αw

(s+0.53αw)(s+1.5αw)

(21b)

δ0

ped

ξ(s) = 0.5014σ0.7475

wrU0

πL1

s+αw(21c)

where:

(.)0denotes CETI inputs to the control system,

ξis white noise,

U0is the mean wind speed,

σv,σware the wind speed standard deviations in the lateral

and vertical directions,

L=53.7 ft is the UH-60 main rotor diameter, and

αw=2U0/L.

The CETI model parameters are taken from Ref. 20 and re-

ported in Table 5for different turbulence levels.

For each data point, to which corresponds a particular pilot

model, twenty 30 sec hover simulations are run. The maxi-

mum position and heading excursions for each data point are

taken as the average maximum excursions across the twenty

simulations. If the explicit uncertainty propagation problem

was to be treated rigorously, white noise would have to be ap-

proximated deterministically using a Karhunen-Lo`

eve (KL)

expansion (Ref. 21). This is because the Koopman operator

approach is limited to parametric uncertainty only as it does

not model diffusion of the augmented state PDF due to pro-

cess noise. The KL expansion of white noise is:

ξ(t)≈r2

T

Nt

∑

i=1

ζi(ω)cosi−1

2πt

T(22)

where:

6

tis the current simulation time,

Tis the total simulation time,

Ntis the number of terms used for the expansion, and

ζi(ω)∼N(0,1)are the independent and identically dis-

tributed (i.i.d.) random variables (or uncertain Fourier coefﬁ-

cients) characterizing the white noise frequency content.

The i.i.d. random variables would have to be included in the

uncertain parameter vector, thus increasing signiﬁcantly the

order of the problem. Another approach would be to ﬁx such

variables and thus excite the system with a deterministic sig-

nal. It is worth noting that if the i.i.d. variables are ﬁxed,

Eq. 22 reduces to a sum of cosines. Future investigations will

consider these approaches.

Once each data point in the uncertainty domain is mapped to

Desired, Adequate, or Inadequate performance, the integrals

of Eqs. 18,20 are solved numerically to ﬁnd the mean perfor-

mance level, and the probability that the pilot-vehicle system

will fall within each performance level. The results for the

Hover MTE are shown for ambient turbulence level in Fig.

4(a), and light turbulence level in Fig. 4(b). The mean perfor-

mance of the pilot-vehicle system falls within the Adequate

performance level for both ambient and light turbulence lev-

els. However, the performance is shown to degrade, proba-

bilistically speaking, with increasing turbulence level. In fact,

the probability to have Adequate performance for ambient tur-

bulence level is about 79%, whereas the probability to have

Adequate performance for light turbulence level is about 55%.

It is worth noting that the proposed methodology is applied to

a pilot-vehicle system where the pilot model closes the loop

around an unaugmented aircraft. However, the methodology

could readily be applied to the case where the pilot model

closes the loop around an augmented aircraft, provided that

the pilot model parameters are derived from the augmented

aircraft dynamics.

Frequency-Domain Speciﬁcations

The goal of this section is to compute the longitudinal axis

expected broken loop gain margin (GM) and phase mar-

gin (PM) (Ref. 10). This leads to the natural deﬁnition of

two cost functions, gGM(Keφ,Keθ,τ0) = GM(Keφ,Keθ,τ0)and

gPM(Keφ,Keθ,τ0) = PM(Keφ,Keθ,τ0). For this particular ex-

ample, the transformations Sdescribed above are the iden-

tify transformations, since the cost functions are deﬁned di-

rectly as functions of the uncertain parameters (rather than on

a transformed set of states and parameters). This means that

USis the identity operator. Once gGM and gPM are computed

over a discretized domain of (Keφ,Keθ,τ0), the expected gain

margin can be found as:

E[GM] =

ZΩ

fKeφ,Keθ,τ0(Keφ,Keθ,τ0)gGM(Keφ,Keθ,τ0)dKeφdKeθdτ0

(23)

where Ωis the support of fKeφ,Keθ,τ0. An analogous equation

can be created for the expected phase margin. As an alterna-

tive, suppose three different score functions are used to assess

(a)

(b)

Figure 4: Uncertain pilot-vehicle system Hover MTE

performance assessment for a) ambient and b) light

turbulence levels.

performance. Each of these is deﬁned as an indicator function

such that:

gDes(Keφ,Keθ,τ0) = (1 if GM(Keφ,Keθ,τ0)∈GMDes

0 else

(24)

where GM(Keφ,Keθ,τ0)∈GMDes is the set of evaluation vec-

tors that is deﬁned to give Desired performance. Analogous

score function deﬁnitions can be created for the Adequate and

7

Table 6: Broken loop gain and phase margins requirements

(Ref. 22).

Performance GM [dB] PM [deg]

Desired 6 45

Adequate 3 35

Figure 5: Uncertain pilot-vehicle system gain and phase

margins performance assessment.

Inadequate categories, as well as for phase margin Desired,

Adequate, and Inadequate performance. Then, the probabil-

ity that the system satisﬁes the Desired speciﬁcations can be

computed as:

E[gDes(Keφ,Keθ,τ0)] =

ZΩ

fKeφ,Keθ,τ0(Keφ,Keθ,τ0)gDes(Keφ,Keθ,τ0)dKeφdKeθdτ0

(25)

Although gain and phase margins speciﬁcations do not exist

for a pilot-vehicle system, the stability margins requirements

were taken from SAE AS94900 (Ref. 22). These stability

margins requirements are shown in Table 6. Figure 5shows

the realizations of the discretized points in the uncertainty do-

main in the pilot-vehicle system gain margin and phase mar-

gin domains, using the uncertainty distributions deﬁned in the

previous section. The pilot-vehicle system mean performance

is Adequate. The analysis suggests that the probability to have

Desired, Adequate, and Inadequate performances are, respec-

tively, 1%, 79%, and 20%.

PERCEIVED VEHICLE HANDLING

QUALITIES

This section illustrates the mapping of parametric uncertainty

in the pilot model to uncertainty in the perceived vehicle han-

Table 7: Pilot opinion rating vs mean square value of pilot

control activity (data extrapolated from Ref. 4)

Pilot Rating RMS(um/Ke)

2 0.0010

3 0.0262

4 0.1459

5 0.2901

6 0.5572

7 1.0703

8 2.1277

dling qualities (or pilot opinion ratings). Consider the frame-

work illustrated in the previous section. Assume that it is pos-

sible to relate quantitative metrics from the time history of the

augmented state to pilot opinion ratings. Then, a similar ap-

proach can be used to map uncertainty in the pilot model un-

certainty to variability in pilot ratings. Deﬁne a score function

such that:

gPRi(¯xeval) = (1 if ¯xeval ∈(¯xeval)PRi

0 else (26)

where ¯xeval ∈(¯xeval)PRiis the set of evaluation vectors that is

deﬁned to give ith pilot rating. Then, the probability that the

vehicle will receive a particular pilot rating can be computed

as:

E[gPRi(~xeval,~pu)] =

ZΩ

fKeφ,Keθ,τ0(Keφ,Keθ,τ0)USgPRi(¯xeval)d~xd~pu(27)

For instance, the pilot workload, which is deﬁned as the root

mean square (RMS) of um/Ke, can be related through the

Cooper-Harper rating scale (Ref. 1) to a pilot rating for a pre-

cision hovering task, according to Refs. 4,18. It follows that,

for each hover simulation described in the previous section, a

particular mean square value of pilot control activity um/Ke

can be computed. Such a value corresponds to pilot opinion

ratings through the relation shown in Fig. 6and Table 7. Once

each data point in the uncertainty domain is mapped to the pi-

lot rating domain, the integrals of Eqs. 26,27 can be solved

numerically to ﬁnd the mean pilot rating for the vehicle and

the probability that the pilot rating will fall within each rating

level.

The results for the precision hovering task are shown for ambi-

ent turbulence level in Fig. 7(a), and for light turbulence level

in Fig. 7(b). The mean pilot rating is shown to degrade from

PR=3 to PR=4 with increasing turbulence level (i.e. ambient

to light). This result is in accord with the literature (Ref. 19).

The approach used in this paper may be used in future investi-

gations to tune pilot model parametric uncertainty such that it

matches pilot rating uncertainty obtained through (physically)

piloted simulations and/or ﬂight testing.

8

Figure 6: Pilot opinion rating vs mean square value of pilot

control activity (re-created from Ref. 4).

CONCLUSIONS

A mathematical methodology for probabilistic handling qual-

ities evaluation using the Koopman operator was developed.

Results were obtained using a quasi-nonlinear rotorcraft sim-

ulation model and a multi-axis structural pilot model. The

Koopman operator was applied to the propagation of the un-

certain parameters in the pilot model to speciﬁc pilot-vehicle

system performance speciﬁcations and to perceived vehicle

handling qualities, including the Hover MTE, broken loop

gain and phase margins, and pilot opinion ratings of work-

load. For each type of speciﬁcation, the relevant state trans-

formation and cost function deﬁnition(s) were identiﬁed and

numerical examples were presented. Based on this work, the

following conclusions can be reached:

1. The Koopman operator provides a convenient framework

for the propagation of pilot model uncertainty to pilot-

vehicle system performance speciﬁcations and to per-

ceived vehicle handling qualities. Speciﬁcally, the Koop-

man operator enables evaluation functions that extract

HQ ratings to be pulled back to the uncertainty domain,

wherein the expected value integral can be easily com-

puted.

2. According to probabilistic assessments of the pilot-

vehicle system performance, the mean performance for

a Hover MTE was shown to be Adequate for both ambi-

ent and light turbulence levels. However, the mean per-

formance is shown to degrade with increasing turbulence

level.

3. According to a probabilistic assessment of the broken

loop gain and phase margins speciﬁcations, the mean

performance of the pilot-vehicle system was shown to

be Adequate. The analysis suggested that the probability

to have Desired, Adequate, and Inadequate performance

was, respectively, 1%, 79%, and 20%.

(a)

(b)

Figure 7: Pilot model uncertainty mapped to mean square

value of control activity and pilot opinion ratings for a)

ambient and b) light turbulence levels.

4. Uncertainty in the pilot model parameters resulted in sig-

niﬁcant variability in pilot-vehicle system performance

both for frequency- and time-domain speciﬁcations. This

result indicates the need for probabilistic assessments in

which it may be stated that the pilot-vehicle system con-

forms to speciﬁc guidelines with a certain probability,

given the uncertainty in the pilot-vehicle closed-loop dy-

namics.

5. Uncertainty in the pilot model was propagated to the pi-

lot control activity, a measure of pilot workload, and re-

lated to pilot opinion ratings for a precision hover task.

The mean pilot rating was shown to degrade from PR=3

to PR=4 with increasing turbulence level (ambient to

9

light), which is in accord with the literature.

6. Although the proposed methodology is applied to a pilot-

vehicle system where the pilot model closes the loop

around an unaugmented aircraft, the same methodology

could be readily extended to the case where the pilot

model closes the loop around an augmented aircraft, pro-

vided that the pilot model parameters are derived from

the augmented aircraft dynamics.

The approach used in this paper may be used in future investi-

gations to tune pilot model parametric uncertainty such that it

matches the pilot rating uncertainty obtained through (physi-

cally) piloted simulations and/or ﬂight testing. The methodol-

ogy will be extended to different MTEs and a larger number of

uncertain parameters in the pilot model. Comparative studies

will be conducted to assess the computational advantages of

the proposed technique with respect to Monte Carlo analysis.

REFERENCES

1. Cooper, G. R., and Harper, R. P., The use of pilot rating

in the evaluation of aircraft handling qualities, NASA TN

D-5153, National Aeronautics and Space Administration,

Washington, DC, 1969.

2. Anon, Aeronautical Design Standard Performance Spec-

iﬁcation, Handling Qualities Requirements for Military

Rotorcraft, ADS-33E-PRF, USAAMCOM, 2000.

3. Doman, D., Optimal control pilot modeling for resolving

Cooper-Harper rating discrepancies, 24th Atmospheric

Flight Mechanics Conference, Portland, OR, August 9-

11, 1999.

4. Hess, R. A., Theory for aircraft handling qualities based

upon a structural pilot model, Journal of Guidance, Con-

trol, and Dynamics, Vol 12, No. 6, pp. 792-797, 1989.

5. Hess, R. A., Structural Model of the Adaptive Human Pi-

lot, Journal of Guidance and Control, Vol. 3, No. 5, pp.

416-423, 1980.

6. Wilson, D., and Riley, D., Cooper-Harper pilot rating

variability, 16th Atmospheric Flight Mechanics Confer-

ence, Boston, MA, August 14-16, 1989.

7. Riley, D., and Wilson D., More on Cooper-Harper pi-

lot rating variability, 17th Atmospheric Flight Mechanics

Conference, Portland, OR, August 20-22, 1990.

8. Perhinschi, M., and Prasad., J. V. R., Analytical investi-

gation of helicopter handling qualities criterion using a

structural pilot model, 20th Atmospheric Flight Mechan-

ics Conference, Baltimore, MD, August 7-10, 1995.

9. Mavris, D., DeLaurentis, D., and Soban, D., Probabilistic

Assessment of Handling Qualities Characteristics in Pre-

liminary Aircraft Design, 36th Aerospace Sciences Meet-

ing and Exhibit, Reno, NV, January 12-15, 1998.

10. Tischler, M. B., Berger, T., Ivler, C. M., Mansur, M.

H., Cheung, K. K., and Soong, J. Y., Practical Methods

for Aircraft and Rotorcraft Flight Control Design: An

Optimization-Based Approach, AIAA Education Series,

VA, April 2017.

11. Saetti, U., J., Horn, J. F., Lakhmani, S., Lagoa, C.,

and Berger, T., Design of Dynamic Inversion and Explicit

Model Following Control Laws for Quadrotor Inner and

Outer Loops, Vertical Flight Society 74th Annual Forum

Proceedings, Phoenix, AZ, May 14-17, 2018.

12. Cooper, J., DeVore, M. D., Reed, A., and Klyde, D.,

Handling Quality Analysis and Control Design Using

Polynomial Chaos Model-based Uncertainty Methods,

Vertical Flight Society 75th Annual Forum Proceedings,

Philadelphia, PA, May 13-16, 2019.

13. Halder, A., and Bhattacharya, R., Dispersion Analy-

sis in Hypersonic Flight During Planetary Entry Using

Stochastic Liouville Equation, Journal of Guidance, Con-

trol, and Dynamics, Vol. 34, No. 2, pp. 459-474, 2011.

14. Meyers, J. J., Leonard, M., Rogers J. D., Gerlach, A. R.,

Koopman Operator Approach to Optimal Control Selec-

tion Under Uncertainty, 2019 American Control Confer-

ence, Philadelphia, PA, July 10-12, 2019.

15. Leonard, L., Rogers, J., and Gerlach, A., Koopman Op-

erator Approach to Airdrop Mission Planning Under Un-

certainty, Journal of Guidance, Control, and Dynamics

(2019): 1-17.

16. Fletcher, J. W., A model structure for identiﬁcation of

linear models of the UH-60 helicopter in hover and for-

ward ﬂight, NASA Technical Memorandum 110362, US-

AATCOM Technical Report 95-A-008, Moffett Field,

CA, 1995.

17. Tobias, E. L. and Tischler, M. B., A Model Stitching Ar-

chitecture for Continuous Full Flight-Envelope Simula-

tion of Fixed-Wing Aircraft and Rotorcraft from Discrete-

Point Linear Models, U.S. Army AMRDEC Special Re-

port RDMR-AF-16-01, April 2016.

18. Miller, D. P., and Vinje E. W., FIXED-BASE FLIGHT

SIMULATOR STUDIES OF VTOL AIRCRAFT HAN-

DLING QUALITIES IN HOVERING AND LOW-SPEED

FLIGHT, UACRL-F910482-12, United Aircraft Research

Laboratories, East Hartford, CT, January 1968.

19. Lusardi, J. A., Control Equivalent Turbulence Input

Model for the UH-60 Helicopter, Doctoral Dissertation,

University of California, Davis, CA, 2004.

20. Blanken, C. L., Lusardi, J. A., Ivler, C. M., Tis-

chler, M. B., H¨

oﬁnger, M. T., Decker, W. A., Berger,

T., and Tucker, G. E., An Investigation of Rotorcraft

Stability-Phase Magin Requirements in Hover, Ameri-

can Helicopter Society 65th Annual Forum Proceedings,

Grapevine, TX, May 27-29, 2009.

10

21. Dutta, P., Halder, A., and Bhattacharya, R., Uncertainty

Quantiﬁcation for Stochastic Nonlinear Systems using

Perron-Frobenius Operator and Karhunen-Lo`

eve Expan-

sion, 2012 IEEE International Conference on Control Ap-

plications, Dubrovnik, Croatia, October 3-5, 2012.

22. Anon, Aerospace - Flight Control Systems - Design, In-

stallation and Test of Piloted Military Aircraft, General

Speciﬁcations For, AS94900, SAE International, Warren-

dale, PA, 2007.

11