Content uploaded by Umberto Saetti

Author content

All content in this area was uploaded by Umberto Saetti on Oct 07, 2020

Content may be subject to copyright.

Explicit Uncertainty Quantiﬁcation for Probabilistic Handling Qualities

Assessment

Umberto Saetti

Postdoctoral Fellow

School of Aerospace Engineering

Georgia Institute of Technology

Atlanta, GA 30313

Jonathan Rogers

Associate Professor

School of Aerospace Engineering

Georgia Institute of Technology

Atlanta, GA 30313

ABSTRACT

Modern ﬂight vehicle development typically involves signiﬁcant uncertainty in aerodynamic loads and vehicle control

responses. In many cases, it may be necessary to consider this uncertainty in preliminary evaluations of stability,

handling qualities (HQ), and performance. This paper introduces a novel methodology for computing these quantities

in a probabilistic framework using the Koopman operator. The algorithm discretizes the uncertainty space and uses

relevant transformations or simulations to map each point to the handling qualities evaluation domain, resulting in

a stability, HQ, and/or performance assessment for each discretized point. The expected value of the stability, HQ,

and/or performance requirement is then computed through an expected value integral with the initial joint probability

density function, yielding a probabilistic assessment with respect to relevant speciﬁcations. The methodology is

used to evaluate handling qualities of a quadrotor Unmanned Aerial System (UAS), for which a multi-loop Dynamic

Inversion (DI) ﬂight control law is developed. The UAS example is evaluated with respect to quantitative handling

qualities speciﬁcations based on scaled Mission Task Elements (MTEs). The proposed explicit UQ approach is shown

to provide a convenient framework for the propagation of parametric uncertainty to stability, HQ, and performance

speciﬁcations with unique advantages over Monte Carlo techniques.

INTRODUCTION

A key requirement in the design of a new aircraft, or modiﬁ-

cation to an existing airframe, is that the aircraft meet speciﬁc

handling qualities, performance, and stability requirements.

In a typical design process, evaluation of these criteria in-

volves the use of linear models derived either through ﬂight

testing or linearization of a high ﬁdelity nonlinear model.

Such linear models are naturally subject to uncertainty, for

instance due to the limited amount of ﬂight data available or

modeling errors inherent in a nonlinear modeling process. Be-

cause the linear model parameters are subject to uncertainty,

any stability, handling qualities, or performance assessments

based on the linear models are subject to uncertainty as well.

Particularly in cases where the linear model uncertainty is

signiﬁcant, it may not be appropriate to make deterministic

judgments about a vehicle’s conformance to handling qual-

ities guidelines. Instead, probabilistic assessments must be

made in which it may be stated that the vehicle conforms to

speciﬁc guidelines with a certain probability, given the un-

certainty in the vehicle closed-loop dynamics. This type of

probabilistic handling qualities assessment may be more ap-

propriate particularly early in the design stages of a vehicle

or airframe modiﬁcation, when aerodynamic parameters are

usually subject to large uncertainty. In light of this, there is a

Presented at the VFS International 76th Annual Forum &

Technology Display, Virginia Beach, VA, October 6–8, 2020. Copy-

right c

2020 by the Vertical Flight Society. All rights reserved.

need for new methodologies that can map the uncertainty in

vehicle model parameters to the resulting uncertainty in han-

dling qualities, stability, and performance assessments.

Despite the need for rigorous techniques in this domain, there

is surprisingly little prior research in the development of prob-

abilistic handling qualities evaluation techniques. Mavris et

al. (Ref. 1) developed a Monte Carlo-based approach to the

probabilistic assessment of ﬁxed-wing handling qualities dur-

ing preliminary design. In the rotorcraft domain, a similar

Monte Carlo-based technique exists through the Control De-

signer’s Uniﬁed Interface (CONDUIT) (Ref. 2) robust han-

dling qualities evaluation tool. While easy to understand and

implement, there are several issues with Monte Carlo-based

approaches. First, the technique suffers from well-known

scalability issues such that the number of samples needed to

generate sufﬁcient accuracy may become prohibitively high

as the dimensionality of the uncertainty increases. This is a

signiﬁcant potential problem in rotorcraft design where nu-

merous aerodynamic parameters are typically subject to un-

certainty. Second, even after Monte Carlo samples have been

propagated, the underlying density must be reconstructed

from these samples using, for instance, kernel-based methods

(which are themselves subject to approximation error). An al-

ternative approach recently explored by Saetti et al. (Refs. 3,4)

used an unscented Kalman ﬁlter to quantify the probability

density associated with handling qualities metrics – however,

the UKF approach is still limited by its scalability to higher

dimensions. Another approach that was recently proposed for

1

applications to rotorcraft was polynomial chaos (Ref. 5), al-

though this technique also suffers from poor scaling in higher

dimensions. It is well known that explicit uncertainty quan-

tiﬁcation approaches, particularly the Frobenius-Perron (FP)

Operator (Ref. 6), may be used to map a probability density

explicitly through a nonlinear transformation with improved

scalability characteristics. Recently, Meyers et al. (Ref. 7) ap-

plied the FP operator and its adjoint, the Koopman operator, to

various optimal decision-making problems under uncertainty,

demonstrating the scalability beneﬁts of these explicit tech-

niques compared to Monte Carlo.

This paper presents a novel methodology for computing sta-

bility, handling qualities, and performance assessments in a

probabilistic framework using the Koopman operator. The al-

gorithm begins with a discretization of the uncertainty space

comprised of the uncertain linear or nonlinear model parame-

ters. Each discretized point is mapped through relevant trans-

formations to the handling qualities evaluation domain, result-

ing in a stability, HQ, and/or performance assessment for each

discretized point (similar to Monte Carlo, except usually with

signiﬁcantly less points). Using this mapping, the handling

qualities evaluation is pulled back to the initial uncertainty do-

main using the Koopman operator. The expected value of the

stability, HQ, and/or performance requirement is then com-

puted through an expected value integral with the initial joint

probability density function (PDF), yielding a probabilistic

assessment with respect to relevant speciﬁcations.

The paper begins with a detailed discussion of the mathemat-

ical background and methodology. Following a description

of the simulation model of the quadrotor Unmanned Aerial

System (UAS) and its ﬂight control law, an example imple-

mentation of the methodology is presented. The UAS han-

dling qualities are evaluated probabilistically with respect to

quantitative speciﬁcations based on scaled Mission Task El-

ements (Ref. 8) that were recently proposed in the rotorcraft

community in Ref. 9.

METHODOLOGY

Deﬁne an aircraft/rotorcraft dynamic model with state vector

xand parameterized by p∈Rpaccording to,

˙

x=f(x,p,u)(1)

where uis the vector of controls. A new augmented state

is deﬁned by combining the state vector with the uncertain

model parameters, and any other components needed to eval-

uate HQ ratings. Deﬁne the set of uncertain model parame-

ters as pu∈Rdwhere pu⊂pand dis the dimension of the

uncertain parameter space. Likewise, pk∈Rp−dis the vector

of known parameters. The augmented state is then given as

¯

xT=xTpT

u. Suppose that a map exists that such that, for

each unique initial (augmented) state vector ¯

x0, a state vector

at a ﬁnal time ¯

xfis generated that may be used to evaluate

handling qualities:

¯

xf=S(¯

x0)(2)

In Eq. 2, the transformation Smay represent various types

of algebraic or dynamic transformations. For instance, in the

evaluation of ADS-33E performance standards, Mission Task

Elements (MTEs) are used to categorize ﬂight performance

into Desired, Adequate, or Inadequate performance (or equiv-

alently, into Level 1, Level 2, or Level 3 performance). In

this case, evaluation of this speciﬁcation requires executing

the MTE from the prescribed initial condition using a se-

lected value of p. Thus, Smay involve propagation of a ﬂight

dynamic model from the prescribed initial condition for the

MTE to the prescribed ﬁnal condition, with the ﬁnal state

being ¯

xf. Other handling qualities, performance, or stabil-

ity metrics may involve other types of dynamic or algebraic

transformations Sto obtain a vector ¯

xf. In subsequent discus-

sion, this ﬁnal state vector will be referred to as ¯

xeval =¯

xffor

clarity, to represent the state vector on which the HQ ratings

can be evaluated.

Given this mapping, a “score” function gis deﬁned that as-

signs a numerical score based on ¯

xeval representing the han-

dling qualities or performance speciﬁcation associated with

the forward mapping of the augmented state ¯

x0. This score

function value may be “pulled back” to the initial condition

domain of the augmented state through use of the Koopman

operator deﬁned as,

USg=USg(¯

xeval ) = g(S(¯

x0)) (3)

The only restriction on gis that it must be ﬁnite, i.e., ||g∞|| <

∞. The score or cost function value is associated with a

certain handling qualities criterion – for instance, a value of

g(¯

xeval ) = 1 may be assigned for ¯

xeval in a particular region

associated with Desired handling qualities, g(¯

xeval ) = 2 for

regions associated with Adequate handling qualities, and so

on. Note that the Koopman operator USin Eq. 3is deﬁned

speciﬁc to a particular transformation S.

This overall process is illustrated in Fig. 1. In Fig. 1(a),

each discretized point in the uncertainty domain is mapped to

a speciﬁc numerical value in the HQ or performance domain

using the forward mapping S. In Fig. 1(b), the score/cost

function associated with each forward-mapped point is pulled

back to the original uncertainty domain, yielding USg. With

this pulled-back score function, the expected value of the han-

dling qualities assessment function gcan be computed given

the joint probability density function on the uncertain param-

eters puaccording to:

E[g(x,p)] = ZΩ

fpu(pu)USg(¯

xeval )dpu(4)

where Ωis the support of the joint probability density of the

uncertain parameters, fpu(pu). This expected value represents

the expected handling qualities evaluation score for this par-

ticular metric signiﬁed by the cost or score function g. For

instance, in the example shown in Fig. 1, calculation of Eq.

4would provide a value between 0 and 1, representing the

probability of the vehicle achieving the Desired level of per-

formance for the particular handling qualities or performance

metric. Note that by deﬁning several binary score functions

for each handling qualities ranking, probability values can be

generated that specify the probability that the vehicle will con-

form to each level. This represents a generalization of the

chance constraint techniques detailed in Refs. 7,10.

2

(a)

(b)

Figure 1: (a) Forward Mapping of Uncertainty Domain to

HQ/Performance Domain (b) Pull-Back of HQ Evaluation

Score to Uncertainty Domain Using Koopman Operator.

SIMULATION MODEL

The probabilistic HQ evaluation methodology is illustrated

through application to a quadrotor UAS. A 12-state model

is used to simulate the ﬂight dynamics of a small-scale UAS

in hover. The stability and control derivatives from a ﬂight-

identiﬁed linear model are combined with nonlinear equa-

tions of motion to produce a quasi-nonlinear simulation model

(Ref. 11). Consider the rigid-body dynamics of the ﬂight-

identiﬁed linear model:

˙

x=Ax +Bu (5)

where:

xare the states relative to the rigid-body dynamics,

uis the control vector,

Acontains the rigid-body stability derivatives, and

Bcontains the rigid-body control derivatives.

The state vector is:

xT=uvwpqr(6)

where u,v,ware the body velocities, and p,q,rare the body

angular rates. The control vector is:

u=

δlat(t−τφ)

δlon(t−τθ)

δcol(t−τz)

δped(t−τψ)

(7)

where δlat,δlon ,δcol,δped are respectively the lateral, lon-

gitudinal, vertical, and directional control inputs (before the

mixer), and τφ,τθ,τz,τψare the time delays associated with

Table 1: Stability derivatives, control derivatives, and time

delays of the ﬂight-identiﬁed quadrotor model (Refs. 3,4).

Parameter Value CR Bound [%]

Xu-0.2568 [1/s] 5.302

Yv-0.3022 [1/s] 6.107

Zw-0.1734 [1/s] 39.72

Lv-0.8287 [rad/(ft s)] 5.943

Lp0 [1/s] -

Mu1.1257 [rad/(ft s)] 5.618

Mq0 [1/s] -

Nr-0.5617 [1/s] 25.19

Zδcol -49.065 [ft/(s2%)] 2.647

Lδlat 33.514 [rad/(s2%)] 3.297

Mδlon 27.919 [rad/(s2%)] 4.155

Nδped 6.0308 [rad/(s2%)] 3.877

τφ0.0565 [s] 4.170

τθ0.0355 [s] 10.73

τz0.0389 [s] 30.03

τψ0.0401 [s] 7.023

the lateral, longitudinal, vertical, and directional dynamics,

respectively. The system matrix containing the rigid-body sta-

bility derivatives is:

A=

Xu0 0 0 0 0

0Yv0 0 0 0

0 0 Zw0 0 0

0Lv0Lp0 0

Mu0 0 0 Mq0

0 0 0 0 0 Nr

(8)

The control matrix containing the rigid-body control deriva-

tives is:

B=

0 0 0 0

0 0 0 0

0 0 Zδcol 0

Lδlat 0 0 0

0Mδlon 0 0

0 0 0 Nδped

(9)

The stability derivatives, control derivatives, and time delays

are taken from the ﬂight-identiﬁed model of Refs. 3,4and are

reported in Table 1.

In order to calculate the aerodynamic perturbation forces and

moments, it is necessary to obtain the state and control pertur-

bations:

∆x=X−X0(10a)

∆u=U−U0(10b)

where:

Xare the nonlinear simulation rigid-body states,

X0are the trim rigid-body states in hover,

Uis the nonlinear simulation control vector, and

U0is the trim control vector in hover.

3

Table 2: Mass and inertia properties of the ﬂight-identiﬁed

quadrotor model (Ref. 3,4).

Parameter Value

m1.1023 [lb]

Ixx 0.0547 [lb-ft2]

Iyy 0.0585 [lb-ft2]

Izz 0.0756 [lb-ft2]

Ixz 0 [lb-ft2]

It is worth noting that the rigid-body perturbation vector does

not include Euler angles or the position since the aerodynamic

forces and moments do not depend explicitly on the Euler an-

gles or the position (Ref. 11). The aerodynamics perturbation

forces and moments are:

∆Faero

∆Maero=M(A∆x+B∆u)(11)

where:

M=

m

m

m

Ixx −Ixz

Iyy

−Ixz Izz

(12)

is the mass matrix containing the mass mand inertia terms

Ixx,Iyy,Izz,Ixz of the quadrotor. The mass and inertia proper-

ties of the quadrotor are reported in Table 2.

The aerodynamic trim forces and moments, based on the trim

Euler angles Φ0,Θ0in hover, are:

Faero0=

gsinΘ0

−gcosΘ0sin Φ0

−gcosΘ0cos Φ0

m(13)

It is worth noting that since the quadrotor is symmetric with

respect to the lateral and longitudinal axes, the trim roll and

pitch angles are zero. The nonlinear simulation gravitational

forces, which are nonlinear with respect to the Euler angles,

are:

Fgrav =

−gsinΘ

gcosΘsin Φ

gcosΘcos Φ

m(14)

The aerodynamics perturbation forces and moments are

summed with the trim and gravitational forces and moments

to obtain the total aerodynamic forces and moments:

F

M=∆Faero +Faero0+Fgrav

∆Maero (15)

These forces and moments are incorporated in the nonlinear

equations of motion in the body-ﬁxed frame to provide the

fuselage 6-DOF dynamics:

˙

U=RV −QW +Fx

m(16a)

˙

V=PW −RU +Fy

m(16b)

˙

W=QU −PV +Fz

m(16c)

˙

P=Izx(Ixx −Iyy +Izz)PQ −(I2

zz −IyyIzz +I2

xz)QR +IzzL+IxzN

IxxIzz −I2

xz (16d)

˙

Q=(Izz −Ixx)PR −Ixz(P2−R2) + M

Iyy

(16e)

˙

R=(I2

xx −IxxIyy +I2

xz)PQ −Izx(Ixx −Iyy +Izz)QR +Ixz L+IxxN

IxxIzz −I2

xz (16f)

˙

Φ=P+QsinΦtan Θ+Rcos Φtan Θ(16g)

˙

Θ=QcosΦ−Rsin Φ(16h)

˙

Ψ=QsinΦsec Θ+Rcos Φsec Θ(16i)

˙

X=UcosΘcos Ψ+V(sin Φsin Θcos Ψ−cos Φsin Ψ)

+W(cosΦsin Θcos Ψ+sin Φsin Ψ)(16j)

˙

Y=UcosΘsin Ψ+V(sin Φsin Θsin Ψ+cos Φcos Ψ)

+W(cosΦsin Θsin Ψ−sin Φcos Ψ)(16k)

˙

Z=−UsinΘ+Vsin Φcos Θ+WcosΦcos Θ(16l)

FLIGHT CONTROL DESIGN

A multi-loop Dynamic Inversion (DI) control law largely

based on Refs. 3,4is designed to enable fully autonomous

ﬂight of the UAS. The schematic of the closed-loop dynam-

ics of the quadrotor UAS is shown in Fig. 2. The outer loop

takes trajectory (position) commands with optional heading

constraints and calculates the pitch and roll attitudes for the

inner loop to track. The natural frequencies and damping ra-

tions of the outer loop command models are given in Table 3.

The inner loop achieves stability, disturbance rejection, an At-

titude Command / Attitude Hold (ACAH) response around the

roll, pitch, and yaw axes, and a position hold response around

the heave axis. The outer and inner loop block diagrams are

shown in Figs. 3and 4, respectively. The outer and inner loop

command model parameters are given in Tables 3and 4, re-

spectively. The PID gains are chosen such that the frequencies

of the error dynamics are of the same order as the command

models (Ref. 12). The numerical values of the PID gains are

shown in Table 5for the outer loop and and in Table ?? for

the inner loop.

4

Figure 2: Schematic of the closed-loop dynamics of the quadrotor UAS.

Figure 3: DI outer loop.

5

Figure 4: DI inner loop.

Table 3: Outer loop command models natural frequency and

damping ratio.

Command ωn[rad/s]ζ

Longitudinal Position 1 0.7

Lateral Position 1 0.7

Table 4: Inner loop command models natural frequency and

damping ratio.

Command ωn[rad/s]ζ

Roll Attitude 10 0.7

Pitch Attitude 10 0.7

Yaw Attitude 2 0.7

Vertical Position 1 0.7

RESULTS

Frequency-Domain Speciﬁcations

This section illustrates the mapping of parametric uncertainty

in the quadrotor UAS model to quantitative handling qualities

Table 5: Outer loop PID gains.

KDKPKI

xcmd 1.61 1.28 0.2

ycmd 1.61 1.28 0.2

Table 6: Inner loop PID gains.

KDKPKI

φcmd 16 128 200

θcmd 16 128 200

ψcmd 4.4 5.6 1.6

zcmd 1.61 1.28 0.2

speciﬁcations deﬁned in the frequency domain. Restricting

the analysis to the lateral axis for simplicity, let the uncertain

parameters be the ﬂight-identiﬁed stability and control deriva-

tives and time delay such that:

pT

u=YvLvLδlat τφ(17)

Each parameter is assumed to be a random variable indepen-

dent of the others. The probability density on each parame-

ter is described by a Gaussian distribution with with mean µi

and standard deviation σi. The mean of each parameter cor-

responds to the nominal identiﬁed value reported in Table 1,

such that µi=Θi, where Θis the vector of identiﬁed parame-

ters. The standard deviation is taken as the squared root of the

diagonal elements of the covariance matrix Pfor the vector of

identiﬁed parameters:

σi=pPii (18)

The covariance matrix is calculated from the identiﬁcation re-

sults as:

P=H−1(19)

6

where His the parameter identiﬁcation Hessian matrix

(Ref. 13). The Hessian matrix is composed by the second-

order partial derivatives of the identiﬁcation ﬁt cost Jwith

respect to the identiﬁed parameters vector:

H=∂2J

∂Θ∂ΘT(20)

The Cramer-Rao (CR) bound of the ith identiﬁed parameter

of the identiﬁed parameters vector is related to the Hessian

matrix in the following way:

CRi=q(H−1)ii (21)

It should be noted that the CR bounds are best expressed as a

percentage of the nominal identiﬁcation values:

CRi=

CRi

Θi×100% (22)

and are reported in this form in Table 1.

As an example of quantitative handling qualities speciﬁca-

tions deﬁned in the frequency domain, consider the Distur-

bance Rejection Bandwidth (DRB) and Disturbance Rejection

Peak (DRP) deﬁned in Ref. 14. For a given realization of the

uncertain parameters pu, the DRB is the frequency at which

the sensitivity function of the hold variable (the roll attitude

φfor the inner loop, and the lateral position yfor the outer

loop) crosses −3 dB from below, and DRP is the peak mag-

nitude of the sensitivity function. This leads to the natural

deﬁnition of two cost functions, gDRB(pu) = DRB(pu)and

gDRP(pu) = DRP(pu). For this particular example, the trans-

formations Sdescribed above are the identity transformations,

since the cost functions are deﬁned directly as functions of the

uncertain parameters (rather than on a transformed augmented

state). This means that USis the identity operator. Once gDRB

and gDRP are computed over a discretized domain of the un-

certain parameters pu, the expected DRB can be found as:

E[DRB] = ZΩ

fpu(pu)gDRB(pu)dpu(23)

where Ωis the support of fpu. Note that since the uncertain pa-

rameters are assumed to be independent, the joint uncertainty

distribution fpuis formed as the product of the Gaussian PDFs

governing each element of pu. An analogous equation can be

created for the expected DRP. As an alternative, suppose three

different score functions are used to assess performance. Each

of these is deﬁned as an indicator function such that:

gDes(pu) = (1 if DRB(pu)∈DRBDes

0 else (24)

where DRBDes is the set of evaluation vectors that is deﬁned

to give Desired performance. Analogous score function deﬁ-

nitions can be created for the Adequate and Inadequate cate-

gories, as well as for DRP Desired, Adequate, and Inadequate

performance. Then, the probability that the system satisﬁes

the Desired speciﬁcations can be computed as:

E[gDes(pu)] = ZΩ

fpu(pu)gDes(pu)dpu(25)

Table 7: DRB and DRP scaled Level 1 and Level 2

requirements.

Performance DRB [rad/s] DRP [dB]

Desired (φ)6.4 5.0

Adequate (φ)3.5 8.0

Desired (y) 1.2 3.0

Adequate (y) 0.6 5.0

Although DRB and DRP speciﬁcations are not well deﬁned

for small-scale UAS, the DRB and DRP requirements are

taken from Ref. 14. In this work, the DRB is scaled accord-

ing to the approach used in Refs. 9,15. The scaling is based

on the scale factor N, which is deﬁned as the ratio between

the motor-to-motor length of the quadrotor (1.08 ft) and the

main rotor diameter of the UH-60 (53.8 ft). This leads to a

scale factor of N=0.0201. The DRB requirements are scaled

by √Nand are shown in Table 7for both the inner and outer

loop.

In the implementation used here, each uncertain parameter di-

mension is discretized into np=10 evenly-spaced grid points

spanning µ±4σ, where µand σare the mean and standard

deviation of each parameter. Since the number of uncertain

parameters is 4, this leads to a total of 104data points. The

number of grid points is chosen to ensure that the numeri-

cal solution of the integrals deﬁned on the uncertainty domain

(Eqs. 23,25) is sufﬁciently accurate. Figure 5shows the re-

alizations of the discretized points in the uncertainty domain

in the DRB and DRP domain, using the uncertainty distribu-

tions deﬁned in the previous section. In this ﬁgure, the red

dots are the mapped values of a given realization of the uncer-

tain parameters to the DRB and DRP domains and the black

cross is the mean of the mapped values. Figure 5(a) shows

that the inner loop mean performance is Adequate with 100%

probability, whereas Fig. 5(b) shows that the outer loop mean

performance is Inadequate with 100% probability.

Time-Domain Speciﬁcations

This section illustrates the mapping of parametric uncertainty

in the quadrotor UAS model to quantitative handling qualities

speciﬁcations deﬁned in the time domain. Consider a more

complex example in which a handling qualities metric is de-

ﬁned over a transformed state and parameter vector. ADS-33E

(Ref. 8) deﬁnes the Lateral Reposition MTE in which a rotor-

craft, starting from hover, must initiate a lateral acceleration

to about 35 knots followed by a deceleration to laterally repo-

sition the rotorcraft in a stabilized hover 400 ft down range

within a speciﬁed time. Although this MTE is deﬁned only

for full-scale rotorcraft, it has recently been shown that MTE

trajectories can be scaled for use in small-scale UAS HQ eval-

uation (Ref. 9). The trajectories are scaled by the square of the

ratio between the maximum velocity of the UH-60 (160 kts)

and the estimated maximum velocity of the quadrotor UAS

7

3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8

DRB [rad/s]

2

3

4

5

6

7

8

9

DRP [dB]

LEVEL 1 = 0%

LEVEL 2 = 100%

LEVEL 3 = 0%

mean

mapping

(a)

0.4 0.6 0.8 1 1.2 1.4 1.6

DRB [rad/s]

0

1

2

3

4

5

6

DRP [dB]

LEVEL 1 = 0%

LEVEL 2 = 0%

LEVEL 3 = 100%

mean

mapping

(b)

Figure 5: Probabilistic assessment of the disturbance

rejection bandwidth and peak for the uncertain quadrotor

system: (a) Inner loop, (b) Outer loop.

(35 ft/s):

Lscale =(VUAS)max

(VUH-60)max 2

(26)

This deﬁnition yields a scale factor of 0.0479 for the vehicle

considered here. It follows that the quadrotor has to reposition

itself by approximately 19 ft from its starting location. The

objective function proposed in Ref. 9to score the mission task

element is the trajectory tracking and aggressiveness (TTA)

score and is given by:

ΦTTA(L) = 200

1+eL(27)

where:

L=wα

α−αG

αB−αG

+wε

ε−εG

εB−εG

(28)

Table 8: TTA performance levels for gentle and aggressive

lateral reposition maneuvers from Ref. 9.

Desired Adequate

Gentle ΦTTA ≥75 75 >ΦTTA ≥70

Aggressive ΦTTA ≥82 82 >ΦTTA ≥77

Table 9: TTA scoring parameters from Ref. 9.

Parameter DRB [rad/s]

wα0.5

wε0.5

αG3

αB0

εG0

εB0.35

wvel 0.7

wpos 0.3

The TTA score has a minimum value of 0 and a maximum

value of 100. Higher values represent better tracking perfor-

mance and more aggressiveness. Table 8shows the Desired

and Adequate TTA scores for the lateral reposition MTE. The

parameters wα,wεare relative weights, whereas αG,αB,εG,

εBare conditioning parameters indicating “good” and “bad”

performance. The aggressiveness term αis deﬁned as:

α=(Vcmd)max

(Vcmd)nom

(29)

where:

(Vcmd)max =(VUAS)max

(VUH-60)max

(30)

is the maximum speed of the scaled MTE. The tracking error

term εis:

ε=wvel

RMSE(vel)

(Vcmd)max

+wpos

RMSE(pos)

Lpath

(31)

where RMSE is the root mean square of the error between the

commanded and measured velocity or position, and Lpath is

the length of the scaled trajectory (i.e. 19 ft). The parameters

used to calculate the TTA score are reported in Table 9. The

scaled lateral repositioning maneuver is shown in Fig. 6for

the vehicle and controller described above. The resulting ag-

gressiveness score, as determined by the outer loop command

ﬁlter for the lateral dynamics, is α=0.6754. The TTA score

is 77.2.

Suppose the quasi non-linear quadrotor simulation model with

state xand uncertain parameters pT

u=YvLvLδlat is used to

evaluate performance against the assigned MTE criteria (i.e.

the TTA score). The MTE can be simulated from an initial

condition x0and speciﬁc set of parameters to obtain a time

history of xthroughout the maneuver. Key values from this

time history needed to evaluate the MTE metrics, such as ve-

locity and position tracking errors, may be extracted from this

8

012345678910

0

5

10

Vy [ft/s]

012345678910

0

10

20

y [ft]

Command

Response

012345678910

0

10

20

30

[deg]

012345678910

Time [s]

-1

0

1

z [ft]

Figure 6: Simulated scaled lateral reposition MTE with

aggressiveness α=0.6754.

time history to produce an evaluation state ¯

xeval. The map-

ping S, which maps ¯

x0to ¯

xeval is thus the dynamic simula-

tion model of the UAS. Given ¯

xeval, a cost or score function

g(¯

xeval)may be deﬁned based on the TTA score which cat-

egorizes the MTE performance into Desired, Adequate, and

Inadequate based on the speciﬁc thresholds listed in Table 8.

Once the simulation model is used to compute USg(¯

xeval)over

a discretized set of points in the initial augmented state space,

the expected score can be computed as:

E[g(¯

xeval)] = ZΩ

fpu(pu)USg(¯

xeval)dpu(32)

As an alternative, suppose three different score functions are

used to assess performance. Each of these is deﬁned as an

indicator function such that:

gDes(¯

xeval) = (1 if ¯

xeval ∈(¯

xeval)Des

0 else (33)

where (¯

xeval)Des is the set of evaluation vectors that is deﬁned

to give Desired performance according to the MTE deﬁnition.

Analogous score function deﬁnitions can be created for the

Adequate and Inadequate categories. Then, the probability

that the system satisﬁes the Desired, Adequate, and Inade-

quate speciﬁcations can be computed as:

E[gDes(¯

xeval] = ZΩ

fpu(pu)USgDes(¯

xeval)dpu(34)

Figure 7shows the realizations of the discretized points in

the uncertainty domain in the TTA score domain, using the

uncertainty distributions deﬁned in the previous section. The

closed-loop quadrotor UAS performance for the lateral repo-

sition MTE for a gentle maneuvering mission (Fig. 7(a))

shows that the mean performance is Desired with a probability

of 100%. However, for the aggressive maneuvering mission

shown in Fig. 7(b), the mean performance is Adequate. The

analysis suggests that the system will have Desired, Adequate,

and Inadequate performances with probabilities 0%, 82.13%,

and 17.87%, respectively.

65

70

75

80

TTA score

LEVEL 1 = 100.00%

LEVEL 2 = 0.00%

LEVEL 3 = 0.00%

mean

mapping

(a)

72

74

76

78

80

82

84

86

TTA score

LEVEL 1 = 0.00%

LEVEL 2 = 82.13%

LEVEL 3 = 17.87%

mean

mapping

(b)

Figure 7: Probabilistic assessment of the TTA score for the

uncertain quadrotor UAS: (a) Gentle maneuvering mission

(b) Aggressive maneuvering mission.

Comparison of Monte Carlo and Koopman Operator Ap-

proaches

The probability that the TTA score falls within the Desired,

Adequate, and Inadequate performance ranges for an aggres-

sive lateral repositioning MTE can also be calculated us-

ing Monte Carlo simulation. The scaled lateral reposition-

ing maneuvers used to obtain the TTA scores are generated

for randomly-sampled lateral stability and control derivatives.

Figure 8compares the absolute error from the probability that

the TTA score falls within the Inadequate (Level 3) perfor-

mance using the Koopman operator to the average absolute

error from 20 executions of Monte Carlo for varying number

of data points. As shown in Fig. 8on average Monte Carlo

is less accurate than the Koopman operator for the same num-

ber of data points. In fact, about 2500 samples are needed to

9

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

Data points

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Absolute error [%]

Absolute error (Koopman)

Mean absolute error (Monte Carlo)

Figure 8: Probability that the TTA score falls within Level 3

performance: comparison between the absolute error using

the Koopman operator and the average absolute error from 20

executions of Monte Carlo for varying number of data points.

reduce the average Monte Carlo error below that of Koopman

with only 1000 data points. While the computational effort of

simulating an additional 1500 data points required by Monte

Carlo to achieve a similar accuracy to Koopman is minimal

for this particular example, for systems involving increased

complexity and/or a greater number of uncertain parameters

(i.e. if all the uncertain stability and control derivatives were

included) this difference may be signiﬁcant.

CONCLUSIONS

A novel methodology for computing stability, Handling Qual-

ities (HQ), and performance assessments in a probabilistic

framework using the Koopman operator has been developed.

The algorithm computes stability, HQ, and performance met-

rics for each point in the discretized uncertainty space us-

ing either algebraic transformations or simulation models.

The expected value of relevant metrics can then be computed

through numerical integration with the parameter uncertainty

distribution. By deﬁning speciﬁc binary cost functions, the

probability that the vehicle achieves a certain HQ level or per-

formance metric may be directly evaluated. The methodology

was used to evaluate the handling qualities of a small quadro-

tor with a multi-loop dynamic inversion controller. Using

scaled Mission Task Elements (MTEs), results showed that

the method can predict the probability of achieving certain

HQ levels using a fairly small number of sample points. Com-

parative studies were conducted to assess the computational

advantages of the proposed technique with respect to Monte

Carlo analysis.

It is worth noting that, typically, the aircraft stability and con-

trol derivatives are ﬂight-identiﬁed at different speeds to ac-

count for the change in aerodynamic properties with speed.

This is not accounted for in this particular model as the maxi-

mum speed reached by the quadcopter is fairly low, and the

dynamics can be approximated with the bare-aircrame dy-

namics in hover. In the case where the stability and control

derivatives were scheduled with airspeed, the uncertain pa-

rameters included in the analysis would change with speed,

which would simply increase the number of uncertain param-

eters that need to be considered (i.e., increasing the dimension

of the uncertainty space). One limitation of the method lies in

the inability to include process noise (i.e. turbulence). This

is because the Koopman operator approach is limited to para-

metric uncertainty only as it does not model diffusion of the

augmented state PDF due to process noise. Future efforts will

focus on extending the methodology to include process noise.

Based on the current work, the following conclusions can be

reached:

1. The Koopman operator provides a convenient framework

for the propagation of parametric uncertainty to stabil-

ity, Handling Qualities (HQ), and performance speciﬁ-

cations. The methodology uses evaluation functions that

extract HQ scores, which can be pulled back to the un-

certainty domain through the use of the Koopman op-

erator. The expected value integral with the parameter

uncertainty distribution can then be computed.

2. The probabilistic method can be used to evaluate HQ,

stability, and/or performance for any air vehicle ranging

from larger helicopters to small UAS, as long as relevant

criteria can be deﬁned.

3. On average Monte Carlo is less accurate than the Koop-

man operator for the same number of data points. The

computational effort of simulating additional data points

required by Monte Carlo to achieve a similar accuracy to

Koopman may be signiﬁcant for systems involving ex-

pensive dynamics and/or a large number of uncertain pa-

rameters. Thus, the Koopman operator approach may

provide signiﬁcant advantages when simulations are ex-

pensive or time-consuming to perform.

REFERENCES

1. Mavris, D., DeLaurentis, D., and Soban, D., “Proba-

bilistic Assessment of Handling Qualities Characteristics

in Preliminary Aircraft Design,” Proceedings of the 36th

AIAA Aerospace Sciences Meeting and Exhibit, Reno,

NV, January 12–15, 1998, DOI: 6.1998-492.

2. Tischler, M. B., Berger, T., Ivler, C. M., Mansur,

M. H., Cheung, K. K., and Soong, J. Y., Practical

Methods for Aircraft and Rotorcraft Flight Control De-

sign: An Optimization-Based Approach, American Insti-

tute of Aeronautics and Astronautics, Inc., 2017, DOI:

10.2514/4.104435.

3. Saetti, U., J., Horn, J. F., Lakhmani, S., Lagoa, C., and

Berger, T., “Design of Dynamic Inversion and Explicit

Model Following Control Laws for Quadrotor Inner and

Outer Loops,” Proceedings of the 74th Annual Forum of

the American Helicopter Society, Phoenix, AZ, May 14–

17, 2018.

10

4. Saetti, U., J., Horn, J. F., Lakhmani, S., Lagoa, C., and

Berger, T., “Design of Dynamic Inversion and Explicit

Model Following Control Laws for Quadrotor UAS,”

Journal of the American Helicopter Society,65, 032006

(2020), DOI: 10.4050/JAHS.65.032006.

5. Cooper, J., DeVore, M. D., Reed, A., and Klyde, D.,

“Handling Quality Analysis and Control Design Using

Polynomial Chaos Model-based Uncertainty Methods,”

Proceedings of the 75th Annual Forum of the Vertical

Flight Society, Philadelphia, PA, May 13–16, 2019.

6. Lasota, A., and Mackey, M. C., Chaos, Fractals, and

Noise: Stochastic Aspects of Dynamics, Springer, Vol. 97,

New York, NY, 2013.

7. Meyers, J. J., Leonard, M., Rogers J. D., Gerlach, A. R.,

“Koopman Operator Approach to Optimal Control Selec-

tion Under Uncertainty,” Proceedings of the 2019 Amer-

ican Control Conference, Philadelphia, PA, July 10–12,

2019, DOI: 10.23919/ACC.2019.8814461.

8. Anon, “Aeronautical Design Standard Performance Spec-

iﬁcation, Handling Qualities Requirements for Military

Rotorcraft,” ADS-33-PRF, U.S. Army Aviation and Mis-

sile Command Aviation Engineering Directorate, Red-

stone Arsenal, AL, 2000.

9. Ivler, C. M., Goerzen, C. L., Wagster, J. A., Sanders, F.

C., Cheung, K. K., and Tischler, M. B., “Control Design

for Tracking of Scaled MTE Trajectories on an IRIS+

Quadcopter,” Proceedings of the 74th Annual Forum of

the American Helicopter Society, Phoenix, AZ, May 14–

17, 2018.

10. Leonard, L., Rogers, J., and Gerlach, A., “Koopman Op-

erator Approach to Airdrop Mission Planning Under Un-

certainty,” Journal of Guidance, Control, and Dynamics,

Vol. 42, (11), 2019, pp. 1–17, DOI: 10.2514/1.G004277.

11. Tobias, E. L. and Tischler, M. B., “A Model Stitch-

ing Architecture for Continuous Full Flight-Envelope

Simulation of Fixed-Wing Aircraft and Rotorcraft from

Discrete-Point Linear Models,” U.S. Army AMRDEC

Special Report RDMR-AF-16-01, April 2016.

12. Horn, J. F., “Non-Linear Dynamic Inversion Control

Design for Rotorcraft,” Aerospace, Vol. 6, (3), 2019, pp.

38, DOI: 10.3390/aerospace6030038.

13. Tischler, M. B., and Rample, R. K., Aircraft and Rotor-

craft System Identiﬁcation, Second Edition, American In-

stitute of Aeronautics and Astronautics, Inc., 2012, DOI:

10.2514/4.868207.

14. Berger, T., Ivler, C. M., Berrios M. G., Tischler, M.

B., and Miller, D. G., “Disturbance Rejection Handling

Qualities Criteria for Rotorcraft,” Proceedings of the 72nd

Annual Forum of the American Helicopter Society, West

Palm Beach, FL, May 17-19, 2016.

15. Alvarenga, J., Vitzilaios, M., Rutherford, M., and Vala-

vanis, K., “Scaled Control Performance Benchmarks and

Maneuvers fro Small-Scale Unmanned Helicopters,” Pro-

ceedings of the 54th IEEE Conference on Decision and

Control, Osaka, Japan, December 15–18, 2015, DOI:

10.1109/CDC.2015.7403168.

11