Content uploaded by Umberto Saetti
Author content
All content in this area was uploaded by Umberto Saetti on Oct 07, 2020
Content may be subject to copyright.
Explicit Uncertainty Quantification for Probabilistic Handling Qualities
Assessment
Umberto Saetti
Postdoctoral Fellow
School of Aerospace Engineering
Georgia Institute of Technology
Atlanta, GA 30313
Jonathan Rogers
Associate Professor
School of Aerospace Engineering
Georgia Institute of Technology
Atlanta, GA 30313
ABSTRACT
Modern flight vehicle development typically involves significant uncertainty in aerodynamic loads and vehicle control
responses. In many cases, it may be necessary to consider this uncertainty in preliminary evaluations of stability,
handling qualities (HQ), and performance. This paper introduces a novel methodology for computing these quantities
in a probabilistic framework using the Koopman operator. The algorithm discretizes the uncertainty space and uses
relevant transformations or simulations to map each point to the handling qualities evaluation domain, resulting in
a stability, HQ, and/or performance assessment for each discretized point. The expected value of the stability, HQ,
and/or performance requirement is then computed through an expected value integral with the initial joint probability
density function, yielding a probabilistic assessment with respect to relevant specifications. The methodology is
used to evaluate handling qualities of a quadrotor Unmanned Aerial System (UAS), for which a multi-loop Dynamic
Inversion (DI) flight control law is developed. The UAS example is evaluated with respect to quantitative handling
qualities specifications based on scaled Mission Task Elements (MTEs). The proposed explicit UQ approach is shown
to provide a convenient framework for the propagation of parametric uncertainty to stability, HQ, and performance
specifications with unique advantages over Monte Carlo techniques.
INTRODUCTION
A key requirement in the design of a new aircraft, or modifi-
cation to an existing airframe, is that the aircraft meet specific
handling qualities, performance, and stability requirements.
In a typical design process, evaluation of these criteria in-
volves the use of linear models derived either through flight
testing or linearization of a high fidelity nonlinear model.
Such linear models are naturally subject to uncertainty, for
instance due to the limited amount of flight data available or
modeling errors inherent in a nonlinear modeling process. Be-
cause the linear model parameters are subject to uncertainty,
any stability, handling qualities, or performance assessments
based on the linear models are subject to uncertainty as well.
Particularly in cases where the linear model uncertainty is
significant, it may not be appropriate to make deterministic
judgments about a vehicle’s conformance to handling qual-
ities guidelines. Instead, probabilistic assessments must be
made in which it may be stated that the vehicle conforms to
specific guidelines with a certain probability, given the un-
certainty in the vehicle closed-loop dynamics. This type of
probabilistic handling qualities assessment may be more ap-
propriate particularly early in the design stages of a vehicle
or airframe modification, when aerodynamic parameters are
usually subject to large uncertainty. In light of this, there is a
Presented at the VFS International 76th Annual Forum &
Technology Display, Virginia Beach, VA, October 6–8, 2020. Copy-
right c
2020 by the Vertical Flight Society. All rights reserved.
need for new methodologies that can map the uncertainty in
vehicle model parameters to the resulting uncertainty in han-
dling qualities, stability, and performance assessments.
Despite the need for rigorous techniques in this domain, there
is surprisingly little prior research in the development of prob-
abilistic handling qualities evaluation techniques. Mavris et
al. (Ref. 1) developed a Monte Carlo-based approach to the
probabilistic assessment of fixed-wing handling qualities dur-
ing preliminary design. In the rotorcraft domain, a similar
Monte Carlo-based technique exists through the Control De-
signer’s Unified Interface (CONDUIT) (Ref. 2) robust han-
dling qualities evaluation tool. While easy to understand and
implement, there are several issues with Monte Carlo-based
approaches. First, the technique suffers from well-known
scalability issues such that the number of samples needed to
generate sufficient accuracy may become prohibitively high
as the dimensionality of the uncertainty increases. This is a
significant potential problem in rotorcraft design where nu-
merous aerodynamic parameters are typically subject to un-
certainty. Second, even after Monte Carlo samples have been
propagated, the underlying density must be reconstructed
from these samples using, for instance, kernel-based methods
(which are themselves subject to approximation error). An al-
ternative approach recently explored by Saetti et al. (Refs. 3,4)
used an unscented Kalman filter to quantify the probability
density associated with handling qualities metrics – however,
the UKF approach is still limited by its scalability to higher
dimensions. Another approach that was recently proposed for
1
applications to rotorcraft was polynomial chaos (Ref. 5), al-
though this technique also suffers from poor scaling in higher
dimensions. It is well known that explicit uncertainty quan-
tification approaches, particularly the Frobenius-Perron (FP)
Operator (Ref. 6), may be used to map a probability density
explicitly through a nonlinear transformation with improved
scalability characteristics. Recently, Meyers et al. (Ref. 7) ap-
plied the FP operator and its adjoint, the Koopman operator, to
various optimal decision-making problems under uncertainty,
demonstrating the scalability benefits of these explicit tech-
niques compared to Monte Carlo.
This paper presents a novel methodology for computing sta-
bility, handling qualities, and performance assessments in a
probabilistic framework using the Koopman operator. The al-
gorithm begins with a discretization of the uncertainty space
comprised of the uncertain linear or nonlinear model parame-
ters. Each discretized point is mapped through relevant trans-
formations to the handling qualities evaluation domain, result-
ing in a stability, HQ, and/or performance assessment for each
discretized point (similar to Monte Carlo, except usually with
significantly less points). Using this mapping, the handling
qualities evaluation is pulled back to the initial uncertainty do-
main using the Koopman operator. The expected value of the
stability, HQ, and/or performance requirement is then com-
puted through an expected value integral with the initial joint
probability density function (PDF), yielding a probabilistic
assessment with respect to relevant specifications.
The paper begins with a detailed discussion of the mathemat-
ical background and methodology. Following a description
of the simulation model of the quadrotor Unmanned Aerial
System (UAS) and its flight control law, an example imple-
mentation of the methodology is presented. The UAS han-
dling qualities are evaluated probabilistically with respect to
quantitative specifications based on scaled Mission Task El-
ements (Ref. 8) that were recently proposed in the rotorcraft
community in Ref. 9.
METHODOLOGY
Define an aircraft/rotorcraft dynamic model with state vector
xand parameterized by p∈Rpaccording to,
˙
x=f(x,p,u)(1)
where uis the vector of controls. A new augmented state
is defined by combining the state vector with the uncertain
model parameters, and any other components needed to eval-
uate HQ ratings. Define the set of uncertain model parame-
ters as pu∈Rdwhere pu⊂pand dis the dimension of the
uncertain parameter space. Likewise, pk∈Rp−dis the vector
of known parameters. The augmented state is then given as
¯
xT=xTpT
u. Suppose that a map exists that such that, for
each unique initial (augmented) state vector ¯
x0, a state vector
at a final time ¯
xfis generated that may be used to evaluate
handling qualities:
¯
xf=S(¯
x0)(2)
In Eq. 2, the transformation Smay represent various types
of algebraic or dynamic transformations. For instance, in the
evaluation of ADS-33E performance standards, Mission Task
Elements (MTEs) are used to categorize flight performance
into Desired, Adequate, or Inadequate performance (or equiv-
alently, into Level 1, Level 2, or Level 3 performance). In
this case, evaluation of this specification requires executing
the MTE from the prescribed initial condition using a se-
lected value of p. Thus, Smay involve propagation of a flight
dynamic model from the prescribed initial condition for the
MTE to the prescribed final condition, with the final state
being ¯
xf. Other handling qualities, performance, or stabil-
ity metrics may involve other types of dynamic or algebraic
transformations Sto obtain a vector ¯
xf. In subsequent discus-
sion, this final state vector will be referred to as ¯
xeval =¯
xffor
clarity, to represent the state vector on which the HQ ratings
can be evaluated.
Given this mapping, a “score” function gis defined that as-
signs a numerical score based on ¯
xeval representing the han-
dling qualities or performance specification associated with
the forward mapping of the augmented state ¯
x0. This score
function value may be “pulled back” to the initial condition
domain of the augmented state through use of the Koopman
operator defined as,
USg=USg(¯
xeval ) = g(S(¯
x0)) (3)
The only restriction on gis that it must be finite, i.e., ||g∞|| <
∞. The score or cost function value is associated with a
certain handling qualities criterion – for instance, a value of
g(¯
xeval ) = 1 may be assigned for ¯
xeval in a particular region
associated with Desired handling qualities, g(¯
xeval ) = 2 for
regions associated with Adequate handling qualities, and so
on. Note that the Koopman operator USin Eq. 3is defined
specific to a particular transformation S.
This overall process is illustrated in Fig. 1. In Fig. 1(a),
each discretized point in the uncertainty domain is mapped to
a specific numerical value in the HQ or performance domain
using the forward mapping S. In Fig. 1(b), the score/cost
function associated with each forward-mapped point is pulled
back to the original uncertainty domain, yielding USg. With
this pulled-back score function, the expected value of the han-
dling qualities assessment function gcan be computed given
the joint probability density function on the uncertain param-
eters puaccording to:
E[g(x,p)] = ZΩ
fpu(pu)USg(¯
xeval )dpu(4)
where Ωis the support of the joint probability density of the
uncertain parameters, fpu(pu). This expected value represents
the expected handling qualities evaluation score for this par-
ticular metric signified by the cost or score function g. For
instance, in the example shown in Fig. 1, calculation of Eq.
4would provide a value between 0 and 1, representing the
probability of the vehicle achieving the Desired level of per-
formance for the particular handling qualities or performance
metric. Note that by defining several binary score functions
for each handling qualities ranking, probability values can be
generated that specify the probability that the vehicle will con-
form to each level. This represents a generalization of the
chance constraint techniques detailed in Refs. 7,10.
2
(a)
(b)
Figure 1: (a) Forward Mapping of Uncertainty Domain to
HQ/Performance Domain (b) Pull-Back of HQ Evaluation
Score to Uncertainty Domain Using Koopman Operator.
SIMULATION MODEL
The probabilistic HQ evaluation methodology is illustrated
through application to a quadrotor UAS. A 12-state model
is used to simulate the flight dynamics of a small-scale UAS
in hover. The stability and control derivatives from a flight-
identified linear model are combined with nonlinear equa-
tions of motion to produce a quasi-nonlinear simulation model
(Ref. 11). Consider the rigid-body dynamics of the flight-
identified linear model:
˙
x=Ax +Bu (5)
where:
xare the states relative to the rigid-body dynamics,
uis the control vector,
Acontains the rigid-body stability derivatives, and
Bcontains the rigid-body control derivatives.
The state vector is:
xT=uvwpqr(6)
where u,v,ware the body velocities, and p,q,rare the body
angular rates. The control vector is:
u=
δlat(t−τφ)
δlon(t−τθ)
δcol(t−τz)
δped(t−τψ)
(7)
where δlat,δlon ,δcol,δped are respectively the lateral, lon-
gitudinal, vertical, and directional control inputs (before the
mixer), and τφ,τθ,τz,τψare the time delays associated with
Table 1: Stability derivatives, control derivatives, and time
delays of the flight-identified quadrotor model (Refs. 3,4).
Parameter Value CR Bound [%]
Xu-0.2568 [1/s] 5.302
Yv-0.3022 [1/s] 6.107
Zw-0.1734 [1/s] 39.72
Lv-0.8287 [rad/(ft s)] 5.943
Lp0 [1/s] -
Mu1.1257 [rad/(ft s)] 5.618
Mq0 [1/s] -
Nr-0.5617 [1/s] 25.19
Zδcol -49.065 [ft/(s2%)] 2.647
Lδlat 33.514 [rad/(s2%)] 3.297
Mδlon 27.919 [rad/(s2%)] 4.155
Nδped 6.0308 [rad/(s2%)] 3.877
τφ0.0565 [s] 4.170
τθ0.0355 [s] 10.73
τz0.0389 [s] 30.03
τψ0.0401 [s] 7.023
the lateral, longitudinal, vertical, and directional dynamics,
respectively. The system matrix containing the rigid-body sta-
bility derivatives is:
A=
Xu0 0 0 0 0
0Yv0 0 0 0
0 0 Zw0 0 0
0Lv0Lp0 0
Mu0 0 0 Mq0
0 0 0 0 0 Nr
(8)
The control matrix containing the rigid-body control deriva-
tives is:
B=
0 0 0 0
0 0 0 0
0 0 Zδcol 0
Lδlat 0 0 0
0Mδlon 0 0
0 0 0 Nδped
(9)
The stability derivatives, control derivatives, and time delays
are taken from the flight-identified model of Refs. 3,4and are
reported in Table 1.
In order to calculate the aerodynamic perturbation forces and
moments, it is necessary to obtain the state and control pertur-
bations:
∆x=X−X0(10a)
∆u=U−U0(10b)
where:
Xare the nonlinear simulation rigid-body states,
X0are the trim rigid-body states in hover,
Uis the nonlinear simulation control vector, and
U0is the trim control vector in hover.
3
Table 2: Mass and inertia properties of the flight-identified
quadrotor model (Ref. 3,4).
Parameter Value
m1.1023 [lb]
Ixx 0.0547 [lb-ft2]
Iyy 0.0585 [lb-ft2]
Izz 0.0756 [lb-ft2]
Ixz 0 [lb-ft2]
It is worth noting that the rigid-body perturbation vector does
not include Euler angles or the position since the aerodynamic
forces and moments do not depend explicitly on the Euler an-
gles or the position (Ref. 11). The aerodynamics perturbation
forces and moments are:
∆Faero
∆Maero=M(A∆x+B∆u)(11)
where:
M=
m
m
m
Ixx −Ixz
Iyy
−Ixz Izz
(12)
is the mass matrix containing the mass mand inertia terms
Ixx,Iyy,Izz,Ixz of the quadrotor. The mass and inertia proper-
ties of the quadrotor are reported in Table 2.
The aerodynamic trim forces and moments, based on the trim
Euler angles Φ0,Θ0in hover, are:
Faero0=
gsinΘ0
−gcosΘ0sin Φ0
−gcosΘ0cos Φ0
m(13)
It is worth noting that since the quadrotor is symmetric with
respect to the lateral and longitudinal axes, the trim roll and
pitch angles are zero. The nonlinear simulation gravitational
forces, which are nonlinear with respect to the Euler angles,
are:
Fgrav =
−gsinΘ
gcosΘsin Φ
gcosΘcos Φ
m(14)
The aerodynamics perturbation forces and moments are
summed with the trim and gravitational forces and moments
to obtain the total aerodynamic forces and moments:
F
M=∆Faero +Faero0+Fgrav
∆Maero (15)
These forces and moments are incorporated in the nonlinear
equations of motion in the body-fixed frame to provide the
fuselage 6-DOF dynamics:
˙
U=RV −QW +Fx
m(16a)
˙
V=PW −RU +Fy
m(16b)
˙
W=QU −PV +Fz
m(16c)
˙
P=Izx(Ixx −Iyy +Izz)PQ −(I2
zz −IyyIzz +I2
xz)QR +IzzL+IxzN
IxxIzz −I2
xz (16d)
˙
Q=(Izz −Ixx)PR −Ixz(P2−R2) + M
Iyy
(16e)
˙
R=(I2
xx −IxxIyy +I2
xz)PQ −Izx(Ixx −Iyy +Izz)QR +Ixz L+IxxN
IxxIzz −I2
xz (16f)
˙
Φ=P+QsinΦtan Θ+Rcos Φtan Θ(16g)
˙
Θ=QcosΦ−Rsin Φ(16h)
˙
Ψ=QsinΦsec Θ+Rcos Φsec Θ(16i)
˙
X=UcosΘcos Ψ+V(sin Φsin Θcos Ψ−cos Φsin Ψ)
+W(cosΦsin Θcos Ψ+sin Φsin Ψ)(16j)
˙
Y=UcosΘsin Ψ+V(sin Φsin Θsin Ψ+cos Φcos Ψ)
+W(cosΦsin Θsin Ψ−sin Φcos Ψ)(16k)
˙
Z=−UsinΘ+Vsin Φcos Θ+WcosΦcos Θ(16l)
FLIGHT CONTROL DESIGN
A multi-loop Dynamic Inversion (DI) control law largely
based on Refs. 3,4is designed to enable fully autonomous
flight of the UAS. The schematic of the closed-loop dynam-
ics of the quadrotor UAS is shown in Fig. 2. The outer loop
takes trajectory (position) commands with optional heading
constraints and calculates the pitch and roll attitudes for the
inner loop to track. The natural frequencies and damping ra-
tions of the outer loop command models are given in Table 3.
The inner loop achieves stability, disturbance rejection, an At-
titude Command / Attitude Hold (ACAH) response around the
roll, pitch, and yaw axes, and a position hold response around
the heave axis. The outer and inner loop block diagrams are
shown in Figs. 3and 4, respectively. The outer and inner loop
command model parameters are given in Tables 3and 4, re-
spectively. The PID gains are chosen such that the frequencies
of the error dynamics are of the same order as the command
models (Ref. 12). The numerical values of the PID gains are
shown in Table 5for the outer loop and and in Table ?? for
the inner loop.
4
Figure 2: Schematic of the closed-loop dynamics of the quadrotor UAS.
Figure 3: DI outer loop.
5
Figure 4: DI inner loop.
Table 3: Outer loop command models natural frequency and
damping ratio.
Command ωn[rad/s]ζ
Longitudinal Position 1 0.7
Lateral Position 1 0.7
Table 4: Inner loop command models natural frequency and
damping ratio.
Command ωn[rad/s]ζ
Roll Attitude 10 0.7
Pitch Attitude 10 0.7
Yaw Attitude 2 0.7
Vertical Position 1 0.7
RESULTS
Frequency-Domain Specifications
This section illustrates the mapping of parametric uncertainty
in the quadrotor UAS model to quantitative handling qualities
Table 5: Outer loop PID gains.
KDKPKI
xcmd 1.61 1.28 0.2
ycmd 1.61 1.28 0.2
Table 6: Inner loop PID gains.
KDKPKI
φcmd 16 128 200
θcmd 16 128 200
ψcmd 4.4 5.6 1.6
zcmd 1.61 1.28 0.2
specifications defined in the frequency domain. Restricting
the analysis to the lateral axis for simplicity, let the uncertain
parameters be the flight-identified stability and control deriva-
tives and time delay such that:
pT
u=YvLvLδlat τφ(17)
Each parameter is assumed to be a random variable indepen-
dent of the others. The probability density on each parame-
ter is described by a Gaussian distribution with with mean µi
and standard deviation σi. The mean of each parameter cor-
responds to the nominal identified value reported in Table 1,
such that µi=Θi, where Θis the vector of identified parame-
ters. The standard deviation is taken as the squared root of the
diagonal elements of the covariance matrix Pfor the vector of
identified parameters:
σi=pPii (18)
The covariance matrix is calculated from the identification re-
sults as:
P=H−1(19)
6
where His the parameter identification Hessian matrix
(Ref. 13). The Hessian matrix is composed by the second-
order partial derivatives of the identification fit cost Jwith
respect to the identified parameters vector:
H=∂2J
∂Θ∂ΘT(20)
The Cramer-Rao (CR) bound of the ith identified parameter
of the identified parameters vector is related to the Hessian
matrix in the following way:
CRi=q(H−1)ii (21)
It should be noted that the CR bounds are best expressed as a
percentage of the nominal identification values:
CRi=
CRi
Θi×100% (22)
and are reported in this form in Table 1.
As an example of quantitative handling qualities specifica-
tions defined in the frequency domain, consider the Distur-
bance Rejection Bandwidth (DRB) and Disturbance Rejection
Peak (DRP) defined in Ref. 14. For a given realization of the
uncertain parameters pu, the DRB is the frequency at which
the sensitivity function of the hold variable (the roll attitude
φfor the inner loop, and the lateral position yfor the outer
loop) crosses −3 dB from below, and DRP is the peak mag-
nitude of the sensitivity function. This leads to the natural
definition of two cost functions, gDRB(pu) = DRB(pu)and
gDRP(pu) = DRP(pu). For this particular example, the trans-
formations Sdescribed above are the identity transformations,
since the cost functions are defined directly as functions of the
uncertain parameters (rather than on a transformed augmented
state). This means that USis the identity operator. Once gDRB
and gDRP are computed over a discretized domain of the un-
certain parameters pu, the expected DRB can be found as:
E[DRB] = ZΩ
fpu(pu)gDRB(pu)dpu(23)
where Ωis the support of fpu. Note that since the uncertain pa-
rameters are assumed to be independent, the joint uncertainty
distribution fpuis formed as the product of the Gaussian PDFs
governing each element of pu. An analogous equation can be
created for the expected DRP. As an alternative, suppose three
different score functions are used to assess performance. Each
of these is defined as an indicator function such that:
gDes(pu) = (1 if DRB(pu)∈DRBDes
0 else (24)
where DRBDes is the set of evaluation vectors that is defined
to give Desired performance. Analogous score function defi-
nitions can be created for the Adequate and Inadequate cate-
gories, as well as for DRP Desired, Adequate, and Inadequate
performance. Then, the probability that the system satisfies
the Desired specifications can be computed as:
E[gDes(pu)] = ZΩ
fpu(pu)gDes(pu)dpu(25)
Table 7: DRB and DRP scaled Level 1 and Level 2
requirements.
Performance DRB [rad/s] DRP [dB]
Desired (φ)6.4 5.0
Adequate (φ)3.5 8.0
Desired (y) 1.2 3.0
Adequate (y) 0.6 5.0
Although DRB and DRP specifications are not well defined
for small-scale UAS, the DRB and DRP requirements are
taken from Ref. 14. In this work, the DRB is scaled accord-
ing to the approach used in Refs. 9,15. The scaling is based
on the scale factor N, which is defined as the ratio between
the motor-to-motor length of the quadrotor (1.08 ft) and the
main rotor diameter of the UH-60 (53.8 ft). This leads to a
scale factor of N=0.0201. The DRB requirements are scaled
by √Nand are shown in Table 7for both the inner and outer
loop.
In the implementation used here, each uncertain parameter di-
mension is discretized into np=10 evenly-spaced grid points
spanning µ±4σ, where µand σare the mean and standard
deviation of each parameter. Since the number of uncertain
parameters is 4, this leads to a total of 104data points. The
number of grid points is chosen to ensure that the numeri-
cal solution of the integrals defined on the uncertainty domain
(Eqs. 23,25) is sufficiently accurate. Figure 5shows the re-
alizations of the discretized points in the uncertainty domain
in the DRB and DRP domain, using the uncertainty distribu-
tions defined in the previous section. In this figure, the red
dots are the mapped values of a given realization of the uncer-
tain parameters to the DRB and DRP domains and the black
cross is the mean of the mapped values. Figure 5(a) shows
that the inner loop mean performance is Adequate with 100%
probability, whereas Fig. 5(b) shows that the outer loop mean
performance is Inadequate with 100% probability.
Time-Domain Specifications
This section illustrates the mapping of parametric uncertainty
in the quadrotor UAS model to quantitative handling qualities
specifications defined in the time domain. Consider a more
complex example in which a handling qualities metric is de-
fined over a transformed state and parameter vector. ADS-33E
(Ref. 8) defines the Lateral Reposition MTE in which a rotor-
craft, starting from hover, must initiate a lateral acceleration
to about 35 knots followed by a deceleration to laterally repo-
sition the rotorcraft in a stabilized hover 400 ft down range
within a specified time. Although this MTE is defined only
for full-scale rotorcraft, it has recently been shown that MTE
trajectories can be scaled for use in small-scale UAS HQ eval-
uation (Ref. 9). The trajectories are scaled by the square of the
ratio between the maximum velocity of the UH-60 (160 kts)
and the estimated maximum velocity of the quadrotor UAS
7
3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8
DRB [rad/s]
2
3
4
5
6
7
8
9
DRP [dB]
LEVEL 1 = 0%
LEVEL 2 = 100%
LEVEL 3 = 0%
mean
mapping
(a)
0.4 0.6 0.8 1 1.2 1.4 1.6
DRB [rad/s]
0
1
2
3
4
5
6
DRP [dB]
LEVEL 1 = 0%
LEVEL 2 = 0%
LEVEL 3 = 100%
mean
mapping
(b)
Figure 5: Probabilistic assessment of the disturbance
rejection bandwidth and peak for the uncertain quadrotor
system: (a) Inner loop, (b) Outer loop.
(35 ft/s):
Lscale =(VUAS)max
(VUH-60)max 2
(26)
This definition yields a scale factor of 0.0479 for the vehicle
considered here. It follows that the quadrotor has to reposition
itself by approximately 19 ft from its starting location. The
objective function proposed in Ref. 9to score the mission task
element is the trajectory tracking and aggressiveness (TTA)
score and is given by:
ΦTTA(L) = 200
1+eL(27)
where:
L=wα
α−αG
αB−αG
+wε
ε−εG
εB−εG
(28)
Table 8: TTA performance levels for gentle and aggressive
lateral reposition maneuvers from Ref. 9.
Desired Adequate
Gentle ΦTTA ≥75 75 >ΦTTA ≥70
Aggressive ΦTTA ≥82 82 >ΦTTA ≥77
Table 9: TTA scoring parameters from Ref. 9.
Parameter DRB [rad/s]
wα0.5
wε0.5
αG3
αB0
εG0
εB0.35
wvel 0.7
wpos 0.3
The TTA score has a minimum value of 0 and a maximum
value of 100. Higher values represent better tracking perfor-
mance and more aggressiveness. Table 8shows the Desired
and Adequate TTA scores for the lateral reposition MTE. The
parameters wα,wεare relative weights, whereas αG,αB,εG,
εBare conditioning parameters indicating “good” and “bad”
performance. The aggressiveness term αis defined as:
α=(Vcmd)max
(Vcmd)nom
(29)
where:
(Vcmd)max =(VUAS)max
(VUH-60)max
(30)
is the maximum speed of the scaled MTE. The tracking error
term εis:
ε=wvel
RMSE(vel)
(Vcmd)max
+wpos
RMSE(pos)
Lpath
(31)
where RMSE is the root mean square of the error between the
commanded and measured velocity or position, and Lpath is
the length of the scaled trajectory (i.e. 19 ft). The parameters
used to calculate the TTA score are reported in Table 9. The
scaled lateral repositioning maneuver is shown in Fig. 6for
the vehicle and controller described above. The resulting ag-
gressiveness score, as determined by the outer loop command
filter for the lateral dynamics, is α=0.6754. The TTA score
is 77.2.
Suppose the quasi non-linear quadrotor simulation model with
state xand uncertain parameters pT
u=YvLvLδlat is used to
evaluate performance against the assigned MTE criteria (i.e.
the TTA score). The MTE can be simulated from an initial
condition x0and specific set of parameters to obtain a time
history of xthroughout the maneuver. Key values from this
time history needed to evaluate the MTE metrics, such as ve-
locity and position tracking errors, may be extracted from this
8
012345678910
0
5
10
Vy [ft/s]
012345678910
0
10
20
y [ft]
Command
Response
012345678910
0
10
20
30
[deg]
012345678910
Time [s]
-1
0
1
z [ft]
Figure 6: Simulated scaled lateral reposition MTE with
aggressiveness α=0.6754.
time history to produce an evaluation state ¯
xeval. The map-
ping S, which maps ¯
x0to ¯
xeval is thus the dynamic simula-
tion model of the UAS. Given ¯
xeval, a cost or score function
g(¯
xeval)may be defined based on the TTA score which cat-
egorizes the MTE performance into Desired, Adequate, and
Inadequate based on the specific thresholds listed in Table 8.
Once the simulation model is used to compute USg(¯
xeval)over
a discretized set of points in the initial augmented state space,
the expected score can be computed as:
E[g(¯
xeval)] = ZΩ
fpu(pu)USg(¯
xeval)dpu(32)
As an alternative, suppose three different score functions are
used to assess performance. Each of these is defined as an
indicator function such that:
gDes(¯
xeval) = (1 if ¯
xeval ∈(¯
xeval)Des
0 else (33)
where (¯
xeval)Des is the set of evaluation vectors that is defined
to give Desired performance according to the MTE definition.
Analogous score function definitions can be created for the
Adequate and Inadequate categories. Then, the probability
that the system satisfies the Desired, Adequate, and Inade-
quate specifications can be computed as:
E[gDes(¯
xeval] = ZΩ
fpu(pu)USgDes(¯
xeval)dpu(34)
Figure 7shows the realizations of the discretized points in
the uncertainty domain in the TTA score domain, using the
uncertainty distributions defined in the previous section. The
closed-loop quadrotor UAS performance for the lateral repo-
sition MTE for a gentle maneuvering mission (Fig. 7(a))
shows that the mean performance is Desired with a probability
of 100%. However, for the aggressive maneuvering mission
shown in Fig. 7(b), the mean performance is Adequate. The
analysis suggests that the system will have Desired, Adequate,
and Inadequate performances with probabilities 0%, 82.13%,
and 17.87%, respectively.
65
70
75
80
TTA score
LEVEL 1 = 100.00%
LEVEL 2 = 0.00%
LEVEL 3 = 0.00%
mean
mapping
(a)
72
74
76
78
80
82
84
86
TTA score
LEVEL 1 = 0.00%
LEVEL 2 = 82.13%
LEVEL 3 = 17.87%
mean
mapping
(b)
Figure 7: Probabilistic assessment of the TTA score for the
uncertain quadrotor UAS: (a) Gentle maneuvering mission
(b) Aggressive maneuvering mission.
Comparison of Monte Carlo and Koopman Operator Ap-
proaches
The probability that the TTA score falls within the Desired,
Adequate, and Inadequate performance ranges for an aggres-
sive lateral repositioning MTE can also be calculated us-
ing Monte Carlo simulation. The scaled lateral reposition-
ing maneuvers used to obtain the TTA scores are generated
for randomly-sampled lateral stability and control derivatives.
Figure 8compares the absolute error from the probability that
the TTA score falls within the Inadequate (Level 3) perfor-
mance using the Koopman operator to the average absolute
error from 20 executions of Monte Carlo for varying number
of data points. As shown in Fig. 8on average Monte Carlo
is less accurate than the Koopman operator for the same num-
ber of data points. In fact, about 2500 samples are needed to
9
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Data points
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Absolute error [%]
Absolute error (Koopman)
Mean absolute error (Monte Carlo)
Figure 8: Probability that the TTA score falls within Level 3
performance: comparison between the absolute error using
the Koopman operator and the average absolute error from 20
executions of Monte Carlo for varying number of data points.
reduce the average Monte Carlo error below that of Koopman
with only 1000 data points. While the computational effort of
simulating an additional 1500 data points required by Monte
Carlo to achieve a similar accuracy to Koopman is minimal
for this particular example, for systems involving increased
complexity and/or a greater number of uncertain parameters
(i.e. if all the uncertain stability and control derivatives were
included) this difference may be significant.
CONCLUSIONS
A novel methodology for computing stability, Handling Qual-
ities (HQ), and performance assessments in a probabilistic
framework using the Koopman operator has been developed.
The algorithm computes stability, HQ, and performance met-
rics for each point in the discretized uncertainty space us-
ing either algebraic transformations or simulation models.
The expected value of relevant metrics can then be computed
through numerical integration with the parameter uncertainty
distribution. By defining specific binary cost functions, the
probability that the vehicle achieves a certain HQ level or per-
formance metric may be directly evaluated. The methodology
was used to evaluate the handling qualities of a small quadro-
tor with a multi-loop dynamic inversion controller. Using
scaled Mission Task Elements (MTEs), results showed that
the method can predict the probability of achieving certain
HQ levels using a fairly small number of sample points. Com-
parative studies were conducted to assess the computational
advantages of the proposed technique with respect to Monte
Carlo analysis.
It is worth noting that, typically, the aircraft stability and con-
trol derivatives are flight-identified at different speeds to ac-
count for the change in aerodynamic properties with speed.
This is not accounted for in this particular model as the maxi-
mum speed reached by the quadcopter is fairly low, and the
dynamics can be approximated with the bare-aircrame dy-
namics in hover. In the case where the stability and control
derivatives were scheduled with airspeed, the uncertain pa-
rameters included in the analysis would change with speed,
which would simply increase the number of uncertain param-
eters that need to be considered (i.e., increasing the dimension
of the uncertainty space). One limitation of the method lies in
the inability to include process noise (i.e. turbulence). This
is because the Koopman operator approach is limited to para-
metric uncertainty only as it does not model diffusion of the
augmented state PDF due to process noise. Future efforts will
focus on extending the methodology to include process noise.
Based on the current work, the following conclusions can be
reached:
1. The Koopman operator provides a convenient framework
for the propagation of parametric uncertainty to stabil-
ity, Handling Qualities (HQ), and performance specifi-
cations. The methodology uses evaluation functions that
extract HQ scores, which can be pulled back to the un-
certainty domain through the use of the Koopman op-
erator. The expected value integral with the parameter
uncertainty distribution can then be computed.
2. The probabilistic method can be used to evaluate HQ,
stability, and/or performance for any air vehicle ranging
from larger helicopters to small UAS, as long as relevant
criteria can be defined.
3. On average Monte Carlo is less accurate than the Koop-
man operator for the same number of data points. The
computational effort of simulating additional data points
required by Monte Carlo to achieve a similar accuracy to
Koopman may be significant for systems involving ex-
pensive dynamics and/or a large number of uncertain pa-
rameters. Thus, the Koopman operator approach may
provide significant advantages when simulations are ex-
pensive or time-consuming to perform.
REFERENCES
1. Mavris, D., DeLaurentis, D., and Soban, D., “Proba-
bilistic Assessment of Handling Qualities Characteristics
in Preliminary Aircraft Design,” Proceedings of the 36th
AIAA Aerospace Sciences Meeting and Exhibit, Reno,
NV, January 12–15, 1998, DOI: 6.1998-492.
2. Tischler, M. B., Berger, T., Ivler, C. M., Mansur,
M. H., Cheung, K. K., and Soong, J. Y., Practical
Methods for Aircraft and Rotorcraft Flight Control De-
sign: An Optimization-Based Approach, American Insti-
tute of Aeronautics and Astronautics, Inc., 2017, DOI:
10.2514/4.104435.
3. Saetti, U., J., Horn, J. F., Lakhmani, S., Lagoa, C., and
Berger, T., “Design of Dynamic Inversion and Explicit
Model Following Control Laws for Quadrotor Inner and
Outer Loops,” Proceedings of the 74th Annual Forum of
the American Helicopter Society, Phoenix, AZ, May 14–
17, 2018.
10
4. Saetti, U., J., Horn, J. F., Lakhmani, S., Lagoa, C., and
Berger, T., “Design of Dynamic Inversion and Explicit
Model Following Control Laws for Quadrotor UAS,”
Journal of the American Helicopter Society,65, 032006
(2020), DOI: 10.4050/JAHS.65.032006.
5. Cooper, J., DeVore, M. D., Reed, A., and Klyde, D.,
“Handling Quality Analysis and Control Design Using
Polynomial Chaos Model-based Uncertainty Methods,”
Proceedings of the 75th Annual Forum of the Vertical
Flight Society, Philadelphia, PA, May 13–16, 2019.
6. Lasota, A., and Mackey, M. C., Chaos, Fractals, and
Noise: Stochastic Aspects of Dynamics, Springer, Vol. 97,
New York, NY, 2013.
7. Meyers, J. J., Leonard, M., Rogers J. D., Gerlach, A. R.,
“Koopman Operator Approach to Optimal Control Selec-
tion Under Uncertainty,” Proceedings of the 2019 Amer-
ican Control Conference, Philadelphia, PA, July 10–12,
2019, DOI: 10.23919/ACC.2019.8814461.
8. Anon, “Aeronautical Design Standard Performance Spec-
ification, Handling Qualities Requirements for Military
Rotorcraft,” ADS-33-PRF, U.S. Army Aviation and Mis-
sile Command Aviation Engineering Directorate, Red-
stone Arsenal, AL, 2000.
9. Ivler, C. M., Goerzen, C. L., Wagster, J. A., Sanders, F.
C., Cheung, K. K., and Tischler, M. B., “Control Design
for Tracking of Scaled MTE Trajectories on an IRIS+
Quadcopter,” Proceedings of the 74th Annual Forum of
the American Helicopter Society, Phoenix, AZ, May 14–
17, 2018.
10. Leonard, L., Rogers, J., and Gerlach, A., “Koopman Op-
erator Approach to Airdrop Mission Planning Under Un-
certainty,” Journal of Guidance, Control, and Dynamics,
Vol. 42, (11), 2019, pp. 1–17, DOI: 10.2514/1.G004277.
11. Tobias, E. L. and Tischler, M. B., “A Model Stitch-
ing Architecture for Continuous Full Flight-Envelope
Simulation of Fixed-Wing Aircraft and Rotorcraft from
Discrete-Point Linear Models,” U.S. Army AMRDEC
Special Report RDMR-AF-16-01, April 2016.
12. Horn, J. F., “Non-Linear Dynamic Inversion Control
Design for Rotorcraft,” Aerospace, Vol. 6, (3), 2019, pp.
38, DOI: 10.3390/aerospace6030038.
13. Tischler, M. B., and Rample, R. K., Aircraft and Rotor-
craft System Identification, Second Edition, American In-
stitute of Aeronautics and Astronautics, Inc., 2012, DOI:
10.2514/4.868207.
14. Berger, T., Ivler, C. M., Berrios M. G., Tischler, M.
B., and Miller, D. G., “Disturbance Rejection Handling
Qualities Criteria for Rotorcraft,” Proceedings of the 72nd
Annual Forum of the American Helicopter Society, West
Palm Beach, FL, May 17-19, 2016.
15. Alvarenga, J., Vitzilaios, M., Rutherford, M., and Vala-
vanis, K., “Scaled Control Performance Benchmarks and
Maneuvers fro Small-Scale Unmanned Helicopters,” Pro-
ceedings of the 54th IEEE Conference on Decision and
Control, Osaka, Japan, December 15–18, 2015, DOI:
10.1109/CDC.2015.7403168.
11
