Modeling and optimizing military air operations

Abstract

Dynamic programming has recently received significant attention as a possible technology for formulating control commands for decision makers in an extended complex enterprise that involves adversarial behavior. Enterprises of this type are typically modeled by a nonlinear discrete time dynamic system. The state is controlled by two decision makers, each with a different objective function and different hierarchy of decision making structure. To illustrate this enterprise, we derive a state space dynamic model of an extended complex military operation that involves two opposing forces engaged in a battle. The model assumes a number of fixed targets that one force is attacking and the other is defending. Due to the number of control commands, options for each force, and the steps during which the two forces could be engaged, the optimal solution for such a complicated dynamic game over all stages is computationally extremely difficult, if not impossible, to propose. As an alternative, we propose an expeditious suboptimal solution for this type of adversarial engagement. We discuss a solution approach where the decisions are decomposed hierarchically and the task allocation is separate from cooperation decisions. This decoupled solution, although suboptimal in the global sense, is useful in taking into account how fast the decisions should be in the presence of adversaries. An example scenario illustrating this military model and our solution approach is presented.

Full text

Page 1

Modeling and Optimizing Military Air Operations
Mariam Faied and Anouck Girard
Abstract—Dynamic programming has recently received sig-
nificant attention as a possible technology for formulating
control commands for decision makers in an extended complex
enterprise that involves adversarial behavior. Enterprises of
this type are typically modeled by a nonlinear discrete time
dynamic system. The state is controlled by two decision makers,
each with a different objective function and different hierarchy
of decision making structure. To illustrate this enterprise, we
derive a state space dynamic model of an extended complex
military operation that involves two opposing forces engaged
in a battle. The model assumes a number of fixed targets
that one force is attacking and the other is defending. Due to
the number of control commands, options for each force, and
the steps during which the two forces could be engaged, the
optimal solution for such a complicated dynamic game over all
stages is computationally extremely difficult, if not impossible,
to propose. As an alternative, we propose an expeditious
suboptimal solution for this type of adversarial engagement.
We discuss a solution approach where the decisions are de-
composed hierarchically and the task allocation is separate
from cooperation decisions. This decoupled solution, although
suboptimal in the global sense, is useful in taking into account
how fast the decisions should be in the presence of adversaries.
An example scenario illustrating this military model and our
solution approach is presented.
I. INTRODUCTION
Since 1916 when Lanchester developed the basic founda-
tion of the erosion models for modern warfare [7], they have
received significant attention [2], [12]. This mathematical
analysis for calculating attrition of forces in an air mission
is very useful in providing a mathematical representation
of event probabilities, and operations. In the beginning and
due to the fact that these models were not expressed using
state space algorithms [1], [11], [6], they were not amenable
for the application of optimal control and dynamic game
theory. The emerging field of modeling and controlling
such an extended complex enterprise [3] seems to provide
an appropriate mechanism for analyzing and formulating
command and control actions for modern military operations.
An enterprise of this type is typically modeled as a highly
complex nonlinear dynamical system that is controlled by
two or more adversarial decision agents. The states are
influenced by the environment and the objective functions.
Recently, state space models of an extended complex military
operation that involves two opposing forces were developed
[5], [4]. Motivated mainly by military Unmanned Air Vehi-
cle (UAV) operations and inspired by these state models,
This work was supported by The Ministry of Higher Education in Egypt,
AFRL and AFOSR under grant number FA 8650-07-2-3744.
Allauthorsarewith theDepartment
ing, The University of Michigan, Ann Arbor, MI 48109-2140 USA
mfaieda@umich.edu
ofAerospaceEngineer-
we propose this paper. This work is different from the
previous models in the command constraints that control
the game, the underlying process description, and also in
the solution methodology. Along with our previous work
in [9], [8], [10], this paper proposes one of the first UAV
coordination algorithms that enable a number of UAVs to
achieve autonomous battle management in an adversarial
environment. In our previous work [9] adversarial UAV
operations and a computationally effective algorithm were
presented for a SEAD (Suppression of Enemy Air Defenses)
type military mission. In [8], the algorithm was represented
in the framework of Dynamic Network of Hybrid Automata
(DNHA). This algorithm was further improved to incorporate
MTL type constraints in [10].
In this paper, our problem includes both combinatorial and
stochastic complexity, and represents the military uncertain
and adversarial environment. More precisely, we consider
a SEAD mission, in which the Blue force and the Red
force engage in multiple stages. The Red force is composed
of multiple targets which have to be engaged individually.
It is assumed that these targets cannot be engaged in any
order, but the set of feasible orderings of the targets will
be calculated. In our model, we assume that the Red force
can counterattack and destroy the Blue force with some
known probability. The Blue force controller, however, can
attack a target with multiple UAVs to increase the probability
of destruction of the target which of course will introduce
the extra cost and risk of employing more UAVs in the
mission. Thus in the beginning of each stage a decision
about which target to attack and which set of UAVs to attack
with should be made. Then, the engagement between the two
forces in this stage will take place according to the command
constraints. In the next stage, and according to which target
has been destroyed in the previous stage, the state of the
game will be determined. This model is dynamic in nature
with state variables whose evolution with time depends on
the choice of control actions by each of the two forces. It
is extended in the sense that the effects of each opposing
force action, and of the environment are explicitly included in
the model. In addition to the battle dynamics, we model the
objective functions of each force and the control commands
which will be calculated at each stage to optimize the state
of the game. Consider a discrete-time system controlled by
two independent decision-makers whose state vector evolves
according to the equation:
xk+1= fk(xk,u1
k,u2
k), k = 0,...,N − 1
kand u2
(1)
where xkis the state vector, u1
control sequences of the two opposing decision makers and
kare the independent
Joint 48th IEEE Conference on Decision and Control and
28th Chinese Control Conference
Shanghai, P.R. China, December 16-18, 2009
ThCIn6.11
978-1-4244-3872-3/09/$25.00 ©2009 IEEE 6274

Page 2

x0is the initial condition at k = 0. Each team decision maker
is trying to optimize a performance index of the form:
Ji= φi
N(N,xN) +
N−1
?
k=0
Li(xk,u1
k,u2
k), i = 0,1,,N − 1
(2)
Where φi
of the process, and Li(xk,u1
time k. Over the interval [0, N], obtaining a game-theoretic
optimal solution for such a system is not only complex
but also the complexity will rise exponentially with the
length of the time horizon N. In order to overcome these
difficulties, we represent our decoupled solution which will
use an off-line general policy for the battle. This policy yields
threshold solutions which can be used on-line with minimal
computational effort.
This manuscript is organized as follows. In section II we
introduce the state space model of this extended complex
military operation. Then in Section III, we introduce the
computational complexity for solving this model. Section
IV describes our solution approach. Later in section V we
propose an example scenario illustrating our approach. The
paper ends with conclusions.
N(N,xN) is the terminal cost incurred at the end
k,u2
k) is the cost incurred at
II. AN EROSION MODEL FOR A MILITARY OPERATION
In our model there are two forces, the attacking force
(Blue force) and the defending force (Red force). The
Blue force consists of “Bombers” as the Master UAV (BM)
and “ Jammers” as the slave UAV (BS). The “Bombers”
are essentially units whose purpose is to attack and suppress
the Red air defenses and the Red units. The “Jammers” can
support the “Bombers” and combine with each other as a
team according to the risk assessment during the battle.On
the other hand, the Red force consists of Red Defense Units
(RDUs) such as surface-to-air missiles (SAMs) in addition to
Red Fixed Targets (RFTs) such as command center, bridges.
A. Unit State Space Representation
In this game, the Blue force is planning to attack and the
Red force is planning to defend. Each unit is fully described
by its state vector. That is:
zm
i(k) =


xm
ym
wm
i(k)
i(k)
i(k)

, m = {BM,BS,RDU,RFT} (3)
where i= 1,2,...,Nm
Here, x and y correspond to the x-coordinate and the y-
coordinate at time step k, respectively; w corresponds to
total number of weapons left at time step k in that unit, and
Nmto the total number of units of m type. Thus for each
moving unit in the theater of operations, we define a three
dimensional state variable. Combining all the state variables
for each type of forces into one vector, we have:


zm(k) =

zm
...
zm
1(k)
Nm(k)

.

(4)
In a similar way, we can write the overall state vector
corresponding to the Blue and Red forces as follows:
?
The RFT units don’t carry any weapons, so size of the
combined state vector for the Blue force as well as the Red
force will be a 3x(NBM+ NBS+ NRDU)+2x(NRFT).
zB(k) =
zBM(k)
zBS(k)
?
and zR(k) =
?
zRDU(k)
zRFT(k)
?
(5)
B. Control Variables
The control variables for each unit are chosen from a finite
set of choices. They are divided into three types:
1) Motion Control dm
of the battlespace as a rectangular grid to provide a compu-
tationally efficient abstraction for motion control. Each unit
can move to another adjacent point on the grid; that is, the
maximum distance traveled at each step is given by:
?
where ∆xm
∆ym
Because we restrict the units to be moving along the “streets”
of the grid, there are four neighborhood locations that each
unit can relocate to, and the option of no movement from
the current location at this time step k .
2) Fire Controls rm
wm
unit decides to fire at time step k , that is rm
at each time step, a unit can fire only at one target of the
opposing forces. Let cm
for unit i of type m at time step k.
3) Team Configuration Control fm
for each force, we have two configurations: Cooperative
teams or Independent units. Every unit in each force has
the option to decide between these two configurations.
i(k): We discretized the geography
dm
i(k) =
∆xm
∆ym
i(k)
i(k)
?
;
(6)
i(k) corresponds to a step in the x-direction and
i(k) corresponds to a realignment in the y-direction.
i(k), cm
i(k): Each unit that has
i(k) > 0 has the option to fire or not to fire. When a
i(k) = 1. Also
i(k) represents the choice of target
i(k): We assume that
fm
i(k) =
?
1 if Cooperative team
0 if Independent unit
?
(7)
Combining all these command variables into one control
vector, we have the following control vector for each unit:
um
i(k) =



∆xm
∆ym
rm
cm
fm
i(k)
i(k)
i(k)
i(k)
i(k)



(8)
For each unit, we have 5-dimensional control vector. The
composite control vector for all units in the Blue and Red
forces is 5x(NBM+NBS+NRDU). Note that the Red Fixed
Target doesn’t have this control command because they are
neither moving nor shooting units.
ThCIn6.11
6275

Page 3

C. Command Constraints
These control variables are subject to different types of
constraints: Logic constraints, and Kinematic constraints.
The Logic constraints are mainly trying to manage the firing
maneuver, and the Kinematic constraints do the same thing
with the moving maneuver. The Logic constraints can be
summarized into:
• Units may not move and fire simultaneously.
δm
i(k) + rm
i(k) ≤ 1.
i(k) is defined as follows:
(9)
In the above relation, δm
δm
i(k) =
?
i(k) is equal to 1 if the unit decides to move
to an adjacent location and is equal to 0 otherwise.
• A unit will not be able to fire at a unit of the other force
unless they are both in range (this constraint is related
to the range of weapons and requires that both units be
??l??squares apart on the grid for the weapon from one
side to reach the target on the other side).
??xX
unit j of type Y unless the distance between them is
less than a predefined distance li.
• Units on either force have a limited number of weapons
and any unit can not fire at step k unless wm
?
• Enemy units can be targeted only by the BM or a
combination of the BM and BS.
• No two units of the same force can fire at the same
target of the opposing force at the same time step k:
0, if ∆xm
1, otherwise
i(k) = 0 and ∆ym
i(k) = 0
(10)
That is, δm
i− xY
j
??+??yX
i− yY
j
??≤ li.
(11)
Eq.(11) represents that a unit i of type X cannot fire at
i(k) > 0.
rm
i(k) =
1, if wm
0, otherwise
i(k) > 0
(12)
NBM
?
i=1
rBM
i
(k) +
NBS
?
i=1
rBS
i
(k) ≤ 1.
(13)
for each Red target,
NRDU
?
i=1
rRDU
i
(k) ≤ 1.
(14)
for each Blue target.
As we can see, some of the Logic constraints are naturally
imposed and some we specified for the game. We use
steerable unicycle Kinematics on the Manhattan grid for our
example application. This level of abstraction is sufficient to
evaluate the architecture and provides fast simulation results.
Any unit will be constrained to move along the streets at a
constant speed. We can describe the motion of these types
of vehicles by the following discrete equations:
?
∆xm
∆ym
i(k)
i(k)
?
=
?
v cosψm
v sinψm
i(k)
i(k)
?
(15)
Here, ∆xm
displacement and the y-displacement on the grid, respec-
tively; v is the linear forward velocity and ψm
orientation of the unit.
i(k) and ∆ym
i(k) correspond to the x-
i(k) is the
D. Model State Space Representation
The state equations for this military model are defined as
?
xm
ym
i(k + 1)
i(k + 1)
?
=
?
xm
ym
i(k)
i(k)
?
+
?
∆xm
∆ym
i(k)βw(k)
i(k)βw(k)
?
(16)
wm
i(k + 1) = wm
i(k) − rm
i(k)
(17)
Nm
i(k + 1) = Nm
i(k) − Lm
i(k)
(18)
Lm
i(k) = Tm
i(k)Am
i(k).
(19)
Eq.(17), (18) and (19) for weapons and unit losses are of
the erosion type, where βm
are weather effects, the number of targeted units, the total
number of units, and the number of lost units respectively.
The term Am
at step k surviving the transition to step k + 1 and Tm
represent the number of targeted units at step k. For this
model, we allow only one-on-one engagement at each time
step. Once the identities of the attacking and the attacked
units are determined from the choice of target controls, this
percentage can be expressed as:
i(k), Tm
i(k), Nm
i(k), and Lm
i(k)
i(k) in (19) is the percentage of units targeted
i(k)
Am
i(k) = 1 − βXY
ij PXY
ij
(k).
(20)
In (20) the term βXY
jth unit of Y acquires the ith unit of X as a target. The
other term in (20), PXY
ij
, is called the “erosion factor” and
can be computed from
ij
represent the probabilities that the
PXY
ij
= 1 − (1 − βw(k)PKXY
ij )
(21)
Eq.(21) represents the probability of the ith unit of X being
destroyed by a weapon fired from the jth unit of Y . The
term PKXY
ij
represents the probability that the ith unit of
X is destroyed by a weapon fired from the jth unit of Y
under ideal weather and terrain conditions. The factor βwin
Eq.(21) and Eq.(16) accounts for weather instability effects.
E. Objective Function
Although in the general model, every unit can have its
own objective function, in this paper, we consider a global
objective functions for all units of one force that take part
in the operation. We define that aggregate objective function
for the Blue force in the following way:
ThCIn6.11
6276

Page 4

JB
k(xk,u1
k,u2
k) = max
uk
E[
NBM
?
i=1
αBMiPBM
uk(k)TBM
i
(k)
+
NBS
?
NRFT
?
NRDU
?
i=1
αBSiPBS
uk(k)TBS
i
(k)
−
i=1
αRFTiPRFT
uk
(k)TRFT
i
(k)
−
i=1
αRDUiPRDU
uk
(k)TRDU
i
(k)]
(22)
The Red force’s objective function can be written in a
similar way. In this equation, Blue team tries to find the
suitable configuration ukto maximize its objective function.
Each objective function is formed of the force’s reward (over
all its units) minus a penalty over the enemy force’s units.
Note that if the weighting coefficients αXi=βXifor every
k in eq.(22), we will have a zero-sum dynamic game.
III. COMPUTATIONAL COMPLEXITY
Discrete-time dynamic games in general suffer from the
curse of dimensionality. If the decision-makers have a finite
set of decision variables at each stage, then the number of
all possible sequences choices over the entire game grows
exponentially with the number of stages. In our problem, we
have a finite set of control variables for every unit in the
battle. This set is determined by the set of allowable moving
controls dm
choice of fire rm
trols fm
for the adjacent locations and the “no movement” control. So
for motion control, we have 5 x (NBM+ NBS+ NRDU)
possible actions. The number of target choices for any force
at time k is equal to the number of the opposite force units.
A unit ith of the Blue force has NRDU+ NRFTchoices
for the next target and vice versa. Thus for the target choice
control cm
choices for the Blue force and (NRDUx (NBM+ NBS))
for the Red force. For the fire choice rm
either force has the option to fire or not to fire, that is 2
x(NBM+ NBS+ NRDU) control choices. Our last control
command is fm
is 2 x (NBM+ NRDU). Thus, the total number of choices
for the ith unit in the Blue force at time step k is equal to
the number of enemy units NRDU+ NRFTthat could be
attacked. For each unit in the Red force, the ith unit in the
Blue force should decide on a fire choice in a space that
is 2x(NRDU+ NRFT), and also on the team configuration
control, which is another 2x(NRDU+NRFT) space choices.
The total number of choices for the ith unit in the Blue force
at time step k is therefore given by:
i(k), the set of choice of target cm
i(k), and the set of team configuration con-
i(k). For motion control, every unit has four choices
i(k), the set of
i(k), we have ((NBM+NBS) x (NRDU+NRFT))
i(k), each unit in
i(k); for this one the command vector length
nB
i(k) = 5 + 4x(NRDU+ NRFT)
(23)
This number will be multiplied by the number of units in the
Blue force, in order to get the total number of choices for the
Blue force in one stage. These numbers of different control
sequences for each force will be multiplied by the number
of stages K, in order to get the total control commands for
the entire game. This exponential growth in the number of
control sequences will lead to an equally huge number of
states that need to be determined. It will therefore become
computationally unfeasible to determine an optimal solution
for the game over its entire duration of K time steps. In the
next section, we will provide our solution approach for this
complicated problem.
IV. SOLUTION APPROACH
The optimal solution for such a composite game over all
stages is difficult, if not impossible, to obtain. In order to re-
duce the computational requirements, instead of maximizing
the objective function over the entire time horizon, we only
consider locally optimal sub-solutions to the problem. We
deal with each control command separately from the others.
After choosing each control command individually, we used
the whole set of control commands together to run the game.
In the following, we will describe each control methodology.
We specify our solution approach in a specialization of the
scenario described above for readability and to provide a
simulation example.
A. Motion Control dm
Assume that all the Blue force assets have the same speed
and all the Red units move at a much lower speed so that
effectively they appear stationary. The control and the state
variables will be updated every time step. The path from a
ith unit of type X in the Blue force to a jth unit of type
Y in the Red force is generated using the shortest path cost
function. That is
???xX
So the ith unit of the Blue force will determine its ∆xi(k+1)
and ∆yi(k+1) sequence according to the path that minimizes
the cost function in eq. (24).
i(k)
JXY
ij
= min
ij
i− xY
j
??+??yX
i− yY
j
??
(24)
B. Fire Control
There are two control variables related to this command.
1) Target Choice cm
choice of the next target for the ith unit of type m at time
step k. The cost function we use for choosing the next target
is a risk assessment cost function. An ith unit of type X of
the Blue force chooses its next target j of type Y according
to the cost function:
i(k): The first one is cm
i(k), the
cXY
ij (k) =
NY
min
j=1hXY
ij (k)
(25)
Where hXY
unit of type Y being targeted by the ith unit of type X.
Eq.(25) means that at each time step k, an ith unit of type
X will target the lowest risk factor jth unit of all the units
ij(k) represents the risk associated with the jth
ThCIn6.11
6277

Page 5

of type Y . That risk factor is calculated for each jth unit of
type Y according to the equation:
hXY
ij
=
?
(aY
j+
UY
?
j=1
hY
j)
(26)
Here aY
shooting ability for the type Y unit. So the risk factor for
the jth unit of type Y is this aY
between all units of the same type, added to the number of
units of type Y that can shoot the ith unit of type X in its
current location. Note that UY⊂ NY, and this number can
be determined according to eq. (11) which represents the fire
distance condition.
2) Fire Choice rm
unit in either force has the ability to shoot. That is:
jis a factor specified a priori to represent the known
j, which is a common factor
i(k): At every decision step k, only one
NBM
?
i=1
ri(k) +
NBS
?
i=1
ri(k) +
NRDU
?
i=1
ri(k) ≤ 1
(27)
According to the above equation, shooting will occur sequen-
tially. The Blue force will start the shooting sequence until
all involved units of different types of the Blue force finish,
then the Red force will have the ability to shoot back.
C. Team Configuration Control fm
All units in either force have two different team configu-
rations to choose from: Cooperative team and Independent
unit. For the Red force, we assume that all of the RDU will
be located around the RFT trying to defend them. So all of
the RDU and the RFT will work as a team until the RFT are
rendered inoperative. When they work as a team, they have a
higher shooting probability. When the RFT turn inoperative,
the RDU automatically change to enter the “Independent
unit” mode. For the Blue force, we define a team as a
grouping of n units which have a common mission, where
n is unity or greater. Note that it is possible for a team to
be composed of only one unit. We assume that if we have
only one unit, it will have more ability to shoot and hide. A
team will have a higher collaborative shooting ability but it
will also be easier to track and shoot it. We use Stochastic
Dynamic Programming (SDP) to solve this trade-off game
according to the equation (22). Here αB is the weighting
coefficient for the Blue force objective function, Pm
the shooting ability for the units of type m according to their
configuration fk, and Tm
by ith unit of type m at time step k. Normally the SDP
methodology yields a general strategy to control the entire
game. Starting from time step k = N, going backward to
k = 0, we find the maximum cost function at this step by
comparing the “Cooperative team” cost function with the
“Independent unit” cost function. Following these steps at
each time step, we obtain a threshold solution. Over the
entire time horizon N, this threshold strategy will control
the choice of fm
PRDU
fk
(k) of the opposite force is higher than the threshold
shooting probability, then the best solution is “Cooperative
i(k)
fk(k) is
i(k) is the number of units targeted
i(k) at each time step. If the shooting ability
TABLE I
DESTRUCTION PROBABILITIES OF THE BLUE AND RED FORCE UNITS IN
THE SCENARIO
Configuration
Cooperative
Independent
CC alive
CC inoperative
BM
0.6
0.5
-
-
BS
0.6
0.5
-
-
RDU
-
-
0.4
0.1
team”. Otherwise the “Independent unit” shooting will yield
the maximum cost function.
V. SIMULATION EXAMPLES
Our scenarios take place on a 10x10 square grid. Each
square on the grid corresponds to 40x40 square miles in
dimensions. The Blue force consists of two Unmanned Air
Vehicle (UAV) units: one Bomber and one Jammer. The
Bomber can operate independently or cooperatively with
the Jammer. The configuration will be adapted based on
the Red force’s adversarial behavior. For all configurations,
UAVs should fulfill and meet the cost function and the
required command specifications. The mission of the Blue
force is to destroy one RFT, which is a Command Center
(CC), that is heavily defended by a number of RDUs. We
consider a zero-sum game, in which the Blue force’s mission
will be considered accomplished when the RFT and the
RDUs are destroyed completely. After relaxing the weapons
command, we assume that when engagement occurs, the
forces continue optimizing their controls online until the goal
is accomplished. We tried different configurations for the
Red force in the simulation. In the next section, we present
a heterogeneous scenario for the Red force.
A. heterogeneous distribution for the Red force assets
We use a configuration as in fig. 1, in which the SAM
sites are located in a heterogeneous distribution around the
command center. In this configuration, the UAVs can start
shooting the command center first from the opening locations
and then go to the SAM sites according to their risk factor.
Since we will not be able to show any animation of the
units’ movement or the unit erosion, the scenario’s initial
conditions are shown in fig. 1.
Table I summarize the destruction probabilities of each
unit in the Blue force and the Red force respectively ac-
cording to its configuration. We assume that the destruction
probability Pm
unit as much as it depends on the unit team configuration. In
the simulation, we assume ideal weather conditions (βw= 1)
and equal weighting coefficients for αXi and βXi (zero-
sum game). All of these probabilities and coefficients can
be adjusted by the top commander.
In the simulation, the controls that govern this engagement
for both sides are calculated each according to its strategy.
For the Red force, as long as the Command Center is
working, the RDUs will defend as a team with higher
detection and shooting capability. By working as a team,
fk(k) of each unit does not depend on the facing
ThCIn6.11
6278