Holistic Methodology for Stochastic Mission Utility Analysis

Abstract

Preliminary analysis and design of a mission has profound effects on its lifecycle; yet, mission engineering is a discipline without unanimous definition and implementation. Current mission design philosophies are inadequate due to disaggregated design methods, incomplete mission phasespaces, exclusion of constitutive relations, curtailed tradespace exploration and unsubstantiated concept solutions. Presented is a new methodology for simultaneously developing, exploring and assessing statistically validated concepts of prospective missions. Stochastic modelling and simulation, providing multivariate mission utility analysis with concurrently integrated tradespace exploration, risk assessment and holistic design facilitates the methodology.

Full text

Holistic Methodology for Stochastic Mission Utility Analysis
Ja’Mar A. Watson
Watson Institute for Scientific Engineering Research
4201 Wilson Boulevard, Suite 110-444
Arlington, VA 22203, USA
Email: jamar.watson@sciengresearch.org
Abstract: Preliminary analysis and design of a mission has profound effects
on its lifecycle; yet, mission engineering is a discipline without unanimous
definition and implementation. Current mission design philosophies are
inadequate due to disaggregated design methods, incomplete mission
phasespaces, exclusion of constitutive relations, curtailed tradespace
exploration and unsubstantiated concept solutions. Presented is a new
methodology for simultaneously developing, exploring and assessing
statistically validated concepts of prospective missions. Stochastic
modeling and simulation, providing multivariate mission utility analysis
with concurrently integrated tradespace exploration, risk assessment and
holistic design facilitates the methodology.
Keywords: Mission Engineering; Mission Utility Analysis; Stochastic
Design; System-of-Systems Engineering; Mission Simulation; Mission
Modeling; Mission Architecture; Analysis of Alternatives; AoA; Concept
Development and Experimentation; CD&E.
Reference to this paper should be made as follows: Watson, J.A. (2017)
‘Holistic Methodology for Stochastic Mission Utility Analysis’, Int. J.
System of Systems Engineering, Vol. 8, No. 2, pp.174–188.
DOI: 10.1504/IJSSE.2017.088447
1. Introduction
Prospective missions are often nebulous with notional mission systems and
conceptual mission architectures. Theatre-level knowledge is often riddled with axioms,
conjectures, misconceptions, and postulates; yet, mission intelligence is never a global
variable in the concept of operations (CONOPS). Mission environments present uncertain
boundary conditions but operational weather is never holistically captured. Such neglect
makes the achievement of pragmatic solutions infeasible. Future missions often require
nascent technologies and/or callow methodologies – variables that are rarely analyzed for
their contribution to mission objectives. This is due to the omission or separation of
tradespace exploration from utility analysis. Additionally, humans present unparalleled
complexity and variable factors; however, the importance of ergonomics in the CONOPS

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
2
is often unappreciated in comparison to the emphasis on mission architecture in concept
definitions. In fact, the inclusion of human factors is mostly nonexistent in applicable
mission phasespaces. Often coupled with budgetary, political and temporal constraints,
current techniques to architect, engineer and analyze missions do not encapsulate these
problems. The lack of resolution stems from design philosophies that do not truly analyze
the mission. Their techniques often ignore utility for representation of one of its
constituents. For that reason, it is unclear if proposed missions under current design
methods actually accomplish their objectives; thus, these solutions are unsubstantiated.
Without concurrent consideration of utility, it is not feasible to conduct meaningful mission
concept development and experimentation (CD&E) or analysis of alternatives (AoA).
Presented is a new methodology providing multivariate mission utility analysis with
concurrently integrated tradespace exploration, risk assessment and holistic design via
stochastic modeling and simulation (M&S). It facilitates simultaneous development,
exploration and assessment while statistically validating the prospective mission concepts.
2. State-of-the-Art
Currently, abstract and disaggregated system design processes plague mission
engineering. Most mission analyses are nothing more than a conglomeration of system
designs and/or mission phases – not a true analysis of the mission. This approach deprives
interoperability of design choices by ignoring the holistic, system-of-systems nature of
mission architectures. Such truncation eliminates emergent concept behaviors and designs,
reducing CD&E to meagre trade studies.
The cost of tradespace exploration in current design methods is inordinately expensive.
Such cost has led to the omission or severe restriction on front-end analysis. This often
attributes to greater mission ambiguity since the preliminary mission design phase
produces a greater differential between concept and operation.
The current reliance on testing and evaluation (T&E), both developmental and
operational, is too great among today’s diversity of missions. Additionally, it is often
impractical or impossible to perform traditional T&E through progressive demonstration
and/or test missions. Such is the case in many defense and space exploration missions, for
example. It is therefore of utmost importance to systematically replicate these processes
during the CD&E front-end analysis. Failure to do so will result in significant reduction of
capabilities and operations simply due to the shortcomings of mission design methods.
Previous research has shown that developing complex missions with rapidly
advancing technology is inefficient when one stage of analysis is nearly complete before
another begins (Friedman and Weber, 2004). This is due to premature stagnation of critical
design variables in preceding stages. More often than not, future missions require nascent
technologies and/or callow mission methodologies, meaning the introduction of concurrent
engineering to the proposed methodology is beneficial to mission design mechanisms.
Concurrent engineering allows mission design to be directed by all knowledge concerning
the architecture and its operational environment.
The lack of efficiency in current mission design methods has become so
overbearing that cost often replaces success as the primary mission driver. This has caused
the loss of capabilities and abandonment of principles governing the acquisition of critical
design variables in preceding stages. More often than not, future missions require nascent
technologies and/or callow mission methodologies, meaning the introduction of concurrent

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
3
engineering to the proposed methodology is beneficial to mission design mechanisms.
Concurrent engineering allows mission design to be directed by all knowledge concerning
the architecture and its operational environment.
The lack of efficiency in current mission design methods has become so
overbearing that cost often replaces success as the primary mission driver. This has caused
the loss of capabilities and abandonment of principles governing the acquisition of mission
programs (Defense Science Board and Air Force Scientific Advisory Board, 2003). The
result of such has introduced failures in balancing schedule, cost, and scope, giving birth
to the triple constraint theory denoting the infeasibility of simultaneous improvement under
current methods (Van Wyngaard et al., 2012). The effects of this theory are potent – even
before consideration of other leading constraints such as risk. Missions in many disciplines
do not have standard cost; therefore, it is difficult to base the efficacy of a new paradigm
on its reduction. This has led to complacency in accepting any cost reduction as indication
of exemplary methods. With absence of knowledge for true minimal cost in mission
achievement, quantification is transferred to scope, which becomes expressed as mission
utility: the ability to satisfy mission goals and objectives.
The analysis of mission utility requires the relaxation of the notion of requirements.
Requirements are the leading driver for increasing the scope-to-cost ratio, but this is only
true if the requirements are expressed functionally with quantitative degrees of satisfaction.
This is philosophically comparable to Pareto’s 80/20 principle, which translates 80% of the
mission’s scope can be accomplished with 20% of the mission’s cost (Vasile et al., 2014).
Historically, requirements are viewed as inflexible and the cheapest mission to accomplish
them is designed irrespective of this ratio. This has single-handedly restrained
advancements in mission design. With the removal of the traditional definition for
requirements, quantification is transferred to measures of effectiveness (MoE) and figures
of merit (FoM), which include the variables of cost, schedule and scope. It also includes
risk, which quantifies a design’s ability to transfer cost into scope. It is therefore mission
utility analysis that serves as the foundation for this philosophy and M&S has become an
essential mechanism for this in mission engineering.
3. Background
For decades, M&S has helped ease the burden of complexity in mission engineering
(Sierhuis et al., 2006) by allowing computers to mimic traditional laboratories with reduced
T&E costs (Rainey et al., 2004). Traditional tests are run in analytical scenarios: one that
displays the operation of a system without doing so in a real or all fully plausible situations.
This introduced the practice of designing for worst-case scenarios of subsystems, not only
dooming the architecture by provoking unnecessarily constrained size, weight and power
budgets, but also producing an architecture built for unrealistic scenarios that may not
occur simultaneously. This realization led to the integration of Monte Carlo methods that
iteratively run tests with varying scenarios to collect data on multiple runs. This is
impractical for entire mission concepts with traditional T&E methods but can be performed
using Monte Carlo simulations for a fraction of the cost while encompassing almost all
likely scenarios (Conway, 2015).
However, these benefits only transferred to the subsystem level making it difficult
to understand how design choices alter mission accomplishment, as well as their impact
across the architecture both in terms of performance and operation. When determining how

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
4
an architecture can achieve a desired outcome in the mission, realistic scenarios need to be
developed in which to test architectural performance. This is mission utility analysis and
the only way to perform it is with a simulation of the mission (Wertz, 2011, 2008). It then
becomes clear that individual Monte Carlo analysis is incomplete and there must be a
simulation of the entire mission. A simulation as such in a realistic environment
incorporating a comprehensive model of the mission concept is absent in current design
methods.
Now that the entire mission is being considered, new issues arise when trying to
use traditional methods. At the architectural level, the problem set contains multitudes of
parameters and AoA of a concept can become too complex to determine the interoperability
and trades between choices. To eliminate ambiguity, tradespace exploration is introduced
to perform the analysis; however, effective AoA performed in a realistic mission
environment is still needed.
The integration known as multi-attribute tradespace exploration with concurrent
design (MATE-CON) can be performed to simultaneously design, explore and evaluate
potential architectural concepts from the tradespace (Ross et al., 2004). In the end a method
for the assessment of utility using an AoA that provides rationale for potential solutions –
utilizing mission objectives to evaluate and compare mission architectures in developing
and refining a mission concept is developed. This method is unparalleled in mission
engineering, where it is becoming practice that systems are brought to conformance with
the simulations instead of the contrary (Conway, 2015).
What is considered a current advancement in MATE-CON is the inclusion of
uncertainty and this is an absolute must in utility simulation of prospective missions (Ross
and Hastings, 2005). The ability of stochastic M&S to both answer and produce emergent
questions plays a crucial role in this addition. The greatest advantage of Monte Carlo
simulations is the ability to give designers knowledge of occurrence likelihoods. With such
a capability, the shortcomings of traditional risk assessment and reliability analysis can be
augmented. Traditional risk assessment and reliability analysis methods have proven to be
unreliable in forecasting failures and uncertainty because they arise from contributions not
modelled in the analysis of proposed mission architectures. The methodology presented in
this paper, incorporating many sources of uncertainty while providing holistic and
comprehensively stochastic analysis of all potential architectural and operational designs
in a realistic mission simulation, is the launch pad for deploying the mission utility analysis
methodology needed to produce ‘faster, better, cheaper’ missions.
4. Objectives
In order to simultaneously address the aforementioned problems the following
objectives are achieved.
Objective 1: Quantify Mission Goals – Facilitating the computation of mission utility
requires the quantification of mission objectives. This is the introduction of implicitly
flexible and functional requirements presented with purposeful generality and freedom of
achievement path as to not inadvertently introduce bias or hinder solution domain. They
are outlined to allow selection of all possible outcomes in satisfying the chosen mission
goals. The objective is to quantify mission goals that replicate the purpose of the mission.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
5
Objective 2: Enable Mission Utility Simulation – The only way to truly assess mission
utility is with a simulation of the mission. Accordingly, it is an objective to simulate the
mission, varying mission concept components that affect mission utility. It is the results
from these mission utility simulations that feed the analytics for AoA and CD&E.
Objective 3: Incorporate Precursor Missions – History does not suggest complex
missions are attempted in the first deployment design. Therefore, an objective of this
methodology is to quantify the effects of potential precursor missions on final mission
utility to determine their necessity.
Objective 4: Incorporate Mission Uncertainty – A substantial challenge of mission
utility simulations for future missions is the potential level of uncertainty. When present,
initial models are undoubtedly dubious. Therefore, the composition of the mission as a
model should be treated as nothing more than a hypothesis accompanied by constituent
percent probabilities of falsehood. These likelinesses of true/false should be updated by
real-time data in the simulation to provide a realistic portrayal of information needs
satisfaction in the mission. The objective is to replicate the unfolding of a live mission
while also being able to determine how initial mission uncertainty affects mission utility.
Objective 5: Incorporate Operational Weather – This is also true for the constitutive
relations that complete comprehensive simulation of the concept. External factors, such as
mission environments and potential operational weather, are critical components to
simulating a realistic mission. It is therefore also an objective to finesse these factors into
the utility analysis.
Objective 6: Incorporate Architectural Uncertainty – Another objective is to capture
the uncertainty in the mission architecture. This can arise due to their current level of
conception or uncertainty in system selection – both of which affect the physical models’
state of systems in the simulated mission. Therefore, inclusion of an exhaustive mission
ensemble is critical with stochastic parameters to replicate this uncertainty. Chaos theory
predicts that even the minutes of changes at mission initiation can have profound impacts
on final mission utility.
Objective 7: Incorporate Human Factors – It is an objective of this methodology to
ensure comprehensive depiction by reflecting human factors. Mission design perspective
and perceived theatre-level knowledge are factors that arise due to humans designing,
occurring in and/or operating the mission. In missions where humans represent mission
systems or payloads, their physiological and psychological effects on the mission must be
captured. The impact of ergonomics on both the design and analysis of the architecture, as
well as the CONOPS, has profound effects on mission utility and the simulation’s
resemblance to a realistic mission.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
6
Objective 8: Incorporate Operational Uncertainty – Mission simulation comes with a
large uncertainty of mission outcomes associated with operational aspects of the model.
The objective is to provide a solution to handle this uncertainty and account for all possible
mission evolutions and outcomes, including not just mission aborts but also a hierarchy of
contingencies. In this approach, it is the aggregated results over all possible mission
phasespace trajectories that represent an accurate calculation of mission utility.
Objective 9: Assess Mission Metrics – Determination of mission utility requires
quantification of mission metrics representative of the mission objectives. Calculation
across all concept alternatives enables a substantiated solution. Therefore, an objective of
this methodology is to assess the mission metrics to determine an appropriate mission
concept through AoA and CD&E.
Objective 10: Architect a New Mission Concept – The beauty of mission utility analysis
is in its ability to provide comprehensively integrated solutions at the early stage of concept
assessment. Identifying often unseen and counterintuitive problem statements allows
effective determination of technological, systematic, architectural and/or operational
deficiencies. Therefore, the objective is to use the results of mission utility analysis to
provide a platform for statically validated mission concept improvements – effectively
giving a quantifiable measure of this methodology’s contribution.
5. Methodology
While there may be many pathways to achieve the objectives outlined, the
following tasks represent the framework of this methodology. Figure 1 provides a visual
representation of the workflow and an introduction to the mechanisms of the methodology:

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
7
Figure 1: Methodology Flow Diagram
Task 1: Define Mission Objectives – Utility is a quantifiable metric of a mission’s ability
to satisfy mission objectives. In order to perform mission utility analysis, the mission
objectives shall be defined, their means of satisfaction identified and their contributions to
the computation of utility mathematically outlined.
Task 2: Define Metrics – Though they are often used interchangeably, two distinguishable
definitions for MoE and FoM are introduced to facilitate mission concept evaluation and
comparison. MoE are measures of satisfaction of task 1’s mission objectives to determine
mission effectiveness. Utility shall be a constituent of the MoE. On the other hand, FoM
are values that do not directly assess a mission’s effectiveness in satisfying mission
objectives but provide value to alternatives across the solution domain. Both the purpose
and the method of calculation shall be defined for these metrics.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
8
Task 3: Model the Mission Arena – The mission arena: the stage on which all mission
simulations occur. The arena shall be completed with inclusion of modelled constitutive
relations such as operational weather. The fidelity of the model shall be sufficient to
provide a statistical significance to mission arena variables that affect mission metrics.
Task 4: Model the Mission Architecture – The modeling shall conform to all mission
systems that make up each potential mission architecture. This means a model shall be
formulated for all considered heritage, state-of-the-art and nascent mission
technologies/systems, as well as the emergent and interoperable behaviors of each potential
architectural conglomeration. In essence, capturing the mission ensemble with applicable
command, control, computers, communications, intelligence, surveillance, and
reconnaissance (C4ISR) interoperability successfully completes modeling of all possible
mission architectures.
Task 5: Define the Operational Concept – The CONOPS shall be defined in a fashion
that facilitates realistic simulation of the modelled mission architecture. The concept shall
integrate the impacts of precursor missions, possibilistic mission events and their
respective contingencies, logistics of mission systems, apportionment of mission
operations, and the mechanisms for achieving utility to satisfy mission objectives.
Therefore, this task establishes a definition for phenomenology of the mission.
Task 6: Establish the Baseline – The first simulation to complete is that of the mission as
currently defined or perceived. This shall serve as the reference mission architecture and
shall establish baseline performance results of the metrics, including mission utility.
Task 7: Define the Tradespace – The definition shall enable AoA of the baseline concept
as well as tradespace exploration with the modelled mission ensemble. This will serve as
identification of the solution domain and enumeration of the multivariate inputs. The
tradespace shall be inclusive of all relevant variable inputs (precursor missions,
architectural components, etc.) and stochastic components of the CONOPS (mission
intelligence, possibilistic mission event outcomes, etc.), as well as components of
constitutive relations (operational weather, applicable external factors and stimuli, etc.).
Comprehensive tradespace definition produces a realistic multivariate mission utility
analysis. When the tradespace exploration is concurrent with the utility analysis, its
variable architectural and operational components of the concept further inherently
produce risk assessment and holistic design in stochastic Monte Carlo simulations.
Task 8: Perform Mission Utility Simulations – This produces all of the data to be
aggregated – the results of which shall culminate in the computation of the metrics defined
in Task 2. In the case of multiple mission designs, this shall include all simulations for
AoA. In the case of a desired improvement to the mission concept, this expands to CD&E,
requiring either an exhaustive or experimentally designed tradespace exploration.
Task 9: Perform Analytics and Enterprise Architecting – Mission simulations shall be
followed by a full spectrum of descriptive, predictive and prescriptive (DP2) analytics of
the simulation datasets, along with enterprise architecting. The descriptive portion shall

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
9
provide data analysis of what occurred in the mission simulations. It shall produce metrics
and sensitivity analyses. This shall culminate in the manufacturing of utility functions for
further rapid, algorithmic computation of mission utility. The predictive portion of the
analytics shall produce predictive mission design trends by deriving mission utility as a
function of the mission’s input design domain and inherent stochastic variables. From this,
it is also possible to analyze additional mission concepts whose synthesis is outside of the
initial tradespace. This further allows refinement of the mission problem statement. The
prescriptive analytics shall provide quantitative instruction for improving the mission
concept, including, but not limited to, statistically guided amendments, additional
investigations, contingencies and preliminary principles for performing the mission.
Prescription seeks to expand and give application to the previous steps of analytics. Finally,
enterprise architecting shall relate the analytics to program level actions for the mission.
This includes potential collaboration between entities, identification of stakeholders,
initiation of research and development and/or technological investments required to
develop the mission concept, as well as a potential roadmap for the realization of the
mission. When integrated with the previous tasks, DP2 analytics enables simultaneous and
statistically validated development, exploration and assessment of the prospective mission.
Task 10: Architect a New Mission Concept – As a final task, particularly in CD&E, a
new mission concept shall be architected from the analytics of task 9. This can be presented
as a singular solution or a set of allocations from the derived Pareto frontier or mission
solution ensemble. In principle it shall equal or exceed the highest fidelity and utility of the
baseline and alternative mission concepts. This effectively gives a quantifiable measure of
this methodology’s contribution to the mission.
6. Metrics
While the included metrics along with utility can be highly individual to each
prospective mission, the most prominent MoE and FoM are presented in this section. For
MoE, two values of the mission have the greatest impact on its worthiness: its measure of
success and utility. While the latter deals exclusively with preference (satisfaction of
selected mission objectives), a measurement of success serves as an unbiased discovery
index. It is unwise to use utility as a single metric in missions with multiple, unequal
objectives when their weighting can be considered highly subjective (Friedman and Weber,
2004). In the event mission objectives change or are not unanimous across all decision-
makers, a second unweighted (although possibly unrealistically unweighted) value
provides perspective to the mission’s usefulness. It is also worth noting that mission
objectives used for utility are satiable but represent minimum levels of acceptable
achievement. A measurement of success can tabulate excess performance. Four derived
values – termed ability, productivity, success and utility – constitute the MoE (Table 1): a
value that directly assesses the mission’s effectiveness in satisfying mission objectives.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
10
Table 1: Measures of Effectiveness
MEASURE OF
EFFECTIVENESS
DEFINITION
MEASUREMENT
Ability
Ability of mission to
achieve utility
Number of objectives achieved
(without duplication)
Productivity
Level of mission yield
Number of objectives achieved
(with duplication)
Success
Ability to satisfy
mission sub-objectives
Percent of mission objective
satisfaction (unweighted)
Utility
Ability to satisfy
mission objectives
Percent of mission objective
satisfaction (weighted)
Table 2: Figures of Merit
FIGURE OF
MERIT
DEFINITION
MEASUREMENT
APPLICATION
Cost
Amount of
resources required
Cost, mission duration,
etc.
Lifecycle cost
estimation
Efficiency
Ratio of utility
attempts to utility
obtainment
Percent of ideal
mission utility
Intelligence and
phenomenology
analysis
Flexibility
Ability to operate
with architectural
variability
Deviation of utility
with change in mission
systems
Risk of pursuing the
concept
Fragility
Ability to result in
LoM from LoS
Percent of LoS that
directly result in LoM
Directs design and
contingencies
Preparedness
Preparation for
MD
Percent obtainment of
needs for MD
identification
Phasespace analysis
Producibility
Readiness to
produce concept
Readiness level of
mission concept
Readiness
Assessment
Resiliency
Ability to maintain
utility with LoS
and MD
Deviation of utility
with LoS and MD
Utility variability
Responsiveness
Ability to address
LoS and MD
Response time, percent
addressment, etc.
Design reliability
Risk
Probability of
LoM
Percent of LoM
outcomes
Limiting Factor
Robustness
Ability to result in
LoM from MD
Percent of MD that
directly result in LoM
Operational
Dependency
Temporalness
Temporal
variability of
utility
Analysis of utility
epoch
Epoch and
Operational Effects
Versatility
Ability to operate
in different
mission designs
Deviation of utility
with CONOPS
variations
Directs design and
contingencies

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
11
Recall, in contrast to MoE, FoM are values that do not directly assess a mission’s
effectiveness but provide value to alternatives across the solution domain. This is critical
in missions where optimum solutions do not necessarily coincide with missions of greatest
success or utility. Without the use of FoM, the objectives-driven process of constructing
programs from top-level objectives has often failed (Conway, 2015). The prominent FoM
are outlined in Table 2 and are used as a quantitative comparison between mission
alternatives. Here, loss of mission (LoM) indicates reaching a premature point in the
mission where further utility obtainment (objectives satisfaction) is no longer possible.
Loss of system (LoS) indicates the loss of a mission system and often times is subdivided
into multiple concept levels (such as loss of payload) or increases mission specificity (such
as loss of crew in human missions). A mission dynamic (MD) indicates the occurrence of
a mission phase or event that alters the mission’s phasespace trajectory.
Cost can be analyzed in any fashion, including systems or lifecycle cost estimation.
These factors, along with the technology readiness level (TRL) of the used technologies,
have a significant relation to the potential monetary cost of the mission. Here, it is assumed
that technologies of lower TRL will require higher cost to develop and mature. This means
the producibility metric is interconnected to that of cost.
The flexibility metric analyses mission performance with changes in architectural
components. While this metric is useful for AoA, it also provides a quantitative value to
the risk of pursuing development of the mission concept. For example, a mission solution
may perform exceptionally well, but only when utilizing ‘technology A’. If, after spending
years and significant resources to begin development of this concept ‘technology A’ no
longer is a feasible or viable option then the mission as a whole could be significantly
compromised or cancelled. Therefore, the interpretation of the flexibility metric is also
interdependent on producibility. Conversely, versatility analyses deviance in utility from
changes in the CONOPS, meaning versatility analyses how utility deviates with the same
mission systems operating in a different mission design.
When determining the root of risk, the metric of fragility is of assistance. It provides
a breakdown of which mission systems contribute to mission failures. In architecting the
new concept, this will determine which components and its dependents require the greatest
attention. In reality, missions are dynamic entities and are not always executed to script.
The metric of robustness determines which MD contribute(s) to LoM. The resiliency metric
is in place to determine how fragility and robustness directly affect mission utility.
There are many dynamics in the mission that are not only known apriori, but are
also preventable. These constitute derivatives of the MD altering mission utility and are
often quantified by tabulating the percent obtainment required to avert perturbations to the
desired or optimal mission phasespace trajectory. This same analysis applies to
responsiveness, which quantifies the ability to recognize and respond to MD derived from
LoS. The efficiency metric determines the differential of ideal and actual mission utility. It
determines the efficiency of mission objectives satisfaction. This derives the percentage of
mission events which are capable of obtaining mission utility but ultimately do not. This
behavior is often an impact of insufficient mission intelligence or an inadequate mission
phenomenology – both of which result in deceptive mission utility obtainment
mechanisms.
Temporalness, which completes the FoM, becomes exceptionally important for
analyzing not only when a mission can proceed but also when the highest utility is obtained.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
12
Analysis of temporal variability in utility may show that a concept only performs well
within certain mission epochs. This provides insight into ways for making the performance
of the mission concept more consistent across varying operational weather and mission
environments.
7. Interpretations
Regardless of the fidelity of the simulation, mission utility analysis can be
significantly hindered or limited if the simulation datasets are misconstrued. When
interpreting the results of stochastic experiments, it is of utmost importance to keep in mind
the nature of probability. Though some of its shortcomings can be negated by the
introduction of possibilistic models, estimative probability and fuzzy logic, its results
should not be confused with the behavior of deterministic nor traditionally stochastic
models. When analyzing mission events in the simulations, lesser values represent a low
probability of occurrence (probabilistic) or a low degree to which occurrence is viewed as
being possible (possibilistic) (Nikolaidis et al., 2003). This, however, does not mean they
are of less priority. Even values on the order of hundredths should not be confused with
impossibility – instead, their occurrence in the simulation is actually confirmation that they
are indeed plausible events and outcomes. Instead, interpretation of stochastic results
should always navigate the decision-making triangle (Figure 2).
Figure 2: Decision-Making Triangle
In essence the cycle states that when a probability (P) of occurrence is analyzed in the
mission, its risk must be understood to direct tradespace (T) selections in addressing the
occurrence. Once this trade is made, an understanding of its scope is required to determine
the impact on utility (U). It is only after this quantification that the true cost of the
occurrence can be understood. For example, consider ‘event A’ at 1% occurrence which
results in LoM and ‘event B’ at 70% occurrence which results in a delay of the mission. It
is of habit to ignore an event such as ‘event A’ because its occurrence is considered rare;
however, its cost of occurrence is significant. A decision-maker will have to decide if a
trade for further reduction to the risk of LoM is of worth, even if it means substantial

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
13
reduction of average utility. On the other hand, ‘event B’ may be the most frequent event
in the simulation but if the cost of occurrence can be considered negligible then the
decision-maker may decide to take no action. This argument may seem trivial in this
example but this train of thought is often forgotten when interpreting the results of
stochastic simulations.
8. Conclusion
Current mission design philosophies are inadequate due to (1) incomplete mission
phasespaces which do not holistically capture mission altering variables; (2) disaggregated
design methods which neglect the interoperability and emergent behaviors present in the
system-of-systems nature of mission architectures; (3) exclusion of constitutive relations
that incorporate mission variables external to the mission architecture; (4) curtailed
tradespace explorations which derive incomplete statistical ensembles of the mission; and
(5) unsubstantiated mission concept solutions which do not quantify mission objectives
satisfaction. Therefore, current mission methods do not truly analyze the mission. This
paper presented a methodology rectifying the aforementioned problems utilizing stochastic
mission utility analysis. Mission objectives are quantified to produce a set of metrics. These
metrics are split between MoE, which directly measure a mission’s ability to satisfy its
objectives and FoM, which quantify the value of alternatives across the tradespace. Mission
simulation is used to provide a platform for mission utility analysis. It requires modeling
of the mission, inclusive of operational weather, human factors and uncertainty at the
system, architectural, mission and enterprise levels. It enables input of mission
architectures across a vast tradespace with rapid mission simulation. This and a
comprehensive replication of mission CONOPS enables quantification of the metrics in
analyzing the mission. Analytics of the simulations’ dataset provides mathematical input
in architecting statistically validated concepts in advancing the mission. This framework
makes it possible to then re-evaluate the mission after development and experimentation
of its mission concepts. The efficacy is therefore achieved by converting high-level mission
objectives into quantifiable mission outcomes for mission engineering, analysis and
architecting.
9. Limitations and Future Work
As with any simulation, particularly in stochastic forecast models relying on
numerical predictions, the results are inherently limited by the nature of probability.
However, this can be augmented by systematic interpretations, as previously discussed.
While subtle, it is the methodology’s need to quantify impacts to utility that often limits its
comprehensive and accurate depiction. Particularly in missions with very limited historical
data, it may be regarded as exceedingly difficult to mathematically capture the complexities
of its entirety. However, being cognisant of equivalent impacts across AoA still provides
value to the utility analysis. It is the CD&E portion of mission utility analysis that is much
more hindered by hidden limitations such as the selected ground rules, assumptions,
delimitations and datasets used to construct the model. All of these aspects, particularly for
an underdeveloped mission, provide uncertain deviations from reality that cannot be
overcome by stochastic techniques. Ultimately, M&S is limited in its capacity by these
aspects, and therefore, so too is the methodology.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
14
However, M&S of missions is a very active area of research. It is anticipated there
will be much future work contributing to the advancement of the methodology and
continuation of mission engineering research. In particular, Monte Carlo and
possibilistic/probabilistic risk assessment have been pinpointed as areas of priority.
Advancement in these fields will certainly address the limitations of the methodology.
10. Applicability
The framework of this methodology applies to anything or anyone seeking to define
the best solution for achieving objectives and goals. In essence, the journey towards
becoming more accomplished is a mission and this methodology presents guiding
principles for engineering mission achievement. This philosophy is applicable to any
domain. In the same way that fields such as operations research and systems engineering
have evolved from an application to definite frameworks and methodologies – it is
foreseeable that the holistically technical approach of mission engineering will carve a
discipline of its own (Sousa-Poza, 2015) and mission engineering will be offered as a
degree program at academic institutions (Sousa-Poza and Old Dominion University, 2017).
Acknowledgements
This research did not receive any specific grant from funding agencies or organizations in
the public, commercial, or not-for-profit sectors.
The author would like to thank all reviewers for their recommendations to this article for
publication.
References
Conway, E. (2015) Exploration and Engineering: The Jet Propulsion Laboratory and the
Quest for Mars, John Hopkins University Press, Baltimore, MD.
Defense Science Board and Air Force Scientific Advisory Board (2003) Acquisition of
National Security Space Programs, US Department of Defense, Washington, DC.
Friedman, S.M. and Weber, R.H. (2004) ‘Evaluating the utility of space systems’, in
Space Modeling and Simulation, Aerospace, El Segundo, CA.
Nikolaidis, E. et al. (2003) ‘Comparison of probabilistic and possibility-based methods
for design against catastrophic failure under uncertainty’, Journal of Mechanical
Design, Vol. 126, No. 3, pp.386–394.
Rainey, L.B., Cloud, D.J. and Crumm, M.D. (2004) ‘Introduction to modeling and
simulation for space systems’, in Space Modeling and Simulation, Aerospace, El
Segundo, CA.
Ross, A.M. and Hastings, D.E. (2005) ‘The tradespace exploration paradigm’, INCOSE.
International Symposium, Rochester, NY.
Ross, A.M., Hastings, D.E., Warmkessel, J. and Diller, N.P. (2004) ‘Multi-attribute
tradespace exploration as front end for effective space system design’, Journal of
Spacecraft and Rockets, Vol. 41, No. 1, pp.20–28.
Sierhuis, M. et al. (2006) ‘Agent-based mission modeling and simulation’, 2006 Spring
Simulation Multiconference, Huntsville, AL.

Holistic(Methodology(for(Stochastic(Mission(Utility(Analysis(
15
Sousa-Poza, A. (2015) ‘Mission engineering’, Int. J. System of Systems Engineering, Vol.
6, No. 3, pp.161–185.
Sousa-Poza, A. and Old Dominion University (2017) ‘Educating the mission engineer’,
Mission Engineering and Analysis, and Integration and Interoperability,
Dahlgren, VA.
Van Wyngaard, C.J., Pretorius, J.H. and Pretorius, L. (2012) ‘Theory of the triple
constraint – a conceptual review’, IEEE International Conference on Industrial
Engineering and Engineering Management, Hong Kong, China, pp.1991–1997.
Vasile, M. et al. (2014) ‘Mission and system design’, in The International Handbook of
Space Technology, Springer, Chichester, UK.
Wertz, J.R. (2008) ‘ORS mission utility and measures of effectiveness’, 6th Responsive
Space Conference, Los Angeles, CA.
Wertz, J.R. (2011) ‘Mission analysis and mission utility’, in Space Mission Engineering,
Microcosm, Hawthorne, CA.