ArticlePDF Available

Holistic Methodology for Stochastic Mission Utility Analysis

  • Watson Institute for Scientific Engineering Research

Abstract and Figures

Preliminary analysis and design of a mission has profound effects on its lifecycle; yet, mission engineering is a discipline without unanimous definition and implementation. Current mission design philosophies are inadequate due to disaggregated design methods, incomplete mission phasespaces, exclusion of constitutive relations, curtailed tradespace exploration and unsubstantiated concept solutions. Presented is a new methodology for simultaneously developing, exploring and assessing statistically validated concepts of prospective missions. Stochastic modelling and simulation, providing multivariate mission utility analysis with concurrently integrated tradespace exploration, risk assessment and holistic design facilitates the methodology.
Content may be subject to copyright.
Holistic Methodology for Stochastic Mission Utility Analysis
Ja’Mar A. Watson
Watson Institute for Scientific Engineering Research
4201 Wilson Boulevard, Suite 110-444
Arlington, VA 22203, USA
Abstract: Preliminary analysis and design of a mission has profound effects
on its lifecycle; yet, mission engineering is a discipline without unanimous
definition and implementation. Current mission design philosophies are
inadequate due to disaggregated design methods, incomplete mission
phasespaces, exclusion of constitutive relations, curtailed tradespace
exploration and unsubstantiated concept solutions. Presented is a new
methodology for simultaneously developing, exploring and assessing
statistically validated concepts of prospective missions. Stochastic
modeling and simulation, providing multivariate mission utility analysis
with concurrently integrated tradespace exploration, risk assessment and
holistic design facilitates the methodology.
Keywords: Mission Engineering; Mission Utility Analysis; Stochastic
Design; System-of-Systems Engineering; Mission Simulation; Mission
Modeling; Mission Architecture; Analysis of Alternatives; AoA; Concept
Development and Experimentation; CD&E.
Reference to this paper should be made as follows: Watson, J.A. (2017)
‘Holistic Methodology for Stochastic Mission Utility Analysis’, Int. J.
System of Systems Engineering, Vol. 8, No. 2, pp.174–188.
DOI: 10.1504/IJSSE.2017.088447
1. Introduction
Prospective missions are often nebulous with notional mission systems and
conceptual mission architectures. Theatre-level knowledge is often riddled with axioms,
conjectures, misconceptions, and postulates; yet, mission intelligence is never a global
variable in the concept of operations (CONOPS). Mission environments present uncertain
boundary conditions but operational weather is never holistically captured. Such neglect
makes the achievement of pragmatic solutions infeasible. Future missions often require
nascent technologies and/or callow methodologies – variables that are rarely analyzed for
their contribution to mission objectives. This is due to the omission or separation of
tradespace exploration from utility analysis. Additionally, humans present unparalleled
complexity and variable factors; however, the importance of ergonomics in the CONOPS
is often unappreciated in comparison to the emphasis on mission architecture in concept
definitions. In fact, the inclusion of human factors is mostly nonexistent in applicable
mission phasespaces. Often coupled with budgetary, political and temporal constraints,
current techniques to architect, engineer and analyze missions do not encapsulate these
problems. The lack of resolution stems from design philosophies that do not truly analyze
the mission. Their techniques often ignore utility for representation of one of its
constituents. For that reason, it is unclear if proposed missions under current design
methods actually accomplish their objectives; thus, these solutions are unsubstantiated.
Without concurrent consideration of utility, it is not feasible to conduct meaningful mission
concept development and experimentation (CD&E) or analysis of alternatives (AoA).
Presented is a new methodology providing multivariate mission utility analysis with
concurrently integrated tradespace exploration, risk assessment and holistic design via
stochastic modeling and simulation (M&S). It facilitates simultaneous development,
exploration and assessment while statistically validating the prospective mission concepts.
2. State-of-the-Art
Currently, abstract and disaggregated system design processes plague mission
engineering. Most mission analyses are nothing more than a conglomeration of system
designs and/or mission phases – not a true analysis of the mission. This approach deprives
interoperability of design choices by ignoring the holistic, system-of-systems nature of
mission architectures. Such truncation eliminates emergent concept behaviors and designs,
reducing CD&E to meagre trade studies.
The cost of tradespace exploration in current design methods is inordinately expensive.
Such cost has led to the omission or severe restriction on front-end analysis. This often
attributes to greater mission ambiguity since the preliminary mission design phase
produces a greater differential between concept and operation.
The current reliance on testing and evaluation (T&E), both developmental and
operational, is too great among today’s diversity of missions. Additionally, it is often
impractical or impossible to perform traditional T&E through progressive demonstration
and/or test missions. Such is the case in many defense and space exploration missions, for
example. It is therefore of utmost importance to systematically replicate these processes
during the CD&E front-end analysis. Failure to do so will result in significant reduction of
capabilities and operations simply due to the shortcomings of mission design methods.
Previous research has shown that developing complex missions with rapidly
advancing technology is inefficient when one stage of analysis is nearly complete before
another begins (Friedman and Weber, 2004). This is due to premature stagnation of critical
design variables in preceding stages. More often than not, future missions require nascent
technologies and/or callow mission methodologies, meaning the introduction of concurrent
engineering to the proposed methodology is beneficial to mission design mechanisms.
Concurrent engineering allows mission design to be directed by all knowledge concerning
the architecture and its operational environment.
The lack of efficiency in current mission design methods has become so
overbearing that cost often replaces success as the primary mission driver. This has caused
the loss of capabilities and abandonment of principles governing the acquisition of critical
design variables in preceding stages. More often than not, future missions require nascent
technologies and/or callow mission methodologies, meaning the introduction of concurrent
engineering to the proposed methodology is beneficial to mission design mechanisms.
Concurrent engineering allows mission design to be directed by all knowledge concerning
the architecture and its operational environment.
The lack of efficiency in current mission design methods has become so
overbearing that cost often replaces success as the primary mission driver. This has caused
the loss of capabilities and abandonment of principles governing the acquisition of mission
programs (Defense Science Board and Air Force Scientific Advisory Board, 2003). The
result of such has introduced failures in balancing schedule, cost, and scope, giving birth
to the triple constraint theory denoting the infeasibility of simultaneous improvement under
current methods (Van Wyngaard et al., 2012). The effects of this theory are potent – even
before consideration of other leading constraints such as risk. Missions in many disciplines
do not have standard cost; therefore, it is difficult to base the efficacy of a new paradigm
on its reduction. This has led to complacency in accepting any cost reduction as indication
of exemplary methods. With absence of knowledge for true minimal cost in mission
achievement, quantification is transferred to scope, which becomes expressed as mission
utility: the ability to satisfy mission goals and objectives.
The analysis of mission utility requires the relaxation of the notion of requirements.
Requirements are the leading driver for increasing the scope-to-cost ratio, but this is only
true if the requirements are expressed functionally with quantitative degrees of satisfaction.
This is philosophically comparable to Pareto’s 80/20 principle, which translates 80% of the
mission’s scope can be accomplished with 20% of the mission’s cost (Vasile et al., 2014).
Historically, requirements are viewed as inflexible and the cheapest mission to accomplish
them is designed irrespective of this ratio. This has single-handedly restrained
advancements in mission design. With the removal of the traditional definition for
requirements, quantification is transferred to measures of effectiveness (MoE) and figures
of merit (FoM), which include the variables of cost, schedule and scope. It also includes
risk, which quantifies a design’s ability to transfer cost into scope. It is therefore mission
utility analysis that serves as the foundation for this philosophy and M&S has become an
essential mechanism for this in mission engineering.
3. Background
For decades, M&S has helped ease the burden of complexity in mission engineering
(Sierhuis et al., 2006) by allowing computers to mimic traditional laboratories with reduced
T&E costs (Rainey et al., 2004). Traditional tests are run in analytical scenarios: one that
displays the operation of a system without doing so in a real or all fully plausible situations.
This introduced the practice of designing for worst-case scenarios of subsystems, not only
dooming the architecture by provoking unnecessarily constrained size, weight and power
budgets, but also producing an architecture built for unrealistic scenarios that may not
occur simultaneously. This realization led to the integration of Monte Carlo methods that
iteratively run tests with varying scenarios to collect data on multiple runs. This is
impractical for entire mission concepts with traditional T&E methods but can be performed
using Monte Carlo simulations for a fraction of the cost while encompassing almost all
likely scenarios (Conway, 2015).
However, these benefits only transferred to the subsystem level making it difficult
to understand how design choices alter mission accomplishment, as well as their impact
across the architecture both in terms of performance and operation. When determining how
an architecture can achieve a desired outcome in the mission, realistic scenarios need to be
developed in which to test architectural performance. This is mission utility analysis and
the only way to perform it is with a simulation of the mission (Wertz, 2011, 2008). It then
becomes clear that individual Monte Carlo analysis is incomplete and there must be a
simulation of the entire mission. A simulation as such in a realistic environment
incorporating a comprehensive model of the mission concept is absent in current design
Now that the entire mission is being considered, new issues arise when trying to
use traditional methods. At the architectural level, the problem set contains multitudes of
parameters and AoA of a concept can become too complex to determine the interoperability
and trades between choices. To eliminate ambiguity, tradespace exploration is introduced
to perform the analysis; however, effective AoA performed in a realistic mission
environment is still needed.
The integration known as multi-attribute tradespace exploration with concurrent
design (MATE-CON) can be performed to simultaneously design, explore and evaluate
potential architectural concepts from the tradespace (Ross et al., 2004). In the end a method
for the assessment of utility using an AoA that provides rationale for potential solutions –
utilizing mission objectives to evaluate and compare mission architectures in developing
and refining a mission concept is developed. This method is unparalleled in mission
engineering, where it is becoming practice that systems are brought to conformance with
the simulations instead of the contrary (Conway, 2015).
What is considered a current advancement in MATE-CON is the inclusion of
uncertainty and this is an absolute must in utility simulation of prospective missions (Ross
and Hastings, 2005). The ability of stochastic M&S to both answer and produce emergent
questions plays a crucial role in this addition. The greatest advantage of Monte Carlo
simulations is the ability to give designers knowledge of occurrence likelihoods. With such
a capability, the shortcomings of traditional risk assessment and reliability analysis can be
augmented. Traditional risk assessment and reliability analysis methods have proven to be
unreliable in forecasting failures and uncertainty because they arise from contributions not
modelled in the analysis of proposed mission architectures. The methodology presented in
this paper, incorporating many sources of uncertainty while providing holistic and
comprehensively stochastic analysis of all potential architectural and operational designs
in a realistic mission simulation, is the launch pad for deploying the mission utility analysis
methodology needed to produce ‘faster, better, cheaper’ missions.
4. Objectives
In order to simultaneously address the aforementioned problems the following
objectives are achieved.
Objective 1: Quantify Mission Goals Facilitating the computation of mission utility
requires the quantification of mission objectives. This is the introduction of implicitly
flexible and functional requirements presented with purposeful generality and freedom of
achievement path as to not inadvertently introduce bias or hinder solution domain. They
are outlined to allow selection of all possible outcomes in satisfying the chosen mission
goals. The objective is to quantify mission goals that replicate the purpose of the mission.
Objective 2: Enable Mission Utility Simulation – The only way to truly assess mission
utility is with a simulation of the mission. Accordingly, it is an objective to simulate the
mission, varying mission concept components that affect mission utility. It is the results
from these mission utility simulations that feed the analytics for AoA and CD&E.
Objective 3: Incorporate Precursor Missions – History does not suggest complex
missions are attempted in the first deployment design. Therefore, an objective of this
methodology is to quantify the effects of potential precursor missions on final mission
utility to determine their necessity.
Objective 4: Incorporate Mission Uncertainty – A substantial challenge of mission
utility simulations for future missions is the potential level of uncertainty. When present,
initial models are undoubtedly dubious. Therefore, the composition of the mission as a
model should be treated as nothing more than a hypothesis accompanied by constituent
percent probabilities of falsehood. These likelinesses of true/false should be updated by
real-time data in the simulation to provide a realistic portrayal of information needs
satisfaction in the mission. The objective is to replicate the unfolding of a live mission
while also being able to determine how initial mission uncertainty affects mission utility.
Objective 5: Incorporate Operational Weather This is also true for the constitutive
relations that complete comprehensive simulation of the concept. External factors, such as
mission environments and potential operational weather, are critical components to
simulating a realistic mission. It is therefore also an objective to finesse these factors into
the utility analysis.
Objective 6: Incorporate Architectural Uncertainty Another objective is to capture
the uncertainty in the mission architecture. This can arise due to their current level of
conception or uncertainty in system selection – both of which affect the physical models’
state of systems in the simulated mission. Therefore, inclusion of an exhaustive mission
ensemble is critical with stochastic parameters to replicate this uncertainty. Chaos theory
predicts that even the minutes of changes at mission initiation can have profound impacts
on final mission utility.
Objective 7: Incorporate Human Factors It is an objective of this methodology to
ensure comprehensive depiction by reflecting human factors. Mission design perspective
and perceived theatre-level knowledge are factors that arise due to humans designing,
occurring in and/or operating the mission. In missions where humans represent mission
systems or payloads, their physiological and psychological effects on the mission must be
captured. The impact of ergonomics on both the design and analysis of the architecture, as
well as the CONOPS, has profound effects on mission utility and the simulation’s
resemblance to a realistic mission.
Objective 8: Incorporate Operational Uncertainty Mission simulation comes with a
large uncertainty of mission outcomes associated with operational aspects of the model.
The objective is to provide a solution to handle this uncertainty and account for all possible
mission evolutions and outcomes, including not just mission aborts but also a hierarchy of
contingencies. In this approach, it is the aggregated results over all possible mission
phasespace trajectories that represent an accurate calculation of mission utility.
Objective 9: Assess Mission Metrics Determination of mission utility requires
quantification of mission metrics representative of the mission objectives. Calculation
across all concept alternatives enables a substantiated solution. Therefore, an objective of
this methodology is to assess the mission metrics to determine an appropriate mission
concept through AoA and CD&E.
Objective 10: Architect a New Mission Concept – The beauty of mission utility analysis
is in its ability to provide comprehensively integrated solutions at the early stage of concept
assessment. Identifying often unseen and counterintuitive problem statements allows
effective determination of technological, systematic, architectural and/or operational
deficiencies. Therefore, the objective is to use the results of mission utility analysis to
provide a platform for statically validated mission concept improvements effectively
giving a quantifiable measure of this methodology’s contribution.
5. Methodology
While there may be many pathways to achieve the objectives outlined, the
following tasks represent the framework of this methodology. Figure 1 provides a visual
representation of the workflow and an introduction to the mechanisms of the methodology:
Figure 1: Methodology Flow Diagram
Task 1: Define Mission Objectives – Utility is a quantifiable metric of a mission’s ability
to satisfy mission objectives. In order to perform mission utility analysis, the mission
objectives shall be defined, their means of satisfaction identified and their contributions to
the computation of utility mathematically outlined.
Task 2: Define Metrics Though they are often used interchangeably, two distinguishable
definitions for MoE and FoM are introduced to facilitate mission concept evaluation and
comparison. MoE are measures of satisfaction of task 1’s mission objectives to determine
mission effectiveness. Utility shall be a constituent of the MoE. On the other hand, FoM
are values that do not directly assess a mission’s effectiveness in satisfying mission
objectives but provide value to alternatives across the solution domain. Both the purpose
and the method of calculation shall be defined for these metrics.
Task 3: Model the Mission Arena – The mission arena: the stage on which all mission
simulations occur. The arena shall be completed with inclusion of modelled constitutive
relations such as operational weather. The fidelity of the model shall be sufficient to
provide a statistical significance to mission arena variables that affect mission metrics.
Task 4: Model the Mission Architecture The modeling shall conform to all mission
systems that make up each potential mission architecture. This means a model shall be
formulated for all considered heritage, state-of-the-art and nascent mission
technologies/systems, as well as the emergent and interoperable behaviors of each potential
architectural conglomeration. In essence, capturing the mission ensemble with applicable
command, control, computers, communications, intelligence, surveillance, and
reconnaissance (C4ISR) interoperability successfully completes modeling of all possible
mission architectures.
Task 5: Define the Operational Concept The CONOPS shall be defined in a fashion
that facilitates realistic simulation of the modelled mission architecture. The concept shall
integrate the impacts of precursor missions, possibilistic mission events and their
respective contingencies, logistics of mission systems, apportionment of mission
operations, and the mechanisms for achieving utility to satisfy mission objectives.
Therefore, this task establishes a definition for phenomenology of the mission.
Task 6: Establish the Baseline – The first simulation to complete is that of the mission as
currently defined or perceived. This shall serve as the reference mission architecture and
shall establish baseline performance results of the metrics, including mission utility.
Task 7: Define the Tradespace – The definition shall enable AoA of the baseline concept
as well as tradespace exploration with the modelled mission ensemble. This will serve as
identification of the solution domain and enumeration of the multivariate inputs. The
tradespace shall be inclusive of all relevant variable inputs (precursor missions,
architectural components, etc.) and stochastic components of the CONOPS (mission
intelligence, possibilistic mission event outcomes, etc.), as well as components of
constitutive relations (operational weather, applicable external factors and stimuli, etc.).
Comprehensive tradespace definition produces a realistic multivariate mission utility
analysis. When the tradespace exploration is concurrent with the utility analysis, its
variable architectural and operational components of the concept further inherently
produce risk assessment and holistic design in stochastic Monte Carlo simulations.
Task 8: Perform Mission Utility Simulations This produces all of the data to be
aggregated – the results of which shall culminate in the computation of the metrics defined
in Task 2. In the case of multiple mission designs, this shall include all simulations for
AoA. In the case of a desired improvement to the mission concept, this expands to CD&E,
requiring either an exhaustive or experimentally designed tradespace exploration.
Task 9: Perform Analytics and Enterprise Architecting – Mission simulations shall be
followed by a full spectrum of descriptive, predictive and prescriptive (DP2) analytics of
the simulation datasets, along with enterprise architecting. The descriptive portion shall
provide data analysis of what occurred in the mission simulations. It shall produce metrics
and sensitivity analyses. This shall culminate in the manufacturing of utility functions for
further rapid, algorithmic computation of mission utility. The predictive portion of the
analytics shall produce predictive mission design trends by deriving mission utility as a
function of the mission’s input design domain and inherent stochastic variables. From this,
it is also possible to analyze additional mission concepts whose synthesis is outside of the
initial tradespace. This further allows refinement of the mission problem statement. The
prescriptive analytics shall provide quantitative instruction for improving the mission
concept, including, but not limited to, statistically guided amendments, additional
investigations, contingencies and preliminary principles for performing the mission.
Prescription seeks to expand and give application to the previous steps of analytics. Finally,
enterprise architecting shall relate the analytics to program level actions for the mission.
This includes potential collaboration between entities, identification of stakeholders,
initiation of research and development and/or technological investments required to
develop the mission concept, as well as a potential roadmap for the realization of the
mission. When integrated with the previous tasks, DP2 analytics enables simultaneous and
statistically validated development, exploration and assessment of the prospective mission.
Task 10: Architect a New Mission Concept As a final task, particularly in CD&E, a
new mission concept shall be architected from the analytics of task 9. This can be presented
as a singular solution or a set of allocations from the derived Pareto frontier or mission
solution ensemble. In principle it shall equal or exceed the highest fidelity and utility of the
baseline and alternative mission concepts. This effectively gives a quantifiable measure of
this methodology’s contribution to the mission.
6. Metrics
While the included metrics along with utility can be highly individual to each
prospective mission, the most prominent MoE and FoM are presented in this section. For
MoE, two values of the mission have the greatest impact on its worthiness: its measure of
success and utility. While the latter deals exclusively with preference (satisfaction of
selected mission objectives), a measurement of success serves as an unbiased discovery
index. It is unwise to use utility as a single metric in missions with multiple, unequal
objectives when their weighting can be considered highly subjective (Friedman and Weber,
2004). In the event mission objectives change or are not unanimous across all decision-
makers, a second unweighted (although possibly unrealistically unweighted) value
provides perspective to the mission’s usefulness. It is also worth noting that mission
objectives used for utility are satiable but represent minimum levels of acceptable
achievement. A measurement of success can tabulate excess performance. Four derived
values – termed ability, productivity, success and utility – constitute the MoE (Table 1): a
value that directly assesses the mission’s effectiveness in satisfying mission objectives.
Table 1: Measures of Effectiveness
Ability of mission to
achieve utility
Number of objectives achieved
(without duplication)
Level of mission yield
Number of objectives achieved
(with duplication)
Ability to satisfy
mission sub-objectives
Percent of mission objective
satisfaction (unweighted)
Ability to satisfy
mission objectives
Percent of mission objective
satisfaction (weighted)
Table 2: Figures of Merit
Amount of
resources required
Cost, mission duration,
Lifecycle cost
Ratio of utility
attempts to utility
Percent of ideal
mission utility
Intelligence and
Ability to operate
with architectural
Deviation of utility
with change in mission
Risk of pursuing the
Ability to result in
LoM from LoS
Percent of LoS that
directly result in LoM
Directs design and
Preparation for
Percent obtainment of
needs for MD
Phasespace analysis
Readiness to
produce concept
Readiness level of
mission concept
Ability to maintain
utility with LoS
and MD
Deviation of utility
with LoS and MD
Utility variability
Ability to address
LoS and MD
Response time, percent
addressment, etc.
Design reliability
Probability of
Percent of LoM
Limiting Factor
Ability to result in
LoM from MD
Percent of MD that
directly result in LoM
variability of
Analysis of utility
Epoch and
Operational Effects
Ability to operate
in different
mission designs
Deviation of utility
Directs design and
Recall, in contrast to MoE, FoM are values that do not directly assess a mission’s
effectiveness but provide value to alternatives across the solution domain. This is critical
in missions where optimum solutions do not necessarily coincide with missions of greatest
success or utility. Without the use of FoM, the objectives-driven process of constructing
programs from top-level objectives has often failed (Conway, 2015). The prominent FoM
are outlined in Table 2 and are used as a quantitative comparison between mission
alternatives. Here, loss of mission (LoM) indicates reaching a premature point in the
mission where further utility obtainment (objectives satisfaction) is no longer possible.
Loss of system (LoS) indicates the loss of a mission system and often times is subdivided
into multiple concept levels (such as loss of payload) or increases mission specificity (such
as loss of crew in human missions). A mission dynamic (MD) indicates the occurrence of
a mission phase or event that alters the mission’s phasespace trajectory.
Cost can be analyzed in any fashion, including systems or lifecycle cost estimation.
These factors, along with the technology readiness level (TRL) of the used technologies,
have a significant relation to the potential monetary cost of the mission. Here, it is assumed
that technologies of lower TRL will require higher cost to develop and mature. This means
the producibility metric is interconnected to that of cost.
The flexibility metric analyses mission performance with changes in architectural
components. While this metric is useful for AoA, it also provides a quantitative value to
the risk of pursuing development of the mission concept. For example, a mission solution
may perform exceptionally well, but only when utilizing ‘technology A’. If, after spending
years and significant resources to begin development of this concept ‘technology A’ no
longer is a feasible or viable option then the mission as a whole could be significantly
compromised or cancelled. Therefore, the interpretation of the flexibility metric is also
interdependent on producibility. Conversely, versatility analyses deviance in utility from
changes in the CONOPS, meaning versatility analyses how utility deviates with the same
mission systems operating in a different mission design.
When determining the root of risk, the metric of fragility is of assistance. It provides
a breakdown of which mission systems contribute to mission failures. In architecting the
new concept, this will determine which components and its dependents require the greatest
attention. In reality, missions are dynamic entities and are not always executed to script.
The metric of robustness determines which MD contribute(s) to LoM. The resiliency metric
is in place to determine how fragility and robustness directly affect mission utility.
There are many dynamics in the mission that are not only known apriori, but are
also preventable. These constitute derivatives of the MD altering mission utility and are
often quantified by tabulating the percent obtainment required to avert perturbations to the
desired or optimal mission phasespace trajectory. This same analysis applies to
responsiveness, which quantifies the ability to recognize and respond to MD derived from
LoS. The efficiency metric determines the differential of ideal and actual mission utility. It
determines the efficiency of mission objectives satisfaction. This derives the percentage of
mission events which are capable of obtaining mission utility but ultimately do not. This
behavior is often an impact of insufficient mission intelligence or an inadequate mission
phenomenology both of which result in deceptive mission utility obtainment
Temporalness, which completes the FoM, becomes exceptionally important for
analyzing not only when a mission can proceed but also when the highest utility is obtained.
Analysis of temporal variability in utility may show that a concept only performs well
within certain mission epochs. This provides insight into ways for making the performance
of the mission concept more consistent across varying operational weather and mission
7. Interpretations
Regardless of the fidelity of the simulation, mission utility analysis can be
significantly hindered or limited if the simulation datasets are misconstrued. When
interpreting the results of stochastic experiments, it is of utmost importance to keep in mind
the nature of probability. Though some of its shortcomings can be negated by the
introduction of possibilistic models, estimative probability and fuzzy logic, its results
should not be confused with the behavior of deterministic nor traditionally stochastic
models. When analyzing mission events in the simulations, lesser values represent a low
probability of occurrence (probabilistic) or a low degree to which occurrence is viewed as
being possible (possibilistic) (Nikolaidis et al., 2003). This, however, does not mean they
are of less priority. Even values on the order of hundredths should not be confused with
impossibility – instead, their occurrence in the simulation is actually confirmation that they
are indeed plausible events and outcomes. Instead, interpretation of stochastic results
should always navigate the decision-making triangle (Figure 2).
Figure 2: Decision-Making Triangle
In essence the cycle states that when a probability (P) of occurrence is analyzed in the
mission, its risk must be understood to direct tradespace (T) selections in addressing the
occurrence. Once this trade is made, an understanding of its scope is required to determine
the impact on utility (U). It is only after this quantification that the true cost of the
occurrence can be understood. For example, consider ‘event A’ at 1% occurrence which
results in LoM and ‘event B’ at 70% occurrence which results in a delay of the mission. It
is of habit to ignore an event such as ‘event A’ because its occurrence is considered rare;
however, its cost of occurrence is significant. A decision-maker will have to decide if a
trade for further reduction to the risk of LoM is of worth, even if it means substantial
reduction of average utility. On the other hand, ‘event B’ may be the most frequent event
in the simulation but if the cost of occurrence can be considered negligible then the
decision-maker may decide to take no action. This argument may seem trivial in this
example but this train of thought is often forgotten when interpreting the results of
stochastic simulations.
8. Conclusion
Current mission design philosophies are inadequate due to (1) incomplete mission
phasespaces which do not holistically capture mission altering variables; (2) disaggregated
design methods which neglect the interoperability and emergent behaviors present in the
system-of-systems nature of mission architectures; (3) exclusion of constitutive relations
that incorporate mission variables external to the mission architecture; (4) curtailed
tradespace explorations which derive incomplete statistical ensembles of the mission; and
(5) unsubstantiated mission concept solutions which do not quantify mission objectives
satisfaction. Therefore, current mission methods do not truly analyze the mission. This
paper presented a methodology rectifying the aforementioned problems utilizing stochastic
mission utility analysis. Mission objectives are quantified to produce a set of metrics. These
metrics are split between MoE, which directly measure a mission’s ability to satisfy its
objectives and FoM, which quantify the value of alternatives across the tradespace. Mission
simulation is used to provide a platform for mission utility analysis. It requires modeling
of the mission, inclusive of operational weather, human factors and uncertainty at the
system, architectural, mission and enterprise levels. It enables input of mission
architectures across a vast tradespace with rapid mission simulation. This and a
comprehensive replication of mission CONOPS enables quantification of the metrics in
analyzing the mission. Analytics of the simulations’ dataset provides mathematical input
in architecting statistically validated concepts in advancing the mission. This framework
makes it possible to then re-evaluate the mission after development and experimentation
of its mission concepts. The efficacy is therefore achieved by converting high-level mission
objectives into quantifiable mission outcomes for mission engineering, analysis and
9. Limitations and Future Work
As with any simulation, particularly in stochastic forecast models relying on
numerical predictions, the results are inherently limited by the nature of probability.
However, this can be augmented by systematic interpretations, as previously discussed.
While subtle, it is the methodology’s need to quantify impacts to utility that often limits its
comprehensive and accurate depiction. Particularly in missions with very limited historical
data, it may be regarded as exceedingly difficult to mathematically capture the complexities
of its entirety. However, being cognisant of equivalent impacts across AoA still provides
value to the utility analysis. It is the CD&E portion of mission utility analysis that is much
more hindered by hidden limitations such as the selected ground rules, assumptions,
delimitations and datasets used to construct the model. All of these aspects, particularly for
an underdeveloped mission, provide uncertain deviations from reality that cannot be
overcome by stochastic techniques. Ultimately, M&S is limited in its capacity by these
aspects, and therefore, so too is the methodology.
However, M&S of missions is a very active area of research. It is anticipated there
will be much future work contributing to the advancement of the methodology and
continuation of mission engineering research. In particular, Monte Carlo and
possibilistic/probabilistic risk assessment have been pinpointed as areas of priority.
Advancement in these fields will certainly address the limitations of the methodology.
10. Applicability
The framework of this methodology applies to anything or anyone seeking to define
the best solution for achieving objectives and goals. In essence, the journey towards
becoming more accomplished is a mission and this methodology presents guiding
principles for engineering mission achievement. This philosophy is applicable to any
domain. In the same way that fields such as operations research and systems engineering
have evolved from an application to definite frameworks and methodologies – it is
foreseeable that the holistically technical approach of mission engineering will carve a
discipline of its own (Sousa-Poza, 2015) and mission engineering will be offered as a
degree program at academic institutions (Sousa-Poza and Old Dominion University, 2017).
This research did not receive any specific grant from funding agencies or organizations in
the public, commercial, or not-for-profit sectors.
The author would like to thank all reviewers for their recommendations to this article for
Conway, E. (2015) Exploration and Engineering: The Jet Propulsion Laboratory and the
Quest for Mars, John Hopkins University Press, Baltimore, MD.
Defense Science Board and Air Force Scientific Advisory Board (2003) Acquisition of
National Security Space Programs, US Department of Defense, Washington, DC.
Friedman, S.M. and Weber, R.H. (2004) ‘Evaluating the utility of space systems’, in
Space Modeling and Simulation, Aerospace, El Segundo, CA.
Nikolaidis, E. et al. (2003) ‘Comparison of probabilistic and possibility-based methods
for design against catastrophic failure under uncertainty’, Journal of Mechanical
Design, Vol. 126, No. 3, pp.386–394.
Rainey, L.B., Cloud, D.J. and Crumm, M.D. (2004) ‘Introduction to modeling and
simulation for space systems’, in Space Modeling and Simulation, Aerospace, El
Segundo, CA.
Ross, A.M. and Hastings, D.E. (2005) ‘The tradespace exploration paradigm’, INCOSE.
International Symposium, Rochester, NY.
Ross, A.M., Hastings, D.E., Warmkessel, J. and Diller, N.P. (2004) ‘Multi-attribute
tradespace exploration as front end for effective space system design’, Journal of
Spacecraft and Rockets, Vol. 41, No. 1, pp.20–28.
Sierhuis, M. et al. (2006) ‘Agent-based mission modeling and simulation’, 2006 Spring
Simulation Multiconference, Huntsville, AL.
Sousa-Poza, A. (2015) ‘Mission engineering’, Int. J. System of Systems Engineering, Vol.
6, No. 3, pp.161–185.
Sousa-Poza, A. and Old Dominion University (2017) ‘Educating the mission engineer’,
Mission Engineering and Analysis, and Integration and Interoperability,
Dahlgren, VA.
Van Wyngaard, C.J., Pretorius, J.H. and Pretorius, L. (2012) ‘Theory of the triple
constraint – a conceptual review’, IEEE International Conference on Industrial
Engineering and Engineering Management, Hong Kong, China, pp.1991–1997.
Vasile, M. et al. (2014) ‘Mission and system design’, in The International Handbook of
Space Technology, Springer, Chichester, UK.
Wertz, J.R. (2008) ‘ORS mission utility and measures of effectiveness’, 6th Responsive
Space Conference, Los Angeles, CA.
Wertz, J.R. (2011) ‘Mission analysis and mission utility’, in Space Mission Engineering,
Microcosm, Hawthorne, CA.
... To clarify the implementation and to demonstrate the applicability of this modeling and simulation (M&S) method for concise space mission utility simulation, the Apollo 11-17 missions to the Moon are backtested using historic datasets. The methodology for the mission utility simulation used in tandem with this modeling approach has been discussed in detail in a previous publication (Watson, 2017). The publication presents the Holistic Methodology for Stochastic Mission Utility Analysis, which describes steps required to complete mission utility simulation. ...
... Risk is the probability of LoM or LoC. Robustness is the ability to result in LoC or LoM from MD. Versatility is the mission's ability to operate in different mission designs with the same mission systems (Watson, 2017). ...
Full-text available
Presented is a stochastic modeling method enabling rapid yet comprehensive space mission utility simulation. The method facilitates multivariate analysis with concurrent tradespace exploration, risk assessment, and holistic design while simultaneously exploring, assessing, and developing statistically validated concepts of prospective space missions. Modeling is achieved through the synergistic integration of statistical mechanics, blackbox, Bayesian, ansatz, and analytics techniques. The method is verified for its ability to accurately depict a space mission and validated for its ability to perform mission utility analysis by backtesting the Apollo 11-17 missions to the Moon through Monte Carlo simulation.
... CD&E is enabled by the Holistic Methodology for Stochastic Mission Utility Analysis, which has been described at length in a previous publication [34]. The publication outlines the methodology in which mission utility simulation quantifies design analysis for decision-makers [35] and introduces the primary measure of effectiveness (mission utility) and figure of merit (risk) used extensively in this CMM CD&E. ...
... Acolytion's Progspexion is utilized for mission utility simulation in this CD&E. It is a software suite of stochastic Monte Carlo mission simulators that concurrently assess, explore, and devel- scriptive (DP2) analytics [34] of Progspexion's simulation dataset facilities concept development for the analyzed concepts. Its mission utility analysis methodology has been verified and validated through backtesting of Apollo 11-17 missions to the Moon in a previous publication [37]. ...
Full-text available
This paper presents the concept development and experimentation of human spaceflight missions to Mars. Utilizing Acolytion's Progspexion, analysis of alternatives is performed inclusive of 18 prominent mission concepts with a vast experimentally designed tradespace exploration of crewed Mars mission architecture constituents. Results indicate that, as of date, no proposed mission concept is capable of achieving acceptable mission utility to risk ratios; however, the research produces 18 statistically guided developments to the concept of crewed Mars missions.
... From the analysis, the metrics of mission utility ( Figure 1) and risk of loss of mission (LoM) and loss of crew (LoC) ( Figure 2) serve as primary parameters for concept design. Mission utility is a weighted percent of mission objectives satisfied in the mission [5] [6]. 4 Therefore, a combination of abort-to-Mars and EoM colonization strategies significantly increases the mission utility. The median mission utility of the concept adopting abort-to-Earth modes is statistically outside the interquartile range (IQR) for the concept when utilizing an abort-to-Mars strategy ( Figure 1). ...
Full-text available
Accumulation of knowledge acquired during mission utility analysis, concept development and experimentation, and analytics of mission simulation datasets is leveraged to statistically architect prospective human spaceflight missions to Mars. These include the Martian Mothership, Martian Space Station, and Mars Accreted Reconnaissance Station (MARS) concepts, as well as an ensemble of the newly architected missions. Utilizing Acolytion's Progspexion, mission utility simulation is indicative of a simultaneous 84% increase in mission utility and 40% reduction in risk in comparison to the human exploration of Mars baseline.
Technical Report
Full-text available
Progspexion is a software suite of mission simulators statistically forecasting both crewed and robotic space exploration missions. The suite utilizes stochastic Monte Carlo simulation of missions to account for variability and uncertainty across a vast tradespace and phasespace, inclusive of multifidelity digital constituents of technologies, payloads, human factors / ergonomics, systems, architectures, environments / weather, and operations in the mission. As such, Progspexion enables a myriad of analyses, from assessment of historic missions to concurrent design and experimentation of prospective mission concepts.
Full-text available
There are numerous forms of systems engineering that have developed since the late 1990s. These are focused on dealing with increasingly complex problems. Most of the methods provide improvements to systems engineering design capabilities, the systems engineering process, or how complex problems must be managed or governed. We argue that an integrated approach that balances design, process and management is needed to avoid the pitfalls of working in complex situations. Mission Engineering has been proposed as the field intended to maintain coherency between systems engineering, operations, and the mission. This paper presents the challenges that Mission Engineering faces to overcome the complexities of the problems that it is to address, as well as principles for its development and execution.
Full-text available
The inability to approach systematically the high level of ambiguity present in the early design phases of space systems causes long, highly iterative, and costly design cycles. A process is introduced and described to capture decision maker preferences and use them to generate and evaluate a multitude of space system designs, while providing a common metric that can be easily communicated throughout the design enterprise. Communication channeled through formal utility interviews and analysis enables engineers to better understand the key drivers for the system and allows for a more thorough exploration of the design tradespace. Multi-attribute tradespace exploration with concurrent design, a process incorporating decision theory into model- and simulation-based design, has been applied to several space system projects at the Massachusetts Institute of Technology. Preliminary results indicate that this process can improve the quality of communication to resolve more quickly project ambiguity and to enable the engineer to discover better value designs for multiple stakeholders. The process is also integrated into a concurrent design environment to facilitate the transfer of knowledge of important drivers into higher fidelity design phases. Formal utility theory provides a mechanism to bridge the language barrier between experts of different backgrounds and differing needs, for example, scientists, engineers, managers, etc. Multi-attribute tradespace exploration with concurrent design couples decision makers more closely to the design and, most important, maintains their presence between formal reviews.
Conference Paper
Full-text available
A simulation environment for agents is presented, enabling agent-based modeling and simulation of people, systems and robots in space exploration missions. The environment allows the analysis and design of mission operation work procedures, communications and interactions between people and systems, co-located or distributed on Earth and in space. The MODAT (Mission Operations Design and Analysis Toolkit) is the integration of NASA Ames' Brahms multiagent modeling and simulation environment, the Mission Simulation Toolkit (MST), a 3-D Visualization and Surface Reconstruction (Viz), plus JPL's Virtual Mission Operations Framework (VMOF) into an agent-based end-to-end mission modeling and simulation environment. This paper describes a work in progress.
This chapter presents different approaches to the design of space missions and in particular the overall integration of systems and mission design. The chapter will start with the relationship between mission analysis and system design and the role of mission analysts in the context of the overall design process.
Although the Jet Propulsion Laboratory in Pasadena, California, has become synonymous with the United States' planetary exploration during the past half century, its most recent focus has been on Mars. Beginning in the 1990s and continuing through the Mars Phoenix mission of 2007, JPL led the way in engineering an impressive, rapidly evolving succession of Mars orbiters and landers, including roving robotic vehicles whose successful deployment onto the Martian surface posed some of the most complicated technical problems in space flight history. In Exploration and Engineering, Erik M. Conway reveals how JPL engineers' creative technological feats led to major Mars exploration breakthroughs. He takes readers into the heart of the lab's problem-solving approach and management structure, where talented scientists grappled with technical challenges while also coping, not always successfully, with funding shortfalls, unrealistic schedules, and managerial turmoil. Conway, JPL's historian, offers an insider's perspective into the changing goals of Mars exploration, the ways in which sophisticated computer simulations drove the design process, and the remarkable evolution of landing technologies over a thirty-year period.
This paper compares probabilistic and possibility-based methods for design under uncertainty It studies the effect of the amount of data about uncertainty on the effectiveness of each method. Only systems whose failure is Catastrophic are considered, where catastrophic means that the boundary between success and,failure is sharp. First, the paper examines the theoretical foundations of probability and possibility focusing on the impact of the differences between the two theories on design. Then the paper compares the two theories on design problems. A major difference between probability and possibility is in the axioms about the union of events. Because of this difference, probability and possibility calculi are fundamentally different and one cannot simulate possibility calculus using probabilistic models. Possibility-based methods tend to underestimate the risk of failure of systems with many failure modes. For example, the possibility of failure of a series system of nominally identical components is equal to the possibility of failure of a single component. When designing for safety, the two methods try to maximize safety in radically different ways and consequently may produce significantly different designs. Probability minimizes the system failure probability whereas possibility maximizes the normalized deviation of the uncertain variables from their nominal values that the system can tolerate without failure. In contrast to probabilistic design, which accounts for the cost of reducing the probability of each failure mode in design, possibility tries to equalize the possibilities of failure of the,failure modes, regardless of the attendant cost. In many safety assessment problems, one can easily determine the most conservative possibilistic model that is consistent with the available information, whereas this is not the case with probabilistic models. When we have sufficient data to build accurate probabilistic models of the uncertain variables, probabilistic design is better than possibility based design. However, when designers need to make subjective decisions, both probabilistic and possibility-based designs can be useful. The reason is that large differences in these designs can alert designers to problems with the probabilistic design associated with insufficient data and tell them that they have more flexibility in the design than they may have known.
Evaluating the utility of space systems
  • S M Friedman
  • R H Weber
Friedman, S.M. and Weber, R.H. (2004) 'Evaluating the utility of space systems', in Space Modeling and Simulation, Aerospace, El Segundo, CA.
Introduction to modeling and simulation for space systems
  • L B Rainey
  • D J Cloud
  • M D Crumm
Rainey, L.B., Cloud, D.J. and Crumm, M.D. (2004) 'Introduction to modeling and simulation for space systems', in Space Modeling and Simulation, Aerospace, El Segundo, CA.
The tradespace exploration paradigm
  • A M Ross
  • D E Hastings
Ross, A.M. and Hastings, D.E. (2005) 'The tradespace exploration paradigm', INCOSE. International Symposium, Rochester, NY.
Educating the mission engineer', Mission Engineering and Analysis, and Integration and Interoperability
  • Sousa-Poza
Sousa-Poza, A. and Old Dominion University (2017) 'Educating the mission engineer', Mission Engineering and Analysis, and Integration and Interoperability, Dahlgren, VA.