ArticlePDF Available

Technical Performance Measurement, Earned Value, and Risk Management: An Integrated Diagnostic Tool for Program Management

Authors:

Abstract and Figures

This research effort, sponsored by the Program Executive Office for Air ASW, Assault, and Special Mission Programs (PEO(A)), is known as the Navy PEO(A) Technical Performance Measurement (TPM) System. A retrospective analysis was conducted on the T45TS Cockpit-21 program and real-time test implementations are being conducted on the Federal Aviation Administration's (FAA) Wide Area Augmentation System (WAAS) program, the Navy's H-1 helicopter upgrade program, and is currently under consideration for other test implementations across the Department of Defense (DoD) and in private industry. Currently-reported earned value data contains invaluable planning and budget information with proven techniques for program management, however, shortcomings of the system are its emphasis on retrospection and lack of integration with technical achievement. The TPM approach, using the techniques of risk analysis and probability, offers a promising method to incorporate technical assessments resulting systematically from technical parameter measurements to derive more discrete management data sufficiently early to allow for cost avoidance. Results obtained from TPM pilot programs, particularly the Cockpit-21 program, support this premise. Several preliminary issues of interest and conclusions are delineated in this paper that demonstrate that the TPM methodology is a powerful integrated diagnostic tool in support of the new paradigm advocating a multidisciplinary approach to program management. It also promises to provide a powerful new tool in proactive risk management.
Content may be subject to copyright.
Integrated TPM, Earned Value and Risk Management Tool - Page 1
Technical Performance Measurement, Earned Value, and Risk Management:
An Integrated Diagnostic Tool for Program Management
Commander N. D. Pisano, SC, USN
Program Executive Office for Air ASW, Assault, and Special Mission Programs (PEO(A))
Abstract
This research effort, sponsored by the Program Executive Office for Air ASW, Assault, and Special
Mission Programs (PEO(A)), is known as the Navy PEO(A) Technical Performance Measurement (TPM)
System. A retrospective analysis was conducted on the T45TS Cockpit-21 program and real-time test
implementations are being conducted on the Federal Aviation Administration’s (FAA) Wide Area
Augmentation System (WAAS) program, the Navy’s H-1 helicopter upgrade program, and is currently
under consideration for other test implementations across the Department of Defense (DoD) and in
private industry.
Currently-reported earned value data contains invaluable planning and budget information with
proven techniques for program management, however, shortcomings of the system are its emphasis on
retrospection and lack of integration with technical achievement. The TPM approach, using the
techniques of risk analysis and probability, offers a promising method to incorporate technical
assessments resulting systematically from technical parameter measurements to derive more discrete
management data sufficiently early to allow for cost avoidance. Results obtained from TPM pilot
programs, particularly the Cockpit-21 program, support this premise.
Several preliminary issues of interest and conclusions are delineated in this paper that demonstrate
that the TPM methodology is a powerful integrated diagnostic tool in support of the new paradigm
advocating a multidisciplinary approach to program management. It also promises to provide a powerful
new tool in proactive risk management.
Introduction.
In recent years the Department of Defense
(DoD) and all segments of the American economy
have been under increasing pressure to change the
way in which business is conducted. This condition
is the result of a number of converging trends and
discrete events--the end of the Cold War, a political
environment skeptical of defense expenditures and
active international involvement, a reduced industrial
base, growing international competition, evolving
quality-focused management methodologies, a
rapidly expanding and innovating Information
Technology (IT) community, pressures on
governments to reduce operations and balance
budgets--that have created an environment of
constant, rapid, and unpredictable change requiring
new management approaches and techniques.
For the government systems program managers
and their teams, this environment creates pressures
that are translated into the need to deliver products
using best value analysis with cost as the overriding
determinant. As a result, information is needed to
allow the manager to make informed trade-off
decisions as early in program execution as possible.
…the cost avoidance window of opportunity is
before the fifteen percent mark in contract
completion.
In addition, any condition threatening the health
of program development must be identified
sufficiently early to allow managers to mitigate those
areas of technical, cost, and schedule risk. The oft-
repeated rule-of-thumb within the DoD and private
industry is that the cost avoidance window of
opportunity is before the fifteen percent mark in
contract completion.1 After this point, the
opportunity for the avoidance of additional resource
consumption is greatly diminished and mitigation
focuses upon the recovery from lost effort and
remaining cost and schedule risk. This and other
research argues for concurrent risk identification and
reduction beginning in the concept phase of
programs and carrying through engineering and
manufacturing development.2
Current acquisition reform initiatives are
rapidly moving program management teams to
adoption of a holistic approach to complex systems
acquisition. Implementation of Integrated Product
Integrated TPM, Earned Value and Risk Management Tool - Page 2
and Process Development (IPPD) technique, with
its focus on integration of program management
activities through multidisciplinary Integrated
Product Teams (IPTs), has established the cultural
and structural framework for systems thinking.
However, few tools exist to support this new
paradigm.
While traditional cost and schedule analysis,
systems engineering, and risk management provide
the program management team with a broad range of
tools, many of these techniques are derived from
separate systems that are viewed in isolation from
one another. Like viewing television, each separate
image flashed before the program manager takes on
an importance and reality of its own, providing little
context relative to other factors. In addition, the
signals from each of these disciplines are being
broadcast over separate channels, and often deliver
contradictory messages.
The perspective of many of these traditional
management control systems is also retrospective in
nature, documenting history rather than providing the
program team with the essential information needed
for day-to-day management. In many Earned Value
Management Systems (EVMS), information is
normally thirty to sixty days old and identify cost and
schedule variances well beyond the window of
opportunity for cost and schedule risk avoidance.
These systems measure work accomplishment
as opposed to technical achievement.3 As a parent I
provide my son with a separate two week allowance
for his school expenses. Under systems that measure
work accomplishment based on time-phased
budgeting, his successful taking of an exam and the
expenditure of all of his money within the allotted
time would earn 100% value regardless of the grade
he achieved. The underlying weakness in this
approach is apparent.
… the basic tenets of the process are the need
for “seamless management tools” that support
an integrated approach … and “proactive
identification and management of risk” for
critical cost, schedule, and technical parameters
(former Secretary Perry’s memo of May 1995)
As it relates to program management, IPPD
guidance implicitly acknowledges these deficiencies.
In former Secretary of Defense Perry’s memo of May
1995, in which he directs the use of IPPD throughout
the DoD, two of the basic tenets of the process are
the need for “seamless management tools” that
support an integrated approach to program
management with the goal of enhancing team
decision-making, and “proactive identification and
management of risk” for critical cost, schedule, and
technical parameters compared against best-in-class
industry benchmarks that provide verification of
actual achievement of technical and business-based
parameters.4
In support of IPPD, and as of this writing, DoD
5000.2-R is being updated to require Integrated
Baseline Reviews (IBRs) within six months after
contract award to ensure reliability in planning and
performance measurement.5 For those program
offices that have conducted an IBR, important insight
has been gained by members of the IPT into the
interrelationships between various management
control systems and processes.
Both the commercial systems engineering and
cost/schedule analysis communities are undergoing
similar change. The proposed revision to EIA/IS-
632, an industry systems engineering standard,
recently reviewed at the annual meeting of the
International Council on Systems Engineering
(INCOSE), includes Technical Performance
Measurement (TPM) as a critical product metric.6 In
addition, within the industry standard EVMS,
technical performance goals are listed as necessary
indicators to be used in order to measure
programmatic progress among its 32 criteria.7
The tools required to support the demands of
this new environment must be those that:
(a) provide an integrated view across programmatic
elements;
(b) support the process of distributed empowerment
implicit in the IPT approach;
(c) logically organize data resulting from systems
engineering, risk management, and earned value
processes;
(d) provide a “real time” indication of contract
performance and future cost and schedule risk;
(e) support the development of systems thinking
within an integrated program model.
The Program Executive Office for Air Anti-
Submarine Warfare, Assault and Special Mission
Programs PEO(A) TPM system is a promising
methodology that addresses these goals and fits well
within the basic tenets of IPPD and acquisition
reform by integrating technical performance with
earned value based upon programmatic risk
assessments and probability. This methodology
provides a flexible framework, based on effective
business practices, that provides government-
contractor program management teams with the
information they need to make informed management
decisions at critical milestones.
Background.
Integrated TPM, Earned Value and Risk Management Tool - Page 3
The TPM project was undertaken in 1991 by
the PEO(A) within the Naval Aviation Systems Team
(NAST). From its earliest inception, the PEO
recognized the need for an integrated approach to
monitor program performance based upon the simple
principle that the solution be of practical utility to the
program office. Consequently, in early 1991, a team
consisting of representatives from each PEO(A)
program office was organized to identify and validate
a process for the integration of cost, schedule, and
technical performance metrics. After several
meetings and off-site planning conferences, this
group generated a requirements specification that
became the basis for the project.
This document identified the need for a
standard process for baseline planning, tracking, and
reporting of technical performance measurements in
a manner similar to, and concurrent with, cost and
schedule metrics. In addition, the document
specified the need for a means of determining cost
and schedule impacts based upon technical
performance. In 1993, the TPM Working Group
selected both a proof-of-concept and commercial-off-
the-shelf (COTS) implementation strategy to achieve
the goals of the requirements specification.
The proper identification of technical
performance parameters (TPPs) and the validity
of technical baseline establishment were seen
early in the project as a key to the proof of
concept.
The proper identification of technical
performance parameters (TPPs) and the validity of
technical baseline establishment were seen early in
the project as a key to the proof of concept. The Air
Deployable Active Receiver (ADAR), a sonobuoy
program, was the first pilot project selected to test the
basic premise that a systematic TPM planning and
tracking function would provide an early warning,
significantly before legacy performance measurement
systems, to be of practical benefit. Insights gained
from this pilot:
First, cost and schedule impact assessments
could not always be clearly determined because
there was not clear linkage between technical
parameters and budgeted work packages via the
Work Breakdown Structure (WBS).
Second, where cost and schedule impact
assessments could be made, the linkage could be
made at a fairly high level within the WBS (level
4) and all work packages could be associated
directly with the parameter.
Third, a statistical association of technical
accomplishment inserted into cost and schedule
could produce calculated impacts amazingly
close to what was eventually experienced.
Fourth, the identification and tracking of
technical performance metrics in a disciplined
and systematic fashion provides significant early
warning of potential problems and their nature.
With the promising results from the ADAR
program, the TPM Project Team selected the Light
Airborne Multipurpose System (LAMPS) Block II
Upgrade, another helicopter program, for its next
pilot. The Block II program was selected based upon
its complexity, its high dollar value, and the high
technical risk inherent in the effort. The TPM team
also decided, concurrent with this pilot, to apply
economic utility theory as the means for determining
the technical metric that would be used for
calculating cost and schedule impacts.
The results from LAMPS Block II were not as
encouraging as in the ADAR pilot but, in hindsight,
of greater value:
First, the technical parameters collected were of
too high a level and did not derive from
disaggregated lower level parameters.
Second, the practical application of the utility
curve assessment approach proved both
impractical and theoretically unsound.
Third, the overall framework of estimating cost
and schedule impacts using the “value” of
technical progress as a foundation was not
flexible enough to exploit the full range of
existing cost estimating techniques. The
framework relied solely upon expert opinion--
the most subjective method in the cost estimating
arsenal--eschewing through the approach both
parametric and industrial engineering
measurement.
While pursuing its proof-of-concept effort, the
TPM Project Team formally surveyed both the
commercial sector and other government program
offices to determine if other methodologies or
products existed that would meet the goals of the
requirements specification. After extensive research,
only one untested commercial product seemed to
show promise. This product, TCS Integra by
Quantitech Inc. of Huntsville, Alabama, was selected
for test implementation and the results from its
retrospective pilot implementation on a PEO(A)
program are presently undergoing review.
By the fall of 1995, with a record of one win
and one loss, and an apparent wrong turn in selecting
economic utility theory as its framework, the TPM
Project Team conducted a thorough reevaluation of
its mission and methodology. It was clear that if the
original goal of the requirement specification was to
Integrated TPM, Earned Value and Risk Management Tool - Page 4
be met, an 80% solution that was well-grounded in
each of the disciplines of systems engineering, risk
management, and cost/schedule analysis would have
to be found.8
The Key: Systems Thinking
and Probability.
In all systems where optimum performance is
necessary or desired, it is useful to establish what
engineers call a negative feedback system. Negative
feedback is the basis for automatic control and
regulation. It can best be understood by a description
of its opposite, which is positive feedback. In
chemistry, positive feedback usually takes the form
of an explosion. In program management, positive
feedback will take the form of a program requiring
greater and greater commitment of resources for
achievement of requirements specifications well
beyond what was originally anticipated.9
Most U.S. Navy ships still use steam as their
main source of propulsion. In the 18th century one
of the roadblocks to the effective use of steam
technology was the inability to control steam
pressure. This inability persisted until James Watt
(1736-1819) and the Watt steam governor came
along. The principle of the governor was to create an
automatic valve that would regulate the flow of steam
to the piston. The trick was to link the valve to the
rotary motion of the engine. The faster the engine
moves, the more the valve shuts down. The slower
the engine moves, the more the valve opens up. The
means used was just as simple and elegant. A pair of
balls on hinged arms spin around using the principle
of centrifugal force. When the balls spin fast, they
rise up on their hinges; when spinning slowly they
hang down. The hinged arms are linked to the steam
throttle.
Our feedback systems must be robust enough to
give us a discrete indication of variability in
progress to allow for adjustments to be made.
With effective tuning, the Watt governor keeps
the engine turning at a constant rate despite
fluctuations from the source of heat. The Watt steam
governor was responsible for the effective use of
steam in industrial production, giving rise to the
industrial revolution, and to the creation of great
navies that could navigate the oceans independent of
the wind.
What the program manager needs is the
equivalent of a Watt steam governor. If we view a
program as a system, what we see are resources as
our inputs (in terms of money, time, and expertise)
with the end item (e.g., a ship, aircraft, or satellite) as
our output. Our feedback systems must be robust
enough to give us a discrete indication of variability
in progress to allow for adjustments to be made.
Preferably, our feedback systems will be negative
ones, but, as we all should know, social systems, of
which a program is one, are more complicated than
our analogy to a relatively simple steam plant.
A program, as an organization, is a type of
complex adaptive system. A complex adaptive
system is one that acquires information about its
environment and its interactions within the
environment, identifies information of importance,
places that information within the context of a
contextual framework, model, or “schema,” and then
acts on the basis of that schema.10 The individual
members of the program office--people--act as
complex adaptive systems themselves and exert a
powerful influence on the selection of both schema
and those adaptive pressures that are used in making
decisions. The extent to which their learning brings
about adaptive or maladaptive behavior will
determine the survival or failure of the
organization.11
In constructing a negative feedback system for
a complex adaptive system, an understanding of the
schema and context in which the system functions is
necessary. Also, the model should be as simple as
possible and only contain those elements absolutely
necessary for approximating reality.
The IPPD technique provides the necessary
schema around which to construct our tools, with its
emphasis on:
decentralized authority through the IPTs,
the renewed importance of cost as the
independent variable,
the use of performance specifications in
acquisitions, and
the emphasis on advance planning and quality as
a by-product of the work performed.
That these business practices are becoming
universal in both government and private industry
also lends us valuable insight.
Stephen Jay Gould, the noted Harvard
polymath, in his book Full House, in using the
disappearance of the .400 hitter from baseball as his
subject to demonstrate increasing excellence of play,
illustrates that complex systems tend to organize
themselves as a set of probable outcomes, often
within a normal distribution. Variation in this
distribution changes over time as members become
familiar with their environment. Gould concludes
that (a) complex systems improve when the best
performers play by the same rules for extended
Integrated TPM, Earned Value and Risk Management Tool - Page 5
periods of time, and (b) as play improves and bell
curves march toward the right wall, variation must
shrink.12 Implicit in these conclusions is the effective
ability of members of the complex system to learn
and adapt. 13
Foundation … With the adoption of earned
value management and critical path scheduling as
industry standards, the foundation was laid in the fall
of 1995 for the TPM Project Team to use the
principles of systems thinking and apply them to the
existing disciplines of cost/schedule control, risk
management, and systems engineering to create an
integrated diagnostic tool. Also, the rapid advances
in desktop computing power, even within the short
life of the project, brought with it the ability to cost-
effectively integrate these concepts.
First choice … The first choice was to ensure
that the methodology was integral to existing
processes involved in planning and tracking program
performance, and supported the cultural and
structural processes established under IPPD. It was
decided that both a TPM process and a practical,
interactive tool would be developed to facilitate an
understanding of technical, cost, and schedule risk
issues.14
Second choice … The second choice to make
was to select the way in which technical performance
impacts would be expressed. The team decided that,
with the emphasis on cost, the industry EVMS would
be used as the user interface. This approach had
worked well on ADAR. This meant that the
Budgeted Cost for Work Performed (BCWP), or
“earned value,” would need to be informed by
technical achievement, but in a way that would lend
credence to the projected impact.
Approaches … Within the risk management
discipline, there are basically two general approaches
for estimating cost impacts--probabilistic and
deterministic. A probabilistic approach is top-down,
based upon the probability of outcomes. The Monte
Carlo analysis model is a good example of a
probabilistic approach. A deterministic approach is
bottom-up, based upon a sequence of causes.
Learning curve estimation is a deterministic method,
though it still possesses a large element of
probability. Probabilistic models are by nature
inexpensive to apply but sometimes lack credibility.
Deterministic models are more expensive to apply
but credibility is also an issue if the work is not
disaggregated properly. Also, as noted above, no
model is ever completely deterministic--the proper
mix between the two approaches must be selected. 15
This last point goes to the heart of the approach
eventually selected. Risk determination is, by nature,
probabilistic. As we noted above, complex systems
tend to organize themselves in a normal distribution
of likely outcomes. As procedures and practices
become standard, the best performers tend to follow
the general trend toward excellence and variation
around the mean shrinks. Other distributions apply
in less mature environments, but a statistical tool
using the assumption outlined above as its basis
should meet the requirement of providing sufficient
early warning of technical perturbations in program
development as long as the technical metrics are
derived systematically, a planning baseline is
established, and technical performance parameters
are disaggregated and properly tied to the WBS.16
Earned Value calculation … With the
assistance of Naval Reserve Unit NAVAIRSYS 1187
under the command of CAPT A. R. Pagnotta, USNR-
R (now retired), a unit consisting of information
technology and systems engineering professionals,
statisticians, and mathematicians, the TPM Project
Team reengineered the TPM method of calculating
technical earned value. Technical earned value
would be the key metric used in recalculating BCWP
and also used in the algorithm to calculate schedule
impacts.
Gantt charts are the standard tracking tool
linking program activities with time. Each TPP
normally has a progress plan assigned to it
based on how the development activities will be
performed.
It is a common commercial practice to segment
work into key product development paths, or
technical progress plans, assignable to specialties
within a function or functions. Gantt charts are the
standard tracking tool linking program activities with
time. Each TPP normally has a progress plan
assigned to it based on how the development
activities will be performed.17
With this in mind, the team selected as its
method for calculating technical earned value:
A 90-50-10 risk profile, that is equivalent to the
probability of successfully achieving the next
TPM milestone [Pr(S)].
The profile is then applied at each assessment
date, which could be monthly, or some other
period.18
This approach isolates technical performance,
in terms of technical achievement and deviation,
from cost and schedule.
The means of establishing the 90-50-10
probability distribution exploits standard risk
management estimation techniques based on analogy,
parametrics, and industrial estimation, and is
constructed concurrent with, and integral to, the
Integrated TPM, Earned Value and Risk Management Tool - Page 6
establishment of the Systems Engineering
Management Plan (SEMP). The methodology of
using technical progress plans and applying a risk
profile against each assessment date is similar to the
establishment of a formal baseline for WBS work
package budgets and schedules. The baselining
issues raised through this process are then of service
during the IBR.
Once again, the application of systems thinking
is instructive in understanding the application of the
probability distribution. A popular analogy concerns
placing a monkey in front of a typewriter. According
to this story, given enough time, the monkey would
eventually produce the collected works of
Shakespeare. Unfortunately for the analogist,
systems, even live ones, do not work this way. Rare
is the sudden act of creation from whole cloth.
Richard Dawkins, the Charles Simonyi Chair of
Public Understanding of Science at Oxford, limited
his simulated computer monkey to producing, in a
single random step, the sentence uttered by Polonius
in the play Hamlet: “Methinks it is like a weasel.”
The odds of getting it in a single step is about 1 in
10,000 million million million million million
million--requiring a longer time to achieve than all of
the time that has expired since the beginning of the
universe. When, however, the monkey used
cumulative progress, built from previous steps in
achievement, the computer built the target phrase in
generation 43 on the first run and in generation 64 on
the second run.19
This example demonstrates that establishing the
probability of successfully achieving the next
technical performance milestone is the proper
approach in deriving a technical earned value. The
probability of successfully achieving an end goal in a
single step is vanishingly small and, if applied in a
methodology, will give us an overstated negative
impact. In our approach, however, each Pr(S)
represents a discrete event along our progress plan
that, when combined with previous scores, gives us
an assessment of the cumulative achievement along
the development path. Breaking down the path into
these discrete probability assessments also has the
effect of isolating and reducing subjectivity.
Development is, after all, an evolutionary process
built on the cumulative effort expended toward the
achievement of eventual program goals.
An Integrated Diagnostic Tool.
Having resolved the major issues of its bottom-
up review, in November 1995 the TPM Project Team
concurrently pursued the development of both a
methodology and software application to achieve the
integration of cost, schedule, and technical
performance. Several additional meetings resulted in
the development of a general framework that would
use the existing internal management methodologies
of prime contractors, as much as practicable, and to
reorganize existing data in a way to achieve the
desired integration.
The final result of these meetings was a
methodology consisting of three phases. Figure 1
provides an overview of the entire methodology.
Plan TPM
Progress Determine
Risk
Program
Requirements Program
Cost & Schedule Government &
Contractor Input
Gov’t/Contractor
Team Select
Technical
Parameters
Weight xxxxxx
Speed xxxxxx
MTBF xxxxxx
Parameter List Progress Plan
100
1
Risk Profile
Figure 1: TPM Methodology Overview
1. Select Technical Parameters. Concurrent with
the formulation of the SEMP, the following criteria
are used to assist in the selection of critical TPPs:
Those that are program cost drivers, reside on
the critical path schedule, or that represent,
based on formal assessment, high risk to the
program.
Once selected, TPPs are organized in a
weighted hierarchy to establish relative
importance and interrelationships.
Linkage is made to the contract WBS. This
last activity is accomplished in order to obtain
a “technical budget baseline,” or the budget
associated with the work packages that are
responsible for a particular parameter’s
developmental success, and that would
ultimately be placed at risk should
performance not meet expectations.
2. Plan TPM Progress. The second phase is to
baseline technical performance measurement through
the establishment of a technical progress plan for
each TPP. The approach to planning and baselining
TPM is virtually the same as that used in baselining
cost and schedule measures with the common goal of
establishing a framework from which to assess actual
progress and measure relative performance. A
disaggregate approach is used to reduce subjectivity
by developing progress plans for lower level TPPs
and applying the cumulative scores from these lower
level plans to higher, summary level, TPPs. Figure 2
Integrated TPM, Earned Value and Risk Management Tool - Page 7
shows a typical progress plan for the weight of a key
component.
Expected
Actual
70
11/01/95
01/21/96
08/10/96
02/01/97
65
60
55
40
35
30
25
45
50
lbs
Sample Progress Plan
(weight - lbs - is being measured)
Figure 2: Sample Progress Plan
In the Gantt chart used as our example, the
technical assessment activity dates are displayed
along the horizontal axis and the units of measure
are on the vertical axis. The straight line at the 32.5
pound mark represents the end goal. The lower
downward sloping line marked by open triangles is
the technical progress plan while the bolder upper
sloping line marked by filled triangles traces the
actual achievement. This component was considered
to be critical to the end item design and a bellwether
of overall aircraft weight.
A TPP may be any function, physical
characteristic, design goal, or other aspect of a
project that has been defined by the
requirements of the program.
Looking at our example in isolation from other
factors and other technical parameters points out the
weakness of a reductionist approach to program
management in which TPMs are collected apart from
other activities. The importance of the technical
variance, that is, the space between the open and
closed triangles is unknown and can easily be
dismissed. Presently, an assessment of this variance
relies too heavily on expert opinion. Consequently,
this process is highly subjective and leaves open the
possibility of a “rubber” technical baseline--one in
which the significance of any variance can be
misinterpreted. This is the importance of establishing
a TPP hierarchy and weighting activities relative to
each other in an IPT environment.
The weight example provided is, of course,
highly simplified and given only as an analogue for
development metrics that may be used to track
technical progress. A TPP may be any function,
physical characteristic, design goal, or other aspect of
a project that has been defined by the requirements of
the program. Both process and product metrics are
candidates that may be selected as TPPs. In software
development, our experience indicates that
qualitative process metrics, such as staffing and error
reporting, are more reliable indicators of poor
technical achievement than more traditional product
metrics such as source lines of code.
3. Determine Risk. The third and last phase is to
apply the 90-50-10 risk profile to each planned value
within the technical progress plan to serve as a
benchmark against actual achievement. Insertion
into earned value is accomplished by applying the
confidence factor as the technical performance score
to calculate a technically informed BCWP. Critical
path schedule impacts are calculated similarly by
applying the confidence factor as the achievement
metric against the portion of the schedule placed at
risk by the technical parameter(s).
70
11/01/95
01/21/96
08/10/96
02/01/97
65
60
55
40
35
30
25
45
50
lbs
50
90
10%
50%
90%
Technical
Variance
Expected
Actual
Figure 3: Progress Plan with Risk Profile
Figure 3 above illustrates the technical
variance for our component weight example. A
probability distribution is applied to every
assessment date on the progress plan, which has
been rotated so that the original y-axis is horizontal.
In this example, the second assessment date is
magnified to show in detail the relationship
established between the actual measures and the
assessed probabilities of success [Pr(S)]. The actual
measure is above the expected value, a potentially
unfavorable condition when measuring weight
reduction. In this case the actual measure falls at the
90 percent confidence level. Interpolation is used to
calculate values that fall on intermediate points
along the slope of the distribution.
In this particular example, a single-sided
probability distribution is applied since our area of
interest is only in the area above the technical
progress plan. In cases where a variance
Integrated TPM, Earned Value and Risk Management Tool - Page 8
significantly above or below the expected value is an
indicator of poor technical achievement, such as in
the case of software problem reports or staffing, a
double-sided distribution is applied to determine the
earned technical value.
Distributions may be customized based upon
any standard analogous, parametric, or industrial
engineering technique. In some cases, a tolerance
band is wider at the beginning of the technical
activity and becomes tighter over time. Should
customization not be applicable in every case, a
default normal distribution is applied that establishes
the 90 percent confidence factor to actuals at 5
percent deviation from expectation, 50 percent
confidence at 10 percent deviation from expectation,
and 10 percent confidence at 20 percent deviation
from expectation.
The technical earned value is inserted into
EVMS by multiplying the confidence level by the
BCWS for the WBS elements associated with the
TPP. For example, if the cumulative BCWS for a
WBS element is $50,000, and the cumulative
technical earned value is 50 percent, then the
calculated BCWP is $25,000. Once BCWP is
calculated, performance indices, variances, and
estimates at complete can be calculated.20
Critical path schedule impacts are also
calculated by using the confidence level metric. This
is accomplished by taking the baseline schedule for
the associated activity and multiplying the portion of
the activity placed at risk by the confidence level
subtracted from 100 percent confidence. For
example, a scheduled activity is 90 days duration. If
the cumulative confidence level is 90 percent and all
90 days of the activity is placed at risk by the
associated technical performance activity, then a 10
percent lost effort is calculated to project a possible
slip of nine days for the activity. Using standard
COTS critical path scheduling tools, impacts for
associated activities can be calculated automatically.
The Cockpit-21 program provided an excellent
analysis environment with several years of Cost
Performance Reports (CPRs) and technical
performance data that included comprehensive
planning and reported actuals.
Proving the Methodology: The T45TS
Cockpit 21 Retrospective Analysis
Once settling on a methodology, questions
arose for the TPM Project Team concerning
interfacing the TPM software tool with EVMS
software packages. Performance Analyzer (PA), the
standard government software tool, proved not to be
Windows capable and did not offer all of the features
desired for calculating earned value impacts. As a
result, the team settled on a Windows-compliant
COTS EVMS tool known as wInsight with which to
establish compatibility. The plan was to fit the TPM
tool later to PA once it became Windows capable.
For the moment, however, wInsight would be the
product used to calculate EVMS impacts.
To prove out the methodology, the TPM Project
Team decided that a retrospective analysis conducted
against actual outcomes on a mature contact would
be the best approach. The exit criteria for a
successful test would be (a) early warning of
technical perturbations in the program before the 15
percent mark in the contract, (b) early warning
significantly before indications from traditional
performance measures, and (c) calculated cost
projections that were statistically significant when
compared against actuals.
At the end of November 1995, the Program
Executive Officer (PEO) gave his approval to the
TPM Project Team to use the T45TS Cockpit 21
project as its proof-of-concept test bed and the
project began in earnest. The Cockpit 21 program
was managed by the Jet Flight Training Program
Office (PMA 273) under PEO(A). The program
involved the development and installation of a digital
cockpit into the T-45A aircraft. Previously, this
aircraft was procured using analog cockpit
instrumentation. The new cockpit has improved
training effectiveness by enhancing the cockpit with
a digital data bus and multi-function displays that
will more closely resemble features on an increasing
number of newer fleet aircraft.
A three-year letter contract was awarded on 29
May 1992 to McDonnell-Douglas (MDA) as the
prime contractor, however, this contract was not
definitized until March 1994. The delay in
definitization was attributable solely to pricing
issues. The system requirements were very well
defined and had no impact on contract definitization.
Smith’s Industries of the United Kingdom was
the subcontractor awarded a major portion of the
software development work for the digital cockpit.
The contract between MDA and Smith’s was Firm
Fixed Price (FFP). After some time for ramp-up,
work on Cockpit 21 development commenced in full
in July 1992.
The Cockpit-21 program provided an excellent
analysis environment with several years of Cost
Performance Reports (CPRs) and technical
performance data that included comprehensive
planning and reported actuals. The program’s
reported cost and schedule information showed good
performance even in the Defense Acquisition
Integrated TPM, Earned Value and Risk Management Tool - Page 9
Executive Summary (DAES) system, but the
products being developed were clearly not meeting
the schedule. Technical perturbations were being
experienced that were not being reflected in the cost
reports.
To ensure that the TPM Project Team could
conduct the study in an objective fashion, certain
information was withheld by the program office.
Specifically, this included the renegotiated contract
ceiling that had recently been settled, and additional
cost risk identified by the prime in negotiations.
Also, only one interview was conducted with the
program technical representative to determine the
risk items under the TPP selection criteria. That
person was asked specifically to express his
judgment and concerns as he recalled them at
contract award. This condition would at least bound
the analysis to a single-blind approach. Otherwise,
only those formally reported metrics were used to
load into the software tool developed to support the
methodology.
The Display Generation WBS element was the
largest single budgeted item reported. Since this
directly related to the software development of the
Cockpit 21 digital displays, known as the Display
Electronics Unit (DEU), the Software Development
Plan (SDP) was reviewed for documentation on
software metrics. A good set of metrics was reported
and these were used in the test set. The SEMP also
contained metrics on Flight Test Problem Reporting
and these were also included. Other metrics were
also found for Reliability, Cooling Requirements,
and Electrical Power. These elements, however, did
not meet the criteria for selection of TPPs and were
not included in the test. In any event, since technical
achievement in these areas exceeded the technical
progress plans, they would not have affected the
results of the study.
Test results … At the end of January 1996, the
test results on the Cockpit 21 program were formally
reported to the PEO. The analysis showed that:
(a) Technical perturbations in software development
containing significant cost risk are first identified
in November 1992 before the 15 percent point in
contract performance.
(b) These results are consistent from November
1992 and throughout 1993, providing an
eighteen month early warning before other
performance measurement techniques began
reporting mitigating activities.
(c) The methodology was discrete enough to
identify the specific activities contributing to
technical cost risk.
(d) Calculated estimates-at-complete using the
wInsight tool were within one standard
deviation of the negotiated contract ceiling of
$68 million and the additional cost risk
identified by the MDA--mitigation instituted by
the prime on the FFP subcontract impacted total
program cost. All elements of the study’s exit
criteria had been met.21
As a result of the positive results of the TPM
study, an internal Peer Review of the Cockpit 21
study results was conducted from February to April
1996. Members for the Peer Review Committee
were assembled from the Naval Aviation Systems
Command (NAVAIR) cost analysis and systems
engineering competencies, PEO(A) program offices,
Defense Systems Management College (DSMC), the
Institute for Defense Analyses (IDA), and private
industry. The TPM Project Team authored a white
paper and submitted their data and results to intense
scrutiny by the group. On 24 April 1996, the peer
review group endorsed TPM as a promising
approach, ratified the plausibility of the TPM
mathematical concepts, and recommended
prospective implementation of the methodology on
future contracts.22
A False Start and A Good Start.
As a result of the positive results from Cockpit
21, and the recommendation for prospective
implementation by the peer review group, the Navy’s
Stand-Off Land Attack Missile-Expanded
Response (SLAM-ER) program sought to
implement the TPM methodology to identify and
mitigate existing cost and schedule perturbations and
to avoid future ones. The program, being 41 percent
complete, was thought to provide a unique
opportunity for applying the methodology both
retrospectively and prospectively. In hindsight,
however, this factor was the main barrier to
implementation.
The retrospective application of a process
improvement tends to be costly and requires changes
to existing procedures. The process also becomes
external to the system and excludes stakeholders
from critical decision-making in the application of
the process--it documents history and nothing more.
As a result, rather than enhancing existing processes,
application of TPM on SLAM-ER would have been
prescriptive on both the program office and the prime
contractor. In addition, requirements for technical
performance data were eliminated from the contract
deliverable items under the aegis of “streamlining”
and the program office was not willing to invest
additional resources in implementing the process.
Integrated TPM, Earned Value and Risk Management Tool - Page 10
Lessons learned … The lessons learned from
SLAM-ER were instructive and led to adjustments in
the manner in which the TPM methodology would be
applied in future. Only new contracts would be
selected for implementation. In addition, a feasibility
study would be conducted prior to commitment of
project resources and costs for implementation would
be identified up-front to the program office. The
project would also use standard risk management
return-on-investment criteria to determine if an
implementation was feasible, and develop other
metrics to determine the methodology’s utility.
In July 1996 the TPM Project Team accepted a
request from the Federal Aviation Administration’s
(FAA) Wide Area Augmentation System (WAAS)
program office to apply TPM as a joint pilot
implementation. The WAAS program is developing
an enhanced Global Positioning System (GPS) to
provide accurate aircraft reporting anywhere on or
near the surface of the earth. The system will
ultimately satisfy the requirements of all phases of
flight, including precision approach and landings.23
The prime contractor is Hughes Information
Systems Company of Fullerton, California. The
contract value, negotiated in October 1996, is
approximately $500 million. The TPM database for
this effort has been populated and implementation
continues with the first reported cost reports. The
initial results are encouraging.
Before contract award the program office
committed itself to ensure that a comprehensive TPM
approach that included the PEO(A) methodology
would be properly implemented and funded. The
TPM planning, which included Hughes personnel
through the construction and verification of TPPs,
progress plans, and probability distributions was
exhaustive and required contractually-defined
deliverables (CDRLs). The approach, because it
involved the substitution of electronic for more labor-
intensive paper-based SEMP deliverables, has had
the effect of reducing effort involved in CDRL
delivery. The WAAS program office also included
TPM-informed information in the Performance
Evaluation Plan (PEP) as a determining factor in the
contract award fee arrangement.
Preliminary conclusions … Some preliminary
conclusions can be drawn from the WAAS
experience. First, the contractor-government
program team have gained insights into the
relationship between cost, schedule, and technical
achievement issues that would not have otherwise
been available. Second, the TPM approach is
compatible with and complementary to existing TPM
and risk assessment methods used by the prime
contractor. Third, the initial results from the first
cost reports indicate that technical perturbations
revealed by TPM reinforce an interactive
environment supportive of the IPT structure. Fourth,
costs and efforts associated solely with the
implementation, while requiring significant advance
planning, are marginal.
Conclusion.
The PEO(A) TPM methodology provides a
promising first step in the integration of cost,
schedule, and technical performance. It achieves this
goal through a flexible and robust methodology that
is of practical use to both government and
commercial program managers and their teams. This
methodology has evolved over time and will continue
to do so as current and future implementations
provide additional insight into proactive technical
risk management.
As an interactive and integrated diagnostic tool,
TPM promises to provide necessary insight into those
issues of importance to IPT members, thereby
supporting the concept of distributed empowerment,
and to provide sufficient early warning of technical
cost risk to allow for early mitigation and cost
avoidance. It also promises to provide an integrated
tool to the business of program management that will
support the new management paradigm of functional
integration and systems thinking.
References
Integrated TPM, Earned Value and Risk Management Tool - Page 11
References
Christensen, David D. “Cost Overrun Optimism:
Fact or Fiction?” Acquisition Review Quarterly 1
(Winter 1994).
Coleman, Charles; Kathryn Kulick, and CDR Nick
Pisano. “NAVAIR PEO(A) Technical Performance
Measurement (TPM) Retrospective Implementation
and Concept Validation on the T45TS Cockpit-21
Program.” (Unpublished white paper) Washington,
DC: April 24, 1996.
Dawkins, Richard. The Blind Watchmaker. New
York: W.W. Norton, 1996.
Department of Defense. Acquisition Management
Policies and Procedures. (Draft DoD Directive
5000.1 and Draft DoD Instruction 5000.2).
Washington, DC: October 1995.
Department of Defense. (Draft DoD Regulation
5000.2-R). Mandatory Procedures for Major
Defense Acquisition Programs (MDAPs) and Major
Automated Information System (MAIS) Acquisition
Programs. Washington, DC: December 1996.
Engineering Industrial Association, EIA 632 Version
0.8. “Processes for Engineering a System.” (January
1997).
Federal Aviation Administration. WAAS Program
Evaluation Plan. (Draft) Washington, DC: 1996.
Ferris, Timothy. The Mind’s Sky: Human
Intelligence in a Cosmic Context. London: Bantam
Press, 1992.
Gell-Mann, Murray. The Quark and the Jaguar.
New York: W.H.Freeman and Company, 1995.
Gould, Stephen Jay. Full House. New York:
Harmony Books, 1996.
Michaels, Jack V. Technical Risk Management.
Upper Saddle River, NJ: Prentice Hall PTR, 1996.
Naval Aviation Systems Command. Peer Review
Group Recommendation. April 24, 1996
NSIA Management Systems Subcommittee, EVMS
Work Team. Industry Standard Guidelines for
Earned Value Management Systems. August 8,
1996.
Secretary of Defense Perry, W. J. “Use of Integrated
Product and Process Development and Integrated
Product Teams in DoD Acquisition.” Washington,
DC: Office of the Secretary of Defense, May 1995.
Simons, Robert. “Control in an Age of
Empowerment.” Harvard Business Review (March-
April 1995).
Under Secretary of Defense (Acquisition &
Technology) Kaminski, P.G. Reengineering the
Oversight and Review Process. Washington, DC:
1995.
Under Secretary of Defense (Acquisition &
Technology) and Assistant Secretary of Defense for
C3I. Rules of the Road: A Guide for Leading
Successful Integrated Product Teams. Washington,
DC: November 1995.
Integrated TPM, Earned Value and Risk Management Tool - Page 12
Endnotes
1 David D. Christensen, “Cost Overrun Optimism:
Fact or Fiction?” Acquisition Review Quarterly 1
(Winter 1994): 29.
2 Jack V. Michaels, Technical Risk Management
(Upper Saddle River, NJ: Prentice Hall PTR, 1996):
9.
3 NSIA Management Systems Subcommittee, EVMS
Work Team, Industry Standard Guidelines for
Earned Value Management Systems (August 8,
1996): 3-10.
4 Secretary of Defense W. J. Perry, “Use of
Integrated Product and Process Development and
Integrated Product Teams in DoD Acquisition”
(Washington, DC: Office of the Secretary of
Defense, May 1995): Attachment 2.
5 Department of Defense Regulation 5000.2-R
(Draft), Mandatory Procedures for Major Defense
Acquisition Programs (MDAPs) and Major
Automated Information System (MAIS) Acquisition
Programs (December 1996): Section 3.3.4.6.
6 Engineering Industrial Association, EIA 632
Version 0.8. “Processes for Engineering a System”
(January 1997): 33-34.
7 NSIA Management Systems Subcommittee,
Industry Standard Guidelines for Earned Value
Management Systems, 2-1.
8 All of the information for this section was
summarized from Charles Coleman, Kathryn Kulick,
and CDR Nick Pisano, “NAVAIR PEO(A) Technical
Performance Measurement (TPM) Retrospective
Implementation and Concept Validation on the
T45TS Cockpit-21 Program,” (Unpublished white
paper), April 24, 1996.
9 Richard Dawkins, The Blind Watchmaker (New
York: W.W. Norton, 1996): 196-198.
10 Murray Gell-Mann, The Quark and the Jaguar
(New York: W.H.Freeman and Company, 1995):
17.
11 Ibid., 298-301.
12 Stephen Jay Gould, Full House (New York:
Harmony Books, 1996): 111-128.
13 Using an athletic activity as a model of
organization for complex social systems is not as
restricted as some may argue. Sufficient evidence in
the field of neuroscience indicates that certain
athletic functioning, such as hitting and throwing,
resides in the premotor cortex of the brain--the same
location used for discriminating patterns, logical
thinking, and abstract thought. See Timothy Ferris,
The Mind’s Sky: Human Intelligence in a Cosmic
Context (London: Bantam Press, 1992): 108-112.
14 For an effective discussion of Interactive Control
Systems see Robert Simons, “Control in an Age of
Empowerment,” Harvard Business Review (March-
April 1995): 80-88.
15 Michaels, 75-78.
16 A complete discussion of aggregation vs.
disaggregation can be found in Michaels, 76.
17 Commanding Officer, NR NAVAIRSYS 1187
memo Ser 075 dated 19 Nov 95.
18 Ibid.
19 Dawkins, 46-49.
20 Coleman, Kulick, and Pisano, 8-15.
21 Ibid., 15-56.
22 Peer Review Group Recommendation dated 24
April 1996.
23 Taken from the Introduction to the WAAS
Program Evaluation Plan draft, undated, 1996.
... Other approaches focus on identifying and developing metrics for performance measurement. For example [66], adapts the notion of technical performance measure (TPM) [53] to the SoS context, proposing a hierarchical metric called SoS performance measure (SPM). However, the focus is strictly on defining a measurement metric, which is not subsequently used for analysis to demonstrate its feasibility. ...
Article
Full-text available
Systems of systems exhibit characteristics that pose difficulty in modelling and predicting their overall performance capabilities, including the presence of operational independence, emergent behaviour, and evolutionary development. When considering systems of systems within the autonomous defence systems context, these aspects become increasingly critical, as constraints on the performance of the final system are typically driven by hard constraints on space, weight and power. System execution modelling languages and tools permit early prediction of the performance of model-driven systems; however, the focus to date has been on understanding the performance of a model rather than determining whether it meets performance requirements, and only subsequently carrying out analysis to reveal the causes of any requirement violations. Moreover, such an analysis is even more difficult when applied to several systems cooperating to achieve a common goal—a system of systems. In this article, we propose an integrated approach to performance prediction of model-driven real-time embedded defence systems and systems of systems. Our architectural prototyping system supports a scenario-driven experimental platform for evaluating model suitability within a set of deployment and real-time performance constraints. We present an overview of our performance prediction system, demonstrating the integration of modelling, execution and performance analysis, and discuss a case study to illustrate our approach.
... The probability distributions used to represent a project's capabilities and estimations regarding each dimension of technical performance are related to TPM tracking, which has long been used in the development of complex systems (e.g., Coleman et al. 1996;DSMC 1990;Kulick 1997;NASA 1995;Pisano 1996) to monitor the status of important attributes of an evolving system design. Figure 6 shows an example of a basic TPM tracking chart, where the estimated maximum range of an aircraft is updated over the course of a project, albeit in hindsight, as new data become available. ...
Chapter
Large, complex, innovative development projects, such as many in the aeronautical industry, occur under conditions of consequential uncertainty, i.e. risk. Risk is manifest with respect to project cost, schedule, and technical performance. These risks need to be managed in relation to each other as well as to the project schedule, budget, and technical requirements. This chapter presents an integrated model to plan and monitor project value in terms of all of these areas. The model complements conventional approaches such as earned value management, which does not directly account for technical performance or any kind of risk. The model allows a user to quantify a project's overall value, the portion of that value at risk, and the respective contributions of cost, schedule, and technical performance to that value and risk..
... In lieu of this, their system utilized a probabilistic approach to determine a "technical earned value" (Pisano, 2002, p.8) by establishing a 90-50-10% confidence risk profile for each TPM to determine the likelihood of successfully achieving the next TPM milestone. This information was then used to calculate cost and schedule performance earned value metrics that incorporated forecasted technical performance variances (Pisano, 2002). A detailed account of how the PEO(A) TPM methodology employed a three stage process with a cost avoidance approach versus a cost recovery approach on the Cockpit 21 program can be found in the white paper titled, Technical Performance Measurement (TPM) Retrospective Implementation and Concept Validation on the T45TS Cockpit-21 Program (Coleman, Kulick, & Pisano, 1996). ...
... For standalone systems, TPMs are often used during development as a leading indicator, providing the PM with confidence that the configuration end item will meet its stated required capabilities. The concept of TPMs was originally developed by the Program Executive Office for Air ASW, Assault, and Special Mission Programs during the mid-90s (Pisano, 1995) in response to the identification of the existence of a gap in information from earned value data that was not tied to technical performance. The International Council on Systems Engineering (INCOSE) has published a technical measurement guide (Roedler & Jones, 2005) that lays out the standard acquisition approach for relating operator requirements to quantifiable and measureable data that can be obtained during development. ...
... They also provide a framework and approach-the risk value method-for measuring the overall technical performance of a product design in terms of its dimensions of performance valued by a customer or market. Each dimension of product technical performance is quantified by a technical performance measure (TPM) or measure of effectiveness (MoE) [e.g., Blanchard, 1997;Coleman, Kulick, and Pisano, 1996;DSMC, 1990;NASA, 1995;Pisano, 1996]. Risk is often quantified as the product of probability of an occurrence and its consequence or impact: ...
Article
Full-text available
In an effort to improve company operations and their results, more firms are applying the principles of “Lean”—not only to manufacturing but also to systems engineering processes. Too often, however, this is done with a shallow understanding of Lean and/or without a systems view, in which case Lean creates new problems and tensions and may not deliver expected results. Lean is not about just minimizing cost, cycle time, or waste. Lean is about maximizing value. In systems engineering or product development (PD), maximizing value may require doing more activities, not fewer. Since a process is a kind of system, a systems view and systems engineering principles are helpful. As the value of a system is more than the value of its individual components, the value of a process is more than the value of its individual activities. Value is driven not only by the presence of necessary (value-adding) activities in the PD process but also by the way those activities work together to ensure that they use and produce the right work products, services, and information at the right time. This paper discusses how value is added in PD through work on activities and the production of deliverables. It integrates findings from several streams of research and provides bases upon which to build improved value models. It shows how the concept of Lean can broaden from asking “What wasteful activities can we stop doing?” to include insights from asking “What helpful activities can we start doing, and when?” © 2003 Wiley Periodicals, Inc. Syst Eng 6: 49–61, 2003. DOI 10.1002/sys.10034
Chapter
The Earned Value Measurement System (EVMS) has become a mainstay in Commercial and Government groups to measure progress and success of a project. EVMS is espoused to be an effective (albeit subjective) measure, but it does not play well with agile development efforts, due to its requirement of static schedules and work plans [55]. Here we introduce a new paradigm for EVMS that will accommodate and be effective in measuring progress and problems within agile development efforts.
Article
A thorough balancing of the costs of conducting verification and validation with the costs of failure promises new possibilities in reducing overall project costs and time-to-market. One goal of the presented SysTest project is to develop an approach to support the strategic verification, validation, and test (VVT) planning. This approach, reflected in the VVT Process Model, supports the trade off between different VVT strategies by calculating the effect of the strategy on cost-, schedule-, and quality-risk. This paper describes the VVT Process Model, which is currently developed, and its application in the development process of a hydraulic pump. Furthermore, its interfaces with other company methods and the corresponding information flows between them are illustrated. To demonstrate its potential, the VVT Process Model is positioned in the verification process.
Article
The rapidly changing environment and asymmetric threats currently encountered on the modern battlefield requires the timely delivery of effective weapons systems. Unfortunately in fiscal year 2008, according to the US Government Accountability Office, research and development costs of the United States Department of Defense major weapons acquisition programs increased 42 percent above original estimates, and delays in initial operational capability deliveries slipped to 22 months. While there are several quantitative methods to estimate acquisition program cost and schedule performance and to identify their risks (e.g., Earned Value Management), the estimation of technical performance and technical risk identification is generally heuristic in nature and based on expert judgment because of limited quantitative data for constructive modeling. The proposed research in this paper expands upon the Technical Risk Index Distribution method developed by Lewis, Mazzuchi and Sarkani by incorporating a performance-based method of mathematically combining quantified expert opinion for technical performance estimation and risk analysis.
Article
Systems engineers are routinely tasked with facilitating the delicate balance between cost, schedule, and technical performance in acquisition programs that are continuously subjected to various outside influences. While there are several quantitative methods to estimate acquisition program cost and schedule performance as well as identify their risks (e.g., Earned Value Management), the estimation of technical performance and technical risk is generally heuristic in nature. In order to monitor the progress of the technical aspects of an acquisition program, the systems engineering discipline utilizes the process of tracking Technical Measures to gain insight into the design and development, to assess risks and issues, and to evaluate the likelihood of realizing objectives. However, with the diversity of so many technical programs, the estimation and risk analysis of technical performance in technology acquisition programs rely on the opinions of experts because the identification and application of relevant quantitative data for constructive modeling is not practical. The Expert-weighted Technical Risk Index methodology proposed in this article introduces a well-established method for mathematically combining expert judgment into the realm of systems engineering to develop predictive progress plans for technical performance estimation and risk analysis. © 2013 Wiley Periodicals, Inc. Syst Eng 17
Article
Technical Performance Measures (TPMs) historically are used to help the Program Manager in predicting if a program is on a path to achieve required performance. When extended over a developmental timeline, TPMs have provided a deterministic approach for predicting expected operational performance. However, TPMs are difficult to derive for a complex acknowledged system of systems (SoS). An SoS Performance Measure (SPM) has been developed that is equivalent in purpose to TPMs and demonstrated for a deterministic state similar to TPM usage. However, reality indicates that many of the SoS component variables have significant uncertainty associated with them during the SoS development. Therefore, to more accurately account for the ability of an SoS to achieve its desired performance, this variability needs to be accounted for. This paper extends the deterministic SPM concept to a stochastic SPM to account for this uncertainty. An Antisubmarine Warfare (ASW) mission example is used, and demonstrates how this extension improves the effectiveness of the SPM methodology. © 2013 Wiley Periodicals, Inc. Syst Eng 17:
Article
Full-text available
Program managers are advocates by necessity, When taken to the extreme, program advocacy can result in the suppression of adverse information about the status of a program gram. Such was the case in the Navy's A-12 Program. In A-12 Administrative inquiry, Beach (1990) speculates that such "abiding cultural problems" were not unique to the Navy. To test that assertion, this paper examines cost overrun data on 64 completed acquisition contracts extracted from the Defense Acquisition Executive Summary database. Cost overruns at various contract completion points are compared with projected final cost overruns estimated by contractor and government personnel. 17% comparison shows that the overruns projected by the contractor and government were excessively optimistic throughout the lives of the contracts examined. These results were found insensitive to contract type (cost, price), contract phase (development, production), the type of weapon system (air, ground, sea), and the military service (Air Force, Army, Navy) that managed the contract.
Article
The tools to reconcile the conflict between creativity and control are discussed. These include the diagnostic control system, belief systems, boundary systems and interactive control systems. Each of these systems has a distinct purpose for managers attempting to harness the creativity of employees. Diagnostic control allows managers to ensure that important goals are being achieved efficiently and effectively. Belief systems encourage individuals to search for new opportunities. Boundary systems establish the rules of the game. Strategic control systems enable managers to focus on strategic uncertainties.
Article
En esta obra, el autor narra su propia historia en el descubrimiento de conexiones entre las leyes básicas de la física y la complejidad y la diversidad del mundo natural. Lo simple: un quark dentro de un átomo. Lo complejo: un jaguar merodeando la jungla, durante la noche. Explorar la relación entre estos dos hechos se convierte en una excitante aventura intelectual.
Department of Defense. Acquisition Management Policies and Procedures
  • Richard Dawkins
  • The Blind
  • Watchmaker
Dawkins, Richard. The Blind Watchmaker. New York: W.W. Norton, 1996. Department of Defense. Acquisition Management Policies and Procedures. (Draft DoD Directive 5000.1 and Draft DoD Instruction 5000.2).
The Mind's Sky: Human Intelligence in a Cosmic Context
  • Timothy Ferris
Ferris, Timothy. The Mind's Sky: Human Intelligence in a Cosmic Context. London: Bantam Press, 1992.
Naval Aviation Systems Command. Peer Review Group Recommendation
  • Jack V Michaels
Michaels, Jack V. Technical Risk Management. Upper Saddle River, NJ: Prentice Hall PTR, 1996. Naval Aviation Systems Command. Peer Review Group Recommendation. April 24, 1996
Earned Value and Risk Management Tool-Page 12
  • Tpm Integrated
Integrated TPM, Earned Value and Risk Management Tool-Page 12