Content uploaded by Todd Little
Author content
All content in this area was uploaded by Todd Little on May 07, 2014
Content may be subject to copyright.
Content uploaded by Todd Little
Author content
All content in this area was uploaded by Todd Little on May 06, 2014
Content may be subject to copyright.
48 IEEE SOFTWARE
Published by the IEEE Computer Society
0740-7459/06/$20.00 © 2006 IEEE
I wanted to quantify some of the conven-
tional wisdom about uncertainty. My com-
pany had collected metrics for three years on
more than 100 commercial software projects,
and I saw the opportunity to mine that data to
expose trends that I could compare to other
industry data. My findings led me to question
aspects of that conventional wisdom.
Background of the project data
Landmark Graphics is a leading vendor of
commercial software for the oil and gas explo-
ration and production market. Landmark has
grown largely via acquisition, and our current
software portfolio includes more than 60 prod-
ucts consisting of more than 50 million lines of
source code. The study reported here looked at
three years of data from 1999 to 2002, during
which Landmark collected data weekly about
all 106 of its software development projects.
For each project, the project manager
recorded several aspects weekly, including sta-
tus, nominal phase, and current estimated de-
livery date. Estimates were updated whenever
the core team reached consensus—approxi-
mately eight times during an average project.
The average project lasted 329 days.
Most teams and project managers were
quite seasoned. The average project manager
had approximately 20 years of industry work
experience. The project manager collected es-
timates from the entire team using the com-
mon industry practice of the “expert method”
(that is, expert judgment using historical data
and analogous projects).
3
The teams generally
didn’t use a formal Delphi estimation tech-
nique, but everyone could give input before
the group reached an estimation consensus.
The corporate culture was such that the team
viewed estimates as targets—similar to
“what’s the earliest date by which you can’t
prove you won’t be finished?”
4
feature
Schedule Estimation and
Uncertainty Surrounding
the Cone of Uncertainty
S
oftware development project schedule estimation has long been a
difficult problem. The Standish C
HAOS Report indicates that only
20 percent of projects finish on time relative to their original
plan.
1
Conventional wisdom proposes that estimation gets better
as a project progresses. This concept is sometimes called the cone of uncer-
tainty, a term popularized by Steve McConnell.
2
The idea that uncertainty
decreases significantly as one obtains new knowledge seems intuitive.
project estimation
Todd Little, Landmark Graphics
Metrics collected
from Landmark’s
projects show that
the estimation
accuracy of project
duration followed
a lognormal
distribution, and the
uncertainty range
was nearly identical
throughout the
project, in conflict
with popular
interpretation
of the “cone of
uncertainty.”
Landmark didn’t follow any formal software
development methodology, although several
projects used iterative milestone development,
and many project teams followed Microsoft So-
lution Framework guidelines.
5
For recording
purposes, we used the four MSF phases: Envi-
sioning, Planning, Developing, and Stabilizing.
Actual versus estimate
Figure 1 shows Landmark project data
along with data from Tom DeMarco for refer-
ence.
6
For the Landmark data, the x-axis shows
the initial estimate of project duration, and the
y-axis shows the actual duration that the proj-
ects required. For DeMarco’s data, I’ve cross-
plotted estimated effort versus actual effort.
The solid line shows the ideal case where the
actual equals the estimate. The data shows sig-
nificant scatter, but by and large, the actual du-
ration or effort was longer than the initial esti-
mate—in some cases significantly longer.
Figure 1 clearly shows a quite similar dis-
tribution scatter of the Landmark data and the
DeMarco data. I recognize that comparing ef-
fort data with duration data might be like
comparing apples and oranges. In our envi-
ronment, and in what I’ve witnessed in other
software product companies, staffing profiles
are relatively constant throughout a project.
So, for our projects, effort was roughly pro-
portional to duration. Thus, I’ll use data to
mean both duration and effort.
In figure 2, I plotted the two data sets from
figure 1 to show a cumulative probability dis-
tribution as a function of the ratio of actual to
estimated data using a log scale for the x-axis.
The solid curves represent a lognormal distri-
bution curve fit through the respective data
points. A lognormal distribution is similar to a
normal distribution in that the frequency dis-
tribution of the log of the data is normal. It’s
common to report values at various probabil-
ity confidences, particularly using p10, p50,
and p90 to represent 10 percent, 50 percent,
and 90 percent confidence respectively.
Cumulative distribution
A cumulative distribution plot shows on
the y-axis the percentage of data samples that
have a value lower than the value on the x-
axis. For example, 20 percent of the projects
from DeMarco’s data were lower than the ini-
May/June 2006
IEEE SOFTWARE 49
Initial estimate
Landmark data
Ideal
DeMarco data
Actual result
Figure 1. Initial
estimated project
duration (or effort)
versus actual duration
(or effort).
0
10
20
30
40
50
60
70
80
90
100
0.1 1.0 10.0
Actual
Estimate
Cumulative distribution (%)
DeMarco data
DeMarco lognormal
Landmark data
Landmark lognormal
Figure 2. The
cumulative distribution
function of the ratio of
actual to estimated
data, using a log scale
for the
x
-axis, for the
Landmark and DeMarco
data.
tial estimate (ratio = 1.0). A normal distribu-
tion has a frequency or probability distribu-
tion in the shape of the classic bell curve.
When plotted as a cumulative distribution
function, it takes on the integral of the bell
curve and shows up as an S-curve.
Figure 3 shows the curve fits displayed as
probability distribution curves. Because this is
plotted on a Cartesian axis, you can see that
the lognormal curve shape is skewed to the
right. Because of this skew, the mean (or ex-
pected value) is greater than the median (p50).
In Landmark’s case, the mean is approximately
2.0 and the median is approximately 1.8. This
seems to validate the old adage “take the initial
estimate and double it” (see the sidebar).
Landmark’s data distribution is nearly iden-
tical to that of DeMarco’s data, and both
demonstrate that the data follows a lognormal
distribution. The data distribution seems to
validate DeMarco’s observed definition: “An
estimate is the most optimistic prediction that
has a non-zero probability of coming true.”
6
A
variation on this definition is to say that a de-
velopment target is an estimate with about a
10 percent chance of being met. Also of inter-
est is the range between p10 and p90. For our
data, the ratio of p90/p10 is 3.25, while for
DeMarco’s data the ratio is 5.2. Barry Boehm
reported that at the Project Concept phase, the
bounding uncertainty was a ratio of 4.
7
Lawrence Putnam, Douglas Putnam, and Don-
ald Beckert published data indicating a range of
plus or minus one standard deviation (p86/p14)
of roughly 3.8.
8
It’s not just good enough to
double the initial estimate—some teams have
found it appropriate to multiply by 3, 4, or
even .
The cone of uncertainty
As I mentioned earlier, the cone of uncer-
tainty seems to imply that estimation uncer-
tainty decreases as a project progresses. Figure
4 shows the cone of uncertainty as Boehm first
50 IEEE SOFTWARE
www.computer.org/software
I’ve seen data from another company’s projects that
showed behavior similar to but somewhat different from the
Landmark data this article discusses. Figure A depicts the cu-
mulative distribution of this company’s projects, plotting the
ratio of actual over initial estimates on a log scale similar to
figure 2. While the tail end greater than p50 resembles figure
2, the early portion of the curve less than p50 looks to be
truncated as a straight line. The curve fit through the data is a
lognormal distribution truncated at 1.0. In this example, the
p50 is approximately 1.1, the p90 is approximately 1.7, and
the ratio p90/p50 is approximately 1.6. This ratio is just
slightly less than the 1.8 that we found from the Landmark data.
I believe that this company used a more conservative esti-
mation policy, coming closer to a p50 estimate rather than a
p10 target. This eliminated the skew but not the overall distri-
bution. A side effect of this conservatism was that because
projects had little incentive to finish early, few did—roughly
half of the projects finished within a very small tolerance of
their initial estimate. The projects that exceeded the
estimate followed a distribution similar to what we observed
with both the Landmark data and Tom DeMarco’s data (
Con-
trolling Software Projects
, Prentice Hall, 1982).
Just Double the Estimate
0
10
20
30
40
50
60
70
80
90
100
0.1 1.0 10.0
Actual duration
Initial estimate
Cumulative distribution (%)
Raw data
Truncated lognormal
Figure A. Data from company X.
021 345678
Probability
DeMarco data
Landmark data
Median: 1.8
Mean: 2.0
Actual
Estimate
Figure 3. The
probability distribution
curve of actual versus
estimated data.
reported it,
7
with relative cost ranges (total-
project-cost multipliers) plotted at multiple
project phases. At the Feasibility phase, the
uncertainty band is a factor of 16 (from 0.25
to 4.0), while at the Concept phase, it has nar-
rowed to a factor of 4 (comparable to our
finding of a factor of 3.25 for the Landmark
data). By the end of the Requirements Specifi-
cation phase, it has decreased to a factor of
2.25. NASA published similar results in
1990.
9
Using Landmark’s weekly data, I cross-
plotted the ratio of actual total duration over
the estimated total duration as a function of
relative time (the ratio of elapsed time over to-
tal actual time). I chose relative time as a
means of normalizing the data so that we
could easily compare it. The result in figure 5
is quite similar to the cone of uncertainty. It’s
not directly comparable to figure 4 because
figure 5’s x-axis represents relative time while
figure 4’s is project phase. The skew to the top
half of the cone is as expected, given that the
initial Landmark estimates are roughly p10.
However, what aspect of uncertainty has
really been reduced? By definition, figure 5
will converge to 1.0 at project completion.
What does it tell us about remaining uncer-
tainty? An alternative view in figure 6 shows
the Landmark data with the ratio of remain-
ing actual duration over remaining estimated
duration, plotted as cumulative distributions
at each phase. It’s surprising that these cumu-
lative-distribution-function curves are nearly
identical for each phase, showing no decrease
in the relative-uncertainty bands. In contrast
to what most people seem to think about un-
certainty decreasing, our data shows that the
remaining uncertainty bands are roughly con-
stant and that the range of uncertainty at all
project stages is approximately a factor of 3 to
4 between p10 and p90.
The pipe of uncertainty
Figure 7 shows a scatter plot of the Land-
mark data along with published data from Mi-
crosoft WinWord 1.0
10
and Microsoft Power-
Point 3.0.
3
The y-axis is the ratio of the actual
remaining duration over the current estimated
remaining duration and is plotted as a function
of relative time for each project at each week.
To the extent that there’s a “cone,” the data
shows an inverted cone behavior. This is al-
most certainly a result of the project manager
May/June 2006
IEEE SOFTWARE 51
Feasibility Concept of
operation
Requirements
spec
Product
design spec
Detail design
spec
Accepted
software
Relative cost range
4.0
2.0
0.5
0.2
1.5
0.6
1.2
0.8
1.0
Figure 4. Barry Boehm’s cone of uncertainty.
7
0.1
1.0
10.0
0 102030405060708090100
Actual duration (%)
Actual total duration
Estimated total duration
Figure 5. Estimation error over time using Landmark’s weekly data.
0
10
20
30
40
50
60
70
80
90
100
0.1 1.0 10.0
Ratio of remaining actual to estimate at each phase
Cumulative distribution (%)
Initial
Start planning
Start development
Start stabilizing
Lognormal
Figure 6. Cumulative distribution by project phase using the same
Landmark data.
holding on to a deadline in hopes that a mira-
cle will occur and the software will release. Fi-
nally the day of reckoning occurs, with no mir-
acle in sight. In the extreme case, the ratio is a
divide-by-zero as the deadline arrives, but sig-
nificant work remains; at this point, the proj-
ect estimate is usually reset. In many cases, this
cycle repeats until the software releases.
Estimation quality factor
I also evaluated the Estimation Quality
Factor, a metric proposed by DeMarco.
6
EQF
measures project estimation accuracy, essen-
tially the estimation error’s reciprocal. Higher
EQFs indicate a higher degree of accuracy in
the overall estimation. An EQF of 5.0 repre-
sents an aggregate estimation error of 20 per-
cent. Landmark’s median EQF was 4.8 com-
pared to the industry median EQF of 3.8
reported by DeMarco.
6
This is additional con-
firmation that the Landmark data represents
an estimation accuracy similar to or slightly
better than the industry average. See the “Es-
timation Quality Factor” sidebar for more on
EQF and the finding that EQF also follows a
lognormal distribution.
52 IEEE SOFTWARE
www.computer.org/software
0
1
10
100
0 10203040506070 .80 90 100
Relative time
Ratio of actual/estimate
Landmark's 100 projects
WinWord 1.0
PowerPoint 3.0
Figure 7. Estimation
ratio versus time
using Landmark and
Microsoft data.
In
Controlling Software Projects
(Prentice Hall, 1982), Tom
DeMarco proposed the Estimation Quality Factor as a tool for
measuring a team’s ability to adjust its project estimates over
the project history. Figure B shows a graphical explanation of
EQF. At the beginning of a project, the team makes an initial
estimate; over time, they might revise the estimate up or down
(the dashed green line). At project completion, the actual
value is known. The estimate’s variation from the actual, inte-
grated over time, is the green area. The slashed area is a rec-
tangle below the actual value and represents the actual value
multiplied by the final duration. The EQF is the ratio of the
slashed area over the green area. The reciprocal of the green
over slashed areas is the estimate’s relative error.
When the estimated quantity is schedule duration, then the
worst case would be to guess completion at the end of the day
and to make that same assumption each day until the product
ships. The green area would then be a triangle, the slashed
area would be a square, and the EQF would be 2.0. Figure C
shows a cumulative distribution function for all our projects of
EQF less this bottom limit (that is, EQF minus 2.0). The data
fits well with a lognormal distribution curve. Approximately 10
percent of Landmark’s projects had EQFs lower than 2.8, half
the projects had EQFs less than the median of 4.8, and 90
percent of the projects had EQFs lower than 11.7. These re-
sults compare to DeMarco’s reported median value of 3.8.
The Estimation Quality Factor
Elapsed time
Value to be estimated
Actual value
Actual end
EQF =
Slashed area
Green area
Initial estimate
Figure B. A graphical explanation of the Estimation
Quality Factor.
0
10
20
30
40
50
60
70
80
90
100
0.1 1 10 100
EQF 2.0
Cumulative distribution (%)
Landmark data
Lognormal curve
Figure C. EQF cumulative distribution for Landmark data.
Analysis of the findings
Although I don’t have definitive evidence to
explain the variation in estimation accuracy I
observed, I’ve identified what I believe are the
primary causes:
■ optimistic assumptions about resource
availability,
■ unanticipated requirements changes brought
on by new market information,
■ underestimation of cross-product integra-
tion and dependency delays,
■ a corporate culture using targets as esti-
mates, and
■ customer satisfaction prioritized over ar-
bitrarily meeting a deadline.
Each of these contributed somewhat to both
the distribution’s variation and the skew to-
ward p10. With these characteristics in mind,
some general interpretations of the data and the
distributions are possible.
The data clearly indicates that Landmark’s
software project duration estimation accuracy
followed a lognormal distribution with an un-
certainty range between p10 and p90 of
roughly a factor of 3 to 4. This distribution
pattern is nearly identical to that found in De-
Marco’s effort estimation data and matches
other authors’ findings. It’s not a new finding
that software estimation follows a lognormal
distribution. In fact, the log axis of the cone
of uncertainty implies this type of distribu-
tion. The PERT (Program Evaluation and Re-
view Technique) model uses a beta distribu-
tion, which has the flexibility to resemble a
lognormal distribution. The use of the beta
distribution appears to have more to do with
its flexible characteristics rather than ob-
served activity.
11
Nonetheless, estimates of es-
poused high confidence of +/– 2 months, or
+/– 20 percent, are still common. Ranges that
are given as +/– a constant time or constant
percent are missing the problem’s exponential
nature.
The data confirmed that the corporate cul-
ture favored using targets as estimates. There’s
really nothing wrong with estimating targets.
That’s probably a good starting point for a
p10 estimate. DeMarco and Tim Lister re-
cently coined this the nanopercent estimate
and recommend it as a basis for determining a
more realistic estimate.
12
Planning a business
around a p10 estimate would be foolish, but if
the pattern I observed is typical of most soft-
ware development organizations, then the full
range of uncertainty could be defined by a p50
estimate of roughly twice p10, and a p90 esti-
mate of roughly three or four times p10.
While the data supports some aspects of the
cone of uncertainty, it doesn’t support the most
common conclusion that uncertainty signifi-
cantly decreases as the project progresses. In-
stead, I found that relative remaining uncer-
tainty was essentially constant over the project’s
life. Although one company’s data might not be
enough to extrapolate to other environments, I
believe that the Landmark data casts doubt on
the reduction of relative uncertainty being a
naturally occurring phenomenon.
Perhaps Landmark has poor estimators or
uses poor estimation techniques? The data
doesn’t demonstrate this. We find something
similar to a p90/p10 ratio of 4 in multiple
places in the literature. Furthermore, the EQF
distribution shows Landmark to be above aver-
age with regard to overall estimation accuracy.
R
educing the range of uncertainty must
be possible, right? Traditional project
management approaches, several of
which are based on a strong belief in the cone
of uncertainty, advocate stronger project con-
trol and greater planning. While controls and
planning are useful, an overly strict focus can
result in attempts to solve the wrong goal.
Shipping on time, to specifications, and within
budget might be meaningless if a competitor is
shipping software that has a greater value to
the market. In that case, the competitor will
win nearly every time, and the prize for
“good” project management might be losing
market share.
Landmark’s measure of success over these
three years had much more to do with cus-
tomer satisfaction and market share than with
meeting knowingly aggressive targets. During
May/June 2006
IEEE SOFTWARE 53
About the Author
Todd Little is a senior development manager for Landmark Graphics. His interests include
the generation of business value through agile software development. He’s a member of the
board of directors of the Agile Alliance and the Agile Project Leadership Network and program
director for the Agile 2006 Conference. He’s also a member of the IEEE Computer Society and
the Society of Petroleum Engineers and a registered Professional Engineer in Texas. He re-
ceived his MS in petroleum engineering from the University of Houston. Contact him at Land-
mark Graphics, PO Box 42806, Houston, TX 77242; tlittle@lgc.com.
these three years, customer satisfaction was
consistently rated very high and steadily in-
creased each year. Market share, revenue, and
profitability grew as well.
Agile and iterative development ap-
proaches acknowledge the presence of uncer-
tainty and adapt to situations rather than try-
ing to control situations to meet the plan.
Continuous feedback, particularly from cus-
tomers or customer representatives, provides
real control and steering. The uncertainty isn’t
eliminated; rather, it’s managed to increase the
probability of meeting market need.
Acknowledgments
I thank Karl Zachry for many fruitful discussions and
for having the determination to implement the Landmark
Portfolio Database that enabled the data capture. I also
thank Stan Rifkin, Hakan Erdogmus, and Craig Larman
for helping me clarify this article’s message.
References
1. CHAOS Report, The Standish Group, 1998.
2. S. McConnell, Rapid Development: Taming Wild Soft-
ware Schedules, Microsoft Press, 1996.
3. M. Jørgensen, “A Review of Studies on Expert Estima-
tion of Software Development Effort,” J. Systems and
Software, vol. 70, nos. 1–2, 2004, pp. 37–60.
4. T. Kidder, The Soul of a New Machine, Little, Brown,
and Co., 1981.
5. M.A. Cusumano and R.W. Selby, Microsoft Secrets, The
Free Press, 1995.
6. T. DeMarco, Controlling Software Projects, Prentice
Hall, 1982.
7. B.W. Boehm, Software Engineering Economics, Prentice
Hall, 1981.
8. L. Putnam, D. Putnam, and D. Beckert, “A Method for
Improving Developers’ Software Size Estimates,”
Crosstalk, 2005, vol. 18, no. 4, pp. 16–19.
9. L. Landis et al., Manager’s Handbook for Software De-
velopment (Revision 1), tech. report SEL-84-101,
NASA Software Eng. Laboratory, 1990.
10. S. McConnell, Software Project Survival Guide, Mi-
crosoft Press, 1998.
11. K.R. MacCrimmon and C.A. Ryavec, “An Analytic
Study of the PERT Assumptions,” J. Operations Re-
search Soc. of America, vol. 12, no. 1, 1964, pp. 16–37.
12. T. DeMarco and T. Lister, Waltzing with Bears, Dorsett
House, 2004.
For more information on this or any other computing topic, please visit our
Digital Library at www.computer.org/publications/dlib.
54 IEEE SOFTWARE
www.computer.org/software
Mid Atlantic (product/recruitment)
Dawn Becker
Phone: +1 732 772 0160
Fax: +1 732 772 0164
Email: db.ieeemedia@ieee.org
New England (product)
Jody Estabrook
Phone: +1 978 244 0192
Fax: +1 978 244 0103
Email: je.ieeemedia@ieee.org
New England (recruitment)
John Restchack
Phone: +1 212 419 7578
Fax: +1 212 419 7589
Email: j.restchack@ieee.org
Connecticut (product)
Stan Greenfield
Phone: +1 203 938 2418
Fax: +1 203 938 3211
Email: greenco@optonline.net
Midwest (product)
Dave Jones
Phone: +1 708 442 5633
Fax: +1 708 442 7620
Email: dj.ieeemedia@ieee.org
Will Hamilton
Phone: +1 269 381 2156
Fax: +1 269 381 2556
Email: wh.ieeemedia@ieee.org
Joe DiNardo
Phone: +1 440 248 2456
Fax: +1 440 248 2594
Email: jd.ieeemedia@ieee.org
Southeast (recruitment)
Thomas M. Flynn
Phone: +1 770 645 2944
Fax: +1 770 993 4423
Email:
flynntom@mindspring.com
Southeast (product)
Bill Holland
Phone: +1 770 435 6549
Fax: +1 770 435 0243
Email: hollandwfh@yahoo.com
Midwest/Southwest (recruitment)
Darcy Giovingo
Phone: +1 847 498-4520
Fax: +1 847 498-5911
Email: dg.ieeemedia@ieee.org
Southwest (product)
Josh Mayer
Phone: +1 972 423 5507
Fax: +1 972 423 6858
Email: jm.ieeemedia@ieee.org
Northwest (product)
Peter D. Scott
Phone: +1 415 421-7950
Fax: +1 415 398-4156
Email: peterd@pscottassoc.com
Southern CA (product)
Marshall Rubin
Phone: +1 818 888 2407
Fax: +1 818 888 4907
Email: mr.ieeemedia@ieee.org
Northwest/Southern CA (recruitment)
Tim Matteson
Phone: +1 310 836 4064
Fax: +1 310 836 4067
Email: tm.ieeemedia@ieee.org
Japan
Tim Matteson
Phone: +1 310 836 4064
Fax: +1 310 836 4067
Email: tm.ieeemedia@ieee.org
Europe (product)
Hilary Turnbull
Phone: +44 1875 825700
Fax: +44 1875 825701
Email: impress@impressmedia.com
ADVERTISER / PRODUCT INDEX MAY/JUNE 2006
Addison-Wesley 11
Agile Conference 2006 Cover 2
Architecture & Design World 2006 Cover 3
Better Software Conference & Expo 2006 Cover 4
IBM Rational Software Development 2006 1
Idea Group 102
Prentice Hall 100
Symantec Press 101
Boldface denotes advertisements in this issue.
Advertising Personnel
Advertiser Page Number
Marion Delaney
IEEE Media, Advertising Director
Phone: +1 415 863 4717
Fax: +1 415 225 5807
Email: md.ieeemedia@ieee.org
Marian Anderson
Advertising Coordinator
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: manderson@computer.org
Sandy Brown
IEEE Computer Society,
Business Development Manager
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: sb.ieeemedia@ieee.org
Advertising Sales Representatives