ArticlePDF Available

Schedule estimation and uncertainty surrounding the cone of uncertainty

Authors:
  • Kanban University

Abstract and Figures

Software development project schedule estimation has long been a difficult problem. The Standish CHAOS Report indicates that only 20 percent of projects finish on time relative to their original plan. Conventional wisdom proposes that estimation gets better as a project progresses. This concept is sometimes called the cone of uncertainty, a term popularized by Steve McConnell (1996). The idea that uncertainty decreases significantly as one obtains new knowledge seems intuitive. Metrics collected from Landmark's projects show that the estimation accuracy of project duration followed a lognormal distribution, and the uncertainty range was nearly identical throughout the project, in conflict with popular interpretation of the "cone of uncertainty"
Content may be subject to copyright.
48 IEEE SOFTWARE
Published by the IEEE Computer Society
0740-7459/06/$20.00 © 2006 IEEE
I wanted to quantify some of the conven-
tional wisdom about uncertainty. My com-
pany had collected metrics for three years on
more than 100 commercial software projects,
and I saw the opportunity to mine that data to
expose trends that I could compare to other
industry data. My findings led me to question
aspects of that conventional wisdom.
Background of the project data
Landmark Graphics is a leading vendor of
commercial software for the oil and gas explo-
ration and production market. Landmark has
grown largely via acquisition, and our current
software portfolio includes more than 60 prod-
ucts consisting of more than 50 million lines of
source code. The study reported here looked at
three years of data from 1999 to 2002, during
which Landmark collected data weekly about
all 106 of its software development projects.
For each project, the project manager
recorded several aspects weekly, including sta-
tus, nominal phase, and current estimated de-
livery date. Estimates were updated whenever
the core team reached consensus—approxi-
mately eight times during an average project.
The average project lasted 329 days.
Most teams and project managers were
quite seasoned. The average project manager
had approximately 20 years of industry work
experience. The project manager collected es-
timates from the entire team using the com-
mon industry practice of the “expert method”
(that is, expert judgment using historical data
and analogous projects).
3
The teams generally
didn’t use a formal Delphi estimation tech-
nique, but everyone could give input before
the group reached an estimation consensus.
The corporate culture was such that the team
viewed estimates as targets—similar to
“what’s the earliest date by which you can’t
prove you won’t be finished?”
4
feature
Schedule Estimation and
Uncertainty Surrounding
the Cone of Uncertainty
S
oftware development project schedule estimation has long been a
difficult problem. The Standish C
HAOS Report indicates that only
20 percent of projects finish on time relative to their original
plan.
1
Conventional wisdom proposes that estimation gets better
as a project progresses. This concept is sometimes called the cone of uncer-
tainty, a term popularized by Steve McConnell.
2
The idea that uncertainty
decreases significantly as one obtains new knowledge seems intuitive.
project estimation
Todd Little, Landmark Graphics
Metrics collected
from Landmark’s
projects show that
the estimation
accuracy of project
duration followed
a lognormal
distribution, and the
uncertainty range
was nearly identical
throughout the
project, in conflict
with popular
interpretation
of the “cone of
uncertainty.”
Landmark didn’t follow any formal software
development methodology, although several
projects used iterative milestone development,
and many project teams followed Microsoft So-
lution Framework guidelines.
5
For recording
purposes, we used the four MSF phases: Envi-
sioning, Planning, Developing, and Stabilizing.
Actual versus estimate
Figure 1 shows Landmark project data
along with data from Tom DeMarco for refer-
ence.
6
For the Landmark data, the x-axis shows
the initial estimate of project duration, and the
y-axis shows the actual duration that the proj-
ects required. For DeMarco’s data, I’ve cross-
plotted estimated effort versus actual effort.
The solid line shows the ideal case where the
actual equals the estimate. The data shows sig-
nificant scatter, but by and large, the actual du-
ration or effort was longer than the initial esti-
mate—in some cases significantly longer.
Figure 1 clearly shows a quite similar dis-
tribution scatter of the Landmark data and the
DeMarco data. I recognize that comparing ef-
fort data with duration data might be like
comparing apples and oranges. In our envi-
ronment, and in what I’ve witnessed in other
software product companies, staffing profiles
are relatively constant throughout a project.
So, for our projects, effort was roughly pro-
portional to duration. Thus, I’ll use data to
mean both duration and effort.
In figure 2, I plotted the two data sets from
figure 1 to show a cumulative probability dis-
tribution as a function of the ratio of actual to
estimated data using a log scale for the x-axis.
The solid curves represent a lognormal distri-
bution curve fit through the respective data
points. A lognormal distribution is similar to a
normal distribution in that the frequency dis-
tribution of the log of the data is normal. It’s
common to report values at various probabil-
ity confidences, particularly using p10, p50,
and p90 to represent 10 percent, 50 percent,
and 90 percent confidence respectively.
Cumulative distribution
A cumulative distribution plot shows on
the y-axis the percentage of data samples that
have a value lower than the value on the x-
axis. For example, 20 percent of the projects
from DeMarco’s data were lower than the ini-
May/June 2006
IEEE SOFTWARE 49
Initial estimate
Landmark data
Ideal
DeMarco data
Actual result
Figure 1. Initial
estimated project
duration (or effort)
versus actual duration
(or effort).
0
10
20
30
40
50
60
70
80
90
100
0.1 1.0 10.0
Actual
Estimate
Cumulative distribution (%)
DeMarco data
DeMarco lognormal
Landmark data
Landmark lognormal
Figure 2. The
cumulative distribution
function of the ratio of
actual to estimated
data, using a log scale
for the
x
-axis, for the
Landmark and DeMarco
data.
tial estimate (ratio = 1.0). A normal distribu-
tion has a frequency or probability distribu-
tion in the shape of the classic bell curve.
When plotted as a cumulative distribution
function, it takes on the integral of the bell
curve and shows up as an S-curve.
Figure 3 shows the curve fits displayed as
probability distribution curves. Because this is
plotted on a Cartesian axis, you can see that
the lognormal curve shape is skewed to the
right. Because of this skew, the mean (or ex-
pected value) is greater than the median (p50).
In Landmark’s case, the mean is approximately
2.0 and the median is approximately 1.8. This
seems to validate the old adage “take the initial
estimate and double it” (see the sidebar).
Landmark’s data distribution is nearly iden-
tical to that of DeMarco’s data, and both
demonstrate that the data follows a lognormal
distribution. The data distribution seems to
validate DeMarco’s observed definition: “An
estimate is the most optimistic prediction that
has a non-zero probability of coming true.”
6
A
variation on this definition is to say that a de-
velopment target is an estimate with about a
10 percent chance of being met. Also of inter-
est is the range between p10 and p90. For our
data, the ratio of p90/p10 is 3.25, while for
DeMarco’s data the ratio is 5.2. Barry Boehm
reported that at the Project Concept phase, the
bounding uncertainty was a ratio of 4.
7
Lawrence Putnam, Douglas Putnam, and Don-
ald Beckert published data indicating a range of
plus or minus one standard deviation (p86/p14)
of roughly 3.8.
8
It’s not just good enough to
double the initial estimate—some teams have
found it appropriate to multiply by 3, 4, or
even .
The cone of uncertainty
As I mentioned earlier, the cone of uncer-
tainty seems to imply that estimation uncer-
tainty decreases as a project progresses. Figure
4 shows the cone of uncertainty as Boehm first
50 IEEE SOFTWARE
www.computer.org/software
I’ve seen data from another company’s projects that
showed behavior similar to but somewhat different from the
Landmark data this article discusses. Figure A depicts the cu-
mulative distribution of this company’s projects, plotting the
ratio of actual over initial estimates on a log scale similar to
figure 2. While the tail end greater than p50 resembles figure
2, the early portion of the curve less than p50 looks to be
truncated as a straight line. The curve fit through the data is a
lognormal distribution truncated at 1.0. In this example, the
p50 is approximately 1.1, the p90 is approximately 1.7, and
the ratio p90/p50 is approximately 1.6. This ratio is just
slightly less than the 1.8 that we found from the Landmark data.
I believe that this company used a more conservative esti-
mation policy, coming closer to a p50 estimate rather than a
p10 target. This eliminated the skew but not the overall distri-
bution. A side effect of this conservatism was that because
projects had little incentive to finish early, few did—roughly
half of the projects finished within a very small tolerance of
their initial estimate. The projects that exceeded the
estimate followed a distribution similar to what we observed
with both the Landmark data and Tom DeMarco’s data (
Con-
trolling Software Projects
, Prentice Hall, 1982).
Just Double the Estimate
0
10
20
30
40
50
60
70
80
90
100
0.1 1.0 10.0
Actual duration
Initial estimate
Cumulative distribution (%)
Raw data
Truncated lognormal
Figure A. Data from company X.
021 345678
Probability
DeMarco data
Landmark data
Median: 1.8
Mean: 2.0
Actual
Estimate
Figure 3. The
probability distribution
curve of actual versus
estimated data.
reported it,
7
with relative cost ranges (total-
project-cost multipliers) plotted at multiple
project phases. At the Feasibility phase, the
uncertainty band is a factor of 16 (from 0.25
to 4.0), while at the Concept phase, it has nar-
rowed to a factor of 4 (comparable to our
finding of a factor of 3.25 for the Landmark
data). By the end of the Requirements Specifi-
cation phase, it has decreased to a factor of
2.25. NASA published similar results in
1990.
9
Using Landmark’s weekly data, I cross-
plotted the ratio of actual total duration over
the estimated total duration as a function of
relative time (the ratio of elapsed time over to-
tal actual time). I chose relative time as a
means of normalizing the data so that we
could easily compare it. The result in figure 5
is quite similar to the cone of uncertainty. It’s
not directly comparable to figure 4 because
figure 5’s x-axis represents relative time while
figure 4’s is project phase. The skew to the top
half of the cone is as expected, given that the
initial Landmark estimates are roughly p10.
However, what aspect of uncertainty has
really been reduced? By definition, figure 5
will converge to 1.0 at project completion.
What does it tell us about remaining uncer-
tainty? An alternative view in figure 6 shows
the Landmark data with the ratio of remain-
ing actual duration over remaining estimated
duration, plotted as cumulative distributions
at each phase. It’s surprising that these cumu-
lative-distribution-function curves are nearly
identical for each phase, showing no decrease
in the relative-uncertainty bands. In contrast
to what most people seem to think about un-
certainty decreasing, our data shows that the
remaining uncertainty bands are roughly con-
stant and that the range of uncertainty at all
project stages is approximately a factor of 3 to
4 between p10 and p90.
The pipe of uncertainty
Figure 7 shows a scatter plot of the Land-
mark data along with published data from Mi-
crosoft WinWord 1.0
10
and Microsoft Power-
Point 3.0.
3
The y-axis is the ratio of the actual
remaining duration over the current estimated
remaining duration and is plotted as a function
of relative time for each project at each week.
To the extent that there’s a “cone,” the data
shows an inverted cone behavior. This is al-
most certainly a result of the project manager
May/June 2006
IEEE SOFTWARE 51
Feasibility Concept of
operation
Requirements
spec
Product
design spec
Detail design
spec
Accepted
software
Relative cost range
4.0
2.0
0.5
0.2
1.5
0.6
1.2
0.8
1.0
Figure 4. Barry Boehm’s cone of uncertainty.
7
0.1
1.0
10.0
0 102030405060708090100
Actual duration (%)
Actual total duration
Estimated total duration
Figure 5. Estimation error over time using Landmark’s weekly data.
0
10
20
30
40
50
60
70
80
90
100
0.1 1.0 10.0
Ratio of remaining actual to estimate at each phase
Cumulative distribution (%)
Initial
Start planning
Start development
Start stabilizing
Lognormal
Figure 6. Cumulative distribution by project phase using the same
Landmark data.
holding on to a deadline in hopes that a mira-
cle will occur and the software will release. Fi-
nally the day of reckoning occurs, with no mir-
acle in sight. In the extreme case, the ratio is a
divide-by-zero as the deadline arrives, but sig-
nificant work remains; at this point, the proj-
ect estimate is usually reset. In many cases, this
cycle repeats until the software releases.
Estimation quality factor
I also evaluated the Estimation Quality
Factor, a metric proposed by DeMarco.
6
EQF
measures project estimation accuracy, essen-
tially the estimation error’s reciprocal. Higher
EQFs indicate a higher degree of accuracy in
the overall estimation. An EQF of 5.0 repre-
sents an aggregate estimation error of 20 per-
cent. Landmark’s median EQF was 4.8 com-
pared to the industry median EQF of 3.8
reported by DeMarco.
6
This is additional con-
firmation that the Landmark data represents
an estimation accuracy similar to or slightly
better than the industry average. See the “Es-
timation Quality Factor” sidebar for more on
EQF and the finding that EQF also follows a
lognormal distribution.
52 IEEE SOFTWARE
www.computer.org/software
0
1
10
100
0 10203040506070 .80 90 100
Relative time
Ratio of actual/estimate
Landmark's 100 projects
WinWord 1.0
PowerPoint 3.0
Figure 7. Estimation
ratio versus time
using Landmark and
Microsoft data.
In
Controlling Software Projects
(Prentice Hall, 1982), Tom
DeMarco proposed the Estimation Quality Factor as a tool for
measuring a team’s ability to adjust its project estimates over
the project history. Figure B shows a graphical explanation of
EQF. At the beginning of a project, the team makes an initial
estimate; over time, they might revise the estimate up or down
(the dashed green line). At project completion, the actual
value is known. The estimate’s variation from the actual, inte-
grated over time, is the green area. The slashed area is a rec-
tangle below the actual value and represents the actual value
multiplied by the final duration. The EQF is the ratio of the
slashed area over the green area. The reciprocal of the green
over slashed areas is the estimate’s relative error.
When the estimated quantity is schedule duration, then the
worst case would be to guess completion at the end of the day
and to make that same assumption each day until the product
ships. The green area would then be a triangle, the slashed
area would be a square, and the EQF would be 2.0. Figure C
shows a cumulative distribution function for all our projects of
EQF less this bottom limit (that is, EQF minus 2.0). The data
fits well with a lognormal distribution curve. Approximately 10
percent of Landmark’s projects had EQFs lower than 2.8, half
the projects had EQFs less than the median of 4.8, and 90
percent of the projects had EQFs lower than 11.7. These re-
sults compare to DeMarco’s reported median value of 3.8.
The Estimation Quality Factor
Elapsed time
Value to be estimated
Actual value
Actual end
EQF =
Slashed area
Green area
Initial estimate
Figure B. A graphical explanation of the Estimation
Quality Factor.
0
10
20
30
40
50
60
70
80
90
100
0.1 1 10 100
EQF 2.0
Cumulative distribution (%)
Landmark data
Lognormal curve
Figure C. EQF cumulative distribution for Landmark data.
Analysis of the findings
Although I don’t have definitive evidence to
explain the variation in estimation accuracy I
observed, I’ve identified what I believe are the
primary causes:
optimistic assumptions about resource
availability,
unanticipated requirements changes brought
on by new market information,
underestimation of cross-product integra-
tion and dependency delays,
a corporate culture using targets as esti-
mates, and
customer satisfaction prioritized over ar-
bitrarily meeting a deadline.
Each of these contributed somewhat to both
the distribution’s variation and the skew to-
ward p10. With these characteristics in mind,
some general interpretations of the data and the
distributions are possible.
The data clearly indicates that Landmark’s
software project duration estimation accuracy
followed a lognormal distribution with an un-
certainty range between p10 and p90 of
roughly a factor of 3 to 4. This distribution
pattern is nearly identical to that found in De-
Marco’s effort estimation data and matches
other authors’ findings. It’s not a new finding
that software estimation follows a lognormal
distribution. In fact, the log axis of the cone
of uncertainty implies this type of distribu-
tion. The PERT (Program Evaluation and Re-
view Technique) model uses a beta distribu-
tion, which has the flexibility to resemble a
lognormal distribution. The use of the beta
distribution appears to have more to do with
its flexible characteristics rather than ob-
served activity.
11
Nonetheless, estimates of es-
poused high confidence of +/– 2 months, or
+/– 20 percent, are still common. Ranges that
are given as +/– a constant time or constant
percent are missing the problem’s exponential
nature.
The data confirmed that the corporate cul-
ture favored using targets as estimates. There’s
really nothing wrong with estimating targets.
That’s probably a good starting point for a
p10 estimate. DeMarco and Tim Lister re-
cently coined this the nanopercent estimate
and recommend it as a basis for determining a
more realistic estimate.
12
Planning a business
around a p10 estimate would be foolish, but if
the pattern I observed is typical of most soft-
ware development organizations, then the full
range of uncertainty could be defined by a p50
estimate of roughly twice p10, and a p90 esti-
mate of roughly three or four times p10.
While the data supports some aspects of the
cone of uncertainty, it doesn’t support the most
common conclusion that uncertainty signifi-
cantly decreases as the project progresses. In-
stead, I found that relative remaining uncer-
tainty was essentially constant over the project’s
life. Although one company’s data might not be
enough to extrapolate to other environments, I
believe that the Landmark data casts doubt on
the reduction of relative uncertainty being a
naturally occurring phenomenon.
Perhaps Landmark has poor estimators or
uses poor estimation techniques? The data
doesn’t demonstrate this. We find something
similar to a p90/p10 ratio of 4 in multiple
places in the literature. Furthermore, the EQF
distribution shows Landmark to be above aver-
age with regard to overall estimation accuracy.
R
educing the range of uncertainty must
be possible, right? Traditional project
management approaches, several of
which are based on a strong belief in the cone
of uncertainty, advocate stronger project con-
trol and greater planning. While controls and
planning are useful, an overly strict focus can
result in attempts to solve the wrong goal.
Shipping on time, to specifications, and within
budget might be meaningless if a competitor is
shipping software that has a greater value to
the market. In that case, the competitor will
win nearly every time, and the prize for
“good” project management might be losing
market share.
Landmark’s measure of success over these
three years had much more to do with cus-
tomer satisfaction and market share than with
meeting knowingly aggressive targets. During
May/June 2006
IEEE SOFTWARE 53
About the Author
Todd Little is a senior development manager for Landmark Graphics. His interests include
the generation of business value through agile software development. He’s a member of the
board of directors of the Agile Alliance and the Agile Project Leadership Network and program
director for the Agile 2006 Conference. He’s also a member of the IEEE Computer Society and
the Society of Petroleum Engineers and a registered Professional Engineer in Texas. He re-
ceived his MS in petroleum engineering from the University of Houston. Contact him at Land-
mark Graphics, PO Box 42806, Houston, TX 77242; tlittle@lgc.com.
these three years, customer satisfaction was
consistently rated very high and steadily in-
creased each year. Market share, revenue, and
profitability grew as well.
Agile and iterative development ap-
proaches acknowledge the presence of uncer-
tainty and adapt to situations rather than try-
ing to control situations to meet the plan.
Continuous feedback, particularly from cus-
tomers or customer representatives, provides
real control and steering. The uncertainty isn’t
eliminated; rather, it’s managed to increase the
probability of meeting market need.
Acknowledgments
I thank Karl Zachry for many fruitful discussions and
for having the determination to implement the Landmark
Portfolio Database that enabled the data capture. I also
thank Stan Rifkin, Hakan Erdogmus, and Craig Larman
for helping me clarify this article’s message.
References
1. CHAOS Report, The Standish Group, 1998.
2. S. McConnell, Rapid Development: Taming Wild Soft-
ware Schedules, Microsoft Press, 1996.
3. M. Jørgensen, “A Review of Studies on Expert Estima-
tion of Software Development Effort,” J. Systems and
Software, vol. 70, nos. 1–2, 2004, pp. 37–60.
4. T. Kidder, The Soul of a New Machine, Little, Brown,
and Co., 1981.
5. M.A. Cusumano and R.W. Selby, Microsoft Secrets, The
Free Press, 1995.
6. T. DeMarco, Controlling Software Projects, Prentice
Hall, 1982.
7. B.W. Boehm, Software Engineering Economics, Prentice
Hall, 1981.
8. L. Putnam, D. Putnam, and D. Beckert, “A Method for
Improving Developers’ Software Size Estimates,”
Crosstalk, 2005, vol. 18, no. 4, pp. 16–19.
9. L. Landis et al., Manager’s Handbook for Software De-
velopment (Revision 1), tech. report SEL-84-101,
NASA Software Eng. Laboratory, 1990.
10. S. McConnell, Software Project Survival Guide, Mi-
crosoft Press, 1998.
11. K.R. MacCrimmon and C.A. Ryavec, “An Analytic
Study of the PERT Assumptions,” J. Operations Re-
search Soc. of America, vol. 12, no. 1, 1964, pp. 16–37.
12. T. DeMarco and T. Lister, Waltzing with Bears, Dorsett
House, 2004.
For more information on this or any other computing topic, please visit our
Digital Library at www.computer.org/publications/dlib.
54 IEEE SOFTWARE
www.computer.org/software
Mid Atlantic (product/recruitment)
Dawn Becker
Phone: +1 732 772 0160
Fax: +1 732 772 0164
Email: db.ieeemedia@ieee.org
New England (product)
Jody Estabrook
Phone: +1 978 244 0192
Fax: +1 978 244 0103
Email: je.ieeemedia@ieee.org
New England (recruitment)
John Restchack
Phone: +1 212 419 7578
Fax: +1 212 419 7589
Email: j.restchack@ieee.org
Connecticut (product)
Stan Greenfield
Phone: +1 203 938 2418
Fax: +1 203 938 3211
Email: greenco@optonline.net
Midwest (product)
Dave Jones
Phone: +1 708 442 5633
Fax: +1 708 442 7620
Email: dj.ieeemedia@ieee.org
Will Hamilton
Phone: +1 269 381 2156
Fax: +1 269 381 2556
Email: wh.ieeemedia@ieee.org
Joe DiNardo
Phone: +1 440 248 2456
Fax: +1 440 248 2594
Email: jd.ieeemedia@ieee.org
Southeast (recruitment)
Thomas M. Flynn
Phone: +1 770 645 2944
Fax: +1 770 993 4423
Email:
flynntom@mindspring.com
Southeast (product)
Bill Holland
Phone: +1 770 435 6549
Fax: +1 770 435 0243
Email: hollandwfh@yahoo.com
Midwest/Southwest (recruitment)
Darcy Giovingo
Phone: +1 847 498-4520
Fax: +1 847 498-5911
Email: dg.ieeemedia@ieee.org
Southwest (product)
Josh Mayer
Phone: +1 972 423 5507
Fax: +1 972 423 6858
Email: jm.ieeemedia@ieee.org
Northwest (product)
Peter D. Scott
Phone: +1 415 421-7950
Fax: +1 415 398-4156
Email: peterd@pscottassoc.com
Southern CA (product)
Marshall Rubin
Phone: +1 818 888 2407
Fax: +1 818 888 4907
Email: mr.ieeemedia@ieee.org
Northwest/Southern CA (recruitment)
Tim Matteson
Phone: +1 310 836 4064
Fax: +1 310 836 4067
Email: tm.ieeemedia@ieee.org
Japan
Tim Matteson
Phone: +1 310 836 4064
Fax: +1 310 836 4067
Email: tm.ieeemedia@ieee.org
Europe (product)
Hilary Turnbull
Phone: +44 1875 825700
Fax: +44 1875 825701
Email: impress@impressmedia.com
ADVERTISER / PRODUCT INDEX MAY/JUNE 2006
Addison-Wesley 11
Agile Conference 2006 Cover 2
Architecture & Design World 2006 Cover 3
Better Software Conference & Expo 2006 Cover 4
IBM Rational Software Development 2006 1
Idea Group 102
Prentice Hall 100
Symantec Press 101
Boldface denotes advertisements in this issue.
Advertising Personnel
Advertiser Page Number
Marion Delaney
IEEE Media, Advertising Director
Phone: +1 415 863 4717
Fax: +1 415 225 5807
Email: md.ieeemedia@ieee.org
Marian Anderson
Advertising Coordinator
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: manderson@computer.org
Sandy Brown
IEEE Computer Society,
Business Development Manager
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: sb.ieeemedia@ieee.org
Advertising Sales Representatives
... These findings offer valuable insight suggesting that IT project performance may not follow a normal distribution. However, Little's [48] study was limited in the sense that it was based on a small sample (e.g., 106 projects) from a single company. Similar to Little [48], there are other studies that have reported fairly large overruns. ...
... However, Little's [48] study was limited in the sense that it was based on a small sample (e.g., 106 projects) from a single company. Similar to Little [48], there are other studies that have reported fairly large overruns. For example, based on interviews with systems development managers Jenkins et al. [34] obtained IT project performance data on 72 systems development projects from 23 organizations and found that the largest cost overrun was 525%. ...
... The probability distribution appeared to be non-Gaussian, possibly with a fat tail, with a mode around 0% (ratio of 1.0). These observations run counter to those of earlier studies which suggested that cost overruns tend to be more common than underruns [13,34,48,54,60]. Contrary to this, our data indicate that overruns and underruns are about equally frequent. ...
Preprint
Full-text available
If managers assume a normal or near-normal distribution of Information Technology (IT) project cost overruns, as is common, and cost overruns can be shown to follow a power-law distribution, managers may be unwittingly exposing their organizations to extreme risk by severely underestimating the probability of large cost overruns. In this research, we collect and analyze a large sample comprised of 5,392 IT projects to empirically examine the probability distribution of IT project cost overruns. Further, we propose and examine a mechanism that can explain such a distribution. Our results reveal that IT projects are far riskier in terms of cost than normally assumed by decision makers and scholars. Specifically, we found that IT project cost overruns follow a power-law distribution in which there are a large number of projects with relatively small overruns and a fat tail that includes a smaller number of projects with extreme overruns. A possible generative mechanism for the identified power-law distribution is found in interdependencies among technological components in IT systems. We propose and demonstrate, through computer simulation, that a problem in a single technological component can lead to chain reactions in which other interdependent components are affected, causing substantial overruns. What the power law tells us is that extreme IT project cost overruns will occur and that the prevalence of these will be grossly underestimated if managers assume that overruns follow a normal or near-normal distribution. This underscores the importance of realistically assessing and mitigating the cost risk of new IT projects up front.
... These findings offer valuable insight suggesting that IT project performance may not follow a normal distribution. However, Little's [48] study was limited in the sense that it was based on a small sample (e.g., 106 projects) from a single company. Similar to Little [48], there are other studies that have reported fairly large overruns. ...
... However, Little's [48] study was limited in the sense that it was based on a small sample (e.g., 106 projects) from a single company. Similar to Little [48], there are other studies that have reported fairly large overruns. For example, based on interviews with systems development managers Jenkins et al. [34] obtained IT project performance data on 72 systems development projects from 23 organizations and found that the largest cost overrun was 525%. ...
... The probability distribution appeared to be non-Gaussian, possibly with a fat tail, with a mode around 0% (ratio of 1.0). These observations run counter to those of earlier studies which suggested that cost overruns tend to be more common than underruns [13,34,48,54,60]. Contrary to this, our data indicate that overruns and underruns are about equally frequent. ...
Article
Full-text available
If managers assume a normal or near-normal distribution of Information Technology (IT) project cost overruns, as is common, and cost overruns can be shown to follow a power-law distribution, managers may be unwittingly exposing their organizations to extreme risk by severely underestimating the probability of large cost overruns. In this research, we collect and analyze a large sample comprised of 5,392 IT projects to empirically examine the probability distribution of IT project cost overruns. Further, we propose and examine a mechanism that can explain such a distribution. Our results reveal that IT projects are far riskier in terms of cost than normally assumed by decision makers and scholars. Specifically, we found that IT project cost overruns follow a power-law distribution in which there are a large number of projects with relatively small overruns and a fat tail that includes a smaller number of projects with extreme overruns. A possible generative mechanism for the identified power-law distribution is found in interdependencies among technological components in IT systems. We propose and demonstrate, through computer simulation, that a problem in a single technological component can lead to chain reactions in which other interdependent components are affected, causing substantial overruns. What the power law tells us is that extreme IT project cost overruns will occur and that the prevalence of these will be grossly underestimated if managers assume that overruns follow a normal or near-normal distribution. This underscores the importance of realistically assessing and mitigating the cost risk of new IT projects up front.
... Why did the project fail? As with many software projects (Little, 2006), the team drastically underestimated how hard it would be to implement the many features in their design and still have a simple, easy-to-use interface. They overestimated the level of effort teachers would be willing to invest in writing lesson plan case studies. ...
Article
Full-text available
Educational Design Research (EDeR) methodologists argue that iteration is a core component of EDeR (McKenney & Reeves, 2018). Iteration is currently defined as a process of gathering more information through actions, such as testing, and using that information to improve the design (Hoadley & Campos, 2022). In this paper, we seek to tighten the definition of iteration to help EDeR teams conduct iterations more effectively. We argue that EDeR teams should organize their research in slices that deliver small but real value to end users while informing the design research. EDeR should pick slices that are: (a) minimal and focused, (b) deployed in a real context, (c) valuable to the end users, and (d) informative to the research. Slicing helps EDeR teams increase ecological validity when they test because it allows testing which is within real-world educational contexts or with the stakeholders who will use and be impacted by the design. Increasing ecological validity of testing is particularly important because EDeR projects tackle highly complex real-world problems with many unknown elements and relational complexity – this means it is challenging to predict what designs will have the desired impact without real-world deployment. Effective iteration through organizing research in slices helps EDeR teams to better support stakeholder goals, develop more impactful theory, and have greater and earlier impact upon education.
... Our data indicate that the estimation accuracy in agile software development does not improve over time, even though timely feedback is provided for each estimation. This finding corresponds with a previous study's results on non-agile projects [48], which found that the uncertainty range was nearly identical throughout the project, contradicting the conventional view of estimation getting better as a project progresses. ...
Article
Full-text available
Effort estimation is an important practice in agile software development. The agile community believes that developers’ estimates get more accurate over time due to the cumulative effect of learning from short and frequent feedback. However, there is no empirical evidence of an improvement in estimation accuracy over time, nor have prior studies examined effort estimation in different development activities, which are associated with substantial costs. This study fills the knowledge gap in the field of software estimation in agile software development by investigating estimations across time and different development activities based on data collected from a large agile project. This study investigated effort estimation in various development activities, including feature development, bug fixing, and refactoring in agile software development. The results indicate that estimation of agile development does not improve over time, as claimed in the literature. Our data also indicate that no difference exists in the magnitude of estimation errors between feature tasks and bug-fixing/refactoring tasks, while bug-fixing and refactoring tasks are overestimated more frequently than feature tasks. This study also contributes to our knowledge about overestimation and underestimation patterns in agile software development.
... Two approaches have emerged for lightweight agile methods (Shore and Warden 2008;Schwaber and Sutherland 2017). The former highlights the importance of risk analysis in the project management process (Little 2006, as an additional element performed by the team leader (Shore and Warden 2008;Schwaber and Sutherland 2017); however, it does not present risk management models that could be used in practice. In contrast, the latter approach treats project risk as a natural element (Highsmith 2009), built into agile methods through transparency, prioritization, an iterative approach, or constant contact with the contracting entity and almost immediate response to changes, whether in requirements, technology, or even scope elements (DeMarco and Lister 2013). ...
Article
Full-text available
A synthesis of literature studies covering the determinants of agile project management methods, risk management processes as well as factors influencing the shaping of project success and failure clearly indicates that in most publications on risk in agile managed projects, the human factor is heavily underestimated at the expense of often excessive favoring of procedures. Meanwhile, after analyzing the risk factors that arise in agile-managed IT projects, it became apparent that in addition to aspects such as technology, hardware, system, or even project schedule and cost, the project team is highlighted, which is also the second concept with the GPM P5 Standard for Sustainability in Project Management. Thus, the purpose of this article is to develop a model for risk management in IT projects. As a result of the empirical research carried out by means of an expert interview (108 experts) and a questionnaire survey (123 respondents), a risk management model was developed and six original risk management areas were identified, describing 73.92% of all risk factors that may occur during the implementation of an IT project. Furthermore, empirical studies confirm that basic processes such as risk factor identification, impact assessment, and key risk factor management are used by managers and/or team leaders during the implementation of IT projects.
... In the light-weight (Schwaber, & Sutherland, 2016;Chrapko, 2015;Shore, & Warden, 2008) agile methods, two approaches have been developed. The first one emphasises the significance of risk analysis in the project management process (Little, 2006) as an additional element executed by the team leader (Shore, & Warden, 2008;Boehm et al., 2002), however, it does not provide risk management models that could be used in practice. The second approach considers project risk as an inherent element (Lasek, & Adamus, 2014;Highsmith, 2007), incorporated in agile methods (Walczak, 2010) through transparency, prioritisation, iterative approach or continuous contact with the contracting party and almost immediate reaction to changes in requirements, technology or even scope elements (DeMarco, & Lister, 2003). ...
Thesis
Full-text available
I metodi Agile sono un insieme di valori, pratiche e principi già utilizzati per la gestione di progetti nel campo dell’Information Technology con ottimi risultati. Potenzialmente, questi metodi si prestano a essere utilizzati in altri ambiti, ma la documentazione di altre applicazioni al di fuori del settore informatico risulta lacunosa (Gustavsson, 2016, p. 114). L’obiettivo di questa tesi è indagare l’utilizzo del framework di sviluppo prodotto Scrum, uno dei metodi Agile più utilizzati e significativi, applicandolo nel campo del design della comunicazione. La scelta di indagare il settore del design della comunicazione è stata fatta per contestualizzare il progetto di ricerca nell’ambito lavorativo del dottorato industriale in cui viene svolto, ovvero lo studio di design e comunicazione Chialab s.r.l. di Bologna. La domanda di ricerca della presente tesi è: Come può Scrum migliorare la pianificazione e la gestione di un progetto di design della comunicazione? Contestualmente viene esplorato il sotto-problema: Come può Scrum migliorare la comunicazione tra design e sviluppo? Le sperimentazioni contenute in questo studio sono state strutturate come delle osservazioni ripetute su dei gruppi di team Scrum (designer, manager e sviluppatori) che, secondo le regole della guida Scrum, sono formati da un massimo di 10 persone più un ristretto numero di stakeholder del progetto (Schwaber & Sutherland, 2020, p. 5). I team osservati si sono occupati di realizzare diverse tipologie di progetti afferenti al design della comunicazione. Nel disegno sperimentale sono state individuate le variabili di eccellenza e le relative metriche che costituiscono il fondamento della presente ricerca. Al termine di ogni esperimento è stato somministrato ai soggetti partecipanti un questionario di autovalutazione rispetto all’uso di Scrum, allo scopo di ottenere dati sia quantitativi che qualitativi. Ulteriori dati sono stati ricavati dalle metriche di performance del team Scrum (la velocity del team, paragrafo 2.3) e tramite schede di osservazione appositamente realizzate per essere utilizzate durante gli incontri con i team previsti dal metodo Scrum (paragrafo 5.3). Da quanto emerso in literature review (capitolo 2), risultano evidenti le lacune di letteratura scientifica sul metodo Scrum e la sua relativa applicazione, sia dal punto di vista quantitativo sia qualitativo della ricerca, rendendo necessaria la creazione di una base di conoscenza coerente. La presente ricerca non si focalizza unicamente sul paragone tra Scrum e metodi esistenti, bensì cerca di capire come poter modificare il metodo Scrum stesso, per poterlo adattare in maniera efficiente al contesto del design della comunicazione. In conclusione di questa tesi, vengono date delle indicazioni sia per applicare il metodo Scrum nei progetti di design della comunicazione, sia per effettuare ulteriori ricerche. In questo modo si potrebbe contribuire al consolidamento di un paradigma di ricerca attualmente poco studiato, attraverso un numero consistente di esperimenti e ai relativi dati raccolti.
Chapter
This chapter looks back at the way technology has been managed historically, the difficulties that this has created, and why it is necessary to rethink the problem as we look to the future. The material was mostly sourced from a review of academic literature which consists of peer-reviewed research findings by others in the broad area of the management and governance of business technology but it also contains some anecdotal evidence from what people in leadership roles are experiencing right now in what is a rapidly changing world.
Article
Context Software projects involve technical and managerial activities, including software estimation. Inaccurate estimates are harmful and improving estimation methods is not enough: we need to understand more of the factors that impact estimates. Objective Our study aims to identify the existing evidence about the factors that affect estimates in software projects when using expert judgment. Method We executed a Systematic Literature Mapping (SLM) based on database and snowballing searches, selecting papers by first reading their titles and abstracts and later reading the full text. Results Researchers investigated a wide range of different factors employing mostly laboratory research strategies and relying primarily on differences of estimates and participants’ perceptions to measure the factors’ effects. Resulting from our analysis, we present the SEXTAMT (Software Estimates of eXperts: A Map of influencing facTors), a map of factors affecting estimates built on three dimensions: project/iteration phase, stakeholders, and type of effect. Conclusion Over the years, researchers have investigated a varied set of factors. Many of them were explored in different studies, employing diverse research strategies. Such studies provide compelling evidence on the elements that influence expert judgment estimates, which can be used to assess and improve everyday estimation in the software industry.
Article
A method that decomposes programming artifacts into elementary units of work that form the size used for model based estimating is presented. The method, also known as parametric or model-based estimating, determines the size of the software elements by breaking them down into common low-level software implementation units (IU). It then creates a model-based first cut estimate using a productivity assumption, the project size, and the critical constraints. The process is simple to implement, flexible, can be tuned with actual project performance data, and fosters developer buy-in by involving them in the estimating process.
Article
This paper presents the results of a mathematical analysis of the standard assumptions used in PERT calculations. The objectives of this analysis were four-fold: 1 to pull together the mathematical aspects of the PERT model, 2 to suggest relevant analytic techniques, 3 to obtain an indication of the magnitude and direction of errors introduced by the assumptions, and 4 to suggest possible modifications and improvements in the model. The first part of the paper analyzes those assumptions relevant to the individual activities. Three possible sources of error are considered here: 1 the beta distribution assumption, 2 the standard deviation assumption and the approximation formula for the mean, and 3 the imprecise time estimates. The second part of the paper deals with the PERT network as a whole and analyzes the calculations underlying the project mean, standard deviation, and probability statements. The concept of relative criticalness is explored for the PERT stochastic model. Techniques for network reduction are outlined.
Book
From the Book:Corporate and commercial software development teams all want solutions for one important problem--how to get their high-pressure development schedules under control. In Rapid Development, author Steve McConnell addresses that concern head-on with overall strategies, specific best practices, and valuable tips that help shrink and control development schedules and keep projects moving. Inside, you'll find: The best rapid-development strategies that can be applied to any project Candid discussions of great and not-so-great rapid-development practices--estimation, prototyping, forced overtime, motivation, teamwork, rapid-development languages, risk management, and many others A list of classic mistakes to avoid for rapid-development projects, including creeping requirements, shortchanged quality, and silver-bullet syndrome Case studies that vividly illustrate what can go wrong, what can go right, and how to tell in which direction your project is headed Rapid Developement is the real-world guide to more efficient applications development
Article
This paper summarizes the current state of the art and recent trends in software engineering economics. It provides an overview of economic analysis techniques and their applicability to software engineering and management. It surveys the field of software cost estimation, including the major estimation techniques available, the state of the art in algorithmic cost models, and the outstanding research issues in software cost estimation.
Article
In recent years the software industry has seen the increasing imposition of structure and discipline on technical development activities in an attempt to improve the efficiency of software development and the reliability of the software produced. The clear emphasis in the modern approach to software engineering is to focus attention on the overall development process and the co-ordination all aspects of software development. This paper examines the principles of managing and successfully controlling software development from a software engineering basis.
Article
This paper provides an extensive review of studies related to expert estimation of software development effort. The main goal and contribution of the review is to support the research on expert estimation, e.g., to ease other researcher’s search for relevant expert estimation studies. In addition, we provide software practitioners with useful estimation guidelines, based on the research-based knowledge of expert estimation processes. The review results suggest that expert estimation is the most frequently applied estimation strategy for software projects, that there is no substantial evidence in favour of use of estimation models, and that there are situations where we can expect expert estimates to be more accurate than formal estimation models. The following 12 expert estimation “best practice” guidelines are evaluated through the review: (1) evaluate estimation accuracy, but avoid high evaluation pressure; (2) avoid conflicting estimation goals; (3) ask the estimators to justify and criticize their estimates; (4) avoid irrelevant and unreliable estimation information; (5) use documented data from previous development tasks; (6) find estimation experts with relevant domain background and good estimation records; (7) Estimate top-down and bottom-up, independently of each other; (8) use estimation checklists; (9) combine estimates from different experts and estimation strategies; (10) assess the uncertainty of the estimate; (11) provide feedback on estimation accuracy and development task relations; and, (12) provide estimation training opportunities. We found supporting evidence for all 12 estimation principles, and provide suggestions on how to implement them in software organizations.