Reference Class Forecasting for Hong Kong’s Major Roadworks Projects

Abstract

Reference class forecasting is a method to remove optimism bias and strategic misrepresentation in infrastructure projects and programmes. In 2012 the Hong Kong government’s Development Bureau commissioned a feasibility study on reference class forecasting in Hong Kong – a first for the Asia-Pacific region. This study involved 25 roadwork projects, for which forecast costs and durations were compared with actual outcomes. The analysis established and verified the statistical distribution of the forecast accuracy at various stages of project development, and benchmarked the projects against a sample of 863 similar projects. The study contributed to the understanding of how to improve forecasts by de-biasing early estimates, explicitly considering the risk appetite of decision makers, and safeguarding public funding allocation by balancing exceedance and under-use of project budgets.

Full text

1
investment projects often experience cost overruns and completion
delays, and subsequent research questions the positive contribution
of infrastructure megaprojects (Flyvbjerg etal., 2003).
1.1 Reference class forecasting method
Reference class forecasting is a method of predicting the future,
through looking at similar past situations and their outcomes.
Reference class forecasting predicts the outcome of a planned
action based on actual outcomes in a reference class of similar
actions to that being forecast. The theories behind reference
class forecasting were developed by Daniel Kahneman and Amos
Tversky. The theoretical work helped Kahneman win the Nobel
Prize in economics.
Kahneman and Tversky (1977) found that human judgement
is generally optimistic owing to overconfidence and insucient
consideration of distributional information about outcomes.
Therefore, people tend to underestimate the costs, completion times
and risks of planned actions, whereas they tend to overestimate the
benefits of those same actions. Such an error is caused by actors
taking an ‘inside view’, where focus is on the constituents of the
specific planned action instead of on the actual outcomes of similar
ventures that have already been completed.
1. Introduction
Investments in public infrastructure have historically been seen
as a key policy to stimulate economic growth. Calderon et al.
(2011) estimated that a 10% increase in infrastructure provision
increases long-term economic output by 1–5%.
A recent report by the McKinsey Global Institute (Dobbs etal.,
2013) argues that deferring infrastructure investments stifles
growth and recovery. The report also estimates that the current
infrastructure stock is on average 70% of a country’s gross
domestic product (GDP) (China 76%), which is seen as indication
that infrastructure spending needs to grow with countries’ GDP
growth targets.In the case of China, the projected GDP growth by
2013–2030 would translate into growing the needed investment
into infrastructure from 6.4% of the GDP to 8.5% of the GDP.
A historic analysis by the Organization for Economic
Co-operation and Development and the International Transport
Forum (OECD/ITF, 2013) showed that countries in western
Europe, North America and Australia used investments into road
infrastructure, although while reducing maintenance spend, to
stimulate economic growth during the 2008–2011 recession.
However, research by Flyvbjerg etal. (2003) has shown that capital
Reference class forecasting for
Hong Kong’s major roadworks
projects
1 Bent Flyvbjerg DTech, DSc, PhD
First BT Professor and Founding Chair of Major Programme
Management, Saïd Business School, University of Oxford, Oxford, UK
2 Chi-keung Hon JP, BSc(Eng), MPA, CEng, FHKIE, FICE, FCIArb
Permanent Secretary for Development (Works), Development
Bureau, Hong Kong, PR China
3 Wing Huen Fok MSc, MA, EngD, CEng, FHKIE, FICE, FIStructE,
FCIHT, FCIArb, FHKIArb, APMP
Director Projects, Advisian, Hong Kong, PR China
Reference class forecasting is a method to remove optimism bias and strategic misrepresentation in
infrastructure projects and programmes. In 2012 the Hong Kong government’s Development Bureau
commissioned a feasibility study on reference class forecasting in Hong Kong – a first for the Asia-Pacific
region. This study involved 25 roadwork projects, for which forecast costs and durations were compared
with actual outcomes. The analysis established and verified the statistical distribution of the forecast
accuracy at various stages of project development, and benchmarked the projects against a sample of 863
similar projects.The study contributed to the understanding of how to improve forecasts by de-biasing
early estimates, explicitly considering the risk appetite of decision makers, and safeguarding public funding
allocation by balancing exceedance and under-use of project budgets.
Proceedings of the Institution of Civil Engineers
http://dx.doi.org/10.1680/jcien.15.00075
Paper 1500075
Received 15/10/2015 Accepted 25/04/2016
Keywords: Economics & finance/Infrastructure planning/
Risk & probability analysis
Civil Engineering
Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
ICE Publishing: All rights reserved
123

Civil Engineering
Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
2
California high speed rail and is currently being used in Europe’s
largest civil engineering projects, Crossrail and High Speed Two.
Daniel Kahneman has called reference class forecasting as
developed by Flyvbjerg and Cowi (2004) ‘the single most important
piece of advice regarding how to increase accuracy in forecasting
through improved methods’ (Kahneman, 2011: p.251).
1.3 Benefits of using reference class forecasting
The UK experience of using reference class forecasting for more
than 10 years has shown that, after applying the method, a number
of projects have been cancelled. Projects that were implemented
after reference class forecasting had been applied have shown
reduced cost overruns in comparison to projects that have not been
de-biased. ‘We are tending now to find that with experience, the
project costs remain very much under control and the optimism
bias added in appraisals is too high against the actual delivery’,
observed a UK Department for Transport ocial (personal
communication, December 2012).
Discussing the value of reference class forecasting with project
managers who have applied the method, the authors learned that
the value of reference class forecasting becomes particularly visible
in the latter half of a project. When project budgets are exhausted
and contingencies spent, which is common for projects without
reference class forecasting, project managers spend most of their
time on renegotiating project scope, raising new funding, reducing
the cost of the outstanding works and answering critique of bad
project management in the media. Typically this point is reached in
the latter half of project delivery.
However, anecdotal feedback from project managers suggests
that projects with an un-biased level of contingencies allow project
managers to focus on what is really important: completing the
project.
2. First reference class forecasting study in
Hong Kong and Asia-Pacific
The idea of reference class forecasting was first introduced
to Hong Kong in late 2010. In September 2012, a study was
commissioned by the Development Bureau (2012) to study the
feasibility of using reference class forecasting in Hong Kong, with
the major roadworks projects of the Highways Department as a
pilot reference class.
The study involved the retrieval of cost and completion time
information of 25 completed projects for statistical treatment, such
that a reference class on cost and completion time forecasting of
roadworks projects could be established (Figure1).The study was
the first of its kind in the Asia-Pacific region.
2.1 Development process of Hong Kong’s public works
projects
All public works projects including roadworks projects are
developed under well-established government procedures.
According to chapter 2 of the Project Administration Handbook
for Civil Engineering Works issued by the Hong Kong SAR
government (2014), or its earlier versions, public works projects
go through various categories of development before construction.
Kahneman and Tversky (1977) concluded that disregard of
distributional information, which is a focus on single-point
estimates, is perhaps the major source of error in forecasting.On
that basis they recommended that forecasters ‘should therefore
make every eort to frame the forecasting problem so as to facilitate
utilising all the distributional information that is available’.In other
words, forecasts should not be confined to the most likely estimate,
but should present the full range of estimates.
Although newer methods such as Monte Carlo simulations
attempt to present a range of outcomes, these simulations are still
the result of an inside view and therefore prone to underestimation.
Kahneman and Tversky (1977) further argue that using distributional
information from previous ventures similar to the one being forecast
is called taking an ‘outside view’. Reference class forecasting is a
method for taking an outside view on planned actions.
Reference class forecasting for a specific project involves the
following three steps
(a) identify a reference class of past, similar projects
(b) establish a probability distribution for the selected reference
class for the parameter that is being forecast
(c) compare the specific project with the reference class
distribution to establish the most likely outcome for the specific
project.
The key benefit of reference class forecasting is that it replaces
assumptions – such as assumed adverse events and assumed
uncertainty in estimates – with data.It should be noted that reference
class forecasting assumes that a project performs no better or worse
than past similar projects.On this basis, adjustments can be made
to the reference class forecast. However, it should be stressed that
adjustments might re-introduce the bias of assumptions back into
the forecast and should only be made based on strong evidence that
the project is indeed better or worse than past projects.
1.2 Reference class forecasting in cost and other
forecasting of major projects
Reference class forecasting has been used by the UK Department
for Transport since 2004 to implement the UK Treasury’s Green
Book Supplementary Guidance: Optimism Bias (HM Treasury,
2013). It has since been employed on all major UK transport
infrastructure projects and programmes. It has also been used
outside the transport sector in a broad variety of projects and
programmes across departments, ranging from the government’s
review of public private partnerships, the renewal of the British
school estate and equipment purchases in the UK healthcare
system to the turn-around programme of Jaguar Land Rover.
Reference class forecasting has also developed into a globally
used method.In 2005, the American Planning Association endorsed
reference class forecasting. Similarly, the Project Management
Institute has included the concept of taking the ‘outside view’ in
its standard for project cost estimation. Like the UK, Denmark has
made reference class forecasting mandatory for large rail and road
projects. Furthermore, reference class forecasting has been used
on individual projects by the governments of Sweden, Switzerland,
Norway, the Netherlands, South Africa and Australia, among
others.
Reference class forecasting was pivotal in the US Government
Accountability Oce’s assessment of the business case of
Offprint provided courtesy of www.icevirtuallibrary.com
Author copy for personal use, not for distribution

Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
Civil Engineering
3
The estimating using risk analysis process is by nature a bottom-
up process based on the ‘inside view’ of the project ocers to
determine the risk values of a certain project. In contrast, the
reference class forecasting provides a data-driven ‘outside view’ of
the risk values of a certain project. First, reference class forecasting
establishes the full distributional information of the risk exposure
of a project.
Second, reference class forecasting provides an uplift factor
to the base estimate to become the project estimate, based on
the risk appetite of the decision maker. This is regarded as a
top-down process since the uplift factor corresponding to the
decision maker’s acceptable risk of budget overrun is derived
from the statistics of comparison between the constant price –
that is, the same time value of the final project cost and the base
estimate of a reference class of similar past projects, and hence
an ‘outside view’.
Both estimating using risk analysis and reference class
forecasting are methods of quantitative risk analysis based on
the risk-free base estimate. Reference class forecasting oers an
‘outside view’ to scrutinise the ‘inside view’ estimating using
risk analysis results. Decision makers can subsequently allocate
contingencies corresponding to their risk appetite.
3. Study process
For this study data were collected by first identifying a list of
past projects each larger than HK$100million (£9million) in final
outturn cost, a criterion set after discussions with the Highways
Department regarding what is actually considered as a large-scale
project.The study aimed to collect data of 25 projects, which was a
trade-o between the statistical validity of the results and the eort
needed to collect the archival data.
Data on the most recently completed 25 projects above
HK$100million were collected.The data included the scope and
timeline of each project as well as base estimates and contingency
estimates at the stages of upgrading to category C, category B and
category A as well as the final cost. All these projects are major
roadworks projects constructed between 1993 and 2011. The
earliest upgrade to category C was in 1986.
The approach to data collection was taken to ensure a
representative view of the projects undertaken by the Highways
Department.The availability of archival records, one could argue,
might have an impact on the performance that a study will find
if badly performing projects fail to document their plans and
progress reports. Thus it has previously been argued, for example
in the international benchmarking, that the results err on the
conservative side.
■ Category C: projects that have been broadly accepted on the
basis of a project definition statement and technical feasibility
statement that has been approved by the Works Branch of the
Development Bureau.
■ Category B: projects that have had resources earmarked
internally, but they have not yet been presented to finance
committee of the Legislative Council for approval and they
cannot incur expenditure other than on pre-construction works
including investigation and design.
■ Category A: projects that are ready in all respects for tenders
to be invited and construction works to proceed, and have been
granted an approved project estimate by finance committee.
When a project is being upgraded to each of these categories, an
estimate of the cost prepared by the respective works department will
be logged into the capital works programme of the government.It
is natural that the cost estimate of a project at category A is much
closer to the final cost of the project than that at category C, as
more details of the project have been worked out when a project
progresses from category C to category B, then to category A.
2.2 Current practice of cost forecasting and potential
use of reference class forecasting
When preparing for the cost estimate of a project at each category
stage, the works department will produce the base estimate based
on quantities of the latest project plan and the most appropriate
rates.
Since mid-1993, the assessment of the contingencies of a
project follows the standard process called ‘estimating using
risk analysis’ as directed by the technical circular issued by the
Works Branch (1993), then a major branch under the government
secretariat.In the process, risk items of a project are identified and
quantified based on experience and knowledge of project ocers
on similar projects.The summation of all risk values becomes the
contingency value to be added to the base estimate of the project
to become its project estimate for logging into the capital works
programme. Thesame process is applied when a project upgrades
to categoryC, category B and category A status.
The UK experience of using
reference class forecasting
for more than 10 years
has shown that, owing to
the method, a number of
projects have been cancelled
Figure 1. Cost and completion time information for 25 completed
Hong Kong road projects was retrieved and analysed

Civil Engineering
Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
4
accuracy was associated with a more realistic estimation of the
base cost, which was reflected in increased unit cost estimates
in real terms. The analysis over time also established a shift in
estimation accuracy after mid-1993, when estimating using risk
analysis was introduced.The shift from before 1993 to after 1993
as shown in Figure2 is nearly statistically significant even with a
small data sample.In order to ensure that the established reference
class is relevant to current planning practices, projects started
before 1993 – that is, pre-estimating using risk analysis – were
excluded from formulating the reference class.
The cost overrun is defined as the actual minus estimated costs
divided by the estimated cost and expressed in percentages. Cost
overruns are calculated based on constant prices. Schedule overrun
is the dierence between actual time to completion and the estimated
time to completion divided by the estimated time to completion.
Estimated time to completion is defined as duration of a
project from the date of the upgrade to category C to the planned
completion date, whereas actual time to completion is defined as
duration from the date of the upgrade to category C to the date
of actual completion. Time-to-completion data were also retrieved
while screening through the project files. Time-to-completion
estimates were available for 22 projects at category C, 23 projects
at category B and 25 projects at category A.
3.2 Formation of uplift curves of the reference class
With all the valid cost and time data, the reference class with the
uplifts for cost and time-to-completion estimates at the upgrade to
category C, category B and category A were established.
First, the cost and time-to-completion overruns at the decision
points were calculated. Second, the uplifts Q were established by
identifying Q(p) = inf{x: p ≤ PR(X≤x)}, where x are the overruns,
p is the probability of a cost overrun and X is a given value of x.
In other words, for all probabilities between 0 and 1 the
maximum overrun was established that was not exceeded in the
historical data.The uplifts are thus comparable to the risk exposure
levels derived from Monte Carlo simulations, which are commonly
used in quantitative risk assessments and which also identify
the risk exposure levels through examining the quantiles of the
distribution.
3.1 Retrieval and adjustment of project data
Data retrieved for the dierent category stages were verified by
cross-checking with available detailed breakdown of the figures of
the respective project, such as checking the consistency of project
estimate with the base estimate plus contingencies at the same
category stage.
For the 25 projects, cost and time estimates were available for 23
projects at category C, 22 projects at category B and 20 projects at
category A.All estimates and final outturn costs were then adjusted
to the same year level using the public works price deflators issued
by the Hong Kong government.
For the projects in which the actual disbursement record
of projects could not be retrieved, statistically established
disbursement profiles are assumed (see Table1). The assumption
of a distribution profile is commonly made to convert year-of-
expenditure cost into constant cost – that is, to remove the eects of
inflation (see for example and Bacon and Besant-Jones (1996)).For
instance, a 3 year project is spending 17% of the total outturn cost
in year 1, 65% in year 2 and 18% in year 3.A 5 year project spends
6% in year 1, 22% in year 2, 43% in year 3, and so on.
An analysis of the adjusted data revealed statistically significant
improvements in forecasting accuracy for dierent vintages of
estimates (compare with Figure2). This improvement in forecasting
Project length
in years
Disbursement profile (as % of total outturn cost in xth year of the project)
1st year 2nd year 3rd year 4th year 5th year 6th year 7th year 8th year 9th year 10th year
1 100%
2 49% 51%
3 17% 65% 18%
4 9% 40% 42% 10%
5 6% 22% 43% 23% 6%
6 4% 13% 32% 33% 14% 5%
7 3% 8% 21% 32% 23% 9% 4%
8 3% 4% 10% 20% 25% 20% 11% 7%
9 3% 4% 10% 20% 25% 20% 11% 4% 3%
10 2% 3% 7% 14% 21% 22% 15% 8% 4% 3%
Table 1. Disbursement profiles
400
200
0
Cost overrun
(actual cost over cost estimated at
upgrade to category C): %
Date of the upgrade to category C
1990 1995 2000
Figure 2. Cost overrun of projects by date of project’s upgrade to
category C (added Loess trendline and 95% confidence interval)
Offprint provided courtesy of www.icevirtuallibrary.com
Author copy for personal use, not for distribution

Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
Civil Engineering
5
Steps 1–3 are repeated for the next project from the data set.
Table 2 shows the results of the validation using categoryC
baseline cost uplifts data available for 18 projects. First,
the project with ID 6736 was excluded from the dataset.
Second, using the remaining 17 projects, a reference class was
constructed. The reference class suggested a +18% P50 uplift
and a +46% uplift for P80. Third, the actual cost performance
of project 6736 was −47%, which means that both the P50
The cost or time-to-completion uplifts at each category stage
are represented by a curve similar to Figures 3 and 4.The figures’
Loess curves – that is, smoothed local regressions – give the uplifts
in cost and time to completion, respectively, for the upgrade to
category C.
Two levels of uplifts that correspond to the risk levels that
project owners are typically willing to take were highlighted. In
large portfolios, where cost overruns in one project can be oset
by cost underruns in other projects, most organisations accept a
50% chance of cost overruns after adding contingencies.In other
words, they aim for 50% certainty of the revised estimate (P50,
for portfolio management). When forecasting individual projects
or projects with a high potential impact on budgets, organisations
typically accept less risk of overruns, for example, only 20%, and
thus choose a higher level of certainty, for example 80% (P80, for
management of individual projects).
Using Figure 3 as an example, at a 20% acceptable chance
of overruns the data suggest an uplift of +44%, whereas at a 50%
acceptable chance of overruns the data suggest an uplift of +13%.The
uplifts are based on the quantity estimate that is the baseline cost in
constant Hong Kong dollar terms, excluding all contingencies added
during the estimating using risk analysis and excluding inflation.
3.3 Validation of the reference class
To test the robustness of the reference class, the ‘leave-one-out
validation’ method was used.The validation works in three steps.
■ Step 1: exclude one project from the data set.
■ Step 2: construct a reference class based on the remaining projects.
■ Step 3: check whether the suggested uplift would have provided
a risk envelope large enough for the excluded project.
Two levels of uplifts that
correspond to the risk levels
that project owners are typically
willing to take were highlighted
Project
no.
P50
uplift
P80
uplift
Actual cost
performance
Would
P50 have
prevented
overrun?
Would
P80 have
prevented
overrun?
6736 +18% +46% −47%  
6757 +14% +41% +47%  
6365 +14% +46% +32%  
6553 +14% +41% +62%  
6580 +18% +46% −37%  
6718 +18% +46% −33%  
6731 +18% +46% −32%  
6759 +18% +46% −7%  
6384 +14% +41% +52%  
642 +18% +46% +14%  
6577 +18% +46% +1%  
6706 +18% +46% −52%  
6541 +14% +41% +68%  
6721 +14% +44% +43%  
6323 +14% +46% +29%  
6694 +14% +46% +31%  
6695 +18% +46% +1%  
6645 +14% +46% +18%  
Table 2. Leave-one-out validation of cost uplifts for category C
baseline cost estimates
80
60
40
20
0
–20
–40
–60
44
13
Required uplift: %
Acceptable chance of overrun: %
020 40 50 60 80
Figure 3. Required cost uplifts for different levels of the acceptable
chance of a cost overrun for category C baseline cost estimates (added
Loess smoother and 95% confidence interval)
400
350
300
250
200
150
100
50
0
–50
75
25
Required uplift: %
Acceptable chance of overrun: %
806040020 50
Figure 4. Required time uplifts for different levels of the acceptable
chance of a delay for category C time-to-completion estimate (added
Loess smoother and 95% confidence interval)

Civil Engineering
Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
6
upgrades to each category stage at both a 50% acceptable chance
of overruns (P50) and a 20% acceptable chance of overruns (P80)
can be summarised in Table3.
The reference class forecasts show that uncertainty reduces over
time in the project estimates. At upgrade to category C the P80
uplift should be +44% for cost and +75% for time to completion;
with the upgrade to category B the uplifts have reduced because of
a reduction in uncertainty of forecasts and should be +34% for cost
and +53% for time to completion. When the project is upgraded to
category A the appropriate uplifts are +14% for cost and +26% for
time to completion.
The results also show that once a project has been upgraded
to category A the median budget is quite accurate: the P50 cost
uplift is –1%. This shows that the median (P50) project estimate at
category A tends to match with the final cost.
4.2 International benchmarking
The available cost and time overruns of Hong Kong’s roadworks
projects were compared to an international benchmark maintained
by researchers working at the University of Oxford.The benchmark
database covers 863 road projects from 34 countries.The baseline
of the benchmark database is the decision to build, sometimes
also called the ‘green light decision’. In discussions with project
planners in Hong Kong it was ascertained that the decision to build
corresponded to the upgrade to category C, because projects that
have been upgraded to category C are rarely abandoned.
As shown in Table 4, the roadworks projects in Hong Kong
had an average cost overrun of +11%, which is lower than the
international benchmark, where the average cost overrun was
+20%. This dierence, however, is not statistically significant
using a non-parametric test. Moreover, the cost overruns were
less frequent in Hong Kong: seven out of ten Hong Kong road-
works projects experienced cost overruns, whereas nine out of
ten projects in the benchmark database had cost overruns (non-
parametric test statistically significant at p = 0.006). This indicates
that there is a bias in the Hong Kong data towards cost overruns;
that is, the likelihood of a cost overrun is higher than the likelihood
of an underrun.
The bias, however, is smaller than the international benchmark.
Moreover, the bias reduces as the estimate progresses towards
construction; at category A estimates are as likely to overrun as
and the P80 envelope would have provided a sucient level of
contingencies for the project.
The table shows that in 9 out of 18 reference classes the P50 uplift
would have been sucient to prevent a cost overrun.In other words,
to validate the reference classes each project is treated as if it is to
be de-risked by the uplifts suggested by a reference class based on
the other projects.In 50% of these validations the uplift would have
been exceeded, which validates that the P50 uplift of this reference
class corresponds to a 50% acceptable chance of overruns.
Similarly, in 14 out of 18 references classes, the P80 uplift would
have provided a sucient contingency to prevent a cost overrun.
This means in 78% of the cases the risk envelope was big enough.
This shows that the P80 uplift corresponds to a 22% acceptable
chance of cost overruns, which is close to the targeted risk level
of 20%.
Taken together, these results show the reference class to be robust
in the sense that the reference class would have been sucient to
prevent cost overruns through the application of uplifts had any of
the projects in the sample been currently in planning.
4. Outcomes of the study
4.1 Uplifts for cost and time-to-completion forecasting
Based on the uplift curves in Figure3 and Figure4 for categoryC
and the respective curves for category B and category A, the results
of the cost and time-to-completion reference class forecast for the
Level of
certainty
Required uplifts
Category C Category B Category A
Base cost upliftaP50 +13% +7% –1%
P80 +44% +34% +14%
Schedule upliftbP50 +26% +22% +8%
P80 +75% +53% +26%
aTotal estimated budget, excluding inflation and excluding bottom-up
contingencies
bMeasured from the date of the estimate to the date of project completions
Table 3. Summary of the reference class forecast uplifts
International benchmark Category C estimates Category B estimates Category A estimates
Average cost overrun +20% +11% +6% –1%
Frequency of cost overruns 9 out of 10 7 out of 10c7 out of 10 5 out of 10
Standard deviation of cost overruns 30% 38% 34% 24%
Average schedule overrun +38% +58%a+30% +18%
Frequency of schedule overruns 6 out of 10 8 out of 10b8 out of 10 9 out of 10
Standard deviation of schedule overrun 85% 103% 39% 29%
Average duration (years) 5.5 8.9d
ap ≤ 0.1
bp ≤ 0.05
cp ≤ 0.01
dp ≤ 0.001
Note: Only category C estimates are directly comparable to the international benchmark
Table 4. International benchmarking of Hong Kong highways data
Offprint provided courtesy of www.icevirtuallibrary.com
Author copy for personal use, not for distribution

Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
Civil Engineering
7
decision maker the distribution suggests a top-down uplift to adjust
the bottom-up estimate.The most basic action would be to allocate
this uplift to the project as a contingency, in other words to increase
the project budget and time. However, there are two issues with this
approach.
First, Parkinson (1958) postulated that if large contingencies are
given to project managers, it is highly likely that they will be used
up. This phenomenon is well documented in capital investment
projects in the public sector. However, this study found no strong
evidence that this is the case for roadworks projects in Hong Kong,
where half the projects in the sample underspent their budget
estimate made at the upgrade to category C.It also means that not
all contingency, which was excluded from the estimates, is used
up in the projects in Hong Kong. This unusual pattern might be
attributable to peculiar cultural dierences in Hong Kong project
management and warrants further research.
Second, as observed in other jurisdictions, for example by Liu
and Napier (2010), inecient allocation of funds comes from
providing contingencies that are later not spent.A benchmark of
the full cost estimate (base cost plus contingencies) found evidence
of pessimism bias also in category C and category B cost estimates
in this sample.
Thus contingencies need to be managed more intelligently
than simply added to project estimates to (a) avoid contingencies
being spent freely and (b) avoid an inecient allocation of funds.
Portfolio management and thus a multi-level governance of
project funds can improve the allocation of budgets while tightly
controlling the contingency drawdown.
A tier contingency scheme could, for example, use the
reference class forecasting to define contract, project and portfolio
contingencies. In such a governance scheme, contract managers
could have a delegated authority over a small contingency, for
example 10%, which is equivalent to approximately P50–P55.The
project manager could have delegated authority of another 10%
contingency, which is up to approximately P60, and at portfolio
level an additional contingency of 20% could be held that covers
a total risk exposure up to P80. Such a multi-layered approach to
contingency management can ensure that project budgets are not
inflated, while contingencies are available if and when they are
needed.
5. Experience gained from the study
Much eort was made in this study to retrieve and verify project
data. These eorts could have been saved if the relevant data were
recorded in a way convenient for the application of the reference
class forecasting method.A standard proforma was developed as
they are to underrun. Conversely, the standard deviation of the cost
overruns in the Hong Kong highway projects was 38%, which was
greater than 30% of the international benchmark. In other words
the Hong Kong highway projects showed greater variability in their
cost overruns.
Time overruns showed a dierent pattern compared to the
benchmark.The average time to completion overrun of Hong Kong
roadworks projects was +58%, whereas the international benchmark
was +38% (non-parametric test not statistically significant, p =
0.089). Again the frequency of the time-to-completion overruns
was statistically significantly higher in Hong Kong (eight out of
ten projects) than in the international benchmark (six out of ten
projects, p = 0.048). This indicates that time estimates are biased
towards underestimation – that is, delays occurring – and, unlike
the cost estimates, this bias does not reduce with estimates closer
to the start of construction. Lastly, the variability of time-to-
completion overruns was higher in Hong Kong (103%) than in the
international benchmark (85%).
Also, Hong Kong roadworks projects had a statistically
significantly longer average duration of 8.9 years compared to
the benchmark of 5.5 years (non-parametric test, p < 0.001). A
summary of the time spent by the studied projects in the dierent
phases is presented in Figure5.It shows that generally more than
half of the project duration was spent before start of the actual
construction, indicating longer lead time required for project
design and planning, public engagement, and so on in the past two
decades.
Finally, the figures showed that both cost and time-to-completion
overruns improved from early estimates to estimates closer to
construction. The average cost overruns decreased from +11% at
the upgrade to category C to –1% at the upgrade to category A.
Similarly, the frequency of cost overruns decreased from seven
out of ten projects to five out of ten projects. Average time-to-
completion overruns also improved over time from +58% at
upgrade to category C to +18% at upgrade to category A.In other
words the data showed that the accuracy of forecasts increased as
the project progresses.
4.3 Adverse consequences of using uplifts and
mitigation through portfolio management
In this study it was argued that reference class forecasting shows
the most un-biased estimate of the risk exposure of a certain
project in its full distribution. For a given risk appetite of the
Portfolio management and
thus a multi-level governance
of project funds can improve
the allocation of budgets
while tightly controlling the
contingency drawdown
Completion
Build
start
Category
A
Category
B
Category
C
• Average 13 months
• Median 4 months
• Average 110 months
• Median 109 months
• Average 40 months
• Median 38 months
• Average 49 months
• Median 44 months
• Average 8 months
• Median 7 months
56% opened at
completion, 24%
opened before, 20%
opened after
Decision
to build
the project
Figure 5. Summary of project duration

Civil Engineering
Reference class forecasting for Hong
Kong’s major roadworks projects
Flyvbjerg, Hon and Fok
8
References
Ansar A, Flyvbjerg B, Budzier A and Lunn D (2014) Should we build more
large dams? The actual costs of hydropower megaproject development.
Energy Policy 69: 43–56.
Bacon RW and Besant-Jones JE (1996) Estimating Construction Costs and
Schedules: Experience with Power Generation Projects in Developing
Countries.The World Bank, Washington, DC, USA, World Bank technical
paper no.WTP 325.
Calderon C, Serven L and Moral-Benito E (2011) Is Infrastructure Capital
Productive? A Dynamic Heterogeneous Approach.The World Bank,
Washington, DC, USA.
Development Bureau (2012) Agreement No.CE 11/2012 (CE):
Implementation of Reference Class Forecasting to Large-scale Public
Works Projects in Hong Kong – Establishment of a Reference Class for
Major Highways Projects – Feasibility Study. Government of the Hong
Kong SAR, Hong Kong, PR China.
Dobbs R, Pohl H, Lin D etal. (2013) Infrastructure Productivity: How to Save
$1 Trillion a Year. McKinsey Global Institute, Seoul, South Korea.
Flyvbjerg B, Bruzelius N and Rothengatter W (2003) Megaprojects and Risk.
Cambridge University Press, Cambridge, UK.
Flyvbjerg B and Cowi (2004) Procedures for Dealing with Optimism Bias in
Transport Planning.UK Department of Transport, London, UK.
Government of the Hong Kong SAR (2014) Project Administration
Handbook for Civil Engineering Works. Government of the Hong Kong
SAR, Hong Kong, PR China.
HM Treasury (Her Majesty’s Treasury) (2013) Green Book Supplementary
Guidance: Optimism Bias. HM Treasury, London, UK. Kahneman D (2011)
Thinking, Fast and Slow. Macmillan, New York, NY, USA.
Kahneman D and Tversky A (1977) Intuitive Prediction: Biases and Corrective
Procedures. Decisions and Designs Inc., McLean, VA, USA.
Liu L and Napier Z (2010) The accuracy of risk-based cost estimation for
water infrastructure projects: preliminary evidence from Australian
projects. Construction Management and Economics 28(1): 89–100.
OECD/ITF (Organization for Economic Co-operation and Development/
International Transport Forum) (2013) Spending on Transport
Infrastructure 1995–2011: Trends, Policies, Data. International Transport
Forum, Paris, France.
Parkinson C N (1958) Parkinson’s Law: The Pursuit of Progress. John Murray,
London, UK.
Works Branch (1993) Technical Circular No.22/93 – Estimating Using Risk
Analysis. Government Secretariat, Hong Kong.
The study also contributed to the discussion of reference class
forecasting and similar methods by suggesting a method to test the
robustness of the reference class forecast and by providing data
from the Asia-Pacific region for the very first time.
Acknowledgements
The authors would like to acknowledge the Development Bureau
and the Highways Department of the Hong Kong government for
the permission to publish this paper.The authors would also like
to thank Alexander Budzier who contributed significantly to the
content of this paper.
part of the study so that all cost and time-to-completion data of
future projects that are relevant to the maintenance, updating and
application of the reference class can be recorded and retrieved
conveniently without significant eort. The concerned project
ocers were trained to record the data into the standard proforma
and to use them in updating the uplift curves of the reference class
at suitable intervals.
The study has provided valuable insights into how projects are
planned and how the maturity of individual schemes increases
during the front-end process.The study has also shown that early
estimates, made when information is scarce and uncertain, can be
de-biased to prevent cost overruns and time-to-completion delays
using a reference class forecasting approach to conservatively bias
otherwise optimistic estimates.
Moreover, the study has shown that sucient data exist in Hong
Kong and that these data can be used to create value-adding insights
for project managers, departments and policy makers. Lastly, the
study has highlighted that, although reference class forecasting
alone is a useful method to de-bias project estimates, further
enhancements can be made by integrating probabilistic thinking
into project incentive schemes and portfolio management regimes.
6. Conclusions and recommendations
Countries invest in public infrastructure with a view to increase
long-term economic growth. However, capital investment projects
often experience cost overruns and time-to-completion delays.The
reference class forecasting method is addressing the overrun issue
by adopting a top-down approach based on the ‘outside view’ of
past similar projects. With the appropriate uplifts to the baseline
cost and time-to-completion estimates, project estimates that
correspond to acceptance level of certainty can be worked out.The
method has been widely and successfully used in Europe, has
been endorsed by several countries and, for the first time, has been
developed for use in the Asia-Pacific region.
In the study, the uplift curves of the reference class of Hong
Kong roadworks projects at various development category stages
were established by relevant data collected from 25 completed
projects. These can be used to determine project estimates with
contingencies at the appropriate level of certainty.
On comparing the data with an international benchmark that is
equivalent to the baseline cost at category C stage, Hong Kong
performed better in cost forecasting on average. For time-to-
completion forecasting, it was found that generally more than half
of the project duration was spent before actual construction, that is
the pre-construction stage.
It is also argued that the current contingency provisions on
a project-by-project basis might be ineciently allocated. If
the contingencies are pooled across a portfolio of projects, the
contingency funds can be used more flexibly and thus more
eciently.
The study oers important contributions to estimation and
forecasting practice. Three implications are documented: how early
project estimates can be de-biasing; how contingency can be set
in a data-driven way that explicitly considers the risk appetite of
decision makers; and how public funds can be safeguarded; that is,
how exceedance and under-use of project budgets can be balanced
through simultaneously governing project contingencies at the
project and portfolio level.
What do you think?
If you would like to comment on this paper, please email up to 200 words
tothe editor at journals@ice.org.uk.
If you would like to write a paper of 2000 to 3500 words about your own
experience in this or any related area of civil engineering, the editor will be
happy to provide any help or advice you need.
Offprint provided courtesy of www.icevirtuallibrary.com
Author copy for personal use, not for distribution