Content uploaded by Caleb Williams
Author content
All content in this area was uploaded by Caleb Williams on May 11, 2020
Content may be subject to copyright.
Simplifying the Expert
Elicitation Process
2019 NASA Cost & Schedule Symposium
Caleb Williams
2
2019 NASA COST & SCHEDULE SYMPOSIUM 2
3
2019 NASA COST & SCHEDULE SYMPOSIUM
SpaceWorks | History in the NASA Cost & Schedule Community
4
2019 NASA COST & SCHEDULE SYMPOSIUM
Expert Elicitation Overview
While the physical sciences are thought to be too
thorough and exact to rely on the judgement of experts,
the field of operations research is more pragmatic…
5
- Dr. Olaf Helmer -
6
2019 NASA COST & SCHEDULE SYMPOSIUM
Overview | What is Expert Elicitation?
▪Expert elicitation is a process for aggregating estimates from a panel of experts
▪It is a useful tool for developing estimates when there exists insufficient data or a significant
degree of uncertainty – i.e., when traditional parametric or analogous approaches aren’t possible
▪Over the past 75 years, expert elicitation has been used to yield fascinating insights into a wide-
range of global problems:
Economics Healthcare Nuclear Defense Public Policy
Global Warming
7
2019 NASA COST & SCHEDULE SYMPOSIUM
Overview | Types of Expert Elicitation
▪A number of both mathematical and behavioral techniques have been developed to facilitate the
creation of precise, aggregated expert estimates
▪Behavioral approaches are focused on bringing the group to a consensus estimate through a
variety of sociological techniques
•These include the Delphi Process, SHELF Process, Trial Roulette Method, etc.
▪Meanwhile, mathematical approaches use statistical methods to aggregate expert estimates
•These include Simple Averages, Distribution Fitting, Opinion Pooling, etc.
Many expert elicitation processes involve a combination of both
behavioral and mathematical aggregation techniques
8
2019 NASA COST & SCHEDULE SYMPOSIUM
Overview | Traditional Expert Elicitation Process (SHELF)
▪Decision Maker
▪Facilitator
▪Normative Experts
▪Domain Experts
▪Stakeholders
▪Analyst
▪Evidence Dossier
▪Training Materials
▪Preliminary Estimate
▪Individual Elicitation Results
▪Group Discussion Notes
▪Consensus Elicitation Results
Evidence
Dossier
Training
Workshop
Expert
Selection
Plausible
Range Individual
Estimate Moderated
Discussion RIO
Estimate Distribution
Fitting
95% Range Tertiles
Quartiles Discrete
Probabilities Normal
Gamma
Beta
LogNormal
Delayed Results Compilation
9
2019 NASA COST & SCHEDULE SYMPOSIUM
Overview | Drawbacks of Traditional Expert Elicitation
Issues in Conventional Approaches
Numerous
Roles &
Artifacts
Complex
Decision
Trees
Specialist
Training
Required
Delayed
Results
Compilation
Cognitive
Heuristics &
Bias
Poorly performed expert elicitation exercises are likely
to be misinterpreted and further hurt the adoption
of more suitable approaches
10
11
2019 NASA COST & SCHEDULE SYMPOSIUM
Simplifying the Process
12
2019 NASA COST & SCHEDULE SYMPOSIUM
Simplifying | A Simplified Approach
▪To make systematic expert elicitation more accessible, and drive further adoption of such
approaches, SpaceWorks has developed a simplified approach to expert elicitation
▪This new approach preserves many of the essential features of more complex elicitation methods,
while simultaneously reducing process overhead, required setup time, and decision points
▪The proposed Simplified Approach incorporates three key elements:
Element Name
Purpose
Modern Collaboration Tools
Enables real-time aggregation of expert responses and facilitates elicitation workshops with distributed teams
Mini
-Delphi Games Helps to mitigate cognitive heuristics and bias while reducing the required number of artifacts and roles
Opinion Pooling
Provides a flexible, statistical way for aggregating expert estimates without adding numerous decision gates
13
2019 NASA COST & SCHEDULE SYMPOSIUM
Simplifying | Key Process Element #1: Modern Collaboration Tools
▪Traditional expert elicitation requires a large number of physical
artifacts (questionnaire responses, individual estimates, etc.)
▪By using modern collaboration tools, such as Google Drive or
SharePoint Online, the Simplified Approach streamlines the
data collection process and creates a single source of record
▪These tools can be used to create simple forms that enable
geographically disaggregated teams to easily complete expert
elicitation exercises
▪Perhaps most importantly, they enable a real-time feedback
loop so there is no delay between estimating rounds
Example Modern Collaboration
Form (Google Drive)
14
2019 NASA COST & SCHEDULE SYMPOSIUM
Simplifying | Key Process Element #2: Mini-Delphi Games
▪Mini-Delphi games employ a similar process to the traditional
Delphi collection method, but in a simpler format
▪The approach is relatively simple:
•The process begins with each participant submitting an anonymous
individual estimate and any assumptions made
•Next, the median collated estimate is displayed and a moderator leads a
short discussion, introducing any assumptions made by participants
•After the discussion period, respondents are asked to submit and updated
estimate
▪This method enables a type of behavioral consensus to be
reached, without forcing an artificial point of agreement
▪By limiting the information flow to the “why” but not the “what”,
Mini-Delphi games help reduce cognitive heuristics and bias
•Similar to traditional Delphi techniques, but with less overhead cost
Individual
Estimate
Moderated
Discussion
Mini-Delphi
Process
Collated
Responses
15
2019 NASA COST & SCHEDULE SYMPOSIUM
Simplifying | Key Process Element #3: Opinion Pooling
▪While behavioral consensus generating techniques are critical to
the overall process, it is unrealistic for expert opinions to
converge on a specific point
▪Opinion pools offer a statistical basis for aggregating expert
opinions beyond behavioral consensus alone
•They additionally offer the ability to weight individual respondent
estimates based on a variety of factors
▪The technique involves turning expert probability distributions (in
this case, a Triangle Distribution) into Cumulative Density
Functions (CDFs), which can then be averaged
▪The Simplified Approach calls only for the use of Linear Opinion
Pools, though more advanced approaches can also be used
Combined Linear Opinion Pool
Respondent CDFs
Raw Respondent Distributions
16
2019 NASA COST & SCHEDULE SYMPOSIUM
Simplifying | Simplified Expert Elicitation Process
▪Facilitator
▪Analyst
▪Subject Matter Experts
▪Evidence Dossier
▪Online Workbook
Mini-Delphi Game
Triangle
Distribution Triangle
Distribution Linear
Workbook
Setup
Evidence
Dossier
Expert
Selection
Individual
Estimate Moderated
Discussion Final
Estimate Opinion
Pool
The goal of the simplified approach is to eliminate
overhead and decision gates in an effort to drive
responsible adoption of expert elicitation
17
18
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study: Reusable Spaceplane
19
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Overview
▪As part of an ongoing study, SpaceWorks was tasked with creating an estimate of turnaround
times for a reusable spaceplane vehicle
▪Six individual experts were identified and asked to participate in the Simplified Approach for expert
elicitation to estimate man-hours required for each task involved in the vehicle turnaround
•Nearly 200 maintenance activities were condensed into 20 individual tasks that encompassed the entirety of
the vehicle’s turnaround operations
Task 2.4
–Perform Servicing/Closeout
2.4.1 Drain and flush fluid systems
2.4.2 Replenish, fill, or verify fluids and gas commodities (including chemical purity)
2.4.3 Lubricate and adjust subsystems as required
2.4.4 Perform cleaning close-out
2.4.5 Remove any access hardware or other non-flight hardware
2.4.6 Perform close-out photography
…
20
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Reusable Spaceplane Turnaround Operations
21
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Results Summary
The Simplified Approach
had a significant impact
on group consensus
and
the actual manhours
estimate
The mean manhours
estimate was reduced by
nearly 15% in Game B,
a sizable shift from the
initial estimate
More importantly, the
interquartile range
shrunk by 27%,
demonstrating a much
greater degree of
respondent consensus
0%
25%
50%
75%
100%
0500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Cumulative Probability
Total Manhours (All Tasks)
0%
25%
50%
75%
100%
0500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Cumulative Probability
Total Manhours (All Tasks)
Respondent #1 Respondent #2 Respondent #3 Respondent #4 Respondent #5 Respondent #6 Opinion Pool
Mean: 2531
IQR: 1026
Mean: 2153
IQR: 746
22
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Variation Across Games
The coefficient of
dispersion provides
insight into how certain
experts are about their
estimate
Respondents in this case
actually had greater
variation in Game B
as compared
to Game A
Despite greater variation
within individual estimates,
the consensus estimate
has a lower coefficient of
dispersion in Game B
-9% -11% +24% +18% +13% +23% -14%
23
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Consensus in Tasks
It can also be useful to
look at the coefficient
of dispersion at
the task level
Across all tasks, the
average coefficient of
dispersion dropped by
over 20% between Game
A and Game B
A lower coefficient of
dispersion at this level
sheds insight into why
Game B overall showed
greater consensus
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Coefficient of Dispersion
Preparation Landing Maintenance Preflight Launch
24
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Individual Respondent Distributions
Game A Game B Opinion Pool
0%
25%
50%
75%
100%
01000 2000 3000 4000 5000
Total Manhours (All Tasks)
0%
25%
50%
75%
100%
01000 2000 3000 4000 5000
Total Manhours (All Tasks)
0%
25%
50%
75%
100%
01000 2000 3000 4000 5000
Total Manhours (All Tasks)
0%
25%
50%
75%
100%
01000 2000 3000 4000 5000
Total Manhours (All Tasks)
0%
25%
50%
75%
100%
01000 2000 3000 4000 5000
Total Manhours (All Tasks)
0%
25%
50%
75%
100%
01000 2000 3000 4000 5000
Total Manhours (All Tasks)
25
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Optimism Bias
▪At the task level, we find that most participants were relatively centered in their estimates, though
some individuals did show a distinct optimism bias
•Estimation preferences were more prevalent in Game A, with four respondents trending optimistic
(responding below the group mean 60% or more of the time)
▪Only two respondents showed an overall personal estimation bias across both games
(responding above or below the group mean more than 60% of the time)
Respondent Game A Game B Overall
Trend
Below
Group Mean Above
Group Mean Below
Group Mean Above
Group Mean
#1 16 414 6 Optimistic
#2 12 811 9 -
#3 15 513 7 Optimistic
#4 12 810 10 -
#5 10 10 10 10 -
#6 911 911 -
26
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Additional Bias
▪The results presented show significant potential for identifying (and potentially correcting)
additional cognitive bias present in the expert elicitation process
▪While not fully explored in this presentation, evaluating the integral of personal respondent
distributions may provide substantive insight into:
•Conservatism Bias
•Leadership Bias
•Group Think
•Default Effect
Measuring and resolving cognitive bias in expert
elicitation is an intriguing field of future research
27
2019 NASA COST & SCHEDULE SYMPOSIUM
Case Study | Worth It?
▪At first glance, it is tempting to simply take the median of the expert’s point estimates
▪As seen in the table below, the simplified approach provides substantially different results, with the
end result varying by nearly 30%
•This is because an aggregate median most likely estimate does not account for the distribution tails
▪Considering the cost of a 30% delta in a cost, schedule, or risk estimate, this approach shows
promise as a low-overhead, high value-add method for conducting expert elicitation
Sum of Median Most Likely Opinion Pool Median Value % Difference
Game A (All Tasks) 1894 2531 -34%
Game B (All Tasks) 1669 2153 -29%
The proposed simplified approach captures many
of the benefits of more thorough methods,
while reducing process complexity
28
29
2019 NASA COST & SCHEDULE SYMPOSIUM
Takeaways
30
2019 NASA COST & SCHEDULE SYMPOSIUM
Takeaways | Conclusions
▪Expert elicitation is a useful tool for developing estimates when there exists insufficient data or a
significant degree of uncertainty
▪Current expert elicitation approaches are relatively complex, require a large number of
roles/artifacts, and can be expensive to conduct properly
▪The proposed simplified approach incorporates many of the essential features of more thorough
elicitation approaches while reducing overhead requirements and decision gates
▪A case study using this new method to estimate turnaround time for a reusable spaceplane
indicated group consensus improved by almost 30%
The proposed approach shows great promise as a low-overhead,
High value-add method for expert elicitation
31
2019 NASA COST & SCHEDULE SYMPOSIUM
Takeaways | Final Thoughts
▪The primary goal of this approach is to drive adoption of responsible expert elicitation methods
•While this method may not provide a more accurate or systematic approach than traditional techniques, it is
less expensive and more accessible
▪Perhaps the most intriguing finding of this study was that this approach may provide the means to
identify cognitive bias present in the expert elicitation approaches
▪If cognitive bias can be identified based on personal respondent distributions, they may also be
able to be measured and resolved
Future research from SpaceWorks on this topic will involve evaluating
techniques for identifying, measuring, and resolving cognitive bias
Authors:
Caleb Williams
Lead Economic Analyst
caleb.williams@spaceworks.aero
Bill Doncaster
Senior Systems Engineer
bill.doncaster@spaceworks.aero
Illustrations by: Davin Gerber, Lorena Carapaica
2