Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared (marketsforforecasting.com)

Abstract

Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare two structured alternatives to traditional meetings: the Delphi technique and prediction markets. Delphi is relatively simple and cheap to implement and has been adopted for diverse applications in business and government since its origins in the 1950s. It can be used for nearly any forecasting, estimation, or decision making problem not barred by complexity or ignorance. While prediction markets were used more than a century ago, their popularity waned until more recent times. Prediction markets can be run continuously, and they motivate participation and participants to reveal their true beliefs. On the other hand, they need many participants and clear outcomes in order to determine pay-offs. Moreover, translating knowledge into a price is not intuitive to everyone and constructing contracts that will provide a useful forecast may not be possible for some problems. It is difficult to maintain confidentiality with markets and they are vulnerable to manipulation. Delphi is designed to reveal panelists’ knowledge and opinions via their forecasts and the reasoning they provide. This format allows testing of knowledge and learning by panelists as they refine their forecasts but may also lead to conformity due to group pressure. The reasoning provided as an output of the Delphi process is likely to be reassuring to forecast users who are uncomfortable with the “black box” nature of prediction markets. We consider that, half a century after its original development, Delphi is under-utilized.

Full text

A PUBLICATION OF THE INTERNATIONAL INSTITUTE OF FORECASTERS
FORESIGHT
The IIF, now in its 27th year, is the leading
non-profit clearinghouse of forecasting
theory, research and practice.
Issue 8  Fall 2007
$40 per issue
The International Journal of Applied Forecasting
www.forecasters.org/foresight
THE ESSENTIAL READ FOR THE PRACTICING FORECASTER
GOOD AND BAD JUDGMENT
IN FORECASTING
Lessons from Four Companies
METHODS TO ELICIT FORECASTS
FROM GROUPS
The Delphi Method
and Prediction Markets
NEW PERSPECTIVES ON THE
COST OF FORECAST ERROR
PHARMACEUTICAL FORECASTING
How to Project Patient Persistency
THE KEYS TO THE
WHITE HOUSE
Forecast for 2008
BAYESIAN MODELS FOR
SHORT TIME SERIES

17 Fall 2007 Issue 8 FORESIGHT
METHODS TO ELICIT FORECASTS FROM GROUPS
DELPHI AND PREDICTION MARKETS COMPARED
Kesten Green, J. Scott Armstrong, and Andreas Graefe
PREVIEW
The Delphi technique is better than traditional
group meetings for forecasting and has some
advantages over another promising alternative
to meetings, prediction markets. In this article,
Kesten, Scott, and Andreas observe the increasing
popularity of Delphi, describe the benefits of using
this method to obtain forecasts from experts,
compare it with prediction markets, and conclude
that Delphi should be used more widely.
J. Scott Armstrong, Professor of Marketing at the Wharton School, University of Pennsylvania, was a
founder of the
Journal of Forecasting
,
International Journal of Forecasting
, and International Symposium on
Forecasting. He is the creator of forecastingprinciples.com and editor of
Principles of Forecasting
(Kluwer,
2001), an evidence-based summary of knowledge on forecasting. In 1996, he was selected as one of the
first six Honorary Fellows by the International Institute of Forecasters. Along with Philip Kotler and Gerald
Zaltman, he was named the Society of Marketing Advances’ Distinguished Marketing Scholar of 2000. For the
past 13 years, he has been writing
Persuasive Advertising: An Evidence-Based Approach
, which he forecasts
will appear in 2008, or 2009, who knows.
Kesten Green is a Senior Research Fellow of the Business and Economic Forecasting Unit, Monash University,
Co-Director of forecastingprinciples.com, and Managing Director of Decision Research Ltd. He has published in
the
International Journal of Forecasting
,
Interfaces
,
International Journal of Business
, and
Foresight
. In recent
years, he has been researching the problem of how best to predict the decisions people will make in conflict
situations. His first paper on the topic was awarded Best Paper for 2002-2003 by the International Institute
of Forecasters. Kesten has become concerned that major government policies are based on poor forecasts,
in particular forecasts of global warming. His audit of climate forecasting methods with Scott Armstrong will
be published later in 2007 in
Energy and Environment
. Prior to his academic career, Kesten spent more than twenty years in
business as a founder of four companies.
Andreas Graefe is a research associate at the Institute for Technology Assessment and Systems Analysis at
the Research Center (Forschungszentrum) Karlsruhe, Germany. He holds a diploma (German equivalent to
a master’s degree) in Economics as well as a diploma in Information Science. In his PhD thesis, Andreas is
researching the applicability of prediction markets for long-term forecasting problems, in particular by comparing
them to the Delphi method.
INTRODUCTION
Muchcanbedonetoimproveupontraditional
group meetings. As Armstrong (2006)
showed,itisdifficulttothinkofastructured
approach (e.g., Delphi, virtual groups,
predictionmarkets)thatwouldnotimproveonthepredictions
anddecisionsmadeintraditionalmeetings.
GeneRowe’sarticleinthisissueofForesight(pp.1116)pres
entsevidence that,incomparison with traditional meetings,
theDelphitechniquecanimproveforecastinganddecision
making.Howdoesitdothat?Ifconductedproperly,Delphi
greatly improves the chances of obtaining unbiased esti
matesandforecaststhattakefullaccountoftheknowledge
and judgment of experts. Delphi is also more convenient
andversatilethanathirdmethodforaggregatingindividual
judgments:predictionmarkets.
Weconsiderthat,halfacenturyafteritsoriginaldevelopment,
Delphiisgreatlyunderutilized.
KEY POINTS
• As structured alternatives to group
meetings, Delphi and prediction markets
can improve organizational efficiency and
effectiveness.
• Delphi can be conducted relatively
cheaply and can be used to speed up, as
well as to replace meetings.
• Freeware is available at
forecastingprinciples.com to help you
implement a Delphi process.
• Delphi can be applied to a greater variety
of problems and is easier to use than
prediction markets.

18 FORESIGHT Issue 8 Fall 2007
HOW DELPHI HAS BEEN USED
TheDelphiprocedurehasbeenaroundsincethelate1950s.
Toassessitsuse,weconductedaGooglesearchfor“Delphi
AND(predictORforecast).”Thisyielded805uniquesites
outofatotalof1.4million,showingthatsomepeoplehave
paidattention.
Using the same keywords, we conducted searches in the
SocialSciencesCitationIndexandtheScienceCitationIndex
Expandedtoassesswhathasbeenhappening toresearcher
interestinDelphiovertheyears.Weidentifiedaltogether65
relevantitems:1fromthe1960s,8fromthe1970s,3from
the1980s,21fromthe1990s,and32sofarthisdecade.
When we searched for “Delphi forecast of” and “Delphi
forecastsof,”wefound42uniqueapplicationsoftheDelphi
technique.Thelargestnumberofthem(43%)werebusiness
applications.Theseincludedforecastsfor:
 theArgentinepowersector
 broadbandconnections
 drybulkshipping
 leisurepursuitsinSingapore
 rubberprocessing
 Irishspecialtyfoodsand
 oilprices.
Forecasts of technology were also popular (36%) these
includedforecastsaboutintelligentvehiclehighwaysystems,
industrial robots, intelligent internet, and technology in
education.Finally,21%ofapplicationswereconcernedwith
broadersocialissuessuchastheurbanfutureofNanaimoin
BritishColumbiaandthefutureoflawenforcement.
We also found nearly 4,000 unique items using a Google
Scholar search for the single word Delphi in titles. This
suggests that the technique is used more widely than just
forforecasting.
We have ourselves employed Delphi for problems like
forecasting prisoner numbers, choosing between regional
development options, predicting outcomes of political
elections, deciding which applicants should be hired for
academicpositions,andpredictinghowmanymealstoorder
atconferenceluncheons.
HOW DELPHI MIGHT BE USED
Delphicanbeusedfornearlyanyprobleminvolvingforecasting,
estimation,ordecisionmaking– aslongas complexityand
ignorancedonotprecludetheuseofexpertjudgment.Inshort,
itcould beusedtoreplacemostfacetofacemeetingsother
thanthoseinvolvingnegotiationsorselling.
Theissue ofignoranceisimportant.Iftheindividualsina
grouparemisinformedaboutatopic,theuseofDelphiwill,
as in a traditional group meeting, only add confidence to
theirignorance.However,uncoveringdisparity amongthe
expertsmighthelptoalertdecisionmakerstothisproblem.
Forexample,in aDelphistudy ofeconomicgrowth, most
participantsbelievedthatsupportforhighereducationwas
a positive factor, while a small minority claimed it was
negative.Thisissueshouldhavebeendecidedbyreference
totheresearchliteratureratherthanbyaskingexperts.
Peoplearenotgoodatthinkingthroughcomplexsituations,
suchasthosethatinvolveseveralroundsofinteractionswith
others. Green andArmstrong (2007) showed that unaided
experts are unable to provide valid forecasts about the
outcomesofnegotiationsandotherconflictsituations.The
Delphiprocesscannotimproveforecastswhentheindividual
panelistsareincapableofprovidingvalidforecasts.
WithDelphi,expertsareaskedtoprovidereasons fortheir
forecastsandtorespondtothepredictionsandjustifications
givenbytheotherexperts.Inourexperience,thisrecordof
argumentation among experts is attractive to those clients
who are skeptical of forecasts from a statistical model.
Hoffmannetal. (2007) observedthatthefindings of their
surveyofexpertopinionsonthedistributionoffoodborne
illnesses in the U.S. were met with skepticism until their
audiencessawthelistofexpertparticipants.
GeneRowe’spaperindicatesthatDelphicanbeexpensive,
but is it expensive in comparison with traditional group
meetings? We like the taximeter solution to meetings:
eachpersonattendingameetingentersa billingrateintoa
computerandthecomputershowsthemountingcostasthe
meetinggrindson.
Delphi can be used for nearly any
problem involving forecasting,
estimation, or decision making
– as long as complexity and
ignorance do not preclude the use
of expert judgment.

19 Fall 2007 Issue 8 FORESIGHT
Whenhighexpertstatusisnotneededtohelpselltheforecasts,
onlymodest expertiseis required.Thismeansthatexpenses
canbekeptlowandthatforecastscanbemaderapidly.
Freeware for conducting Delphi sessions is available at
forecastingprinciples.com(underSoftware).Whenthefore
castquestionisclearandpanelistsarecooperative,thesoft
warehelpstheadministratortocompleteasessioninquick
time.Thesoftwareisusedtocompilequestions,storealist
of potential panelists and their email addresses, send ap
pealstopanelists,andcompileresponses.Thesoftwarealso
providesguidanceon how to useDelphi.The directors of
forecastingprinciples.comcontinue toincrease theflexibil
ityoftheDelphisoftwaretoallowgreatercustomization.
Onewaytoreducethecostoftraditionalgroupmeetingsisto
useDelphiprocedureswithinthemeeting,aprocessknown
asMiniDelphiorestimatetalkestimate.Thisalsohelpsto
ensurethatpeopleprovidetheirestimatesduringthemeeting.
Tofurtherspeeduptraditionalmeetings,GordonandPease
(2006)developedRealTimeDelphi,awebbasedapproach
that automatically aggregates participants’ judgments and
allows them to reassess their positions. RealTime Delphi
appearspromising,butithasnotyetbeenevaluated.
DELPHI VS. PREDICTION MARKETS
In recent years, there has been a resurgence of interest in
prediction markets, which were quite popular in the late
1800s and early1900s (Rhode & Strumpf, 2004). In her
BusinessWeekarticle,King(2006)claimedthatatleast25
companieshadstartedtoexperimentwithpredictionmarkets.
Theforecastshaveproventobeaccurateinlimitedteststo
date.AninternalmarketatHewlettPackardonfutureproduct
sales,forexample,beattheofficialforecastsofthecompany
in6 outof 8events (Chen& Plott,2002). Researchersare
alsodoingmoreinthisarea,andinresponsetothisinterest
theJournalofPredictionMarketswaslaunchedin2007.
Prediction markets are similar to Delphi in that they are
both methods for aggregating diverse opinions. Little is
knownabouttherelativeaccuracyofforecastsfromthetwo
approaches,althoughbothdomuchbetterthanunstructured
groupmeetings.
Participants in prediction markets buy and sell contracts.
Thesecontractspromiseapayoffifaneventoccurs.Intheir
entryonpredictionmarketsfortheNewPalgraveDictionary
ofEconomics(2nded.),WolfersandZitzewitz(2006)provide
a useful summary of the method. They tabulated three
different types of contracts: binary options, index futures,
andspread betting.Eachisdesignedtoprovidea different
kindof forecast.In thecase ofa binaryoption market,the
priceatwhichacontractmostrecentlytraded(oranaverage
of the most recent prices) is interpreted as the market’s
assessmentoftheprobabilitythattheevent willoccur.For
example,supposeacontractwillpay$1intheeventthat
Britainwithdrawsmorethan 50% ofhertroopsfrom Iraq
beforethe endof2007 andnothingifBritaindoes not.If
thecontractlasttradedat22cents,themarket’sassessment
isthatthelikelihoodofthatwithdrawaleventis0.22.
Prediction markets have a number of advantages over
traditionalmeetings:
 Participantsaremotivatedbytheanticipationofprofitto
revealtheirtruebeliefsandtoparticipateoveralongperiod
oftime.
Marketscanberuncontinuouslyandtherebyinstantlyand
automaticallyincorporatenewinformationintotheforecast.
Participantsthemselveschoosetotakepartiftheythinkthat
theirprivateinformationhelpsthem deriveabetter forecast
thantheonethatisimpliedbythecurrentmarketprice.
Usingapredictionmarkettypicallyrequiresthatthesituation’s
outcomewilleventuallybeknown.Withoutaclearoutcome,
suchasthepercentageof votesgainedby acandidate,the
salesfigures foragiventimeperiod,orthe annualgrowth
inGDP,participantscouldnotbeappropriatelyrewardedor
punished.Littleisknownabouthowwellpredictionmarkets
performforeventswhoseoutcomesmaypotentiallynotbe
knownorcannotbeclearlydeterminedatall.Furthermore,
events that have long time horizons pose problems, as
participantsmayhavetowaitforyearsuntiltheirpayoffcan
bedetermined.
Delphihastheseadvantagesoverpredictionmarkets:
[1] Itcanbe usedfora much broaderrangeof problems,
sincethereisnoneedtojudgetheoutcomeofasituationin
ordertodeterminepayoffsforparticipants.
[2] Many people lack the understanding of how markets
workorhowtotranslatetheirexpectationsintomarketprices.
ItiseasierforpeopletorevealtheiropinionsinDelphi.

[3] It can be challenging, if not impossible, to formulate
someproblemsascontractsinpredictionmarkets.Itiseasier

20 FORESIGHT Issue 8 Fall 2007
toaddresscomplexissuesandtoobtainpredictionsbyasking
directquestionsofaDelphipanel.
[4] ItiseasiertomaintainconfidentialitywithDelphi.For
markets, it may be morally objectionable to benefit from
tradingon theoutcomeofcriticalissues.Forexample, the
policyanalysismarketsetupbytheDARPAtopredictevents
like regime changes in the Middle East or the likelihood
of terrorist attacks was cancelled one day after it was
announced (Looney,2004). Concerns may also arise over
theuseofmarketswithinbusinesses,forexampletodecide
whomtohireorfire,orwheretheforecastmaydemotivate
participantswhoarealsoemployeesofthebusiness.
[5] Prediction markets are vulnerable to speculative
attacksmountedinordertomanipulatetheresults.ADelphi
administrator,onthe otherhand, canchoosepanelistswho
are likely to reveal their true beliefs and exclude extreme
values from the calculation of the Delphi forecast either
directlyorbycalculatingamedianratherthanamean.
[6] The opportunity to provide comments or reasons for
judgments allows Delphi participants to introduce new
ideasintothe discussion.And thetransparentexchange of
knowledgeallowsexpertstolearnwhileparticipatinginthe
Delphiprocess.
[7] Such an exchange also reveals information that has
already been taken into account. This helps Delphi panels
avoid two undesirable features of predictions markets: the
inefficiencyofeach participant independentlysearchingfor
informationandtheoccurrence of cascades.Acascadeis a
cumulativeand excessiveprice movementthat occurswhen
someparticipants,assumingthatashiftinpriceisduetonew
information,react,leadingotherstoreacttothereactions.
[8] Delphirequiresonly5to20expertswhohaveagreedto
participateandshouldtherefore be superiortothinmarkets
(thosewithfewparticipants)wheretheincentivetotrade,and
therebyrevealinformation,isweak(Abramowicz,2004).
CONCLUSIONS
Insum,webelievethatDelphishouldbemuchmorewidely
used than it is today. It should replace many traditional
meetings. Provided that it does not drive out other valid
structuredmethods,itisunlikelytocauseharmandwilllikely
improveforecastinganddecisionmaking–andthusincrease
theefficiencyandeffectivenessofyourorganization.
CONTACT
Kesten Green
Business and Economic Forecasting Unit,
Monash University
kesten@kestencgreen.com
J. Scott Armstrong
The Wharton School, University of Pennsylvania
armstrong@wharton.upenn.edu
Andreas Graefe
Institute for Technology Assessment
and Systems Analysis
graefe@itas.fzk.de
REFERENCES
Abramowicz,M.B.(2004).Informationmarkets,administrative
decisionmaking,andpredictivecostbenefitanalysis,University
ofChicagoLawReview,71,9331020.
Armstrong,J.S.(2006).Howtomakebetterforecastsanddeci
sions:Avoidfacetofacemeetings,Foresight:TheInternational
JournalofAppliedForecasting,Issue5,38.
Chen,K.Y.&Plott,C.R.(2002).Informationaggregation
mechanisms:Concept,designandimplementationforasales
forecastingproblem,SocialScienceWorkingPaperNo.1131,
CaliforniaInstituteofTechnology,Pasadena.
Gordon,T.&Pease,A.(2006).RTDelphi:Anefficient,“round
less”almostrealtimeDelphimethod,TechnologicalForecasting
andSocialChange,73,321333.
Green,K.C.&Armstrong,J.S.(2007).Thevalueofexpertisefor
forecastingdecisionsinconflicts,Interfaces,37,287299.
Hoffmann,S.,Fischbeck,P.,Krupnick,A.&McWilliams,M.
(2007).Elicitationfromlarge,heterogeneousexpertpanels:Using
multipleuncertaintymeasurestocharacterizeinformationquality
fordecisionanalysis,DecisionAnalysis,4(2),91109.
King,R.(2006).Workers,placeyourbets,BusinessWeek,August
3,http://www.businessweek.com/technology/content/aug2006/
tc20060803_012437.htm
Looney,R.E.(2004).DARPA’spolicyanalysismarketforintel
ligence:Outsidetheboxoroffthewall?InternationalJournalof
IntelligenceandCounterIntelligence,17,405419.
Rhode,P.W.&Strumpf,K.S.(2004).Historicalpresidentialbet
tingmarkets,TheJournalofEconomicPerspectives,18,127141.
Rowe,G.(2007).AguidetoDelphi,Foresight:TheInternational
JournalofAppliedForecasting,Issue8,1116.
Wolfers,J.&Zitzewitz,E.(2006).Predictionmarketsintheory
andpractice,NBERWorkingPaper12083,http://bpp.wharton.
upenn.edu/jwolfers/Papers/PredictionMarkets(Palgrave).pdf