ArticlePDF Available

Abstract

Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare two structured alternatives to traditional meetings: the Delphi technique and prediction markets. Delphi is relatively simple and cheap to implement and has been adopted for diverse applications in business and government since its origins in the 1950s. It can be used for nearly any forecasting, estimation, or decision making problem not barred by complexity or ignorance. While prediction markets were used more than a century ago, their popularity waned until more recent times. Prediction markets can be run continuously, and they motivate participation and participants to reveal their true beliefs. On the other hand, they need many participants and clear outcomes in order to determine pay-offs. Moreover, translating knowledge into a price is not intuitive to everyone and constructing contracts that will provide a useful forecast may not be possible for some problems. It is difficult to maintain confidentiality with markets and they are vulnerable to manipulation. Delphi is designed to reveal panelists’ knowledge and opinions via their forecasts and the reasoning they provide. This format allows testing of knowledge and learning by panelists as they refine their forecasts but may also lead to conformity due to group pressure. The reasoning provided as an output of the Delphi process is likely to be reassuring to forecast users who are uncomfortable with the “black box” nature of prediction markets. We consider that, half a century after its original development, Delphi is under-utilized.
A PUBLICATION OF THE INTERNATIONAL INSTITUTE OF FORECASTERS
FORESIGHT
The IIF, now in its 27th year, is the leading
non-profit clearinghouse of forecasting
theory, research and practice.
Issue 8 Fall 2007
$40 per issue
The International Journal of Applied Forecasting
www.forecasters.org/foresight
THE ESSENTIAL READ FOR THE PRACTICING FORECASTER
GOOD AND BAD JUDGMENT
IN FORECASTING
Lessons from Four Companies
METHODS TO ELICIT FORECASTS
FROM GROUPS
The Delphi Method
and Prediction Markets
NEW PERSPECTIVES ON THE
COST OF FORECAST ERROR
PHARMACEUTICAL FORECASTING
How to Project Patient Persistency
THE KEYS TO THE
WHITE HOUSE
Forecast for 2008
BAYESIAN MODELS FOR
SHORT TIME SERIES
17 Fall 2007 Issue 8 FORESIGHT
METHODS TO ELICIT FORECASTS FROM GROUPS
DELPHI AND PREDICTION MARKETS COMPARED
Kesten Green, J. Scott Armstrong, and Andreas Graefe
PREVIEW
The Delphi technique is better than traditional
group meetings for forecasting and has some
advantages over another promising alternative
to meetings, prediction markets. In this article,
Kesten, Scott, and Andreas observe the increasing
popularity of Delphi, describe the benefits of using
this method to obtain forecasts from experts,
compare it with prediction markets, and conclude
that Delphi should be used more widely.
J. Scott Armstrong, Professor of Marketing at the Wharton School, University of Pennsylvania, was a
founder of the
Journal of Forecasting
,
International Journal of Forecasting
, and International Symposium on
Forecasting. He is the creator of forecastingprinciples.com and editor of
Principles of Forecasting
(Kluwer,
2001), an evidence-based summary of knowledge on forecasting. In 1996, he was selected as one of the
first six Honorary Fellows by the International Institute of Forecasters. Along with Philip Kotler and Gerald
Zaltman, he was named the Society of Marketing Advances’ Distinguished Marketing Scholar of 2000. For the
past 13 years, he has been writing
Persuasive Advertising: An Evidence-Based Approach
, which he forecasts
will appear in 2008, or 2009, who knows.
Kesten Green is a Senior Research Fellow of the Business and Economic Forecasting Unit, Monash University,
Co-Director of forecastingprinciples.com, and Managing Director of Decision Research Ltd. He has published in
the
International Journal of Forecasting
,
Interfaces
,
International Journal of Business
, and
Foresight
. In recent
years, he has been researching the problem of how best to predict the decisions people will make in conflict
situations. His first paper on the topic was awarded Best Paper for 2002-2003 by the International Institute
of Forecasters. Kesten has become concerned that major government policies are based on poor forecasts,
in particular forecasts of global warming. His audit of climate forecasting methods with Scott Armstrong will
be published later in 2007 in
Energy and Environment
. Prior to his academic career, Kesten spent more than twenty years in
business as a founder of four companies.
Andreas Graefe is a research associate at the Institute for Technology Assessment and Systems Analysis at
the Research Center (Forschungszentrum) Karlsruhe, Germany. He holds a diploma (German equivalent to
a master’s degree) in Economics as well as a diploma in Information Science. In his PhD thesis, Andreas is
researching the applicability of prediction markets for long-term forecasting problems, in particular by comparing
them to the Delphi method.
INTRODUCTION
Muchcanbedonetoimproveupontraditional
group meetings. As Armstrong (2006)
showed,itisdifficulttothinkofastructured
approach (e.g., Delphi, virtual groups,
predictionmarkets)thatwouldnotimproveonthepredictions
anddecisionsmadeintraditionalmeetings.
GeneRowe’sarticleinthisissueofForesight(pp.1116)pres
entsevidence that,incomparison with traditional meetings,
theDelphitechniquecanimproveforecastinganddecision
making.Howdoesitdothat?Ifconductedproperly,Delphi
greatly improves the chances of obtaining unbiased esti
matesandforecaststhattakefullaccountoftheknowledge
and judgment of experts. Delphi is also more convenient
andversatilethanathirdmethodforaggregatingindividual
judgments:predictionmarkets.
Weconsiderthat,halfacenturyafteritsoriginaldevelopment,
Delphiisgreatlyunderutilized.
KEY POINTS
As structured alternatives to group
meetings, Delphi and prediction markets
can improve organizational efficiency and
effectiveness.
Delphi can be conducted relatively
cheaply and can be used to speed up, as
well as to replace meetings.
Freeware is available at
forecastingprinciples.com to help you
implement a Delphi process.
Delphi can be applied to a greater variety
of problems and is easier to use than
prediction markets.
18 FORESIGHT Issue 8 Fall 2007
HOW DELPHI HAS BEEN USED
TheDelphiprocedurehasbeenaroundsincethelate1950s.
Toassessitsuse,weconductedaGooglesearchfor“Delphi
AND(predictORforecast).”Thisyielded805uniquesites
outofatotalof1.4million,showingthatsomepeoplehave
paidattention.
Using the same keywords, we conducted searches in the
SocialSciencesCitationIndexandtheScienceCitationIndex
Expandedtoassesswhathasbeenhappening toresearcher
interestinDelphiovertheyears.Weidentifiedaltogether65
relevantitems:1fromthe1960s,8fromthe1970s,3from
the1980s,21fromthe1990s,and32sofarthisdecade.
When we searched for “Delphi forecast of” and “Delphi
forecastsof,”wefound42uniqueapplicationsoftheDelphi
technique.Thelargestnumberofthem(43%)werebusiness
applications.Theseincludedforecastsfor:
 theArgentinepowersector
 broadbandconnections
 drybulkshipping
 leisurepursuitsinSingapore
 rubberprocessing
 Irishspecialtyfoodsand
 oilprices.
Forecasts of technology were also popular (36%) these
includedforecastsaboutintelligentvehiclehighwaysystems,
industrial robots, intelligent internet, and technology in
education.Finally,21%ofapplicationswereconcernedwith
broadersocialissuessuchastheurbanfutureofNanaimoin
BritishColumbiaandthefutureoflawenforcement.
We also found nearly 4,000 unique items using a Google
Scholar search for the single word Delphi in titles. This
suggests that the technique is used more widely than just
forforecasting.
We have ourselves employed Delphi for problems like
forecasting prisoner numbers, choosing between regional
development options, predicting outcomes of political
elections, deciding which applicants should be hired for
academicpositions,andpredictinghowmanymealstoorder
atconferenceluncheons.
HOW DELPHI MIGHT BE USED
Delphicanbeusedfornearlyanyprobleminvolvingforecasting,
estimation,ordecisionmaking– aslongas complexityand
ignorancedonotprecludetheuseofexpertjudgment.Inshort,
itcould beusedtoreplacemostfacetofacemeetingsother
thanthoseinvolvingnegotiationsorselling.
Theissue ofignoranceisimportant.Iftheindividualsina
grouparemisinformedaboutatopic,theuseofDelphiwill,
as in a traditional group meeting, only add confidence to
theirignorance.However,uncoveringdisparity amongthe
expertsmighthelptoalertdecisionmakerstothisproblem.
Forexample,in aDelphistudy ofeconomicgrowth, most
participantsbelievedthatsupportforhighereducationwas
a positive factor, while a small minority claimed it was
negative.Thisissueshouldhavebeendecidedbyreference
totheresearchliteratureratherthanbyaskingexperts.
Peoplearenotgoodatthinkingthroughcomplexsituations,
suchasthosethatinvolveseveralroundsofinteractionswith
others. Green andArmstrong (2007) showed that unaided
experts are unable to provide valid forecasts about the
outcomesofnegotiationsandotherconflictsituations.The
Delphiprocesscannotimproveforecastswhentheindividual
panelistsareincapableofprovidingvalidforecasts.
WithDelphi,expertsareaskedtoprovidereasons fortheir
forecastsandtorespondtothepredictionsandjustifications
givenbytheotherexperts.Inourexperience,thisrecordof
argumentation among experts is attractive to those clients
who are skeptical of forecasts from a statistical model.
Hoffmannetal. (2007) observedthatthefindings of their
surveyofexpertopinionsonthedistributionoffoodborne
illnesses in the U.S. were met with skepticism until their
audiencessawthelistofexpertparticipants.
GeneRowe’spaperindicatesthatDelphicanbeexpensive,
but is it expensive in comparison with traditional group
meetings? We like the taximeter solution to meetings:
eachpersonattendingameetingentersa billingrateintoa
computerandthecomputershowsthemountingcostasthe
meetinggrindson.
Delphi can be used for nearly any
problem involving forecasting,
estimation, or decision making
– as long as complexity and
ignorance do not preclude the use
of expert judgment.
19 Fall 2007 Issue 8 FORESIGHT
Whenhighexpertstatusisnotneededtohelpselltheforecasts,
onlymodest expertiseis required.Thismeansthatexpenses
canbekeptlowandthatforecastscanbemaderapidly.
Freeware for conducting Delphi sessions is available at
forecastingprinciples.com(underSoftware).Whenthefore
castquestionisclearandpanelistsarecooperative,thesoft
warehelpstheadministratortocompleteasessioninquick
time.Thesoftwareisusedtocompilequestions,storealist
of potential panelists and their email addresses, send ap
pealstopanelists,andcompileresponses.Thesoftwarealso
providesguidanceon how to useDelphi.The directors of
forecastingprinciples.comcontinue toincrease theflexibil
ityoftheDelphisoftwaretoallowgreatercustomization.
Onewaytoreducethecostoftraditionalgroupmeetingsisto
useDelphiprocedureswithinthemeeting,aprocessknown
asMiniDelphiorestimatetalkestimate.Thisalsohelpsto
ensurethatpeopleprovidetheirestimatesduringthemeeting.
Tofurtherspeeduptraditionalmeetings,GordonandPease
(2006)developedRealTimeDelphi,awebbasedapproach
that automatically aggregates participants’ judgments and
allows them to reassess their positions. RealTime Delphi
appearspromising,butithasnotyetbeenevaluated.
DELPHI VS. PREDICTION MARKETS
In recent years, there has been a resurgence of interest in
prediction markets, which were quite popular in the late
1800s and early1900s (Rhode & Strumpf, 2004). In her
BusinessWeekarticle,King(2006)claimedthatatleast25
companieshadstartedtoexperimentwithpredictionmarkets.
Theforecastshaveproventobeaccurateinlimitedteststo
date.AninternalmarketatHewlettPackardonfutureproduct
sales,forexample,beattheofficialforecastsofthecompany
in6 outof 8events (Chen& Plott,2002). Researchersare
alsodoingmoreinthisarea,andinresponsetothisinterest
theJournalofPredictionMarketswaslaunchedin2007.
Prediction markets are similar to Delphi in that they are
both methods for aggregating diverse opinions. Little is
knownabouttherelativeaccuracyofforecastsfromthetwo
approaches,althoughbothdomuchbetterthanunstructured
groupmeetings.
Participants in prediction markets buy and sell contracts.
Thesecontractspromiseapayoffifaneventoccurs.Intheir
entryonpredictionmarketsfortheNewPalgraveDictionary
ofEconomics(2nded.),WolfersandZitzewitz(2006)provide
a useful summary of the method. They tabulated three
different types of contracts: binary options, index futures,
andspread betting.Eachisdesignedtoprovidea different
kindof forecast.In thecase ofa binaryoption market,the
priceatwhichacontractmostrecentlytraded(oranaverage
of the most recent prices) is interpreted as the market’s
assessmentoftheprobabilitythattheevent willoccur.For
example,supposeacontractwillpay$1intheeventthat
Britainwithdrawsmorethan 50% ofhertroopsfrom Iraq
beforethe endof2007 andnothingifBritaindoes not.If
thecontractlasttradedat22cents,themarket’sassessment
isthatthelikelihoodofthatwithdrawaleventis0.22.
Prediction markets have a number of advantages over
traditionalmeetings:
 Participantsaremotivatedbytheanticipationofprofitto
revealtheirtruebeliefsandtoparticipateoveralongperiod
oftime.
Marketscanberuncontinuouslyandtherebyinstantlyand
automaticallyincorporatenewinformationintotheforecast.
Participantsthemselveschoosetotakepartiftheythinkthat
theirprivateinformationhelpsthem deriveabetter forecast
thantheonethatisimpliedbythecurrentmarketprice.
Usingapredictionmarkettypicallyrequiresthatthesituation’s
outcomewilleventuallybeknown.Withoutaclearoutcome,
suchasthepercentageof votesgainedby acandidate,the
salesfigures foragiventimeperiod,orthe annualgrowth
inGDP,participantscouldnotbeappropriatelyrewardedor
punished.Littleisknownabouthowwellpredictionmarkets
performforeventswhoseoutcomesmaypotentiallynotbe
knownorcannotbeclearlydeterminedatall.Furthermore,
events that have long time horizons pose problems, as
participantsmayhavetowaitforyearsuntiltheirpayoffcan
bedetermined.
Delphihastheseadvantagesoverpredictionmarkets:
[1] Itcanbe usedfora much broaderrangeof problems,
sincethereisnoneedtojudgetheoutcomeofasituationin
ordertodeterminepayoffsforparticipants.
[2] Many people lack the understanding of how markets
workorhowtotranslatetheirexpectationsintomarketprices.
ItiseasierforpeopletorevealtheiropinionsinDelphi.
[3] It can be challenging, if not impossible, to formulate
someproblemsascontractsinpredictionmarkets.Itiseasier
20 FORESIGHT Issue 8 Fall 2007
toaddresscomplexissuesandtoobtainpredictionsbyasking
directquestionsofaDelphipanel.
[4] ItiseasiertomaintainconfidentialitywithDelphi.For
markets, it may be morally objectionable to benefit from
tradingon theoutcomeofcriticalissues.Forexample, the
policyanalysismarketsetupbytheDARPAtopredictevents
like regime changes in the Middle East or the likelihood
of terrorist attacks was cancelled one day after it was
announced (Looney,2004). Concerns may also arise over
theuseofmarketswithinbusinesses,forexampletodecide
whomtohireorfire,orwheretheforecastmaydemotivate
participantswhoarealsoemployeesofthebusiness.
[5] Prediction markets are vulnerable to speculative
attacksmountedinordertomanipulatetheresults.ADelphi
administrator,onthe otherhand, canchoosepanelistswho
are likely to reveal their true beliefs and exclude extreme
values from the calculation of the Delphi forecast either
directlyorbycalculatingamedianratherthanamean.
[6] The opportunity to provide comments or reasons for
judgments allows Delphi participants to introduce new
ideasintothe discussion.And thetransparentexchange of
knowledgeallowsexpertstolearnwhileparticipatinginthe
Delphiprocess.
[7] Such an exchange also reveals information that has
already been taken into account. This helps Delphi panels
avoid two undesirable features of predictions markets: the
inefficiencyofeach participant independentlysearchingfor
informationandtheoccurrence of cascades.Acascadeis a
cumulativeand excessiveprice movementthat occurswhen
someparticipants,assumingthatashiftinpriceisduetonew
information,react,leadingotherstoreacttothereactions.
[8] Delphirequiresonly5to20expertswhohaveagreedto
participateandshouldtherefore be superiortothinmarkets
(thosewithfewparticipants)wheretheincentivetotrade,and
therebyrevealinformation,isweak(Abramowicz,2004).
CONCLUSIONS
Insum,webelievethatDelphishouldbemuchmorewidely
used than it is today. It should replace many traditional
meetings. Provided that it does not drive out other valid
structuredmethods,itisunlikelytocauseharmandwilllikely
improveforecastinganddecisionmaking–andthusincrease
theefficiencyandeffectivenessofyourorganization.
CONTACT
Kesten Green
Business and Economic Forecasting Unit,
Monash University
kesten@kestencgreen.com
J. Scott Armstrong
The Wharton School, University of Pennsylvania
armstrong@wharton.upenn.edu
Andreas Graefe
Institute for Technology Assessment
and Systems Analysis
graefe@itas.fzk.de
REFERENCES
Abramowicz,M.B.(2004).Informationmarkets,administrative
decisionmaking,andpredictivecostbenefitanalysis,University
ofChicagoLawReview,71,9331020.
Armstrong,J.S.(2006).Howtomakebetterforecastsanddeci
sions:Avoidfacetofacemeetings,Foresight:TheInternational
JournalofAppliedForecasting,Issue5,38.
Chen,K.Y.&Plott,C.R.(2002).Informationaggregation
mechanisms:Concept,designandimplementationforasales
forecastingproblem,SocialScienceWorkingPaperNo.1131,
CaliforniaInstituteofTechnology,Pasadena.
Gordon,T.&Pease,A.(2006).RTDelphi:Anefficient,“round
less”almostrealtimeDelphimethod,TechnologicalForecasting
andSocialChange,73,321333.
Green,K.C.&Armstrong,J.S.(2007).Thevalueofexpertisefor
forecastingdecisionsinconflicts,Interfaces,37,287299.
Hoffmann,S.,Fischbeck,P.,Krupnick,A.&McWilliams,M.
(2007).Elicitationfromlarge,heterogeneousexpertpanels:Using
multipleuncertaintymeasurestocharacterizeinformationquality
fordecisionanalysis,DecisionAnalysis,4(2),91109.
King,R.(2006).Workers,placeyourbets,BusinessWeek,August
3,http://www.businessweek.com/technology/content/aug2006/
tc20060803_012437.htm
Looney,R.E.(2004).DARPA’spolicyanalysismarketforintel
ligence:Outsidetheboxoroffthewall?InternationalJournalof
IntelligenceandCounterIntelligence,17,405419.
Rhode,P.W.&Strumpf,K.S.(2004).Historicalpresidentialbet
tingmarkets,TheJournalofEconomicPerspectives,18,127141.
Rowe,G.(2007).AguidetoDelphi,Foresight:TheInternational
JournalofAppliedForecasting,Issue8,1116.
Wolfers,J.&Zitzewitz,E.(2006).Predictionmarketsintheory
andpractice,NBERWorkingPaper12083,http://bpp.wharton.
upenn.edu/jwolfers/Papers/PredictionMarkets(Palgrave).pdf
... Kada se radi o predviđanju u oblasti nauke i tehnologije, treba imati na umu da dinamika razvoja i promena koje utiču na razvoj istih toliko izražena da je samim tim neizvesnost velika da nismo uvek u mogućnosti da tačno predvidimo određene promene tj dešavanja. [1] Kada govorimo o izboru stručnjaka onda je veoma važno pomenuti i stepen njihove stručnosti odnosno kompetetnosti jer je to jedna od slabih tačaka ove metode. Stručnjaci moraju biti upoznati sa temom predviđanja jer će u suprotnom to samo dodati veću grešku u sprovođenju metode. ...
... Vreme ostvarenja događaja : Veličine koji su dati u jednačinama (16) , 1 r , 2 r , 3 r , 4 r su koeficijenti po PERT metodi. Iz jednačine (17) dobijamo da je vrednost koeficijenata za normalnu raspodelu : ...
... It has been developed as a systematic, interactive forecasting method that relies on a team of experts. Two structured alternatives to traditional meetings, the Delphi method and Prediction Markets, have been compared in [5]. ...
... Our novel evaluation model together with the structured communication process is an effective method to carry out technology forecasting and capability evaluation projects. If desired, the proposed evaluation model can be used as a tool to get started in a larger evaluation project that uses dedicated or more complex methods like the Delphi method [3]- [5]. The general evaluation method is flexible for applications in small or large projects that can also use other formal techniques. ...
Conference Paper
Full-text available
We present a method to assess the effects of new technologies on the future development of military capabilities. Assessing technologies without any connection to capabilities is insufficient to assist the decision-making of selecting development projects or planning future procurement programs. Instead, a method to connect technologies to capabilities is needed. To this end, we propose a novel evaluation model to help carry out the evaluation process most easily. The evaluation model describes the effects of technological development on military capabilities through the application of different systems. Although our use cases consider military applications, the same questions are present in other fields in the private and public sectors. The focus of this study is to present the method while modelling results are provided only for illustrative purposes.
... The literature indicates that the customer satisfaction index [10] for smart services was proposed on a survey questionnaire developed through several rounds utilizing the Delphi method [33]. The expert assessment was conducted using a ten-point measurement scale. ...
Article
Full-text available
As a part of the food industry, the dairy industry is one of the most important sectors of the process industry, keeping in mind the number of employees in that sector, the share in the total industrial production, and the overall value added. Many strategies have been developed over time to satisfy customer needs and assess customer satisfaction. This paper proposes an innovative model based on adaptive neuro-fuzzy inference system (ANFIS) and elements of the ACSI (American customer satisfaction index) for assessing and monitoring the level of customer satisfaction in a dairy manufacturing company where there are no large seasonal variations. In terms of an innovative approach, the base of fuzzy logic rules is determined by applying the fuzzy Delphi technique for the application of the ANFIS algorithm and assessment of customer satisfaction. The verification of the model is delivered by testing a real sample from a company of the dairy industry. As decisions on the strategic company level may be impacted by customer satisfaction, the company management should choose the most precise methodology for customer satisfaction assessment. The results are compared with other methods in terms of mean absolute deviation (MAD), mean squared error (MSE), and mean absolute percentage error (MAPE). Results show that ANFIS outperformed other methods used for assessing the level of customer satisfaction, such as case-based reasoning and multiple linear regression.
... A. The concept of financial failure prediction Before delving into the concept of financial failure, it is necessary to look at the researchers' opinions about the term "prediction" in general. Both Duran and Gregory see prediction as knowing what will happen in the future by making likely occurrences (Armstrong, 2011). Taqz also went in the same direction, but he defined prediction simply as the process of "predicting the future." ...
Article
Full-text available
Study the prediction of failure is an exciting topic in time of deteriorating economy. Banks compromise an important pilar within any economy. Altman model for prediction of financial failure is well-known application amongst the academicians. Current study test the impact of adopting IFRS on the accuracy of Altman model by testing the output of the model before and after imposing the IFRS application in the Iraqi private banks in 2017. The major finding of the study is that adopting IFRS enhanced the model prediction accuracy toward much plausible results.
... The Delphi survey methodology includes obtaining opinions from a structured group of individuals, which is the reason for the robustness of this methodology compared to when using unstructured groups [41]. Additionally, the Delphi method is considered a powerful tool for analysing market trends and forecasting sector evolution [42]. ...
Article
Full-text available
There is an increasing evidence for the significant value and potential of Medicinal and Aromatic Plants (MAPs) worldwide. Among other non-wood forest products, MAPs are considered a key element of sustainable forest management and economic development. As part of Mediterranean cultural heritage, these plants are a major driver of rural tourism, in many areas representing an important raw material for various bio-based industrial sectors. Besides their economic value, MAPs enhance social integration and maintain gender balance as harvesting and processing MAPs is clearly a female dominated task. Despite the prominent contribution of MAPs to local development, conservation of biodiversity and the development of the traditional Mediterranean food system, many challenges and knowledge gaps could potentially place the sector’s development at risk. The aim of this work is to examine the present situation of the MAPs sector in the most productive countries around the Mediterranean and to identify future challenges and priority actions to develop the MAPs sector in the region. To do so, a supply chain analysis was performed to identify the main stakeholders involved. Various experts from Croatia, France, Greece, Spain, and Tunisia participated in the SWOT analysis and the Delphi approach employed in this study. The results highlight the main challenges facing the sector in Mediterranean countries. Five groups of challenges are identified, related mainly to certification and labelling, life quality and wellbeing, market development, research development, and transforming and processing. To overcome these challenges, a set of actions is validated by the interviewed experts with the aim of improving marketing strategies and including various innovations related to political, legal, organisational, and institutional frameworks.
... 8,9 It has been widely used for research and has certain advantages over other structured forecasting approaches. 10,11 The primary users of the clinical case definition for the post-COVID-19 condition will include patients, relatives and caregivers, clinicians, researchers, advocacy groups, policy makers, health and disability insurance providers, and media. We therefore aimed to have a diverse representation of participants, including clinicians with expertise in a variety of disciplines such as quality improvement and research, patients who have had COVID-19 and its mid-term and longer-term effects, researchers, policy makers, and others from countries representing all WHO regions and World Bank income levels. ...
Article
People with COVID-19 might have sustained postinfection sequelae. Known by a variety of names, including long COVID or long-haul COVID, and listed in the ICD-10 classification as post-COVID-19 condition since September, 2020, this occurrence is variable in its expression and its impact. The absence of a globally standardised and agreed-upon definition hampers progress in characterisation of its epidemiology and the development of candidate treatments. In a WHO-led Delphi process, we engaged with an international panel of 265 patients, clinicians, researchers, and WHO staff to develop a consensus definition for this condition. 14 domains and 45 items were evaluated in two rounds of the Delphi process to create a final consensus definition for adults: post-COVID-19 condition occurs in individuals with a history of probable or confirmed SARS-CoV-2 infection, usually 3 months from the onset, with symptoms that last for at least 2 months and cannot be explained by an alternative diagnosis. Common symptoms include, but are not limited to, fatigue, shortness of breath, and cognitive dysfunction, and generally have an impact on everyday functioning. Symptoms might be new onset following initial recovery from an acute COVID-19 episode or persist from the initial illness. Symptoms might also fluctuate or relapse over time. A separate definition might be applicable for children. Although the consensus definition is likely to change as knowledge increases, this common framework provides a foundation for ongoing and future studies of epidemiology, risk factors, clinical characteristics, and therapy.
Book
Full-text available
In der Dissertation wird der Frage nachgegangen, welche globalen bildungspolitischen Maßnahmen erforderlich sind, um auch bislang exkludierten Menschen den Kompetenzerwerb zu ermöglichen, der benötigt wird, eine positive User Experience in benutzergenerierten, digitalen Lernumgebungen auszubilden, damit sie an der modernen Weltgesellschaft selbstbestimmt teilhaben können. Zu diesem Zweck wurden Castells ‘Netzwerkgesellschaft’ und Csikszentmihalys ‘Theorie der optimalen Erfahrung’ als analytische Grundlagen zur Einordnung der sozialen Netzwerk-Aktivitäten herangezogen. Dies ermöglichte es, unter Rückgriff auf aktuelle Lerntheorien, Kompetenzdebatten, ökonomische Analysen des Bildungssystems und User Experience-Forschungen, einige individuelle und gesamtgesellschaftliche Voraussetzungen abzuleiten, um in der Netzwerkgesellschaft konstruktiv überleben zu können. Mit Blick auf unterschiedliche sozio-kulturelle Bedingungen für persönlichen Flow im ‘space of flows’ liessen sich schließlich differenzierte Flow-Kriterien entwickeln, die als Grundlage für die Operationalisierung im Rahmen einer Real-Time Delphi (RTD)-Studie mit einem internationalen Expertinnen-Panel dienen konnten. Ziel war es, bildungspolitische Ansatzpunkte zu finden, den bislang Exkludierten bis zum Jahre 2020 erste Rahmenbedingungen zu bieten, damit sie potentiell teilhaben können an der Gestaltung der zukünftigen Netzwerkgesellschaft. Das Ergebnis der Expertinnen-Befragung wurde unter Rückgriff auf aktuelle Global und Educational Governance-Studien und das Einflusspotenzial der Zivilgesellschaft auf den Digital Divide reflektiert. Vor diesem Hintergrund konnten abschließend vier bildungspolitische Verlaufsszenarien entworfen werden, die es ermöglichen könnten, bis 2020 die Kluft zu den global Exkludierten wenigstens etwas zu schließen.
Article
In the last few years, breakthroughs in computational and experimental techniques have produced several key discoveries in the science of networks and human collective intelligence. This review presents the latest scientific findings from two key fields of research: collective problem-solving and the wisdom of the crowd. I demonstrate the core theoretical tensions separating these research traditions and show how recent findings offer a new synthesis for understanding how network dynamics alter collective intelligence, both positively and negatively. I conclude by highlighting current theoretical problems at the forefront of research on networked collective intelligence, as well as vital public policy challenges that require new research efforts.
Negative Results
Full-text available
Stress Management is primarily a best to keep the stress of employees under control. One should know how to control stress at work place and in the family. Stress arises due to a number of factors such as work pressure, long duration of work, gender discrimination, sexual harassment, family issues, lack of monetary benefits at the work site etc. In this studypollster has examined two important elements such as managing stress and disturbances to employees due to stress. Work performance also is badly hit due to failure to address the stress related problems. This study has brought both unique data and data of the public domain. Primary Sources of Information has been obtained from employees at IT companies. The well-structured questionnaire regarding the field of the study was circulated among 150 employees thereby data were collected. The data of community domain were obtained in the form of tome, research writing, manuscript being published periodically and etc. Explorative Research Design has been adopted in the study. Purposive Sampling Technique was used and total sample size is 150 IT Employees from selected IT Companies in Chennai City such as Cognizant, TCS and Infosys.
Article
Full-text available
When financial columnist James Surowiecki wrote The Wisdom of Crowds, he wished to explain the successes and failures of markets (an example of a "crowd") and to understand why the average opinion of a crowd is frequently more accurate than the opinions of most of its individual members. In this expanded review of the book, Scott Armstrong asks a question of immediate relevance to forecasters: Are the traditional face-to-face meetings an effective way to elicit forecasts from forecast crowds (i.e. teams)? Armstrong doesn't believe so. Quite the contrary, he explains why he considers face-to-face meetings a detriment to good forecasting practice, and he proposes several alternatives that have been tried successfully.
Article
Full-text available
In important conflicts such as wars and labor-management disputes, people typically rely on the judgment of experts to predict the decisions that will be made. We compared the accuracy of 106 forecasts by experts and 169 forecasts by novices about eight real conflicts. The forecasts of experts who used their unaided judgment were little better than those of novices. Moreover, neither group’s forecasts were much more accurate than simply guessing. The forecasts of experienced experts were no more accurate than the forecasts of those with less experience. The experts were nevertheless confident in the accuracy of their forecasts. Speculating that consideration of the relative frequency of decisions across similar conflicts might improve accuracy, we obtained 89 sets of frequencies from novices instructed to assume there were 100 similar situations. Forecasts based on the frequencies were no more accurate than 96 forecasts from novices asked to pick the single most likely decision. We conclude that expert judgment should not be used for predicting decisions that people will make in conflicts. When decision makers ask experts for their opinions, they are likely to overlook other, more useful, approaches.
Article
Full-text available
In important conflicts, people typically rely on experts' judgments to predict the decisions that adversaries will make. We compared the accuracy of 106 expert and 169 novice forecasts for eight real conflicts. The forecasts of experts using unaided judgment were little better than those of novices, and neither were much better than simply guessing. The forecasts of experts with more experience were no more accurate than those with less. Speculating that consideration of the relative frequency of decisions might improve accuracy, we obtained 89 forecasts from novices instructed to assume there were 100 similar situations and to ascribe frequencies to decisions. Their forecasts were no more accurate than 96 forecasts from novices asked to pick the most likely decision. We conclude that expert judgment should not be used for predicting decisions that people will make in conflicts. Their use might lead decision makers to overlook other, more useful, approaches.
Article
The authors have recently developed a new approach to performing a Delphi study that does not involve the use of sequential “rounds” and as a result, greatly improves the efficiency of the process and shortens the time to perform such studies. This paper describes this process, RT Delphi, and illustrates its use in a decision-making application drawn from the Millennium Project of the American Council for the United Nations University. The illustrative application involves setting priorities among strategies for dealing with anticipated terrorist activities that might be initiated by a single deranged individual.
Article
Decision analysts are frequently called on to help inform decision makers in cases involving considerable uncertainty. In such situations, expert elicitation of parameter values is frequently used to supplement more conventional research. Expert elicitations typically rely on small panels of experts. However, in cases where the information needed for risk management must draw on a broad range of disciplines or types of professional backgrounds and experience, a larger, more heterogeneous expert panel is needed. In this paper we develop a formal protocol and a suite of uncertainty measures for this work. The protocol uses formal survey methods to take advantage of variation in individual expert uncertainty and heterogeneity among experts as a means of quantifying and comparing sources of uncertainty about parameters of interest. We illustrate the use of this protocol with an expert elicitation on the distribution of foodborne illness in the United States across foods. In the survey, experts are asked to attribute illnesses associated with one of eleven major foodborne pathogens to the consumption of one of eleven categories of food. Results show how the distributions of multiple measures of uncertainty (e.g., agreement of experts and uncertainty in knowledge), made feasible by use of a large panel of experts, can help identify which of several types of risk management actions may be most appropriate.
Article
FutureMAP, a project of the Defense Advanced Research Projects Agency, was to involve experiments to determine whether information markets could improve Defense Department decisionmaking. Information markets are securities markets used to derive information from the prices of securities whose liquidation values are contingent on future events. The government intended to use such a market to assess the probabilities of terrorist events, including potential political assassinations. The indelicacy of this potential application contributed to a controversy leading to the cancellation of the program. In this article, Professor Abramowicz assesses whether information markets, in theory, could be useful to administrative agencies and concludes that information markets could help discipline administrative agency predictions if a number of technical hurdles such as the danger of manipulation can be overcome. Because the predictions of well-functioning information markets are objective, they function as a tool that exhibits many of the same virtues in predictive tasks that cost-benefit analysis offers for normative policy evaluation. Both approaches can help to overcome cognitive errors, thwart interest group manipulation, and discipline administrative agency decisionmaking. The article suggests that the two forms of analysis might be combined to produce a "predictive cost-benefit analysis." In such an analysis, an information market would predict the outcome of a retrospective cost-benefit analysis, to be conducted some years after the decision whether to enact a particular policy. As long as the identity of the eventual decisionmaker cannot be anticipated, predictive cost-benefit analysis estimates how an average decisionmaker would be expected to evaluate the policy. Because the predictive cost-benefit analysis assessment does not depend on the identity of current agency officials, they cannot shade the numbers to justify policies that the officials prefer for i
Article
Prediction Markets, sometimes referred to as "information markets", "idea futures" or "event futures", are markets where participants trade contracts whose payoffs are tied to a future event, thereby yielding prices that can be interpreted as market-aggregated forecasts. This article summarizes the recent literature on prediction markets, highlighting both theoretical contributions that emphasize the possibility that these markets efficiently aggregate disperse information, and the lessons from empirical applications which show that market-generated forecasts typically outperform most moderately sophisticated benchmarks. Along the way, we highlight areas ripe for future research.
Article
When we rely on the judgment of experts to help produce our forecasts, the key issues are how to get the appropriate information from our consultants and how to get a forecast if we are using multiple experts. Gene Rowe describes the Delphi method, tells what it offers to the forecaster, and explains what the pitfalls are in its implementation. Copyright International Institute of Forecasters, 2007
Article
This paper analyzes the large and often well-organized markets for betting on U.S. presidential elections that operated between 1868 and 1940. Four main points are addressed. First, we show that the market did a remarkable job forecasting elections in an era before scientific polling. Second, the market was fairly efficient, despite the limited information of participants and active attempts to manipulate the odds. Third, we argue political betting markets disappeared largely because of the rise of scientific polls and the increasing availability of other forms of gambling. Finally, we discuss lessons this experience provides for the present.