I address commentators’ concerns about the research reported in my paper. These concerns do not threaten the conclusion that role-playing should be preferred ahead of game theory and unaided judgement for forecasting decisions in conflicts. I provide additional evidence and argument that the relative forecasting accuracy of game theory is a legitimate subject for research. I discuss non-forecasting uses for game theory and suggest that, without forecasting validity, such applications may be ill-founded. Replication of the Green research (Green, K. C. (2002) International Journal of Forecasting 18, 321–344) by game-theory advocates would be valuable. Extending the research with forecasts for more conflicts would allow greater confidence in recommendations to managers. Extensions should aim to increase the variety of conflicts so that managers can match research findings with their own forecasting problems. More data may allow researchers to identify conditions that favour particular forecasting methods and to estimate the effects of variations in conflict descriptions and decision options.
International Journal of Forecasting 18 (2002) 389–395 www.elsevier.com/locate /ijforecast
E mbroiled in a conflict: who do you call?
Kesten C. Green
School of Business and Public Management
Victoria University of Wellington and Managing Director
Decision Research Ltd
New Zealand
I address commentators’ concerns about the research reported in my paper. These concerns do not threaten the conclusion
that role-playing should be preferred ahead of game theory and unaided judgement for forecasting decisions in conflicts. I
provide additional evidence and argument that the relative forecasting accuracy of game theory is a legitimate subject for
research. I discuss non-forecasting uses for game theory and suggest that, without forecasting validity, such applications may
be ill-founded. Replication of the Green research (Green, K. C. (2002) International Journal of Forecasting
, 321–344)
by game-theory advocates would be valuable. Extending the research with forecasts for more conflicts would allow greater
confidence in recommendations to managers. Extensions should aim to increase the variety of conflicts so that managers can
match research findings with their own forecasting problems. More data may allow researchers to identify conditions that
favour particular forecasting methods and to estimate the effects of variations in conflict descriptions and decision options.
2002 International Institute of Forecasters. Published by Elsevier Science B.V. All rights reserved.
Conflict; Evaluation; Expert opinion; Forecasting; Game theory; Judgement; Prediction; Role-playing; Simulated interaction;
to determine the usefulness of methods for
I respond to the commentators by answering forecasting decisions in conflicts among small
four broad questions. First, was the research numbers of decision makers. My hope was that
objective achieved? Second, was the objective the findings would make it possible to offer
biased against game theory? Third, what should useful advice to managers on choosing a fore-
be the objectives for future research? Fourth, casting method for problems of this type.
given what we know now, which forecasting
approach should a manager who is embroiled in
a conflict adopt?
1 .1.
Research design
1 . Was the research objective achieved? Armstrong (2002) evaluated my research and
The purpose of conducting the research was concluded that it did well when measured
against traditional scientific criteria. Arm-
*Tel.: 164-4-499-2040; fax: 164-4-499-2080.
E-mail address
kesten.green@vuw.ac.nz (K.C. Green). strong’s evaluation addressed most of the con-
0169-2070/02 /$ – see front matter 2002 International Institute of Forecasters. Published by Elsevier Science B.V. All rights reserved.
PII: S0169-2070(02)00020-1
390 K
Green /International Journal of Forecasting
18 (2002) 389
cerns of the other commentators over the design for the three that were described with knowl-
and implementation of the research. edge of the outcomes, were substantially more
After completing the study I reviewed it accurate than game-theorist forecasts and un-
using the evaluation principles presented in aided-judgement forecasts. In the former case
Armstrong (2001b). The study fell short of the there was a 54 percent error reduction versus
ideal for three of the 32 principles. The first of game-theorist forecasts and in the latter a 45
these failings was not to have used objective percent error reduction. Moreover, the conflict
tests of the assumptions inherent in the conflict descriptions written by Armstrong and col-
descriptions. Truly objective tests for descrip- leagues (Artists Protest, Distribution Channel,
tions of human interaction are not possible. and 55% Pay Plan) were not written in the
Nevertheless, it is good practice to ask people expectation that they would be used to test the
who are involved in a conflict to review and accuracy of game theorists’ forecasts. There
comment on the verity of descriptions and the were substantial error reductions between game-
completeness of decision options, to ask experts theorist and role-play forecasts for these con-
to do the same, and to test the material with flicts (44 percent) and for those I had written
research participants to ensure they interpret the (56 percent).
material in the way that the author intends. The second of the failings was to have
Although procedures fell short of the ideal, conducted only limited testing of the assump-
there is no reason to suppose that any weaknes- tions inherent in the conflict for construct
ses in the descriptions or decision options validity. Ideally, one could ask a number of
favoured any one forecasting method over any independent domain experts to produce descrip-
other. A summary of the review is available on tions of a conflict and generate decision options
the Internet at kestencgreen.com. in brainstorming sessions. The forecaster or
Bolton (2002), Erev, Roth, Slonim, and Bar- researcher could then observe the effect of the
ron (2002), and Goodwin (2002) took issue descriptions on the forecasts obtained. In prac-
with the fact that for four of the six conflict tice, I did not have the resources to do this and
situations the outcome was known to the author the evidence of Armstrong’s (2001a) research
of the description. Bolton (2002) rightly pointed and my own is that accurate forecasts can be
out that an author must decide what information obtained using one description written by a
to include in the description and implies that an single independent observer.
author who knows the outcome may write a The third of the failings was not to have
materially different description to one that established whether potential clients understand
would be written without this knowledge. Al- the methods. Ideally, one would survey a di-
though using conflict descriptions written before verse sample of managers about this. Arm-
the outcome was known is desirable, I did not strong, Brodie, and McIntyre’s (1987) survey
have the resources to obtain role-play and provided some indirect evidence on the accep-
unaided-judgement forecasts for more new
situations. Having said this, is there any evi-
The Panalba situation is excluded from this analysis. As it
dence that remedying this lapse might have was described, the situation did not involve interaction
affected my conclusions? The answer is ‘no’. between the parties in conflict and hence the Panalba
Both role-play forecasts for the two conflicts role-plays were not ‘‘simulated interactions’’. See Arm-
that were described by authors without knowl- strong (2002) for a discussion on the term and on the
edge of the outcomes, and role-play forecasts exclusion of Panalba from analysis.
Green /International Journal of Forecasting
18 (2002) 389
tability of various conflict forecasting methods. method for forecasting real conflicts. Goodwin
Research using the Nurses Dispute was sup- (2002) found evidence that some game theory
ported by a New Zealand government depart- researchers regard game theory a legitimate
ment. Officials found the role-play forecasts forecasting method. Erev et al. (2002) suggested
sufficiently useful to commission a second that there are arenas in which game-theory
study. models produce accurate forecasts. Shefrin
(2002) implicitly agreed that game theory can
be a legitimate forecasting method when he
1 .2.
Implementation suggested that game theory’s forecasting ac-
curacy would be improved by taking better
Bolton (2002) cautioned against the possible account of human behaviour. Armstrong
existence of ‘lab effects’, but did not draw (2002)—using the analogy of medical re-
specific conclusions about their existence or search—made the point that predictive validity
otherwise in my research. Armstrong (2002) is an important test of the value of a theory. To
also addressed this question and found that Bolton (2002) and to Wright (2002), on the
identifiable biases favour forecasts by game other hand, testing the relative accuracy of game
theorists. Other, unidentified, biases might be theorists’ forecasts is a little like entering an
present, but it is clear from the findings pre- animal physiologist in the Grand National
sented in Table 1 in my paper and in the Steeplechase.
analysis presented in Armstrong’s (2002) Table Since writing my paper I have continued to
1, that any unknown bias would have to be very search for evidence on the use and accuracy of
large for its removal to eliminate the advantage game theory for forecasting. Goodwin (2002)
in accuracy that the role-play method enjoys. found evidence for both on the Decision In-
‘Was game theory used?’ is a thorny question sights Incorporated website. I asked Decision
that elicited a variety of responses from the
Insights founder Bruce Bueno de Mesquita for
commentators. As I state in my paper, for my evidence on the relative accuracy of his ex-
purpose game theory was ‘what game theorists pected utility model for forecasting conflicts. He
do when faced with practical forecasting prob- referred me to Stanley Feder of Policy Futures,
lems’. My appeal to game-theorists stated that Frans Stokman of the University of Groningen,
the purpose of the research was to compare and to Stokman and Bueno de Mesquita (1994).
game theory with other forecasting methods for I sent e-mail messages to both Feder and
conflicts. Why would these experts not have Stokman. Stokman did not reply. Feder referred
used the skills of their discipline? me to Feder (1987) and to the work of Fraser
and Hipel (1984) on the use of game theory to
forecast conflict outcomes . Feder (1987) com-
2 . Was the research objective biased against pared the accuracy of forecasts using the Bueno
game theory? de Mesquita model with the accuracy of the
2 .1.
Game theory is not intended for
forecasting real conflicts
or is suitable only
Personal communication with Bruce Bueno de Mesquita
for some (senior fellow at the Hoover Institution as well as founder
of Decision Insights) 4 December, 2001.
The commentators were divided on whether Personal communication with Stanley Feder, 5 December,
game theory should be regarded as a legitimate 2001.
392 K
Green /International Journal of Forecasting
18 (2002) 389
unaided judgement forecasts of experts who had
2 .2.
Game theory is useful for describing
provided the input data for the model. He found situations
that both methods were 90 percent accurate. Some of the commentators discuss the value
Stokman and Bueno de Mesquita (1994) of game theory for applications other than
compared the accuracy of forecasts of sixteen forecasting, implying that game theory can be
European Community decisions from the Bueno useful irrespective of its forecasting accuracy. It
de Mesquita expected utility model with the was not the objective of my research to investi-
accuracy of forecasts from alternative, ‘com- gate the non-forecasting uses of game theory.
promise exchange’, models. All models used the Nevertheless, the use of game theory to describe
same variables, but differed in assuming conflict situations (Bolton, 2002) and to design markets
(expected utility model), or exchange (alter- (Erev et al., 2002; Shefrin, 2002) warrants
native models). The Bueno de Mesquita model’s responses.
forecasts were more accurate on average than Bolton (2002) suggested that role-play fore-
those from the other models on the basis of the casters need the guidance of game theory in
measures that were used by the authors. The writing descriptions. Game theory was not used
differences in accuracy between the Bueno de in designing the material for any of the six
Mesquita model forecasts and the forecasts from conflicts used in my research and yet an abso-
the best of the alternative models were not lute majority of role-play decisions matched the
great, however. More importantly for the objec- actual decision for five of the six conflicts. It is
tive of my research, the authors do not compare nevertheless possible that the use of game
the accuracy of the model forecasts with the theory for this purpose may lead to an increase
accuracy of forecasts from experts using un- in the accuracy of forecasts.
aided judgement—the approach that is typically
adopted for such problems.
I inspected Fraser’s publication list and
2 .3.
Game theory is useful for designing
found one title which included the word ‘fore- markets
cast’: Fraser (1986). Although both this work
and Fraser and Hipel (1984) promote the game- Erev et al. (2002) and Shefrin (2002) dis-
theoretic technique ‘conflict analysis’ as a fore- cussed the value of game theory for designing
casting method for any conflict situation, neither markets. In a mechanical world, this would not
offered evidence on comparative forecasting involve forecasting: the designer would set the
accuracy. Fraser did not respond to my email rules confident that the range of possible be-
message of 5 December. haviour and outcomes were circumscribed.
While opinions on the use of game theory as The spectrum auctions in the United States
a forecasting method are diverse, some re- have been seen as a showcase for game theory.
searchers do believe this to be a legitimate It was used by market designers and particip-
subject for research. Moreover, practitioners use ants, and billions of dollars were at stake.
game theory for forecasting. The relative per- Shefrin’s (2002) description of the auctions and
formance of game theory for forecasting con- their aftermath suggests that the outcomes were
flicts is clearly an important issue. mixed, at best. There was more money raised
than had been anticipated. On the face of it, this
was a bonus for tax-payers. But tax-payers are
also investors, employees, and users of the
At http://www.openoptions.com/publications.htm on 5
December 2001. carriers’ services. Outcomes mentioned by Shef-
Green /International Journal of Forecasting
18 (2002) 389
rin (2002) include massive losses and plummet- proach will not produce accurate forecasts of
ing stock prices among carriers, defaults by and these conflicts.
bankruptcy of successful bidders, layoffs, and An alternative approach to that of Erev et al.
ongoing litigation and negotiation. Perhaps the (2002) would be to compare the accuracy of
spectrum could have been allocated in better forecasting methods using conflicts belonging to
ways. Why were the undesirable outcomes not categories that are understood by, and relevant
anticipated by the game-theorist consultants? to, managers. For example, conflicts that are
diverse in arena (ie, industrial, commercial,
civil, political, diplomatic, military) and parties
(ie, individuals, organisations, governments).
3 . What should be the objectives for future Categories such as these would allow managers
research? and practitioners to match their forecasting
problems with research findings.
3 .1.
As Armstrong (2002) suggested, replication
3 .3.
Conict description
by other researchers, particularly by game-
theory experts, would be valuable. Will conflict descriptions written by indepen-
dent authors tend to lead to the same forecast?
3 .2.
Classication of conicts
and extension A formal framework, such as that proposed by
Bolton (2002), may help to improve the validity
Armstrong (2002) suggested studying the of descriptions.
conditions under which each forecasting method Does the framing of decision options affect
is most accurate. Both Erev et al. (2002) and forecasts? Goodwin (2002) raised concerns
Goodwin (2002) discussed the possible exist- about the effect of decision options on strategy
ence of a subset of conflicts that are particularly formulation. Possible decisions can be identified
amenable to game-theoretic forecasting. by, for example, asking protagonists and ex-
Erev et al. (2002) proposed using a classifica- perts, and by conducting brainstorming sessions.
tion scheme from which to draw representative In practice, role-players, or those using other
samples of conflicts with which to compare the methods to forecast, may propose alternative
forecasting accuracy of methods. This is an decisions. Experts or protagonists could judge
interesting idea, and it is good practice to use a the viability of any such forecast decisions.
variety of conflicts to test the methods (Princi- These are all matters for further research.
ple 13.14, Armstrong, 2001c), but what does Does the number of decision options affect
‘representative’ mean in the context of real forecast accuracy? There currently is no clear
conflicts? Classification criteria are, in this evidence on this question.
context, arbitrary. The Erev et al. (2002) criteria
are also abstract, and the authors do not attempt
to match the six conflicts used in my research
3 .4.
Conict experts
with six from their own classification scheme.
Shefrin (2002) did attempt matching, drawing The unaided-judgement forecasts described in
parallels between the Dollar Auction game from Green (2002) were largely obtained from naive
the game-theory literature and the two conflicts judges (mostly students) whereas game theory
55% Pay Plan and Nurses Dispute. He con- forecasts were obtained from experts. Arm-
cluded that the ‘traditional’ game-theoretic ap- strong (2002) suggested asking experts on
394 K
Green /International Journal of Forecasting
18 (2002) 389
conflicts or on forecasting, who are not game- on conflict forecasting accuracy to be estimated
theory experts, to provide forecasts. for these methods.
Wright (2002) cited evidence that expert
judgement can only be expected to be of more
3 .5.
Understanding role-play
value than the judgement of novices when the
subject of judgement has both ecological va-While other commentators were chiefly
lidity for the expert and learning is possible. He concerned with establishing how or whether
suggested that neither requirement holds for the game theory can better be applied to forecasting
game-theory experts and postulates this as an conflicts, Armstrong (2002) and Wright (2002)
explanation for their performance. The first both argued for expending more research effort
requirement (ecological validity) is equivalent on role-playing.
to asking ‘is this person an expert in this eld’. Although, in contrast to Wright (2002), Shef-
Game theorists are experts in the study of rin (2002) suggested that the quirks of human
conflicts. It seems reasonable to assume that the behaviour can be incorporated into game theory
conflicts used in the research had ecological (game theorists can learn), he speculated that
validity for them. In fact, the game-theorist role-play forecasts will tend to be more accurate
participants were unique in that they volun- because the method uses actual interactions
teered their services, as experts, for forecasting between real people subject to emotion and
these conflicts. The expertise of someone who errors of judgement.
was familiar with New Zealand industrial rela-
tions might have been more relevant for fore-
casting the Nurses Dispute, but this is not the 4 . So, who do you call?
same as saying that the expertise of a game
theorist who is not familiar with New Zealand Research needs to be done on forecasting
industrial relations is irrelevant to this problem. decisions in conflict situations. Nevertheless,
Wright (2002) suggested that his second compelling evidence exists that role-playing
requirement (the possibility of learning) was not will provide more accurate forecasts than other
met because the novelty of the conflicts pre- methods, and the cost is modest. Role-playing is
cludes learning being brought to bear on the a successful and efficient way for forecasters to
forecasting problems. If he is right in this, the take real human behaviour into account.
role of game theorists, and of other conflict
experts, is limited.
Each role-play forecast requires several par- A cknowledgements
ticipants, whereas forecasts from the other
methods were mostly obtained from individuals I am grateful for the contributions of the
working in isolation. Collaboration requires commentators: J. Scott Armstrong, Greg Barron,
forecasters to justify their forecasts to their Gary Bolton, Paul Goodwin, Ido Erev, Alvin
fellows and allows forecasts to be combined. Roth, Hersh Shefrin, Robert Slonim, and
Both justification and combining tend to in- George Wright. I am also grateful to Scott
crease the accuracy of judgmental forecasts Armstrong and Paul Goodwin for helpful sug-
(Stewart, 2001). Obtaining collaborative and gestions on drafts of this reply, and to Scott
individual unaided-judgement and game-theory Armstrong and George Wright for organising
forecasts would allow the effect of collaboration the special section on this topic.
Green /International Journal of Forecasting
18 (2002) 389
Fraser, N. M. (1986). Political and social forecasting using
R eferences conflict analysis. European Journal of Political
(2), 203–222.
Armstrong, J. S. (2001a). Role playing: a method to Fraser, N. M., & Hipel, K. W. (1984). Conict analysis
forecast decisions. In Armstrong, J. S. (Ed.), Principles models and resolutions. New York: North-Holland.
of forecasting
a handbook for researchers and prac-Goodwin, P. (2002). Forecasting games: can game theory
titioners. Norwell, MA: Kluwer Academic Publishers, win? International Journal of Forecasting,
, 369–
pp. 15–30. 374.
Armstrong, J. S. (2001b). Evaluating forecasting methods. Green, K. C. (2002). Forecasting decisions in conflict
In Armstrong, J. S. (Ed.), Principles of forecasting
asituations: a comparison of game theory, role-playing,
handbook for researchers and practitioners. Norwell, and unaided judgement. International Journal of Fore-
MA: Kluwer Academic Publishers, pp. 443–472. casting,
, 321–344.
Armstrong, J. S. (2001c). Standards and practices for Shefrin, H. (2002). Behavioral decision making, forecast-
forecasting. In Armstrong, J. S. (Ed.), Principles of ing, game theory, and role play. International Journal
a handbook for researchers and practition-of Forecasting,
, 375–382.
ers. Norwell, MA: Kluwer Academic Publishers, pp. Stewart, T. R. (2001). Improving reliability of judgmental
679–732. forecasts. In Armstrong, J. S. (Ed.), Principles of
Armstrong, J. S. (2002). Assessing game theory, role forecasting
a handbook for researchers and practition-
playing, and unaided judgement. International Journal ers. Norwell, MA: Kluwer Academic Publishers, pp.
of Forecasting,
, 345–352. 81–106.
Armstrong, J. S., Brodie, R. J., & McIntyre, S. H. (1987). Stokman, F. N., & Bueno de Mesquita, B. (1994). Models
Forecasting methods for marketing: review of empirical of exchange and of expected utility maximisation: a
research. International Journal of Forecasting,
, 335– comparison of accuracy. In Bueno de Mesquita, B., &
376. Stokman, F. N. (Eds.), European community decision
Bolton, G. E. (2002). Game theory’s role in role-playing. making
and comparisons. New
International Journal of Forecasting,
, 353–358. Haven, CN: Yale University Press, pp. 214–228.
Erev, I., Roth, A. E., Slonim, R. L., & Barron, G. (2002). Wright, G. (2002). Game theory, game theorists, university
Predictive value and the usefulness of game theoretic students, role playing and forecasting ability. Interna-
models. International Journal of Forecasting,
, 359– tional Journal of Forecasting,
, 383–387.
Feder, S. A. (1987). Factions and Policon: new ways to
analyze politics. Studies in Intelligence,
(1), 41–57.
... 34. Green, K C (2002), "Embroiled in a conflict: who do you call?", International Journal of Forecasting Vol 18, No 3, pages 389-395. ...
Full-text available
Much of São Paulo’s urban expansion is driven by the development of informal settlements on its periphery, which includes the catchment areas that provide important environmental services such as open space and catchments for drinking water reservoirs. In such areas, governance of land, water services and water resources, traditionally administered separately, are in fact interdependent. A role-playing game was built to explore the interactions between different actors (mayor, water company, district representatives and landowners of different types) in land tenure insecurity, water and sanitation infrastructure and pollution in the periphery of São Paulo. An assessment of the game sessions revealed the different points of view of local and institutional actors about the main issues and the possible negotiation arrangements. Institutional actors misunderstood the hierarchy and diversity of community actors’ preoccupations, which were related more to land tenure, access to public transport and other aspects such as health and education rather than to water and sanitation infrastructure. Institutional actors also failed to recognize how access to infrastructure and land tenure is shaped by power-based relationships. While community actors appreciated the game and felt that they had learned from the experiment, institutional actors were more critical. These discrepancies call into question the potential effectiveness of the legal, technical and institutional solutions that the institutional actors are promoting to address the pollution issue.
... What exactly that means, however, is not as clear as I would like since most of the reported assessments are not explicit about how they measured accuracy. At least one critic points out, for instance, that the 1995 CIA evaluation by Feder reports that this model is accurate about 90 percent of the time but so too were the government analysts who provided the input data (Green 2002). Green contends that the right measure is whether the model outperforms the experts. ...
A new forecasting model, solved for Bayesian Perfect Equilibria, is introduced. It, along with several alternative models, is tested on data from the European Union. The new model, which allows for contingent forecasts and for generating confidence intervals around predictions, outperforms competing models in most tests despite the absence of variance on a critical variable in all but nine cases. The more proximate the political setting of the issues is to the new model's underlying theory of competitive and potentially coercive politics, the better the new model does relative to other models tested in the European Union context.
... Eight were about animal, rather than human, behaviour and three were concerned with artificial intelligence or similar. Two were of work by me that is described in this document Green, 2002b) and five were commentaries on my work. ...
Full-text available
There has been surprisingly little research on how best to predict decisions in conflicts. Managers commonly use their unaided judgement for the task. Game theory and a disciplined use of analogies have been recommended. When tested, experts using their unaided judgement and game theorists performed no better than chance. Experts using structured analogies performed better than chance, but the most accurate forecasts were provided by simulated interaction using student role players. Twenty-one game theorists made 98 forecasts for eight diverse conflicts. Forty-one experts in conflicts made 60 solo forecasts using structured analogies and 96 solo forecasts using unaided judgement (a further seven provided collaborative forecasts only) while 492 participants made 105 forecasts in simulated interactions. Overall, one-in-three forecasts by game theorists and by experts who did not use a formal method were correct. Forecasters who used structured analogies were correct for 45 percent and forecasts from simulated interactions were correct for 62 percent of forecasts. Analysis using alternative measures of accuracy does not affect the findings. Neither expertise nor collaboration appear to affect accuracy. The findings are at odds with the opinions of experts, who expected experts to be more accurate than students regardless of the method used.
Technical Report
Full-text available
"Negotiation is all around us, all the time and at all levels. It makes up an important part of our daily lives, whether business or private. Let's take a simple example: in a marriage partnership, each of the partners needs to bring a willingness to live with both the similarities and the differences of the couple - a situation of constant negotiation. But when we argue with our neighbours about the garden gnome with flashing lights or the cherry tree on the boundary line between the two properties, then there is no negotiation, each wants what he wants and is not willing to make concessions. If one of the parties is intent on having his demands satisfied in full, he puts the other in the position of loser. Often enough, a court has to decide on a matter that could have been easily sorted out over a glass of beer across the garden fence - in a word, with negotiation. I get the cherries, you get the gnome. And business affairs are unthinkable without negotiation: just like before the court, it is less a matter of what we are entitled to, and more of what we can negotiate." © 2008 by Koninklijke Brill NV, Leiden, The Netherlands. All rights reserved.
In this book the social scientist and economist Professor Dr. Raymond Saner draws on his long years of experience as a negotiation adviser, teacher, trainer, researcher and university lecturer to show that two thirds of negotiation practice is learnable. The author treats the different aspects of negotiation practice in a way that is useful to both academics and practitioners, such that the general laws and principles gradually become evident as and of themselves. © 2012 by Koninklijke Brill NV, Leiden, The Netherlands. All rights reserved.
Philip E. Tetlock and I agree that forecasting tools are best evaluated in peer-reviewed settings and in comparison not only to expert judgments, but also to alternative modeling strategies. Applying his suggested standards of assessment, however, certain forecasting models not only outperform expert judgments, but also have gone head-to-head with alternative models and outperformed them. This track record demonstrates the capability to make significant, reliable predictions of difficult, complex events. The record has unfolded, contrary to Tetlock's contention, not only in government and business applications, but also in numerous peer-reviewed publications containing hundreds of real-time forecasts. Moreover, reliable prediction is achieved while avoiding significant false-positive or false-negative rates.
Full-text available
This paper reviews the empirical research on forecasting in marketing. In addition, it presents results from some small scale surveys. We offer a framework for discussing forecasts in the area of marketing, and then review the literature in light of that framework. Particular emphasis is given to a pragmatic interpretation of the literature and findings. Suggestions are made on what research is needed.
Full-text available
All judgmental forecasts will be affected by the inherent unreliability, or inconsistency, of the judgment process. Psychologists have studied this problem extensively, but forecasters rarely address it. Researchers and theorists describe two types of unreliability that can reduce the accuracy of judgmental forecasts: (1) unreliability of information acquisition, and (2) unreliability of information processing. Studies indicate that judgments are less reliable when the task is more complex; when the environment is more uncertain; when the acquisition of information relies on perception, pattern recognition, or memory; and when people use intuition instead of analysis. Five principles can improve reliability in judgmental forecasting: 1. Organize and present information in a form that clearly emphasizes relevant information.
Full-text available
Ideally, forecasting methods should be evaluated in the situations for which they will be used. Underlying the evaluation procedure is the need to test methods against reasonable alternatives. Evaluation consists of four steps: testing assumptions, testing data and methods, replicating outputs, and assessing outputs. Most principles for testing forecasting methods are based on commonly accepted methodological procedures, such as to prespecify criteria or to obtain a large sample of forecast errors. However, forecasters often violate such principles, even in academic studies. Some principles might be surprising, such as do not use R-square, do not use Mean Square Error, and do not use the within-sample fit of the model to select the most accurate time-series model. A checklist of 32 principles is provided to help in systematically evaluating forecasting methods.
Full-text available
Green's study (Int. J. Forecasting (forthcoming)) on the accuracy of forecasting methods for conflicts does well against traditional scientific criteria. Moreover, it is useful, as it examines actual problems by comparing forecasting methods as they would be used in practice. Some biases exist in the design of the study and they favor game theory. As a result, the accuracy gain of game theory over unaided judgment may be illusory, and the advantage of role playing over game theory is likely to be greater than the 44% error reduction found by Green. The improved accuracy of role playing over game theory was consistent across situations. For those cases that simulated interactions among people with conflicting roles, game theory was no better than chance (28% correct), whereas role-playing was correct in 61% of the predictions.  2002 International Institute of Forecasters. Published by Elsevier Science B.V. All rights reserved.
Full-text available
Can game theory aid in forecasting the decision making of parties in a conflict? A review of the literature revealed diverse opinions but no empirical evidence on this question. When put to the test, game theorists’ predictions were more accurate than those from unaided judgement but not as accurate as role-play forecasts. Twenty-one game theorists made 99 forecasts of decisions for six conflict situations. The same situations were described to 290 research participants, who made 207 forecasts using unaided judgement, and to 933 participants, who made 158 forecasts in active role-playing. Averaged across the six situations, 37 percent of the game theorists’ forecasts, 28 percent of the unaided-judgement forecasts, and 64 percent of the role-play forecasts were correct.
Green (International Journal of Forecasting, 18, 321–344) considers one method of testing the predictive value of game theory for conflict situations, and finds that role-playing does better. I discuss a second method, one that combines game theory and role-playing. This method has already been used with success to solve practical business problems. I argue game theory will have to play a critical part if role-playing is to be reliable for forecasting conflict outcomes. Existing research that combines game theory and experimental economics holds important lessons for the design of role-playing exercises.
Green [Int. J. Forecasting (2002)] reports that in certain settings predictions made by game theorists can be outperformed by the outcome of a short role playing exercise. Goodwin [Int. J. Forecasting (2002)] argues that this does not imply that game theoretic analysis cannot be useful. The current paper discusses two types of observations that support this assertion. First, there are many important settings in which game theoretic models have high forecasting power. Two examples: the aggregate outcome of entry job markets, and the outcome of repeated interactions are summarized here. The second observation concerns the possibility of objectively forecasting the predictive value of specific models (and methods) on particular domains. To increase our understanding of the value of role playing, we suggest that future research focus on estimating the predictive value of this method using a random selection of problems from a well defined set.
This comment argues that game theory is limited in its ability to analyse and forecast the outcomes of one-off, real-world, conflict situations. It also argues that we have no reason to expect that game theorists would be any better at such forecasts than university students because the conflicts studied by Green did not, on the face of it, have ecological validity for his game theorist participants. Nevertheless, students and game theorists have their own, individual experiences of real-life conflicts and their resolutions. My thesis is that when individuals are enmeshed in role-play simulations of conflicts the relevance of this prior learning becomes salient and can be utilised for the accurate forecasting of the outcomes of the conflicts.
I present evidence to suggest that studying the use of game theory in prediction is a legitimate area of research and suggest ways in which game theory might be used to make or support predictions. Green’s study predominately assesses the accuracy of predictions by game theorists (who may have made informal use of game theory concepts) rather than predictions obtained from formal game theory models. I argue that the accuracy of predictions derived from such models is likely to be contingent on the characteristics of the conflict and provide a partial taxonomy of these characteristics, together with their hypothesised effects. I also argue that it would be worth investigating the potential use of game theory as an aid to obtaining probabilistic predictions.