Content uploaded by Kesten Green
Author content
All content in this area was uploaded by Kesten Green on Jul 28, 2015
Content may be subject to copyright.
Content uploaded by Kesten Green
Author content
All content in this area was uploaded by Kesten Green on May 29, 2014
Content may be subject to copyright.
FORECASTING DECISIONS IN CONFLICTS:
ANALOGY, GAME THEORY, UNAIDED JUDGEMENT, AND SIMULATION
COMPARED
by
Kesten Charles Green
A thesis
submitted to the Victoria University of Wellington
in fulfilment of the
requirements for the degree of
Doctor of Philosophy
in Management
Victoria University of Wellington
4 September, 2003
2
Abstract
There has been surprisingly little research on how best to predict decisions in conflicts.
Managers commonly use their unaided judgement for the task. Game theory and a
disciplined use of analogies have been recommended. When tested, experts using their
unaided judgement and game theorists performed no better than chance. Experts using
structured analogies performed better than chance, but the most accurate forecasts were
provided by simulated interaction using student role players. Twenty-one game theorists
made 98 forecasts for eight diverse conflicts. Forty-one experts in conflicts made 60 solo
forecasts using structured analogies and 96 solo forecasts using unaided judgement (a
further seven provided collaborative forecasts only) while 492 participants made 105
forecasts in simulated interactions. Overall, one-in-three forecasts by game theorists and
by experts who did not use a formal method were correct. Forecasters who used
structured analogies were correct for 45 percent and forecasts from simulated
interactions were correct for 62 percent of forecasts. Analysis using alternative measures
of accuracy does not affect the findings. Neither expertise nor collaboration appear to
affect accuracy. The findings are at odds with the opinions of experts, who expected
experts to be more accurate than students regardless of the method used.
Keywords: accuracy, analogy, conflict, expert opinion, forecasting, game theory,
unaided judgement, role playing, simulated interaction, simulation, structured analogies.
Acknowledgements: I am grateful for the help of four groups of unpaid research
participants. First, the Delphi panel of conflict management experts who rated criteria
for selecting conflict forecasting methods and rated methods on the basis of those
criteria. The panel were: Julie Douglas, Tom Fiutak, Michael Hudson, Jessica Jameson,
David Matz, W. Bruce Newman, and Simon Upton.
Second, I am grateful for the help of the five people who rated for usefulness the
decision options provided for the conflicts used in this research. They were: Allen Jun,
Diana Lin, Margot Rothwell, Dinah Vincent, Philip Wrigley.
Third, the 48 experts who provided forecasts using unaided judgement or structured
analogies. They were: Barry Anderson, Corrine Bendersky, Constant Beugre, Lisa
Bolton, JosJ Cancelo, Nihan Cini, David Cohen, Serghei Dascalu, Nikolay Dentchev,
3
Ulas Doga Eralp, Miguel Dorado, Erkan Erdil, Jason Flello, Paul Gaskin, Andrew
Gawith, David Grimmond, George Haines, Claudia Hale, Michael Kanner, John Keltner,
Daniel Kennedy, Oliver Koll, Rita Koryan, Talha Köse, Tony Lewis, David Matz, Bill
McLauchlan, Kevin Mole, Ben Mollov, W. Bruce Newman, Konstantinos Nikolopoulos,
Dean G. Pruitt, Perry Sadorsky, Greg Saltzman, Amardeep Sandhu, Deborah Shmueli,
M<rta Somogyvári, Harris Sondak, Dana Tait, Scott Takacs, Dimitrios Thomakos, Ailsa
Turrell, Bryan Wadsworth, James Wall, Daniel Williams, Christine Wright, Becky
Zaino, and one other, who asked to remain anonymous. I am also grateful to Nimet
Beriker for asking four of his conflict management graduate students to participate, and
to Geoff Allen and the Board of the International Institute of Forecasters for their
support for an initiative to recruit Institute members as Research Associates and for
access to the list of Associates.
Fourth, the 21 game-theory experts who provided forecasts. They were: Manel Baucells,
Emilio Calvo, Gary Charness, Bereket Kebede, Somdeb Lahiri, Massimiliano Landi,
Andy McLennan, Holger Meinhardt, Claudio Mezzetti, Hannu Nurmi, Andre Rossi de
Oliveira, Ronald Peeters, Alex Possajennikov, Eleuterio Prado, Maurice Salles, Giorgos
Stamatopoulos, Tristan Tomala, Yelena Yanovskaya, Shmuel Zamir, JosJ Zarzuelo,
Anthony Ziegelmeyer. Seven game-theory experts provided helpful comments on the
research. They were: Peter Bennett, Pierre Bernhard, Steven Brams, Vito Fragnelli,
Herbert Gintis, Harold Houba, Marc Kilgour.
I am grateful to Scott Armstrong, Julie Douglas, James Edmondson, Don Esslemont,
Paul Goodwin, Jackie Kaines Lang, and Zane Kearns for their help in testing materials
used in the research and for providing useful suggestions on the writing. Don Esslemont
also made useful suggestions on the writing of this document. I thank Joanne Silberstein
and Shane Kinley of the New Zealand Department of Labour who commissioned
research from me that provided access to information on one of the conflicts used in this
research and provided funding to pay for student role players needed for two of the
conflicts. I was fortunate in being able to talk to the principal participants in these
conflicts and am grateful for their patient responses to my many questions. Mike Hanson
and Russell Taylor are two of these people, while the participants in the other conflict
prefer to remain anonymous.
I thank Pat Walsh for his support. The groundwork for this thesis is, in part, based on
research funded by the Public Good Science Fund administered by the Foundation for
Research Science and Technology (FRST Contract: Vic 903). The contract is
administered by Raymond Harbridge and Pat Walsh. I also thank the many other
academic staff of Victoria University of Wellington, Massey University, and UCOL who
generously made class time available for my research or who provided opportunities for
recruiting participants. In particular, I thank my supervisors, Urs Daellenbach and John
Davies, for their help and advice, Bob Cavana for giving me a chance to resume my
studies, and Vicky Mabin for the opportunity to make a start on my research programme.
My examiners, Paul Goodwin, John Haywood, and Marcus O’Connor provided useful
suggestions for improvements to this document. I am grateful for their suggestions and
for their having agreed to take on the task of examining my work.
My research was inspired by the work of J. Scott Armstrong, and I am grateful to him
for his unstinting support and interest.
Finally, I thank my wife, Annsley, who encouraged me to start on this path and who
took delight in my triumphs, and my children, Hester and Charles, for being there.
4
Contents
Lists of tables, figures, and formulae 7
1. Introduction 10
1.1 Outline 10
1.2 Conflict forecasting methods 12
1.2.1 Unaided judgement 12
1.2.2 Game theory 13
1.2.3 Structured analogies 15
1.2.4 Simulated interaction 17
1.2.5 Survey of experts’ accuracy expectations 19
1.3 Objectives: motivation and implications 21
1.3.1 Overview 21
1.3.2 Estimate relative performance of methods 24
1.3.3 Assess generalisability of findings 25
1.3.4 Assess appeal to managers 27
1.3.5 Summary of objectives 28
2. Prior evidence on methods 29
2.1 Unaided judgement 29
2.2 Game theory 30
2.2.1 Others’ reviews 30
2.2.2 Social Science Citation Index search 31
2.2.3 Internet search 32
2.2.4 Appeal for evidence 32
2.2.5 Personal communications 33
2.2.6 Search findings 35
2.3 Structured analogies 37
2.3.1 Social Science Citation Index search 37
2.3.2 Internet search 38
2.3.3 Appeal for evidence 39
2.3.4 Personal communications 40
2.3.5 Search findings 41
5
2.4 Simulated interaction 42
2.4.1 A review 42
3. Research programme 44
3.1 Approach 44
3.2 Conflict forecasting methods described 45
3.2.1 Unaided judgement 45
3.2.2 Game theory 45
3.2.3 Structured analogies 46
3.2.4 Simulated interaction 47
3.3 Conflict selection and description 48
3.3.1 Conflicts selected 48
3.3.2 Conflict diversity 56
3.3.3 Material provided to participants 62
3.4 Data collection – forecasts 66
3.4.1 Data sources 66
3.4.2 Unaided judgement – novices 67
3.4.3 Unaided judgement and structured analogies – experts 70
3.4.4 Game theory – experts 77
3.4.5 Simulated interaction – novices 80
3.4.6 Summary and implications 86
3.5 Data collection – opinions 88
4. Findings 92
4.1 Relative performance of methods 92
4.1.1 Effect of method on accuracy 92
4.1.2 Effect of method on forecast usefulness 106
4.2 Generalisability 109
4.2.1 Effect of collaboration on accuracy 109
4.2.2 Effect of expertise on accuracy 111
4.3 Appeal to managers 129
4.3.1 Selection criteria weights 129
4.3.2 Method ratings 132
4.3.3 Likely use of methods 135
6
5. Discussion, conclusions, and implications 137
5.1 Discussion and conclusions 137
5.1.1 Relative accuracy 138
5.1.2 Generalisability 147
5.1.3 Appeal to managers 153
5.2 Implications for researchers 153
5.3 Implications for managers 159
Appendices
1Application of forecasting method evaluation principles
165
2Conflict descriptions and questionnaires provided to game theorist participants 168
3Zenith Investment questionnaires provided to participants:
Unaided judgement (novice, expert), structured analogies (expert), and
simulated interaction (novice)
193
4Information Sheet and Informed Consent form 198
5Text of email appeal for unaided-judgement participants (IACM solo version) 200
6
Text of email appeal for structured-analogies participants (IACM solo version)
201
7Text of email appeal for game-theorist participants
202
8Game theorist responses: A copy of Appendix 3 from Green (2002a)
203
9Delphi panel appeal and part 1:
Rating the importance of criteria for selecting forecasting methods 204
10 Delphi panel part 2:
Rating the forecasting methods against the selection criteria 211
11
Delphi panel part 3:
Likelihood that methods would be used or recommended by panellists
224
12
Number of forecasts, by conflict, method, and forecast decision
225
13
Comparison of Brier scores and PFAR scores
226
14 Assessment of a priori judgements of predictability: Approach and response 233
15 Questionnaire for obtaining forecast usefulness ratings 235
16
Delphi panellists’ ratings of conflict forecasting method criteria
239
17
Delphi panellists’ ratings of forecasting methods against criteria
243
References 248
7
Lists of tables, figures, and formulae
Tables
1Experts’ expectations of forecasting methods’ accuracy (#1)
20
2Forecasting method evaluation principles
22
3Research objectives: For reasonable methods, investigate the effect of…
28
4Content of articles found in searches for evidence on the relative accuracy of
game-theoretic forecasts of decisions in real conflicts 35
5Content of articles found in searches for evidence on the relative accuracy of
analogical forecasts of decisions in real conflicts 41
6Armstrong’s (2001a) evidence on the accuracy of simulated-interaction
decisions and unaided judgement forecasts by students
43
7Classification of conflicts: Nature of the parties
57
8Classification of conflicts: Arena of the conflict
59
9Classification of conflicts: Game theorist preference 61
10 Questionnaire content by treatment 64
11
Sources of forecast accuracy data
66
12
Organisation contact lists and email lists that were sent appeals
70
13
IACM responses by allocated treatment
72
14 Sources of expert (non-game theorist) participants 73
15 Unaided-judgement and structured-analogies forecasts by experts:
Number of forecasts 74
16
Unaided-judgement and structured-analogies forecasts by experts:
Median time taken to forecast
75
17
Probabilistic unaided-judgement and structured-analogies forecasts by experts
76
18
Forecasts by game theory experts: Median time taken to forecast
79
19 Forecasts from simulated interaction: Time taken to forecast, in minutes 85
20 Summary of data collection 86
21
Accuracy of solo-experts’ forecasts, and forecasts from simulated-interaction
by novices [Reproduced]
93
[139]
22
Probability forecast accuracy ratings of solo-experts’ forecasts by forecasting
method and derivation of probabilities
100
23
Accuracy of forecasts: Percent error reduction vs chance (PERVC)
102
24 Accuracy of forecasts: Percent error reduction vs unaided judgement (PERVUJ)105
8
25 Accuracy of forecasts: Average usefulness rating out of 10 108
26
Effect of collaboration on experts’ forecast accuracy
109
27
Characteristics of structured-analogies forecasts and forecasters by
collaboration
110
28
Accuracy of experts’ and novices’ unaided-judgement forecasts
111
29 Effect of experience on the accuracy of experts’ unaided-judgement forecasts 113
30 Forecaster characteristics associated with accurate and inaccurate unaided-
judgement forecasts by experts 114
31
Effect of experience as a game theorist on the accuracy of game-theorist
forecasts
116
32
Game-theory experience of game-theorist forecasters by accuracy of forecasts
116
33
Accuracy of structured-analogies forecasts by experience
118
34 Forecaster characteristics associated with accurate and inaccurate structured-
analogies forecasts 119
35 Solo-experts’ confidence in their forecasts 120
36
Accuracy of experts’ forecasts by forecaster confidence
121
37
Forecaster confidence associated with accurate and inaccurate forecasts
122
38
Accuracy of forecasts by source of analogy
124
39 Accuracy of forecasts by quality and by quantity of analogies 125
40 Forecast accuracy by source and quantity of analogies 126
41
Accuracy of experts’ forecasts by time taken
127
42
Importance ratings of criteria for selecting a forecasting method:
Yokum and Armstrong (1995) vs Delphi panel
131
43
Delphi panel’s ratings of conflict forecasting methods by forecasting method
selection criteria
133
44 Likelihood that Delphi panellists would use or recommend methods for their
next important conflict forecasting problem 135
45 Experts’ expectations of forecasting methods’ accuracy (#2) 140
46
Unexplained relationship between number of decision options and error rates
143
47
Effect of assignment of probabilities on average error measures for many
forecasts
227
48
Deriving probabilities from structured analogies data using a rule
229
49 Brier and PFAR scores for cases in which solo experts provided probabilistic
forecasts, by derivation of probabilities and forecasting method 230
50 Forecasting problem for each conflict 234
9
Figures
1Rules for choosing a single-decision forecast from a set of up to five analogies
that have been rated for similarity to a target conflict
96
2A priori predictability rating question 233
Formulae
1Aggregate rating for method m
90
2Brier score (BS)
97
3Probabilistic forecasting accuracy rating (PFAR)
98
4Percentage error reduction vs chance (PERVC)102
5Percentage error reduction vs unaided judgement (PERVUJ)104
10
1. Introduction
If you can look into the seeds of time,
And say which grain will grow and which will not,
Speak then to me, who neither beg nor fear
Your favours nor your hate.
Shakespeare (1606), Banquo to the witches.
Like Banquo, who consulted the “weird sisters”, modern managers often wish to know
how a conflict will unfold. Whether a conflict is industrial, commercial, civil, political,
diplomatic, or military, predicting the decisions of others can be difficult. Yet it is
important that managers plan for likely eventualities and seek effective strategies. Errors
in predicting the decisions of others can lead to needless strikes, losses, protests,
reversals, wars, and defeats. This research addresses the problem of choosing the best
method for forecasting decisions made in conflicts: particular conflicts that involve
interaction between few parties.
Conflicts are complex and hence decisions in conflict situations can be difficult to
predict. The complexity of conflicts is highlighted by, for example, the number and
variety of aspects Raiffa (1982) described in his attempt to characterise them.
Paraphrased, Raiffa’s conflict characteristics are: number of parties, cohesion of parties,
likelihood of iteration, possibility of linkage, number of issues, need for agreement, need
for ratification, possibility of threats, constraints on time, binding of agreement, arena of
negotiation, norms of parties, and possibility of intervention. Raiffa himself describes
the characteristics as a “partial classification” (p. 11).
1.1 Outline
This thesis replicates and extends the research described in Armstrong (2001a).
Armstrong presented evidence on the accuracy of forecasts of decisions in conflicts from
two methods: unaided judgement and role playing. The participants in the research were
primarily university students. Armstrong sought evidence on the relative accuracy of a
third method, forecasts by game theorists, but was unable to find any.
11
In my work, I have followed Armstrong’s (2001e) recommendations on evaluating
forecasting methods. First, I describe my search for evidence on the relative accuracy of
forecasts from reasonable alternative methods for forecasting decisions in conflicts.
Second, I describe my research and present my findings. Third, I assess the
generalisability of the findings. Finally, I draw on my findings to make
recommendations for managers that, if adopted, will lead to improvements in the
accuracy of forecasts of decisions in conflicts.
Document structure
There are five chapters in this document. In this, the first chapter, I describe the methods
that are used or have been recommended for forecasting decisions in conflicts. I then
describe the objectives of my research and their motivation, and address the implications
of the objectives for my research programme.
In chapter 2, I describe my search for empirical evidence on the accuracy of forecasts
from four conflict forecasting methods and present the findings of my search.
In chapter 3, I discuss the methodology of my empirical research, and describe my
research programme in detail. The chapter includes detailed descriptions of the four
forecasting methods that I compared, the conflicts that I used and how I chose them, and
how I collected forecasts and opinions from participants.
In chapter 4, I present my findings on the relative performance of the four methods. I
examine the effect on forecast accuracy of forecaster expertise and of collaboration
between forecasters. I also present my findings on the likely appeal to managers of the
methods I examined.
Finally, in chapter 5, I draw conclusions about the relative performance of the different
conflict forecasting methods and about other influences on forecast accuracy. Some of
these conclusions will be surprising to experts and managers. The chapter includes
discussion of implications and limitations of the research, as well as suggestions on
further research and recommendations to managers on choosing and implementing
forecasting methods for conflicts.
12
1.2 Conflict forecasting methods
Forecasting methods are often chosen because of popularity (frequency of use by
practitioners), or on the basis of expert judgement. Armstrong, Brodie, and McIntyre
(1987) surveyed forecasting practitioners on their use of six methods for forecasting
decisions in conflicts and for their assessment of the usefulness of these methods. The
authors intended these to be an exhaustive list of methods for forecasting decisions in
conflicts (personal communication from J. S. Armstrong, 29 August 2001).
The methods that were included in the Armstrong et al. (1987) survey were: unaided
judgement, intentions of other parties, game theory, statistical analysis of analogies,
role-playing, and field experiments. Singer and Brodie (1990) evaluated the face validity
of theories of and approaches to analysing business competition. The authors suggested
that the findings of their evaluation were broadly in accord with the stated forecasting
method preferences of respondents to the Armstrong et al. (1987) survey. They
concluded that expert judgement and role playing were associated with superior
approaches, and that game theory extensions appeared worthy of further research.
In this section, I consider unaided judgement and the intentions of other parties under the
heading of “unaided judgement”, game theory under the heading of “game theory”,
statistical analysis of analogies under the heading of “structured analogies” and role
playing and field experiments under the heading of “simulated interaction”. I suggest
that incorporating the avowed intentions of others is a common aspect of unaided
judgement, and treat the two as one method. I have not included field experiments in my
research. I present findings from the Armstrong et al. (1987) survey, together with other
opinions on the usefulness of the methods.
1.2.1 Unaided judgement
The term “unaided judgement” is intended to be self-explanatory – it is judgement
without recourse to a formal forecasting method. For the purposes of my research,
unaided judgement is what managers or forecasters use when they are asked to forecast
decisions in real conflicts, but do not use a particular method.
13
It is clear from the literature that managers mostly rely on their own judgement for
forecasting decisions in conflicts; either entirely or in conjunction with the judgemental
predictions of others who know about the situation. In some situations it may be
practical to ascertain the judgements of the other party or parties to a conflict, in the
form of their avowed intentions. For example, in his political manifesto, Mein Kampf,
Hitler outlined the policies he would later pursue as German dictator (Drabble, Ed.,
1995). Managers can incorporate such information into their own judgemental forecasts
of the behaviour of another party.
Expert judgement was used for forecasting competitive action by 85 percent of
organisations in the Armstrong et al. (1987) practitioner survey. More than 90 percent of
the forecasting and marketing experts surveyed endorsed expert judgement for this
purpose. Singer and Brodie (1990) observed that expert judgement “plays a major role as
a forecasting technique because there is no comprehensive unified theory from which
formal or analytic techniques might be derived” (p. 86). Unaided judgement is thus a
benchmark against which other conflict forecasting methods must be judged.
1.2.2 Game theory
Hargreaves Heap and Varoufakis (1995) describe game theory as being underpinned by
three key assumptions about the parties in conflict. These assumptions are that the
parties are (a) instrumentally rational, (b) know this, and (c) know the rules. In order to
forecast decisions that will be made in a real conflict, a game theorist might (1) develop
a new model (or adapt on old one) based on rules and utilities deduced from knowledge
of the conflict, (2) use judgement informed by knowledge of game theory, or (3) use
some combination of modelling and judgement. Experts are employed for their expertise
and so, for the purpose of my research, game theory is the method used by game-theory
experts when they are asked to forecast decisions made in real conflicts.
It seems reasonable to suppose that game theory could help practitioners to forecast
more accurately than they would if they relied on unaided judgement because, for
example, the discipline of the approach should help to counter judgemental biases.
Nalebuff and Brandenburger (1996, p. 8) wrote:
14
By presenting a more complete picture of each ... situation, game theory
makes it possible to see aspects of the situation that would otherwise have
been ignored. In these neglected aspects, some of the greatest opportunities
... are to be found.
McAfee and McMillan (1996) made the bolder statement that game theory “is to show
how people behave in various circumstances” (p. 172). The Sveriges Riksbank (Bank of
Sweden) Prize in Economic Sciences in Memory of Alfred Nobel was awarded in 1994
to three game theorists: John C. Harsanyi, John F. Nash, and Reinhard Selton. A press
release from Kungliga Vetenskapsakademien The Royal Swedish Academy of Sciences
(1994) stated:
… non-cooperative game theory… has had a great impact on economic
research. The principal aspect of this theory is the concept of equilibrium,
which is used to make predictions about the outcome of strategic interaction.
Game theorists “hope to produce a complete theory and explanation of the social world”
(Bullock and Trombley, Eds., 1999). Goodwin (2002) found the authors of two of a
convenience sample of six introductory game theory textbooks (Dixit and Skeath, 1999;
and Hargreaves Heap and Varoufakis, 1995) claimed that the method has value for
prediction or explanation. Binmore (1990) puts prediction first in a list of the aims of
game theory. The authors of a recent edition of a textbook on corporate strategy
(Johnson and Scholes, 2002) stated “Game theory provides a basis for thinking through
competitors’ strategic moves in such a way as to pre-empt or counter them” (p. 354).
Game theory was recommended by some experts in the Armstrong et al. (1987) survey.
It was used in nearly 10 percent of the surveyed organisations.
While Nalebuff and Brandenburger (1996) and Bullock and Trombley (Eds.) (1999), for
example, made optimistic claims for game theory, Shubik (1975, p. xi) described as
“peculiarly rationalistic” the assumptions behind formal game theory:
It is assumed that the individuals are capable of accurate and virtually
costless computations. Furthermore, they are assumed to be completely
informed about their environment. They are presumed to have perfect
perceptions. They are regarded as possessing well-defined goals. It is
assumed that these goals do not change over the period of time during which
the game is played.
15
Shubik suggested that while game theory may be applicable to actual games (such as
backgammon or chess), and may even be useful for constructing a model to approximate
an economic structure such as a market, “it is much harder to consider being able to trap
the subtleties of a family quarrel or an international treaty bargaining session” (1975, p.
14).
The claims made for game theory by some authors, the recommendations of experts to
use game theory, the evidence of game theory’s use by forecasting practitioners, and
controversy over the usefulness of game theory are all reasons to ask whether the
method can provide managers with useful predictions for real conflicts.
1.2.3 Structured analogies
The entry on “analogy” in the Forecasting Dictionary (Armstrong, 2001g) stated: “A
resemblance between situations as assessed by domain experts. A forecaster can think of
how similar situations turned out when making a forecast for a given situation”. The
structured-analogies method is described in the online version of the Forecasting
Dictionary1 as involving
…domain experts selecting situations that are similar to a target situation,
describing the similarities and differences, and providing an overall
similarity rating for each similar (analogous) situation. The outcomes of the
analogous situations are then used to forecast the outcome of the target
situation. The analogous situations’ outcomes can be weighted to forecast a
target situation decision or used to assign probabilities to possible decisions.
This is the approach that I adopted.
Analogous information has been shown to improve forecast accuracy in forecasting
tasks other than forecasting decisions in conflicts. For example, Efron and Morris (1977)
show that forecasts of an individual baseball player’s final batting average are more
accurate when the player’s early-season average is heavily weighted by the league
average than are forecasts based on the individual’s early-season average alone.
Kahneman and Tversky (1982) recommend a similar procedure for adjusting “intuitive”
1 http://morris.wharton.upenn.edu/forecast/dictionary, 12 August, 2002.
16
numerical forecasts (they use the example of sales of a book) towards the average for a
reference class (say, cookbooks by television celebrity cooks).
People who are asked to use their judgement to make a prediction for a situation may
think of analogous situations. Neustadt and May (1986) provided examples of analogies
being used by decision-makers to forecast the decisions of others in conflicts such as the
Cuban missile crisis. The authors suggested the use of analogies may in many situations
have led to inaccurate predictions with serious consequences. They attributed instances
of forecast inaccuracy, in part, to an ill-disciplined or uncritical use of analogies, and
recommended a more formal use of analogies to improve accuracy.
Analogies have been used in a formal way to forecast the distant future. For example, in
“The Railroad and the Space Program: An Exploration in Historical Analogy” (Mazlish
(Ed.), 1965) the authors used a single historical analogy “as a device to assist us in
forecasting... the impact of the space program on society” (p. v). Glantz (1991) explored
the use of analogies for, inter alia, forecasting societal responses to climate change.
Khong (1992) examined the evidence for and against the view implicit in Neustadt and
May (1986) that analogies are used by policymakers for analysis, and not solely for
advocacy and justification. Khong argued that the Neustadt and May view was
supported by the evidence. In particular, Khong argued that the favoured analogies of
decision-makers and advisors provided the best explanation of the decisions made by the
US administration early in the Vietnam war. He also suggested that analogies are not
used very well because policymakers tend to cling to readily accessible analogies and to
reject disconfirming evidence, rather than because they do not adhere to formal
processes. Nevertheless, the use of formal procedures for forecasting has been shown to
increase experts’ accuracy (for example Armstrong, 2001b; Collopy, Adya and
Armstrong, 2001; Harvey, 2001; MacGregor, 2001; Rowe and Wright, 2001; and
Stewart, 2001).
The use of analogies was recommended by Armstrong (2001c) for forecasting problems
where similar situations can be identified. Armstrong has also suggested (2001a) that
extrapolating from analogies may be useful for forecasting decisions in conflicts, but
pointed out that novel situations and novel strategies will lack obvious analogies – that
is, similar situations cannot be identified.
17
More than half of the experts surveyed by Armstrong et al. (1987) agreed that a formal
analysis of analogies should be useful. Statistical analysis of analogous situations was
the second most popular method for forecasting competitor actions – being used by 58
percent of organisations.
The common use of analogies for forecasting decisions in conflicts is sufficient reason to
ask whether the method can help provide managers with useful predictions for conflicts.
1.2.4 Simulated interaction
Experiments in the field can be used to predict decisions in conflicts. Although 40
percent of experts in the Armstrong et al. (1987) survey recommended experimentation,
the method was not popular with practitioners. I do not examine field experiments in this
research.
Laboratory experiments, in the form of role playing, can substitute for field experiments
by simulating a conflict using people who are not party to the conflict. Role playing is
likely to be cheaper than field experiments, and the risk of alerting rivals is reduced.
Role playing is described in the online Forecasting Dictionary (op. cit., 12 August 2002)
as “a technique whereby people play roles to understand or predict behavior”. As the
entry suggests, role playing is a technique that is applicable to problems beyond those
considered here. To avoid confusion, the use of role playing to simulate the interactions
of small numbers of parties whose roles are likely to lead to conflict is referred to as
“simulated interaction” (online Forecasting Dictionary, op. cit., 12 August 2002). I have
used the term “simulated interaction” in the balance of this document in preference to
the term “role playing”, except in cases of direct quotations or where the term “role
playing” is more appropriate.
Discussions of the usefulness and realism of simulated interaction are a feature of the
game-theory literature. The method is often contrasted with the limitations of game
theory in this context. Nalebuff and Brandenburger (1996, p. 62), for example, noted that
it is both important and difficult to appreciate the perceptions of other parties. They
18
suggested that managers might “ask a colleague to role-play by stepping into [another]
player’s shoes” (p. 63) in order to gain a better appreciation. The role-play outcomes of
contrived situations are commonly used by game-theory researchers as the behavioural
benchmark against which their hypotheses are tested. Pioneer of experimental
economics, Vernon Smith, wrote “Theories based upon abstract conditions make no
predictions… I see no way for game theory to advance independently of experimental
(or other) observations” (1994, p. 121).
Shubik (1975) covered similar ground when he wrote of simulated interaction that “an
extremely valuable aspect of operational gaming is the perspective gained by viewing a
conflict of interests from the other side. Experience gained in playing roles foreign to
one’s own interests may provide insights hard to obtain in any other manner” (p. 9). He
also pointed out game theory’s lack of realism relative to simulated interaction (gaming):
In summary we should suggest that many of the uses of gaming are not
concerned with problems which can be clearly and narrowly defined as
belonging to game theory. Environment-poor experimental games come
closest to being strict game theory problems. Yet even here, features such as
learning, searching, organising, are best explained by psychology,
social-psychology, management science, and other disciplines more relevant
than game theory (p. 17).
And on the same topic, Schelling (1961, p. 47) observed that
Part of the rationale of game organization [simulated-interaction
experiments] is that no straightforward analytical process will generate a
‘solution’ to the problem, predict an outcome, or produce a comprehensive
map of the alternative routes, processes, and outcomes that are latent in the
problem.
In contrast, simulated interactions
…do generate these complexities and, by most reports, do it in a fruitful and
stimulating way.
Surprisingly, although simulated interaction was recommended by most of the
forecasting experts in the Armstrong et al. (1987) survey, and there is evidence available
that the method provides more accurate forecasts than can be obtained from unaided
judgement, it is not often used in practice.
Armstrong’s (2001a) evidence on the accuracy of students’ simulated-interaction
decisions relative to the accuracy of students’ unaided-judgement forecasts and the
19
recommendations of experts are good reasons to include the method of simulated
interaction in a comparison of conflict forecasting methods.
1.2.5 Survey of experts’ accuracy expectations
In order to obtain formal data on experts’ expectations of the accuracy of different
conflict forecasting methods examined in this research, Professor Armstrong (personal
communication, 2002) surveyed academics and students attending a talk at Lancaster
University on 24 April 2002. He obtained responses from 27 people. Before asking
participants for their expectations, Armstrong described the forecasting methods and
their implementation in my research. He also described five of the conflicts I had used
and told the audience that by choosing at random from the decision options provided for
these conflicts, one could expect to be correct 28 percent of the time. Armstrong
repeated this procedure in a talk to Harvard alumni (responses from 18 business
executives) on 7 May 2002. I followed the same procedure in a talk to practitioners,
which was organised by the New Zealand Centre for Conflict Resolution, on 17 July
2002 (responses from 12 people). I repeated the procedure in a talk to educators at the
Royal New Zealand Police College on 19 July 2002 (responses from five people).
Overall, these various experts expected the unaided judgement of novices to be no better
than chance (Table 1). They expected a modest improvement in accuracy if novices were
used as role players, rather than as forecasters. Finally, the experts expected experts to
be more accurate than novices, regardless of the methods used.
20
Table 1
Experts’ expectations of forecasting methods’ accuracy a
Percent correct (number of responses)
Method Actual b Expectation c Difference
Unaided judgement (by novices) 27 (139) 30 (60) 3
Simulated interaction (using novices) 61 (75) 40 (60) -21
Unaided judgement (by experts) 50 (62)
Game theory (by experts) 50 (60)
Structured analogies (by experts) 50 (61)
Simulated interaction (using experts) 50 (57)
a
Forecasts for conflicts: Artists Protest, Distribution Channel, 55% Pay Plan, Nurses
Dispute, and Zenith Investment.
b
Findings from Armstrong (2001a) for Artists Protest, Distribution Channel, and 55%
Pay Plan except for 13 unaided judgement findings from Green (2002a): Artists
Protest (1 correct / n=8); Distribution Channel (1/5). Findings for Nurses Dispute and
Zenith Investment from Green (2002a).
c
Median expectation for the five conflicts listed in note “a”.
There is evidence available on the accuracy of novices’ forecasts and novices’
simulated-interaction decisions from Armstrong (2001a). As my findings for these are
similar to Armstrong’s, I have provided aggregated figures in Table 1. On the basis of
this evidence, experts were right in their expectations that novices would be no better
than chance when they forecast using unaided judgement. They were wrong, however, in
supposing that simulated interaction using novice role-players would offer little gain in
accuracy over unaided judgement by novices. It is interesting that the experts’
expectations were wrong on this, as findings of dramatic improvements in accuracy
when novices simulated, rather than predicted, were published fifteen years ago
(Armstrong, 1987).
The survey of expectations supports the need for empirical research by showing expert
opinion to be a poor guide on the relative accuracy of forecasts from different conflict
forecasting methods.
21
1.3 Objectives: motivation and implications
1.3.1 Overview
Evaluating forecasting methods
Singer and Brodie (1990) wrote that forecasting competitors’ behaviour “has not
received much attention in the forecasting literature… there is little guidance to
practitioners as to which forecasting methods to use” (p. 75). The purpose of my
research was to make useful recommendations to managers who face the problem of
forecasting decisions made in real conflicts. In order to achieve this purpose, it was
necessary to remedy the lack of evidence, identified by Singer and Brodie (1990), by
conducting research.
The purpose dictated the research task, which was to evaluate reasonable alternative
forecasting methods for conflicts. Principles for evaluating forecasting methods were
provided by Armstrong (2001e) (Table 2). I used these principles to guide the selection
and framing of research objectives, and the design of the research programme and of this
document.
22
Table 2
Forecasting method evaluation principles a
A/ Using reasonable alternatives
1Compare reasonable forecasting methods
B/ Testing assumptions
1Use objective tests of assumptions
2Test assumptions for construct validity
3Describe conditions for generalisation
4Match tests to the problem
5Tailor analysis to the decision
C/ Testing data and methods
1Describe potential biases
2Assess reliability and validity of data
3Provide easy access to data
4Disclose details of methods
5Do clients understand [and accept] the methods?
D/ Replicating outputs
1Use direct replication to identify mistakes
2Replicate studies to assess reliability
3Extend studies to assess generalisability
4Conduct extensions in realistic situations
5Compare with forecasts from different methods
E/ Assessing outputs
1Examine all important criteria
2Specify criteria in advance
3Assess face validity of methods & forecasts
4Adjust error measures for scale
5Ensure error measures are valid
6Ensure error measures insensitive to difficulty
7Ensure error measures are unbiased
8Ensure error measures are insensitive to outliers
9Do not use R2 to compare models
10Do not use RMSE
11Use multiple error measures
12Use ex ante tests for accuracy
13Use statistical significance to test only reasonable models
14Use ex post tests for policy effects
15Obtain large samples of independent forecast errors
16Conduct an explicit cost-benefit analysis
a Based on Exhibit 10 “Evaluation principles checklist” Armstrong (2001e, p. 465)
23
Appendix 1 provides a summary of how I addressed each of the principles in my
research.
Estimate relative performance of methods
Accuracy is generally rated the most important criterion for selecting a forecasting
method (Yokum and Armstrong, 1995) and is the principal criterion I consider.
Consequently, the primary objective of my research was to estimate the relative
accuracy of forecasts from reasonable forecasting methods – ones that were in use or
recommended by experts. In this context, an accurate forecast of a decision made in a
conflict is one that matches the decision actually made. For example, a decision may be
made to reject a pay offer, resist a take-over bid, disrupt a community, change an
allegiance, support a rebellion, or plan an invasion.
Assess generalisability and appeal to managers
My secondary objectives were to assess (a) the generalisability of the ranking of
forecasting methods by relative accuracy and (b) the likely appeal to managers of the
forecasting methods.
In order to assess generalisability, I investigated whether forecaster collaboration and
forecaster expertise affected the principal findings on the relative accuracy of forecasts
from the four methods.
Assessing the different methods’ appeal to managers is critical to the purpose of making
practical recommendations to managers. There would be no point in making
recommendations on forecasting method selection if the recommendations were not
accepted because of some overlooked selection criterion used by managers.
The data collection methods I used to address my research objectives are described in
chapter 3.
24
1.3.2 Estimate relative performance of methods
Estimate effect of forecasting method on forecast accuracy
Armstrong (2001f) has conjectured that the accuracy of forecasts from a method is
related to the realism with which that forecasting method allows forecasters to model the
target situation (Principle 7.2, p. 695). I suggest that unaided judgement allows the least
realism and simulated interaction the most. Game theory models conflicts using abstract
mathematical analogies, whereas the structured-analogies method uses real analogous
conflicts. I suggest that structured analogies will, therefore, allow more realistic
modelling than will game theory. A distinction could be made between forecasting
methods that rely on thinking and analysis by a forecaster, and those that rely on
simulating. It seems reasonable to assume that simulation will tend to result in greater
realism than thinking and analysing, particularly when forecasting a conflict that may
involve several rounds of direct interaction between two or more parties (Armstrong,
2001a). These conjectures are, however, contrary to the opinions of experts (Table 1). In
order to test the hypothesis of realism, I compare the accuracy of forecasts from the four
methods I consider using percent correct, and other measures.
Accuracy is likely not only to be a function of the forecasting method employed, but also
of how well the method is implemented. I describe how the methods are implemented,
but do not attempt to quantify how well they were implemented, as it would be
impossible to distinguish quality of implementation from the effects of the methods
without a more extensive research programme.
Estimate effect of forecasting method on forecast usefulness
In carrying out analysis on forecast accuracy, I use measures of accuracy that assign no
or negative value to forecasts that do not match the actual outcome. It is not necessarily
the case, however, that such forecasts are valueless. From a manager’s point of view, a
forecast of a decision that is similar to the decision that actually occurs will be more
valuable (useful) than a forecast of a decision that turns out to be substantially different
25
to the actual decision. I investigate the possibility that analysis based on forecast
usefulness may lead to different conclusions about the relative merits of the forecasting
methods.
1.3.3 Assess generalisability of findings
I extended prior research to assess the effects on forecasting accuracy of (1)
collaboration between forecasters for methods other than simulated interaction, and (2)
the expertise of forecasters.
Estimate the effect of collaboration on forecasting accuracy
The simulated-interaction method involves several participants generating each forecast,
and Armstrong’s (2001a) accuracy data for unaided-judgement forecasts were from pairs
of participants. Collaboration requires forecasters to justify their forecasts to their
fellows and allows forecasts to be combined. Both justification and combining tend to
increase the accuracy of judgemental forecasts (Stewart, 2001). I examine the effect of
collaboration on conflict forecasting accuracy for the methods unaided judgement and
structured analogies.
Estimate the effect of expertise on forecast accuracy
Armstrong’s (2001a) unaided-judgement forecast accuracy data were largely obtained
from novices (mostly students) and he obtained simulated-interaction forecast accuracy
data largely using student role players. Experts expect experts to be more accurate
forecasters than novices (Table 1), although research by Armstrong (1980; 1991) and
Tetlock (1992) suggest that this may not be so. I obtained game-theoretic and structured-
analogies forecasts solely from experts as game-theoretic forecasting requires a
knowledge of game theory and structured-analogies forecasting requires a knowledge of
conflicts similar to a target conflict. On the other hand, it is feasible to ask novices to use
their unaided judgement to forecast decisions in conflicts. Thus my first test of the effect
of expertise was to assess whether non-game theorists who were experts in conflicts,
26
forecasting, judgement, or decision making tended to provide unaided-judgement
forecasts that were more accurate than those provided by students.
Non-game theorists might or might not be domain experts in regard to particular
conflicts. For example, a conflict expert might have industrial relations expertise and
provide forecasts for conflicts in the industrial relations arena. My second test of the
effect of expertise was to assess whether experts who had more experience with conflicts
similar to a target conflict were more accurate than those who had less experience with
similar conflicts.
My third test of the effect of expertise was to assess whether experts who had more
years of conflict management experience were more accurate than those who had fewer
years of such experience.
The source of analogies might have a bearing on their usefulness for forecasting using
structured analogies. For example, analogies from direct experience may tend to lead to
forecasts that are more accurate than those from indirect experience such as those from
informal accounts, current affairs, history, or literature. If this were not the case, reading
the newspaper and studying history are likely to be good substitutes for direct experience
with this approach to forecasting. In my fourth test of the effect of expertise, I examined
the effect of analogy source on forecast accuracy. I also examined the effect of the
quantity and quality of analogies provided by forecasters on forecast accuracy.
Simulated interactions using role players who are similar to the real protagonists in a
conflict may provide more accurate forecasts than those that use student role players.
Surprisingly, however, the limited evidence that is available suggests that casting has
little effect on simulated-interaction forecast accuracy (Armstrong, 2001a). Moreover, in
practical applications of simulated-interaction forecasting, as in this research, the cost in
time and money of obtaining representative role players is likely to limit their use. For
these reasons, I have not examined the effect of casting on simulated-interaction forecast
accuracy.
Those who complete several conflict forecasting tasks may be better or worse than they
would have been had they forecast a single conflict. The data I have collected does not
27
allow me to distinguish between such an effect and any self-selection bias that might
exist, and so I have not examined this matter.
1.3.4 Assess appeal to managers
The accuracy of forecasts from a method, although it is the most important, is not the
only criterion used by managers to select a forecasting method. Yokum and Armstrong
(1995) summarised the importance given by researchers, educators, practitioners, and
decision-makers to thirteen forecasting method selection criteria. In order of importance
to the Yokum and Armstrong participants, these are (slightly paraphrased from
Armstrong, 2001c, p. 369):
1. Accuracy
2. Timeliness in providing forecasts
3. Cost savings resulting from improved decisions
4. Ease of interpretation
5. Flexibility
6. Ease in using available data
7. Ease of use
8. Ease of implementation
9. Ability to incorporate judgemental input
10. Reliability of confidence intervals
11. Development cost (computer, human resources)
12. Maintenance cost (data storage, modifications)
13. Theoretical relevance
Armstrong (2001c) suggested three additional criteria. These are:
1. Ability to compare alternative policies
2. Ability to examine alternative environments
3. Ability to learn (experience leads forecasters to improve procedures)
It seemed plausible that managers might weight the criteria differently when selecting a
method for one particular forecasting purpose rather than another. In order to allow for
this possibility, I obtained from experts criteria weights for the specific task of selecting
methods for forecasting decisions in conflicts. I also obtained ratings for the four
methods against the criteria.
28
1.3.5 Summary of objectives
The five objectives of my research on forecasting decisions in conflicts are summarised
in Table 3 under the three broad headings of performance, generalisability, and appeal.
Table 3
Research objectives:
For reasonable methods, investigate the effect of…
Performance Ê method on relative forecast accuracy
Ë method on relative forecast usefulness
Generalisability Ì collaboration on relative forecast accuracy
Í expertise on relative forecast accuracy
Appeal Î method characteristics on appeal to managers
29
2. Prior evidence on methods2
I sought empirical evidence of the relative accuracy of forecasts of decisions in real
conflicts for each of the four forecasting methods I examine in this research.
For the methods of game theory and structured analogies, I searched the Social Science
Citation Index (SSCI) and the internet. I also sent appeals to relevant email lists asking
for evidence and I communicated with leading researchers. The findings of these
searches are presented at the end of the relevant sections.
2.1 Unaided judgement
Decision-makers can be subject to serious biases, or “blind spots”. For example,
decision-makers who are involved in a conflict tend to give “insufficient consideration
of the contingent decisions of others”, as is evidenced by phenomena such as winner’s
curse and non-rational escalation of commitment (Zajac and Bazerman, 1991, p. 50).
Researchers have demonstrated that unaided judgements can be biased by the role of the
person making the judgement. Babcock, Loewenstein, Issacharoff, and Camerer (1995)
asked participants to estimate a “fair” judgement in a dispute between two parties.
Participants were given the role of lawyer for the complainant or for the defendant
before being presented with identical briefing material. The estimates of “complainant
lawyers” were, on average, higher than those of “defendant lawyers”. The researchers
found that the two groups had interpreted the same briefing material in different and
self-serving ways. Similarly, participants who took on the roles of either “cost analyst”
or “sales analyst” in research by Cyert, March, and Starbuck (1961) produced divergent
forecasts from identical sets of numbers, depending on their role. Statman and Tyebjee
(1985) replicated this research with consistent results.
The foregoing evidence suggests that a manager wanting a forecast for a conflict may
benefit from asking people who are not involved in the conflict for their judgement on
2 A version of this literature review was published in Green (2002a). The article did not
include evidence on analogies, nor did it contain the findings of SSCI and Internet
searches conducted in 2002.
30
the likely outcome. Independent judges are also, however, subject to influences that lead
to inaccurate forecasts. Experts, in particular, may be subject to overconfidence (Arkes,
2001), for example, or to biases resulting from the use of common and well-documented
judgemental heuristics (for example, Bazerman, 1998).
Armstrong (2001a) provided evidence on the accuracy of independent judges’ forecasts
of decisions in conflicts. He found that student research participants performed no better
than chance when exercising their unaided judgement to predict decisions made in
conflicts in which they were not involved. Tetlock (1999) found that experts’ (area
specialists’) predictions of the outcomes of political conflicts in the Soviet Union, South
Africa, Kazakhstan, the European Monetary Union, Canada, the US presidential race of
1992, and the Persian Gulf crisis of 1990-91 were “only slightly more accurate than
chance” (p. 351).
Overall, the evidence suggests that unaided judgement is unlikely to be a valid and
reliable method for predicting decisions in conflicts.
2.2 Game theory
2.2.1 Others’ reviews
Despite more than half a century of research, there is little evidence on the predictive
validity of game theory for decisions made in real conflicts. In a review of all game
theory articles published in the leading US operations research and management science
journals, Reisman, Kumar, and Motwani (2001) found an average of less than one article
per year involved a real-world application. In a review of Nalebuff and Brandenburger’s
book Co-opetition (1996), Armstrong (1997) wrote “I have reviewed the literature on the
effectiveness of game theory for predictions and have been unable to find any evidence
to directly support the belief that game theory would aid predictive ability” (p. 94).
Evidence that is available tends to be indirect and incomplete, typically comparing