Article

Forecasting Methods for Conflict Situations

Abstract and Figures

Policymakers need to know whether prediction is possible and, if so, whether any proposed forecasting method will provide forecasts that are substantially more accurate than those from the relevant benchmark method. An inspection of global temperature data suggests that temperature is subject to irregular variations on all relevant time scales, and that variations during the late 1900s were not unusual. In such a situation, a "no change" extrapolation is an appropriate benchmark forecasting method. We used the UK Met Office Hadley Centre's annual average thermometer data from 1850 through 2007 to examine the performance of the benchmark method. The accuracy of forecasts from the benchmark is such that even perfect forecasts would be unlikely to help policymakers. For example, mean absolute errors for the 20- and 50-year horizons were 0.18 � oC and 0.24 � oC respectively. We nevertheless demonstrate the use of benchmarking with the example of the Intergovernmental Panel on Climate Change's 1992 linear projection of long-term warming at a rate of 0.03 � oC per year. The small sample of errors from ex ante projections at 0.03 � oC per year for 1992 through 2008 was practically indistinguishable from the benchmark errors. Validation for long-term forecasting, however, requires a much longer horizon. Again using the IPCC warming rate for our demonstration, we projected the rate successively over a period analogous to that envisaged in their scenario of exponential CO2 growth--the years 1851 to 1975. The errors from the projections were more than seven times greater than the errors from the benchmark method. Relative errors were larger for longer forecast horizons. Our validation exercise illustrates the importance of determining whether it is possible to obtain forecasts that are more useful than those from a simple benchmark before making expensive policy decisions.
Content may be subject to copyright.
Chapter 8, pp. 157-176 of Judgmental Forecasting, G. Wright and P. Ayton (eds.),
John Wiley & Sons, Ltd, 1987. Reprinted with permission.
Forecasting Methods for Conflict Situations
J. Scott Armstrong
The Wharton School
University of Pennsylvania
In 1975, a consortium sponsored by the Argentine government tried to purchase the stock of the British-
owned Falkland Islands Company, a monopoly that owned 43 percent of the land in the Falklands, employed 51 per
cent of the labor force, had a monopoly on all wool exports, and operated the steamship run to South America. The
stockholders were willing to sell especially because the Argentine consortium was reportedly willing to pay “almost
any price.” But the British government stepped in to prevent the sale, (Murray N. Rothbard, as quoted in The Wall
Street Journal, 8 April 1982). In my opinion, the actual solution in the Falklands War left both sides worse off than
before. In contrast, a sale of the Falklands would have benefited both sides in the short run, and, as companies
seldom wage shooting wars, this would probably have been a good long-range solution. Apparently, Britain did not
predict how the Argentine generals would act when it blocked the sale, and the Argentine generals did not predict
how Britain would respond when they occupied the islands. Accurate forecasting by each side in this situation might
have led to a superior solution.
This study examines the evidence on alternative procedures that can be used to forecast outcomes in
conflict situations. I first define what is meant here by conflict situations. Next, I describe alternative forecasting
methods. This is followed by a presentation of hypotheses on which method is more appropriate. The evidence is
reviewed in two stages: first the prior research, then research that we have done.
Conflict Situations
In this study, conflict situations are those where two or more parties have opposing objectives, differing
strategies, or competing claims to a given resource. Differences in objectives occur, for example, when the seller is
trying to get a high price for a product while the buyer wants a low price. An example involving different strategies
would be the conflict involved among groups in New Zealand over the issue of whether the All Blacks rugby team
should have been allowed to play in South Africa; all of the parties were in favor of freedom and dignity, but they
disagreed about strategies to achieve these objectives. Examples of conflict over resources include that between
Britain and Argentina over the Falklands, between competitors such as Hertz and Avis, between labor and
management, between parties involved in attempts to take over a company, or between buyer and seller.
The situation becomes more difficult to predict when large changes or unusual events occur. These could
be due to changes in the environment or they could be brought about by actions of one of the parties. In such cases,
it is difficult to learn from experience. This chapter is concerned with predictions in situations with large changes.
Forecasting Methods
A variety of methods can be used to forecast in conflict situations. Brief descriptions are provided here for
some of the more important methods.
Expert opinion
People who have had relevant experience in similar situations should be able to make useful predictions.
Therefore, Argentine generals might be expected to make accurate predictions about the actions by Britain. Avis
executives can forecast actions by Hertz. Expert opinion is especially relevant for small changes and for changes
well within the experience of the experts.
2
Experts in human behavior could make predictions about the outcomes of situations. For this, they would draw upon
the empirical studies that have been done on conflict resolution. Especially qualified would be those experts who
specialize in negotiation processes.
Game theory
In contrast to expertise on the situation, one might use experts in game theory. These experts could try to
translate information about actual situations into a game theory framework in order to predict the outcome. A key
issue here is whether the game theorist can obtain sufficient relevant information for practical situations.
Intentions surveys
One possibility is to ask participants how they will act in a given situation. The advantage here is that in
addition to having information about the environment, they also understand their own motivation. On the negative
side, they may be unwilling to reveal their true intentions. Projective tests (“What would your friend do in such a
case?”) might be considered, but the chief difficulty would arise in getting participants to respond to the survey.
Furthermore, participants may lack insight as to how they would behave when large changes occur.
Extrapolation by analogies
By examining analogous situations, one may be able to predict for a new situation. For example, the issue
of fluoridation of water supplies has led to conflict in many communities in the USA, so the outcome of a new case
could be predicted by examining similar cases (e.g. “In what percentage of these similar cases did the proponents of
fluoridation win?”). Analogous situations also allow one to assess alternative strategies. Gamson (1975) examine
violence as a strategy of social protest in a variety of situations (e.g., women's suffrage). Regrettably, violence
generally proved to be a good strategy for protesters. Extrapolation by analogies is less relevant, however, for large
changes, new strategies, or new situations.
Laboratory experiments
Key features of a conflict situation might be translated into a laboratory experiment. The greater degree of
control in the laboratory must be weighed against the loss of realism. The laboratory experiment is common in
marketing research, for example, as seen in the use of simulated stores.
Field experiments
Field experiments are appropriate in some situations and their use increases realism. Thus, one could
experiment with the fluoridation issue in a few cities in one state before trying to extend it to other cities. Different
strategies could be examined in different cities. As a practical matter, field experiments are widely used in marketing
when new products are tested in certain geographical areas. The disadvantages of the field approach are that the
experiment itself may change the outcome of the event that is of primary interest, there is a loss of secrecy, expenses
are high, and the other parties in the conflict ma act differently during the experiments. The latter behavior
frequently occurs when firms test-market a new product. Another disadvantage is that experiments are not feasible
in all situations; for example, could Argentina attack only 5 percent of the Falkland Islands? Or could they try a one-
week experimental attack?
Role playing
Role-playing offers some of the advantages of experimentation while overcoming some of the
disadvantages. This section goes into detail about role-playing.
Subjects are given a specific role to play, and they are asked to interact with subjects who have different
roles. For example, Busch (1961) described how Lockheed used role-playing to predict outcomes of proposals to its
major customers. The nature of the conflict is described, and variations in responses must be possible by at least one
party. Perhaps the term “interactive behavioral simulation” would be more descriptive than “role playing.”
Nevertheless, most of the research has been done under the label of role -playing.
3
Research in psychology has demonstrated that role-playing has some validity (see Greenwood (1983) for a
review). But this term covers substantially more than that described above. Alternative definitions include opinions
surveys where the question is worded, “What would you do [in a given situation] assuming you had the following
role . . . ?” Also, the term “role playing” has been used where subjects are informed about the hypotheses and their
role before they participate in a laboratory experiment. The same term has also been employed when a subject is
asked to assume a role and then to imagine how he would behave in a given situation. Our interest is primarily in
what has been called “active role playing” because this seems to reflect reality better in conflict situations. For the
remainder of this chapter, however, I will abbreviate this and simply refer to “role playing.”
Different disciplines use terms other than active role playing. In political science and in the military,
interactive behavioral simulations are called “games”; in personnel psychology, they are called “job samples” (or
“work samp les”); in law, the term is “mock trials”; in psychiatry, it is “psychodrama”; and in sociology, it is “socio-
drama.” The existence of different definitions of the term “role playing,” and of different words to mean role
playing, make it difficult to discover the relevant literature. To avoid further confusion, I provide an operational
definition of the type of role playing proposed for predicting the outcome of conflict situations. This definition
involves each step in the attempt to simulate the interaction. It is necessary to decide who would be involved
(“casting”), how the situation should be described (‘describing”), how the role playing should be conducted
(“administering”), and how the results should be interpreted (“coding”). More detail is provided on each step in the
following paragraphs.
Casting
It seems desirable that those involved in the role playing be similar to the people they represent. This would
involve background, attitudes, and objectives. Interestingly, however, much of the prior research has used random
assignment of students to roles, and the results have been described as realistic (e.g., Zimbardo, 1972; Armstrong,
1977). Furthermore, Mandel’s review of the research on political role playing, which he refers to as gaming, led him
to conclude that similar results have been obtained whether one use experts or novices in the problem at hand (see
Mandel, 1977, pp. 614, 617, and 624). My advice on casting, then, is to obtain similar subjects if the cost to do so is
low; otherwise obtain “somewhat” similar subjects. Students have been used for most research done to date, and this
has been adequate for the most part.
Describing the situation
The description of the conflict situation should include information about current positions, exp ectations
about future relationships, and the processes that the parties expect to use in resolving their conflict (negotiating on
position, negotiating on principle, striking, violence, etc.). Obviously, the descriptions of the situations could be
extensive. Consider, for example, situations such as the Bay of Pigs incident, as described in Janis' book (1982),
Groupthink.
I suggest using short descriptions, preferably less than two typed pages. The use of simple descriptions and
short sessions receives some support from Elstein et al. (1978); comparisons of elaborate (high fidelity) simulations
of doctor/patient interaction were not superior to simple (low fidelity) simulations in their study. Brief descriptions
are also less expensive. Some role-playing simulations have been elaborate and time-consuming. Mandel (1977, p.
625) says that the Pentagon has spent up to $5000 for a single session. However, much role playing has been
relatively inexpensive and based on sessions lasting less than one hour.
The preparation of the situation description requires a good understanding of the situation as well as much care and
effort. Pre-testing is needed to ensure that the written description is clear and effective. Small changes in the
description might affect the outcome.
The description of the situation can lead to bias, especially in research using retrospective situations.
Certain items may be apparent after the fact, and these may favor one or another of the techniques. To guard against
unintended biases, it is desirable to use more than one description, and to compare the role-playing outcomes; this
adds to the costs, however.
4
In some cases, it may help to make the surroundings realistic. This might involve having the participants
dress appropriately (as in Janis and Mann's (1965) role play between doctor and patient) or selecting a realistic
setting (as in Zimbardo's (1972) prison simulation).
When feasible, the possible outcomes should be specified for the role players. This will aid in coding the
results. Of course, if the alternatives are not obvious, it may be better to leave them open; otherwise, an additional
source of bias is introduced.
Describing the roles
Role players should be encouraged to improvise when necessary, in an effort to make the session realistic.
Role players can be asked to act as they themselves would act, given their role and the situation. Alternatively, they
could be asked to act as they believe the persons they represent would act. As shown in Kipper and Har-Even
(1984), differences in this orientation can lead to substantial differences in outcomes. It is not clear which approach
is better.
Administering the session
Lacking evidence that preparation should be extensive, we suggest it be brief (e.g., ten minutes) because
this will be less expensive. To help people adjust well to their roles, it is helpful to have more than one person
represent a given party; these people can then confer to prepare for their role. Additional time, perhaps ten minutes,
should be allowed for this group preparation.
Once the administrator starts the role play, the players should stay in their roles. They should not step out of
their roles to ask questions or to discuss aspects of the case.
Coding
Ideally, each role-playing session should lead to a definite conclusion. This outcome would then be used as
the prediction. For example, if a given offer by management to the union would lead to a strike in 9 out of 10 role-
playing sessions, one would predict a 90 percent chance of a strike. Sometimes, however, the role -players will not
reach a conclusion. In such cases, the participants would be asked to state what they think the outcome of their
negotiations would have been.
The above guidelines have been inferred from previous research. To a great extent, however, the guidelines
are based on my judgment. It is difficult to claim that any of the elements in this list are vital to the design. Table 1
summarizes the role-playing procedure.
5
Table 1. Active role-playing procedure for conflict situations
Casting Find subjects “somewhat similar” to actual participants.
Describing the situation
Furnish brief, but accurate, descriptions. Specify possible outcomes.
Describing the roles
Improvise (for realism).
Stay in role at all times.
Act as the participants themselves would act, or act as the person being played would act.
Administering the session
Hold short sessions (less than one hour).
Allow for a brief preparation (ten minutes).
Prepare a realistic setting (dress, location, etc.).
Coding Use actual outcome from the role-playing as prediction.
Ask participants to state what the outcome would have been if outcome is no reached.
Hypotheses
Our hypotheses on the most accurate forecasting method would favor the use of extrapolation from
analogies if ample data were available on many highly similar situations. But for the situations with which we are
concerned large changes this possibility seldom exists. Lacking such data, we expect that experimentation would
be the most accurate. However, experimentation is expensive, sometimes obtrusive, and frequently not feasible.
Active role-playing has a number of desirable features. It is more realistic than all of the above competing
methods, except for the field experiment; of particular importance is the fact that role-playing allows for an
examination of the interaction among parties. Unlike the field experiment, one can maintain secrecy. Finally, it is
much less expensive than the field experiment. In a sense, role playing might be viewed as a low-cost approach to
experimentation.
Prior Evidence
As noted earlier, the use of role playing goes back many years. Goldhamer and Speier (1959) reported that
Germany used it in 1929 to plan war strategy. It also has a long history in psychology and law. Despite this, little
evidence is available on its predictive validity. Mandel's (1977) review of decades of the use of gaming in politics
led him to conclude that no comprehensive systematic review was available. Similarly, little evaluation has been
done in the legal profession (Gerbasi et al., 1977). For example, while IBM used a “shadow” jury to predict the
responses of an actual jury in the California Computer Products vs. IBM trial (Cooper, 1977), the IBM chairman,
Frank T. Carey, said that no systematic evaluation of this methodology was done, nor were company officials aware
of any (personal communication, 12 June 1978).
Kerr et al. (1979) stated that few: studies in psychology examined the methodology of role playing and its
validity. Greenwood (1983) reached the same conclusion when discussing active role playing. To review evidence
on the relative accuracy of role playing for predictive outcomes in conflict situations, I examined the Social Science
Citation Index for the Social Sciences from 1978 through 1984 (using the terms “role playing” and “socio -drama”). I
also wrote to researchers who had published empirical work m this area.
Numerous studies attest to the face validity of role playing. The evidence suggest that it provides realistic
results. For example, in Orne et al. (1968), observers could not distinguish between subjects who were hypnotized
and those who were role-playing a hypnotic trance. Zimbardo's (1972) simulation of a prison was so realistic that it
was terminated prematurely for fear someone might harm a “prisoner.”
6
Evidence was sought on comparisons of active role playing versus other methods. These comparisons could
be in either real or contrived situations. Real situations provide higher external validity, but the controls are fewer
and the costs are higher. Contrived situations, such as laboratory experiments, may have little relevance to the real
world.
Primary interest focused on evidence from “prospective” studies (i.e., those where the result had not yet
occurred). In addition, however, “retrospective’ studies were also examined. Even when it is possible to disguise
retrospective events, they present a sampling problem. That is, it makes no sense to select uninteresting and obvious
situations. But when selecting interesting cases, it is likely that they are interesting because of unusual outcomes.
Because unusual outcomes are typically based on unaided judgment, the use of judgment would be disadvantageous
when compared with other methods of forecasting retrospective events. That is, we are saying, “here is a forecast
where judgment did not work.”
The various types of evidence are summarized and rated in Table 2.
Table 2. Types of situations in which to assess role-playing
Retrospective Prospective
Contrived * **
Actual ** ****
* = weak evidence and **** = strong evidence
Contrived retrospective
Owing to concern over the rights of subjects (Vinacke, 1954), much of it spurred by the Milgram studies of
blind obedience in the 1960s, psychologists turned to role playing as an alternative to deceptive experiments. In
doing so, one concern they addressed was whether role playing would yield results similar to those derived from
experiments. Some of these studies involved forecasting the experimental (contrived) outcome.
At first glance, there seemed to be much favorable evidence for role playing. However, when I considered
only the type of forecasting relevant to conflict situations, few studies were left (see Greenwood (1983) for a review)
The majority of the studies on role playing did not involve any interaction. Most did not present an alternative model
for comparison. Finally, in some studies, different stimulus materials were used for role-playing than for alternative
techniques.
Some evidence did seem relevant, however. Willis and Willis (1970) had subjects’ role play an experiment
on conformity. They were told the design of the experiment and the hypotheses, and then were asked to respond as if
it were a real experiment. While the main effect was similar to that in the actual experiment, an interaction effect
was missed. Of course, this result lends itself to alternative explanation (e.g., knowledge of the hypothesis might
lead thr subjects to respond in a certain way).
In summary, role playing does seem to provide results that are similar to those from experiments. However,
the evidence for this conclusion is weak, and little of the evidence pertains to conflict situations.
Contrived prospective
7
Kerr et al. (1977) conducted an experiment to compare “real” and “mock” juries. The “real jurors” were led
to believe that their verdict counted in a case involving an academic violation at a university. On a pre-deliberation
questionnaire (in their role, but prior to the enactment of the jury deliberations) 48 percent of the 117 mock jurors
concluded that the defendant was guilty; these conclusions might be regarded as “intentions” to vote. For six-person
juries, assuming the initial majority prevails, it may be inferred that 40 percent of the juries would have reached a
guilty verdict. However, only 8 percent (1 of 12) of the juries that reached a unanimous verdict concluded that the
defendant was guilty. The mock jury results were similar in that none of the ten juries reached a guilty verdict. Thus,
role playing seemed to be superior to intentions, but again the evidence was weak.
Actual retrospective
In a political study known as the East Algonian exercise, Crow and Noel (1965) examined a conflict
between a strong country and a weak country. The countries were the United States and Mexico (personal
communication from Crow.) The situation was role-played by 96 groups to reach a decision for the Mexican
president ranging from 1, a peaceful response, to 11, a warlike response. Historians claim that the optimal decision
would have been a peaceful one (a “1”or a “2”) and that, in their opinion, this was the obvious choice. However, 57
percent of the role-playing groups reached a 4 or 5, a fairly belligerent response that corresponded to the actual
decision. That decision proved to be a bad one for Mexico because it lost Texas. Unfortunately, Crow and Noel
(1965) did not report on opinions in the East Algonian exercise. However, if we treat historians' opinions as
predictions of the “right” decision, role playing led to substantially more accurate predictions of reality.
Armstrong (1977) asked subjects to play the roles of seven members of the board of directors of the Upjohn
Corporation. They were told that an unbiased group of medical scientists, after 20 years of study, were unanimously
recommending that Panalba, an Upjohn drug with harmful side effects, be removed from the market. The board was
given 45 minutes to agree on one of the following five decisions:
(1) Recall Panalba immediately and destroy.
(2) Stop production of Panalba immediately but allow what's made to be sold.
(3) Stop all advertising and promotion of Panalba, but provide it for those doctors who request it.
(4) Continue efforts to market Panalba most effectively until sale is actually banned.
(5) Continue efforts to market Panalba most effectively and take legal, political, and other necessary
actions to prevent the authorities from banning Panalba.
Of the 57 groups that played the role faced by Upjohn directors, none decided to remove the drug from the
market. Furthermore, 79 percent decided to take decision 5. In fact, Upjohn also chose decision 5. (See Mintz (1969)
for descriptions of the events in this case.) The role-playing predictions in the Upjohn case differed substantially
from the predictions of non-role-players of what they would do in this situation. Only 2 percent of 71 respondents to
an interview said they themselves would select decision 5, and over half said they would choose decision 1 (to
remove the drug from the market). In opinion surveys, 41 percent of the 46 respondents at a meeting of the Hawaii
Economic Association predicted that Upjohn would select decision 5, while 17 percent of 18 students at the
Christchurch Polytechnic (New Zealand) predicted such a decision. These results are promising for role playing.
Actual prospective
Halberstram (1973, pp. 558-56) described a situation where high-ranking officers in the United States
military role-played the strategy of bombing North Vietnam. In this role play, the limited bombing strategy failed to
achieve the military objectives for the U.S. unlimited bombing had some military advantages, but overall it was
inferior to the “no bombing” strategy. Later the decision was made by the U.S. President and his advisors, who had
not participated in the role playing. Their expert opinion on the best strategy was limited bombing; this is widely
regarded as a poor decision since, as predicted by role playing, the strategy failed.
Since 1908, Washington and Lee University had run mock political conventions to select a Presidential
candidate for the party that is not in office. The convention is usually held in early May, about two or three months
prior to the actual convention. (A description of the procedure and a summary of the historical results can be
obtained from Washington and Lee University, Lexington, VA 24450). Including the Mondale nomination in 1984,
the convention has been correct on 13 of 18 candidates. Public opinion polls have been conducted since 1936; these
8
provide a comparable record of accuracy (Runyon et al., 1972; Gallup Opinion Index for 1972 and 1976; Harris
Survey for 1980 and 1984). The candidate who was leading in the poll that was conducted on about the same date as
the Washington and Lee convention won the nomination on 8 of 12 occasions. During this period the convention
was also correct on 8 of 12 occasions. The two approaches agreed on seven occasions, and when they did, the
prediction was always correct.
Boorman (1982) found that a batter of tests, one of which was role playing, was superior to a more
traditional battery for predicting success in training for 57 soldiers.
Studies on Opinions vs. Role Playing
We have obtained additional evidence using actual retrospective and prospective studies. A brief report on some of
this evidence was provided in Armstrong (1985). Below, I provide a description of the situations, subjects,
experimental treatment, administration, and results.
Situations
Descriptions for each situation were obtained from published accounts. The “Distribution Plan” and “Dutch
Artists” were actual retrospective situations. One actual prospective situation involved negotiations between owners
and players in the National Football League (NFL); we call the situation the “Football Negotiation.” Two-page
background descriptions were presented to the subjects for each of the three situations. In addition, the subjects
received a set of closed-ended questions designed to cover the range of possible outcomes.
The “Distribution Plan” describes a 1961 plan by the Ace Manufacturing Company (actually the Philco
Corporation) to sell major appliances in supermarkets. Customers at participating supermarkets would turn in their
cash register tapes, and 51 percent of their total purchases would be deducted from the monthly installment payment
for any appliance purchased. The payment of the discount was to be split between the manufacturer and the
supermarket on a sliding scale. Philco was interested in predicting whether the proposed plan would be accepted by
the supermarkets. (This case was taken from Berg (1970, pp. 87-131).)
“Dutch Artists” is based on a situation faced by the Netherlands government as reported by Newman
(1982). Artists in “Histavia” (the Netherlands) staged a sit-in at the country's major art museum in an effort to gain
additional benefits in the form of support for artists who are not able to sell their artwork.
“Football Negotiations” describes the conflict faced by the National Football League's (NFL) Players
Association and the owners of the teams. This situation was not disguised because the outcome had not yet occurred.
Our information was based on reports published on February 1, 1982 (Boyle, 1982; Kirshenbaum, 1982). No
negotiations had taken place prior to this time. The existing contract was scheduled to expire July 15, 1982. The
NFL Players Association said they would demand 55 percent of the football clubs' gross revenue to be used for
players' wages, bonuses, pensions, and benefits.
Subjects
The 96 subjects were students at the Wharton School, Un iversity of Pennsylvania, including 27
undergraduates and 69 MBA students. Fifty- Pennsylvania, including 27 undergraduates and 69 MBA students.
Fifty-seven percent of the subjects received partial credit for a course taught by the author, while the other 43
percent volunteered their time. No honoraria were paid.
The subjects were asked if they could identify the situation. Only one subject claimed to recognize one of
the disguised situations. Results from this pair, in the opinions treatment, were dropped. (Interestingly, that subject's
prediction for this case was incorrect.)
9
Experimental treatment1
Subjects were scheduled in two groups of two people each during an 80-minute session. Upon arrival at the
testing site, they were randomly paired and given the following instructions:
Today we will be examining decision-making to predict the outcomes of future events
involving conflict situations. You have been separated into groups of two. We are
going to go through three situations, one at a time. Each group is required to read,
understand, and then discuss each situation. Each group is to operate independently.
The order in which the situations were presented was randomized across sessions. Some subjects failed to
appear at a few of the sessions. When only three subjects arrived, one of the volunteers was asked to leave and the
remaining two subjects were assigned to the “opinions” group.
With the exception of the no-show sessions, the sessions were randomly assigned to either “opinions” or
“role playing.” The treatment for each group is described next.
Opinion sessions
Participants were split into random pairs, based on time of arrival, and were seated at opposite ends of a
room large enough so they could not overhear each other. Each pair was given the same situations, one at a time.
The following instructions were given:
Please read through the case I've given you. Once you understand the situation, each
group should discuss the issues involved. All relevant information is given. Please don't
be constrained by what's on the paper. I've given both facts and impressions in the text.
It's important to distinguish the two and extrapolate information as needed. Your primary
objective is to reach a consensus on the questions. Are there any questions?
After reaching a consensus, they were to choose among the responses to the questions that most closely
matched their prediction of the outcome. Although any one situation had a total time limit of 60 minutes, all of the
groups were able to complete the situations in the time allotted (80 minutes in all).
Role-playing groups
Owing to the time limits, only two of the situations were presented to the role playing groups. In addition to
the background information for these groups, each pair was randomly assigned to the role of one of the two parties
in the conflict (e.g., owners or players in the NFL). These roles (available from the author) apparently provided little
relevant information beyond that which the subject would have after reading the background information. Our intent
was to make the situation sound realistic and to indicate that the person believed strongly in the group's position. We
gave the following instructions to each group:
Please read through the case I've given you. You are to take on roles of the individuals
involved. Here are your role assignments [pass out roles]. I know some of you know each
other, but for now make it in a professional sense. Act as those people in your roles
would act; you are making decisions as those people.*2 Once you understand the
situation and have taken on the roles, conduct a meeting to discuss the situation with your
peer. Act as you would act if you were in the role which is assigned.* I've given both
facts and impressions in the text. It's important to distinguish the two and extrapolate
information as needed. Don't fill out the response sheet I've given you until after the role-
playing session. Feel free to use it to focus your discussion. You are interested in global
issues and the overall outcome of the situation; don't get bogged down in details and side
1 Copies of the materials used in the experiments are available from the author.
2 Unfortunately, the role instructions were not clearly specified in this experiment, as shown bythe two sentences
with asterisks.
10
issues. Improvise as necessary, but do not step out of your role the entire time this
session is being conducted. You are to prepare to meet your adversary. You will meet in
ten minutes and try to negotiate a conclusion to the situation. Are there any questions?
After reading and preparation time of no more than 20 minutes, the two pairs met at a conference table in
the center of the room. In the Philco Distribution situation, the role players were told they were meeting at the
supermarket chain's headquarters. For the Dutch Artists, the meeting was held “in the museum where the artists
were conducting the sit-in.” The Football Negotiation was said to be “in the meeting room of a luxury hotel in New
York City.”
The role play lasted until consensus or until the 60-minute time limit was reached. At the end of the role
play, each pair separated and answered questions based on the experience. They were instructed not to mark what
they personally thought would happen but, rather, to state the consensus as they saw it. If no consensus had been
reached, they were asked to state what they thought would have happened had their meeting been allowed to run to
its conclusion. Each role -playing session provided two responses, one from each group. While the two groups
interacted during the study, the perception of the outcome was reached independently. Each questionnaire was
treated as a separate response. As with the opinion technique, the “pair” represented the unit of analysis. For the
role-playing technique, each pair predicted the outcome of their interaction. (This procedure is expected to overstate
the statistical significance of the findings.)
Did the experimental design provide role-players with critical information not given to opinion subjects?
To test this, we gave role descriptions to 19 subject pairs (descriptions were identical to those given to role players
except that these were written in the third person). We called this group of subjects “role-aware.” They were asked
to discuss the situation from the perspective of the decision-makers described in the role material.
Administration
Most of the opinion and role-playing sessions were administered by Harry Walker during one week in
April 1982. Role-aware sessions were conducted in 1982, 1983, and 1985. One administrator was present at all
times during the sessions. Procedures used for the opinion and role-play groups are summarized in the Appendix.
Results
The relative accuracy of the role playing and the opinions are summarized in Table 3. Role playing was
significantly more accurate than opinions for the Dutch Artists and the Football Negotiations. Summing across the
three situations, opinions were correct on 6 out of 67 of the predictions (9 percent), while role playing was correct
on 16 of 24 predictions (67 percent), ignoring the two no-answer groups.
Table 3. Role-playing vs. opinions in predicting outcomes
(Entries represent predictions by pairs.)
Opinions
Situation
Prediction
same as
actual? Normal Role
awarea Total
Role
pl ay Statistical
significanceb
Philco Distribution No
Yes 15
0 18
1 33
1
2
8 p < 0.01
Dutch Artists No
Yes
No answer
13
1
0
4
0
0
17
1
0
2
2
2
p < 0.10
Football Negotiation No
Yes 11
4 0
0 11
4
4
6 p < 0.10
a The role-aware pairs were administered by Lisa Elliott and Elizabeth Schindler.
b One-tailed Fischer Exact Test, but overstated due to lack of significance in the role-playing pairs.
11
Additional evidence on the Football Negotiation was obtained from an insurance broker, James Silesky, of
Alexander and Alexander in Minneapolis (personal communication). His firm offered strike insurance to NFL
players. On April 16, one player purchased $20,000 coverage for a $1000 premium. According to the agent, this
premium was arrived at by “gut feeling.” It implies less than 5 percent chance of a strike. Our role-playing
predictions were significantly different than this market opinion (p < 0.01).
A summary of the preceding evidence on actual situations provides much support for role -playing. As
shown in Table 4, role playing was correct for 70 percent of the predictions versus 20 percent for expert opinions
(the predictive accuracy of opinions is typical of what one might expect to be obtained by chance).
Table 4. Actual situations: role-playing vs. opinions
Percentage of correct predictions
(sample sizes)a
Situation Conflict between Chance Opinion Role play
Retrospective
USA-Mexico
Political Conventions
Panalba
Philco Distribution
Dutch Artists
Two countries
Political candidates
Stockholder & consumer
Manufacturer & retailer
Government & interest group
18
33b
20
25
16
0
67
34
3
6
( 1)
(12)
(64)
(33)
(18)
57
67
79
80
50
(96)
(12)
(57)
(10)
( 4)
Prospective
North Vietnam bombing
Football negotiations
Two countries
Employees & owners
33
33
0
27
( 1)
(15)
100
60
( 1)
(10)
Unweighted average 25 20
70
a The sample sizes represent the number of predictions.
b Assuming two leading candidates and “all others.”
The importance of interaction
Role-playing provides vivid information to the participants on how they feel about the actions by others
and how the others react to their actions. These cycles of action and reaction are difficult for one to imagine without
role playing (or actual experience).
To examine the importance of the interaction, we obtained more data on the “role-aware” versus the
“Opinions only” by presenting the Dutch Artists and the Philco Distribution situations to 25 undergraduates at the
University of Canterbury in New Zealand in July 1985. The results, in Table 5, show no difference due to
knowledge of the roles. This is consistent with the hypothesis that the interaction is important to the role-playing
process. It also suggests that the roles did not add relevant information.
12
Table 5. Opinions vs. role-aware (individual subjects)
Percentage correct predictions
(number of predictions)
Opinions Role-aware
Philco Distribution
Dutch Artists
25 (12)
0 (13)
25 (13)
0 (12)
Costs
In addition to examining the accuracy of the predictions, we also examined costs. For the three situations
summarized in Table 3, subjects spent an average of 48 minutes for role playing as against 22 minutes for opinions.
Clearly, role playing is more expensive, as shown by our rough estimates in Table 6. Nevertheless, the potential
gains in accuracy should justify the costs in many situations because role playing predictions can be obtained using
only 50 people-hours. If the subjects are not doing the task as part of their job, we recommend that they be given a
small honorarium.
Table 6. Costs for role play vs. opinions (for 10 pairs of subjects)
People hours
Task Personnel Opinions Role-play
Preparation
Schedule
sessions
Testing
Researcher
Clerical
Administrator
Subjects
17
3
2a
8
Total 30
20
5
10
15
50
a All subjects could be run at one time for opinions.
Summary
From a theoretical viewpoint, role-playing offers advantages over opinions for predicting the outcomes in conflict
situations. It facilitates a realistic examination of the interaction among the parties. This is expected not only to
improve accuracy, but also to provide a better understanding of the dynamics, which in turn might lead to the
development of new strategies. Prior research provided modest evidence suggesting that role-playing is more
accurate than opinions for predicting outcomes. Our experiments added support from two more retrospective
situations and one prospective situation. The cumulative evidence from the seven actual situations found role-
playing to be correct for 70 percent of the predictions versus 20 percent for the opinions only. The interaction
seemed to be a critical element in the superiority of role-playing. However, we have not yet ruled out alternative
explanations, and much of the evidence is flawed.
Although role-playing is more expensive than opinions, in absolute terms the cost is small. The apparent
gains in accuracy are more likely to be worth the cost for important conflict situations. The substantial
improvements possible from role-playing suggest this area to be ideal for further research. Situations where the
outcome has not yet occurred are of particular interest because of the bias associated with the selection and
description of retrospective situations.
Acknowledgments
Harry S. Walker and Elizabeth Schindler conducted some of the role -playing sessions. Rod Brodie and Tom
Gilmore provided comments on earlier drafts. The Department of Business Administration at the University of
Canterbury in Christchurch, New Zealand, provided partial support.
13
Appendix
Opinion test procedure
1. Randomly assign four subjects into groups of two based on arrival (1st person group A, 2nd group B,
etc.).
2. Separate into groups to opposite sides of the room.
3. Read general instructions and answer questions.
4. Read opinion instructions and answer questions.
5. Distribute text of one situation to each person.
6. Distribute response questionnaire.
7. Monitor that there is no cross-group interaction and that the group discusses the case before
responding.
8. Make sure task is completed in 60 minutes
9. Ask if anyone recognized the situation and mark their answer sheet if yes.
10. Repeat 5 through 9 for the rest of the situations.
11. Tell the subjects not to discuss situations or test procedure with anyone else, and dismiss with a
thank you.
Role-playing procedure
1. Randomly assign four subjects into groups of two based on arrival time (1st person group A, 2nd
person group B).
2. Separate groups to opposite sides of the room.
3. Read general instructions and answer questions.
4. Read role-playing instructions, pass out roles, and answer questions.
5. Distribute text of one situation to each person.
6. Distribute response questionnaire.
7. Rearrange the furniture so table is in the center of the room with two chairs on either side.
8. Make sure groups prepare to meet each other.
9. Bring groups together at the table when they're prepared (no longer than 20 minutes).
10. Describe where the meeting is held.
11. Allow the role-play to continue for up to 60 minutes or until a consensus is reached or discussion on
the event to be predicted ceases. (The time will vary substantially depending on the task.)
12. Separate groups and ask them to force the decision or projected decision into one of the responses on
the questionnaire.
13. Ask if anyone recognized the situation and mark their answer sheet if yes.
14. Repeat 5 through 13 for the next situation.
15. Tell the subjects not to discuss the situation or test procedure with anyone else, and dismiss with a
thank you.
References
Armstrong, J. Scott (1977) “Social irresponsibility in management,” Journal of Business Research, 5, 185-213.
Armstrong, J. Scott (1985) Long-Range Forecasting: From Crystal Ball to Computer, 2nd ed.. New York: Wiley-
Interscience.
Berg, Thomas L. (1970) Mismarketing: Case Histories of Marketing Misfires. New York: Doubleday.
Borman, Walter C. (1982) “Validity of behavioral assessment for predicting military recruiter,” Journal of Applied
Psychology, 67, 3-9.
Boyle, Robert H. (1982) “The 55% solution,” Sports Illustrated, 1 February, 30.
14
Busch, G. A. (1961) “Prudent-manager forecasting,” Harvard Business Review, 39, 57-64.
Cooper, Ron (1977) “Shadow jury used by IBM at hearings in big anti-trust case,” The Wall Street Journal, 3
February, 7.
Crow, Wayman J. and Noel, Robert C. (1965). “The valid use of simulation results,” mimeo, Western Behavioral
Sciences Institute, 1150 Silverado Street, La Jolla, California 92037.
Elstein, Arthur S., Shulman, L. S. and Sprafka, S. A. (1978) Medical Problem Solving: An Analysis of Clinical
Reasoning. Cambridge, Mass.: Harvard University Press.
Gamson, William A. (1975) The Strategy of Social Protest . Homewood, Ill.: Dorsey Press.
Gerbasi, Kathleen C., Zuckerman, M. and Reis, H. T. (1977) “Justice needs a new blindfold: A review of mock jury
research,” Psychological Bulletin, 84, 323-345.
Goldhamer, Herbert and Speier, Hans (1959) “Some observations on political gaming,” World Politics, 12, 71-83.
Greenwood, John D. (1983) “Role-playing as an experimental strategy in social psychology,” European Journal of
Social Psychology, 13, 235-254.
Halberstam, David (1973) The Best and the Brightest. London: Barrie & Jenkins.
Janis, Irving L. (1982) Groupthink: Psychological Studies on Policy Decisions and Fiascoes, 2nd ea., Boston:
Houghton-Mifflin.
Janis, Irving L. and Mann, Leon (1965) “Effectiveness of emotional role-playing in modifying smoking habits and
attitudes,” Journal of Experimental Research in Personality, 1, 84-90.
Kern, Jeffrey M., Miller, C. and Eggers, J. (1983) “Enhancing the validity of role-play tests: A comparison of three
role-play methodologies,” Behavior Therapy, 14, 482-492.
Kerr, Norbert L., Nerenz, D. R. and Herrick, D. (1979) “Role-playing and the study of jury behavior,” Sociological
Methods and Research, 7, 337-355.
Kipper, David A. and Har-Even, Dov (1984) “Role-playing techniques: The differential effect of behavior
simulation interventions on the readiness to inflict pain,” Journal of Clinical Psychology, 40, 936-941.
Kirshenbaum, Jerry (1982) “Right destination, wrong track,” Sports Illustrated, 1 February, 7.
Mandel, Robert (1977) “Political gaming and foreign policy making during crises,” World Politics, 29, 610-625.
Mintz, Morton (1969) “FDA and Panalba: A conflict of commercial and therapeutic goals,” Science, 165, 875-881.
Mixon, Don (1972) “Instead of deception,” Journal for the Theory of Social Behaviour, 2, 145-177.
Newman, Barry (1982) “Artists in Holland survive by selling to the government,” The Wall Street Journal, 7
January, 1.
Orne, Martin, T., Sheehan, P. W. and Evans, F. J. (1968) “Occurrence of post-hypnotic behavior outside the
experimental setting,” Journal of Personality and Social Psychology, 9,189-196.
Runyon, John H., Verdini, J. and Runyon, S. S. (1971) Source Book of American Presidential Campaign and
Election Statistics, 1948-1968. New York: Frederick Ungar.
Vinacke, W. E. (1954) “Deceiving subjects,” American Psychologist, 9, 155.
15
Willis, Richard H. and Willis, Y. A. (1970) “Role-playing versus deception: An experimental comparison,” Journal
of Social Psychology, 16, 472-477.
Zimbardo, Philip (1972) “The pathology of imprisonment,” Society, 9 (April), 4-8.
... As a way to go, they recommend decision simulation as a means to make consequences vivid, and this should take place before value elicitation. They mention three methods: role-playing (Armstrong, 1987), outcome psychodrama (Janis and Mann, 1977) and scenario planning. Scenario planning is a well known method where one creates detailed descriptions of possible alternative future states of the world. ...
Article
This paper discusses the paradigm of multi-criteria decision analysis (MCDA), and relates it to other disciplines. It concludes that MCDA needs a larger, not smaller, emphasis on values and subjectivity to increase rationality in decision-making. The paper bases the argument on a conciliation of ethics, philosophy, neuro-psychology and management paradigms. It observes that the MCDA ‘mindset’ relates to consequentialism, as opposed to virtue ethics and rule based ethics. Virtues and rules play an important role in practical decision-making, however. Findings in neuro-psychology show that reliable decision-making requires emotions. Elicitation of emotions is therefore required in MCDA value trade-off processes. This leads to a concept of emotional rationality, which defines rationality as a four-dimensional concept that includes well-founded values and breaks radically with common notions of rationality. Virtues do not easily lend themselves to value trade-off, but questions of virtue usually creates strong social emotions, as opposed to the feebler global emotions that may arise in connection conventional trade-off of end values. The conclusion is that MCDA should not be shy of subjectivity and emotion, but instead put more emphasis on it to increase rationality. A part of this challenge is how to deal with questions of virtue in decision-making. Copyright
Article
This work presents the Delphi method, in its Policy Delphi variant, as an efficient mechanism for carrying out consultations regarding regulatory actions that affect professional bureaucracies, and also, in the last analysis, for forecasting and constructing their future. The Delphi application presented here incorporates some specific characteristics which are aimed at making the plurality of opinions in the organisation under analysis visible, facilitating reasoned debate, minimising the risk of manipulation by study leaders, and maximising the generalised acceptance of the results by the whole collective. This study was performed on the university community of the Basque Country, prior to the enactment of the Act which was to organise the Basque university system. Its results, which were accepted as being representative of the real dissensus existing in the community, constituted a valuable input for the final drawing up of the Act, and for its acceptance without conflict. This Delphi application cannot be classified within its traditional field of exploratory forecasting, but it can be situated within the normative sphere, geared towards the construction of a desired future.
Article
D uring the past five years the Social Science Division of The RAND Corporation has been developing a procedure for the study of foreign affairs that we call “political gaming.” This article gives a brief description of the technique and some of our observations about its utility.
Article
This article evaluates one means—political gaming—for coping with the distorted processes and perceptions that are present in foreign policy making during crises. Political games are exercises in which teams representing national governments meet and discuss crisis situations presented in scenarios. American foreign policy makers have engaged in this activity since the late logo's at the RAND Corporation, M.I.T., the Pentagon, and the C.I.A. Several hypotheses are developed on the changes in decisionmaking processes generated through political gaming, and on the nature of international perceptions during crises, as reflected through political gaming. These hypotheses are evaluated by means of data from the only unclassified professional-level games on international crises (those at RAND and M.I.T.), from a series of student games conducted at Yale, and from insights gained by the author's direction of two C.I.A. games. The results show that political gaming is indeed effective in improving decision making during crises, and they introduce some new aspects into accepted wisdom about international perceptions during crises.
Article
Review of J. Scott Armstrong's 1978 book on forecasting. Click on the DOI link above to read the review
Article
Widespread use of simulated trials and mock juries to study jury behavior has stimulated concern for the external validity ofsuch simulations. A study is reported which examined the role-playing nature of mock jury deliberation. Subjects were either given typical role-playing instructions, or were led to believe that they were deciding an actual student discipline case. The two conditions did not differ significantly on their group or individual verdicts, sentence recommendations, deliberation time, or estimated social decision schemes. The significance and limitations of these results for the study of jury behavior are discussed.
Article
Methods for enhancing the external validity of role-plays were evaluated. The heterosocial performance of 81 male undergraduates in an unobtrusive criterion situation were compared to performances on three content-identical role-plays. The typical role-play requested testees to simulate their behavior in a situation described by the testor. The replication and specification role-plays requested testees to replicate their behavior in the specified criterion situation. On the specification role-play, testees were also told of specific dependent measures which were important to simulate accurately. Results indicated that the specification role-play yielded stronger test-criterion relationships than the other two tests, with the replication role-play tending to be superior to the typical procedure. Only the typical role-play elicited significantly higher levels of performance than the criterion situation, supporting assertions that such tests assess response capabilities as opposed to naturalistic performances.