ArticlePDF Available

How to Make Better Forecasts and Decisions: Avoid Face-to-Face Meetings



When financial columnist James Surowiecki wrote The Wisdom of Crowds, he wished to explain the successes and failures of markets (an example of a "crowd") and to understand why the average opinion of a crowd is frequently more accurate than the opinions of most of its individual members. In this expanded review of the book, Scott Armstrong asks a question of immediate relevance to forecasters: Are the traditional face-to-face meetings an effective way to elicit forecasts from forecast crowds (i.e. teams)? Armstrong doesn't believe so. Quite the contrary, he explains why he considers face-to-face meetings a detriment to good forecasting practice, and he proposes several alternatives that have been tried successfully.
Electronic copy available at: copy available at:
Fall 2006 Issue 5 FORESIGHT 3
PREVIEW: When financial columnist James Surowiecki wrote The Wisdom of Crowds, he wished to explain the
successes and failures of markets (an example of a “crowd”) and to understand why the average opinion of a crowd is
frequently more accurate than the opinions of most of its individual members. In this expanded review of the book, Scott
Armstrong asks a question of immediate relevance to forecasters: Are the traditional face-to-face meetings an effective
way to elicit forecasts from forecast crowds (i.e. teams)? Armstrong doesn’t believe so. Quite the contrary, he explains
why he considers face-to-face meetings a detriment to good forecasting practice, and he proposes several alternatives
that have been tried successfully.
J. Scott Armstrong, Professor of Marketing at the Wharton School, University of Pennsylvania, was a
founder of the Journal of Forecasting, International Journal of Forecasting, and International
Symposium on Forecasting. He is the creator of, and editor of Principles
of Forecasting (Kluwer 2001), an evidence-based summary of knowledge on forecasting. In 1996, he
was selected as one of the first six Honorary Fellows by the International Institute of Forecasters. Along
with Philip Kotler and Gerald Zaltman, he was named the Society of Marketing Advances’ Distinguished
Marketing Scholar of 2000. For the past 13 years, he has been writing Persuasive Advertising: An
Evidence-Based Approach, which he forecasts will appear in 2007.
In The Wisdom of Crowds, Surowiecki claims that the collective thinking of many individuals, acting alone,
contains wisdom. He concludes that traditional face-to-face meetings, which prevent forecasters from acting alone,
yield poor decisions and inaccurate forecasts.
We do have guidelines on how to run meetings effectively, but it is rare to find group leaders who use them. Yet
traditional meetings persist partly because people falsely believe that they are necessary for aggregating opinions.
There are better alternatives, including Markets, Nominal Groups, and Virtual Teams. I describe how these work and
report the evidence on their value.
Some argue that face-to-face meetings are needed in certain contexts, such as when it is important to gain
commitment to decisions, but they have no evidence to back up these assertions.
Electronic copy available at: copy available at:
4 FORESIGHT Issue 5 Fall 2006
Every week I hear people complain about meetings.
What would happen to your organization if it
became difficult to have face-to-face meetings?
To this end, some organizations hold meetings in rooms
without chairs. Some impose limits on the length of the
session or the number of people who attend.
But what if an organization went further and penalized
people for spending time in meetings? Or required that
the meeting have a clear-cut payoff? As part of assessing
the results, management could provide a visible taxi-
style meter that uses attendees’ billing rates to show the
meeting’s costs. Or what if management abolished face-
to-face meetings entirely?
The Wisdom of Crowds
I have been thinking about the need for face-to-face meetings
for some time now. Recently, I have been spurred on by
The Wisdom of Crowds (Surowiecki, 2004), a delightful yet
exasperating book. It is delightful because the writing is
so clever and contains descriptions of interesting research
studies, many of which were new to me; it is exasperating
because it is not well organized, but the writing is so clever
that one may not notice the gaps in logic. Nevertheless, the
book’s major conclusion is important:
Traditional meetings yield poor
decisions and inaccurate forecasts.
Dave Barry summarized this conclusion in fewer words:
“If you had to identify, in one word, the reason that the
human race has not achieved, and never will achieve, its
full potential, that word would be meetings.” Apparently
Barry’s quote hit a nerve; a Google search for his conclusion
turned up almost 600 relevant sites (out of 10,000 total
sites) in July 2006.
The term “crowds” in the title of Surowiecki’s Wisdom
of Crowds is unfortunate. He claims that the collective
thinking of many individuals, when acting alone, contains
wisdom. Crowds act together, and they do not have wisdom.
A more descriptive title would have been “The Superiority
of Combined Independent Anonymous Judgments.”
The book has been widely reviewed on Amazon, with
comments from over 200 readers who have provided
a bimodal ratings distribution. The negative reviewers
fell into two classes: those who were upset at the basic
conclusions and those who were upset at the gaps in logic.
The experts priced this book at $25, but the crowd’s price
for a new copy in May 2006 was $10. If you enjoy books
like Who Moved my Cheese? and Jack Welch’s Winning,
you are unlikely to enjoy The Wisdom of Crowds. But it will
make you reconsider your assumptions about meetings. At
least it had that strong effect on me.
Face-to-face Meetings Could Be Effective
We do have guidelines on how to run meetings effectively.
This was well summarized over four decades ago by
Norman R. F. Maier. His research showed how group
leaders could make effective use of people’s information.
His book (Maier, 1963) provides evidence-based principles
for running meetings. Exhibit 1 provides a summary of
guidelines that draws heavily on Maier’s research.
Unfortunately, it is rare to find group leaders who use
Maier’s advice. In my 46-year career, I can remember only
Exhibit 1.
• Use time budgets. Allocate time to discuss various topics and provide ample slack time.
• Be problem centered. Keep your discussion focused on a problem. Avoid looking for excuses or seeking to blame others.
• Record suggestions. Keep track of all suggestions for solving a problem or making sense of an issue so that each
suggestion may be explored fully.
• Explore. Explore a number of suggestions for addressing an issue. Probing and evaluative questions can then be asked.
How would that strategy work out? Do I understand the issue, or do I need to search out more information? Am I
mistaken in my assumptions about the issue? What are the advantages or disadvantages of each proposal? Is there a
way to combine suggestions to generate a better solution?
• Protect people. Protect individuals from personal attacks and criticism, especially if they present minority or divergent
viewpoints. Avoid saying, “That’s a bad idea.
• Understand and resolve differences. Once ideas have been generated, encourage dissent. Understand differences of
opinions within the group and attempt to resolve them.
Fall 2006 Issue 5 FORESIGHT 5
a handful of business students, academic administrators,
or business executives who have run meetings effectively.
Productive meetings are possible but rare.
Why do people persist in holding face-to-face meetings?
First, we are social animals; many of us enjoy the interaction
with others in a face-to-face setting. Second, managers like
the control that meetings give them over others; they can
see that others are coming together at their commands.
Third, people believe that meetings are effective; managers
believe that they are doing something useful when they
meet (although they often do not have the same opinion
about meetings among their blue-collar workers).
A fourth reason is that people falsely believe that by merely
aggregating opinions without a face-to-face meeting, one
would get a decision or forecast that is only average. The
scientist Sir Francis Galton dispelled such a belief in 1878.
He showed that by averaging portraits of women, the
resulting portrait was judged not average looking but rather
more beautiful than all the component portraits. Larrick
and Soll (2006), in a clever series of experiments, showed
that among highly intelligent subjects (MBA students at
INSEAD), most did not understand that the error of the
group-average judgment is almost always smaller than the
error of the average person in a group. More surprising to
them was that the group-average judgment is sometimes
better than the best judgment.
The Case Against Face-to-Face Meetings
Face-to-face meetings are expensive to schedule and run.
They might involve travel costs or come at inconvenient
times, when attendees are busy or tired. Time is wasted
when people come late, talk about irrelevant topics, or
leave early.
Meetings are also subject to many types of biases. How
loudly do people talk? How deep are their voices? What do
the people look like? How is the furniture arranged? How
are people dressed? What is each person’s body posture?
Who has the power? How does the group leader guide the
meeting? Does the group nurture dissent? Do people have
preconceived positions on the topic at hand?
Some attendees are so concerned about what they want to
say that they do not listen to what others are saying. Some
are so intent on listening that they have no time to think.
Some feel the need to restate their positions. Few people
take notes; therefore they soon forget what happened in the
meeting and are unable to develop useful action plans.
Not surprising then is the prevalence of studies showing that,
compared with other methods of aggregating opinions (such
as using the average of a set of independent judgments),
the simple act of meeting face-to-face harms forecasting
and decision making, although the people involved in these
experiments typically do not believe the results.
Interestingly, the findings for forecasting and decision
making are similar to those studies that involve groups
generating creative ideas. As shown in the research review
by Gallupe et al. (1991), individuals produce more creative
suggestions than groups do, even if the groups are well run.
There are two conditions under which independent
judgments should be combined. First, the experts must have
useful information about the topic of interest; combining
ignorance does not lead to wisdom. Second, participants
must represent diversity of knowledge. The key word is
“knowledge.” For example, it makes little sense to include
experts because of differences in looks, heights, weights,
religions, races, genders, and so on. In fact, Stewart’s (2006)
meta-analysis of 26 tests found a small negative relation
between team members’ demographic heterogeneity and
group performance.
6 FORESIGHT Issue 5 Fall 2006
Decision making and forecasting can be improved to the
extent that
• people state opinions independently, and
• opinions are aggregated objectively, using
a predetermined mechanical scheme.
The implication of the above research is that managers need to
be creative in finding ways to use the knowledge effectively
in a group while preventing members from meeting face-
to-face. This will improve forecasting and decision making.
It will also save time and money. Fortunately, modern
technology has provided useful alternatives.
Alternatives to Face-to-Face Meetings:
Markets, Nominal Groups, and Virtual Teams
There are a number of ways to implement alternatives
to face-to-face meetings. I will discuss three: markets,
nominal groups, and virtual teams.
Markets (Prediction Markets, Information Markets, or
Betting Markets)
Experts and nonexperts alike bet on outcomes. These
markets are common in finance and sporting events.
People receive feedback only through prices and volume
of trading.
In The Wisdom of Crowds, Surowiecki describes the use of
markets for prediction. Their superiority has been shown
by studies in financial markets since the 1920s. Although
investors do not meet, they observe the outcomes of
actions by others and draw on related information to
make their decisions.
Outside of finance and sports, there have been few
comparative studies on the value of prediction markets.
The future looks promising, however. Surowiecki reports
that some companies are using prediction markets for
new-product sales. Since predictions for such problems
are typically made in traditional meetings, I would expect
prediction markets to produce more accurate forecasts.
I hope that The Wisdom of Crowds will lead some companies
to consider the use of prediction markets. Technology
should not pose a barrier. Some organizations will delegate
a person to set up a betting market for sporting events.
Nominal Groups (Including Delphi)
In nominal groups, judgments are collected from a group
of experts and are summarized by a group facilitator.
Surowiecki relied on suggestive and interesting (but
indirect) evidence on the value of nominal groups. He
failed to effectively use the wisdom of the crowds of
forecasting researchers in his search for evidence on the
value of simply combining judgments. In Armstrong
(2001), I summarized 11 comparative empirical studies
on the value of combining judgmental forecasts, which, to
my knowledge, was an exhaustive listing of such studies.
The median error of the group average in these studies was
12.2% less than that of the average expert’s error.
The Delphi technique goes beyond nominal groups. It
involves an anonymous collection of expert judgments
by mail, Internet, or written responses. Feedback on the
responses is provided to the experts, who then repeat this
exercise for at least two rounds.
Rowe and Wright (2001) found that Delphi improved
accuracy over traditional groups in five of the studies,
harmed accuracy in one, and was inconclusive in two.
Using an alternative benchmark, they found that it was
more accurate than one-round expert surveys for 12 of 16
studies, with two ties and two cases in which Delphi was
less accurate. For these 24 comparisons, Delphi improved
accuracy in 71% of the cases and harmed it in 12%. I
was unable to find any published studies that compared
prediction markets with Delphi.
Freeware for Delphi is provided at forecastingprinciples.
com. Usage of this freeware has increased substantially in
recent years.
Virtual Teams
Virtual teams, enabled largely by the Internet, have several
advantages. They allow for a freer flow of information than
do markets or nominal groups. Members of virtual teams
can use mail, e-mail, and Web sites. Phone calls are used
only in emergencies, and conference calls are not used.
These procedures remove some biases (for example, body
language and modes of speech). They allow time for people
to think before responding, and they provide a record of
what was accomplished
Despite the growing popularity of virtual teams, I was
unable to find comparative studies on the value of these
groups. However, based on related research summarized
by Surowiecki, I expect that virtual teams would be much
more effective than face-to-face groups, but less effective
than prediction markets and Delphi. Consistent with this,
it is important to gain commitment to decisions. With
respect to the third condition, one must be concerned
not only with the quality of a decision but also with its
acceptability. Would the feeling of involvement in a
decision more likely lead to acceptance when the group
has made a forecast or decision?
Some papers have suggested that meetings are useful
when the situation is complex and the solutions are not
obvious. While this suggestion has some intuitive appeal,
tests of this concept have failed, according to Dennis and
Kinney (1998). I doubt that such meetings are effective,
given the evidence that (1) people can understand complex
material better and faster when it is written (Chaiken &
Eagly, 1976); (2) people in groups are poor at generating
creative approaches; (3) many participants have difficulty
performing complex analyses in the presence of others;
and (4) groups are not tolerant of creative solutions.
Although I have circulated my paper for comments from
e-mail lists and from other researchers, I have been
unable to obtain evidence to support the use of face-to-
face groups under these or any other conditions. Some
people have responded with their opinions that meetings
are useful or that many managers like meetings. There
was one paper that provided promising results for face-
to-face meetings, but the findings were not clear. Some
people responded that they could not think of evidence
favoring face-to-face meetings.
Such sessions may meet people’s needs for socializing.
Magne Jørgensen (personal communication) mentioned
one company that did away with face-to-face meetings
for their projects, replacing them with e-mail messages.
To satisfy people’s needs for meeting and talking, they
sponsored social events.
Action Steps
Perhaps the first step is damage control. Reduce the
number of meetings, the length of meetings, and the
number of people invited. Post a chart on the group’s
homepage to track the people-hours (and their associated
costs) that are consumed by the meetings. Ask the group
leader to use Maier’s guidelines for meetings. In addition,
ask attendees to summarize the actions they have taken
after each meeting.
If people in your organization do not know how to respond
without meetings, you can bring them together in a room
Fall 2006 Issue 5 FORESIGHT 7
Ganesan, Malter, and Rindfleisch (2005), in a study on
new-product development, found that e-mail was superior
to face-to-face meetings with respect to new-product
creativity and development speed.
A Prediction Case
Can you predict the results of the following experiment?
To solicit useful feedback on research studies, a group of
160 experts was provided with research papers, one paper
per expert. The experts were randomly divided into two
treatment groups. In group A, ten sets of eight experts
participated in 80-minute meetings, where authors of the
ten studies presented their papers and addressed questions.
(Each group heard only one study.) In group B, each subject
in the nominal groups of eight experts worked alone and
without interruption for 80 minutes on one of the ten papers.
These experts wrote comments in the margins of the papers.
In effect, the intent was to have equal amounts of time spent
on each paper. Which treatment, A or B, produced more
useful suggestions? In which treatment did the authors of
the study use the suggestions more effectively?
Unfortunately, there is little research to establish which of
the mechanical methods of combining expert judgments
are most creative, most accurate, least expensive, and most
acceptable. For example, I have found that no published
empirical comparisons have been made among prediction
markets, Delphi, and virtual teams. In fact, the above
study has not been conducted. Based on related research,
however, I assume that Treatment B (nominal groups)
would be superior to Treatment A (traditional groups) in
terms of producing useful, accurate, and creative ideas. I
also assume that, in Treatment B, the acceptance rate by the
authors of the papers would be much higher.
Are Face-to-Face Meetings
Useful Under Some Conditions?
The evidence against face-to-face meetings is extensive,
and I have made no attempt to provide a complete summary
here. I did, however, attempt to contact all authors whose
work I cite in order to ensure that I have referenced the
information properly. My primary concern is to find
evidence that favors face-to-face meetings.
Are there conditions under which meetings contribute
to forecasting or decision making? I speculate on three
possibilities. The first is when the experts cannot read.
The second is when very small groups, perhaps two
people, may be able to work effectively. The third is when
8 FORESIGHT Issue 5 Fall 2006
Dennis, A. R. & Kinney, S. T. (1998). Testing media richness
theory in new media: The effects of cues, feedback, and task
equivocality, Information Systems Research, 9(3), 256-274.
Gallupe, R. B., Bastianutti, L. M. & Cooper, W. H. (1991).
Unlocking brainstorms, Journal of Applied Psychology, 76(1),
Ganesan, S., Malter, A. J. & Rindfleisch, A. (2005). Does
distance still matter? Geographic proximity and new product
development, Journal of Marketing, 69, 44-60.
Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining
opinions: Misappreciation of the averaging principle, Manage-
ment Science, 52, 111-127.
Maier, N. R. F. (1963). Problem Solving Discussions and Con-
ferences. New York: McGraw Hill. (Out of print, but used copies
are available.)
Rowe, G. & Wright, G. (2001). Expert opinions in forecast-
ing: The role of the Delphi technique, in J. S. Armstrong (Ed.),
Principles of Forecasting, Boston: Kluwer Academic Publishers,
Stewart, G. L. (2006). A meta-analytic review of relationships
between team design features and team performance, Journal of
Management, 32, 29-54.
Surowiecki, J. (2004). The Wisdom of Crowds, New York:
Valacich, J. S., Alan, R. D. & Connolly, T. (1994). Idea genera-
tion in computer-based groups: A new ending to an old story,
Organizational Behavior and Human Decision Processes, 57,
Acknowledgements: Useful suggestions were provided by Mon-
ica Adya, Fred Collopy, Kesten Green, and Magne Jørgensen, as
well as by Foresight editors.
Contact Info:
J. Scott Armstrong
The Wharton School, University of Pennsylvania
and then use structured procedures that simulate nominal
groups, as described by Aiken and Vanjani (2003). For
example, you could ask for a short “time-out” during
a meeting and ask everyone to write his or her ideas.
Software is available for conducting structured meetings,
and these products have proved useful (Valacich et al.,
1994) and have been gaining acceptance in organizations.
For example, Briggs et al. (1998) reported that electronic
brainwriting (individual idea generation) has been used by
several million people in over 1,500 organizations around
the world.
Individuals can also take action. My approach is to ask the
person who calls a meeting to describe the problem and
to inquire whether it would be useful to ask participants
to provide written suggestions rather than to attend the
meeting. The leader nearly always says yes and takes my
proposal in a positive way. This approach makes it easier
for people to absorb my suggestions and my reasoning
while it reduces their desire to argue against me (because
I am not there).
We rely heavily on face-to-face meetings, which are more
expensive than alternative approaches, even though it is
difficult to find evidence that supports their use. Although
evidence-based principles exist for running face-to-face
meetings effectively, they are used so rarely that we
must turn to more practical solutions. In fact, a pattern of
evidence suggests that prediction markets, nominal groups,
and virtual teams allow for a more effective use of a group’s
collective wisdom. Technology has enhanced the value of
these approaches.
Aiken, M. & Vanjani, M. B. (2003). Comment distribution in
electronic poolwriting and gallery writing meetings, Communi-
cations of the International Information Management Associa-
tion, 3(2), 17-36.
Armstrong, J. S. (2001). Combining forecasts, in J. S. Armstrong
(Ed.), Principles of Forecasting, Boston: Kluwer Academic
Publishers, 417-439.
Briggs, R.O., Nunamaker, J. F. Jr. & Sprague, R. H. Jr. (1998).
1001 Unanswered Research Questions in GSS, Journal of Man-
agement Information Systems, 14(3), 3-21.
Chaiken, S. & Eagly, A. H. (1976). Communication modality as
a determinant of message persuasiveness and message compre-
hensibility, Journal of Personality and Social Psychology, 34,
Fall 2006 Issue 5 FORESIGHT 9
always welcome Scott Armstrong’s contributions to the
discourse on forecasting. He has contributed more to our
understanding of the field than almost anyone else in
recent decades. In responding to James Surowiecki’s book The
Wisdom of Crowds, Scott considers the ubiquitous forecasting
meeting. He concludes that “the simple act of meeting face-
to-face harms forecasting and decision making.”
We have all attended unproductive meetings. But is it the
sessions themselves that are the problem or the inefficient
way they are run? Scott addresses this question by referring
to some guidelines from Norman Maier. Would meetings
still be useless if leaders followed Maier’s advice? Could
we make forecasting meetings more productive?
Drawing on my experience in attending many sales
forecasting meetings, I wish to suggest:
Forecasting meetings are really NOT about forecasting.
It might be termed a “forecasting meeting,” and its
identifiable output might be a final forecast, but the real
benefit is something quite different. I believe that meetings
have at least five purposes.
The Five Purposes of
Sales Forecasting Meetings
First, a meeting usually produces a final forecast that is
used by operations, finance, and procurement groups.
Second, a meeting produces a commitment to act. The
organization’s schedule for planning, procurement,
workforce numbers, and production are all based on this
commitment. Management must foster a sense of ownership
among members of the forecasting team.
Third, a meeting provides the important benefit of information
sharing. For example, a session of a major Mexican foods
producer considered the sales forecast for salsa. Based on
time-series information and the knowledge of existing
contracts, the product manager suggested a reasonable
estimate. The production manager then commented that
he was having difficulty in sourcing quality tomatoes, and
the purchasing manager questioned whether the new salsa
bottles would be available. On the basis of this input, the
final forecast was very different from the original.
Fourth, a meeting allows stakeholders the opportunity to be
heard. For example, a major global manufacturer of health
and hygiene products needed to revise a sales forecast.
The meeting was attended by the product manager, the
warehouse manager, the sales director, the production
manager, and the marketing director. The product manager
suggested a forecast for one of her major products. The
warehouse manager jumped up and exclaimed that he
refused to accept this forecast. His end-of-year bonus
was based on the value of the stock in his warehouse,
and he believed that the forecast would hurt his chances
of earning the bonus. The forecast was not adjusted, but
the contentious quantity of product was transferred out
of the warehouse. With an increasing emphasis on key
performance indicators (KPIs), the ability to be heard
becomes increasingly important.
Fifth, a meeting should result in a forecast that is linked to
the planning and budgeting cycle. Some might say that this
linkage takes us out of the realm of forecasting. But we
are essentially estimating, planning, and budgeting for the
same phenomena. This reconciliation process can be quite
tricky. To reflect plans and budgets, managers often adjust
forecasts during meetings. The March meeting of a major
cosmetics manufacturer dealt with the sudden realization
that to accept the next quarter’s forecast would mean that
the company would be underachieving its budget by some
20%. So the product manager was told to bring back an
“acceptable” forecast. Management also implied that the
revised forecast needed to be “close to the mark!”
Marcus O’Connor is a Professor
of Business Information Systems
at the University of Sydney.
His interest in forecasting was
stimulated while engaging in his
passion for surfing. Why could
other surfers pick the right wave
to ride when he couldn’t? He has
since spent many years exploring
the ability to engage with the future. Chiefly, that has involved
sitting in as an observer in sales forecasting meetings. He
enjoyed that task immensely and believes he now understands
some of the main drivers of those meetings. But, sadly, his
ability to pick the right wave has never improved.
10 FORESIGHT Issue 5 Fall 2006
Despite these five functions, some claim that the forecasting
process can be implemented through e-mail. My contention is
that a meeting gives us a better opportunity to develop ideas
sequentially, to share information, and to be creative. This is
an empirical proposition that deserves attention and research.
What Does the Research Show?
Forecasting meetings might be seen as a waste of time.
They may not produce more accurate forecasts. In fact,
there is some evidence that meetings compromise forecast
accuracy (Lawrence et al., 2000). However, a close look at
some of this research suggests that the forecasts were better
characterized as targets. In these cases, there was a strong
incentive to skew the forecasting numbers based on the cost
function appropriate to the business.
Scott Armstrong contends that “the research findings (both
direct and indirect) support the conclusion that people should
not meet face-to-face.” He cites some of the research on the
benefits of averaging and bootstrapping, and he touts the lure
of prediction markets. (Incidentally, how would a prediction
market be used in a sales-forecasting environment?) I do not
wish to address this research, but I will respond to Scott’s
challenge that we nominate contrarian views and research. I
would like to mention two articles of interest. First, in Ang &
O’Connor (1991), we compared four consensus forecasting
techniques used in meetings:
• a forecasting meeting in which a consensus forecast
was accepted,
• the same meeting, but a participant had produced his or
her own forecast prior to the session,
• a simple average of individual estimates (which Scott
seems to recommend), and
• a nominal group technique (another Scott-
recommended method).
One technique seemed superior: the forecasting meeting in
which one attendee had produced his or her own forecast
prior to the meeting. This was especially true for high-
difficulty time series. Moreover, we were able to isolate
the reason for the superior forecasts. The improvement
came from the discussion in meetings! This technique was
actually borrowed from an international organization that
produces household consumer products.
The second piece of research is relatively recent.
Rockenbach et al. (2006) were concerned with the
optimality of team decisions in an investment setting (a
guided meeting, in their case). These authors compared
team decisions with individual decisions. The results from
the meetings were more consistent with portfolio selection
theory, and the team decisions accumulated surprisingly
greater value at a significantly lower risk. Perhaps some
meetings are useful.
Scott cites and extends the work of Norman Maier,
suggesting six principles for the conduct of effective
meetings: (1) use time budgets; (2) be problem centered;
(3) record suggestions; (4) explore; (5) protect people;
and (6) understand and resolve differences. The last two
principles address why forecasting meetings are needed.
The real problem is the way that meetings are conducted.
Ineffective meetings lack definable objectives, have
ambiguous outcomes, or lack definable KPIs. Sales
forecasting meetings have a number of key advantages.
They have well-defined objectives. Simple validation
KPIs can be applied to the output. These meetings are
held regularly, so continual improvements can be made
to the structuring process. But I recognize the problems
associated with the lack of a good meeting chairperson.
Also, there is a tendency to get bogged down while
producing forecasts for products for which a statistical
method would have sufficed.
I welcome the debate on the efficacy of sales forecasting
meetings. To my mind, these meetings have important
organizational and social consequences. Could these live
sessions be replaced by e-mail messages or virtual groups?
Perhaps. But e-mail has become a major burden for many
organizations. Some firms are even configuring their
e-mail systems to restrict employee access.
Ang, S. & O’Connor, M. (1991). The effect of group interaction
processes on performance in time series extrapolation, Interna-
tional Journal of Forecasting, 7, 141-149.
Lawrence, M., O’Connor, M. & Edmundson, R. (2000). A field
study of sales forecasting accuracy and processes, European
Journal of Operations Research, 122, 151-160.
Rockenbach, B, Sadrich, A. & Mathauschek, B. (2006). Teams
take better risks, Journal of Economic Behavior and Organiza-
tion, at press.
Contact Info:
Marcus O’Connor
University of Sydney
Fall 2006 Issue 5 FORESIGHT 11
IS THE KEY by Joe Smith
Joe Smith is the Director of Forecasting
and Revenue Management Solutions
fo r Coca-C ola Ent erpri ses Inc .
North American Business Unit.
He has successfully facilitated the
design and implementation of CCE’s
standard, internally collaborative
forecasting system. Joe’s career at
CCE spans over seventeen years
with experience in strategic planning,
category management, promotion control and sales
analysis. Joe graduated from Eastern Michigan University
in 1987 with a BBA in Finance with a concentration in
investment analysis. His passion for applying mathematics,
planning processes and consensus building has led him
to his most recent pursuits in building enterprise-level
planning support systems.
When I was approached by the editor of
Foresight to write a book review, I was more
than happy to participate. I was told that the
book was The Wisdom of Crowds, by James Surowiecki.
I logged onto Amazon and ordered it the same day. As a
practicing business professional serving as the director of
forecasting and revenue management solutions for a $20
billion corporation, and as one who is an active reader of
business books, I thought this book would be a really good
fit for me. I am constantly challenging my team to convert
academic ideas into pragmatic policies for the company.
Most companies have little tolerance for new theories until
they are successfully applied. So I set out not only to write
a review but also to flush out the implications of the book
for our company processes.
My organization, Coca-Cola Enterprises, has been actively
transforming its approach to forecasting and business
planning. In our own way, we have been leveraging “the
wisdom of crowds” by creating an internally collaborative
forecasting process. We request input from over two
thousand employees every week – key account managers,
distribution center managers, and business unit staff.
We centrally aggregate this input into official forecasts.
We continually strive for greater levels of participant
cooperation, although I must confess that obtaining such
cooperation is always a challenge!
The Wisdom of Crowds brings to life the well-understood
precept that “two heads are better than one” – or that
many heads are better than one. Of course these sayings
are true if you can figure out how to get the heads to
communicate. Surowiecki recognizes the significance of
this challenge, and he provides dozens of examples of how
organizations have improved internal communication. The
common thread of the solution is getting a diverse group
of knowledgeable, cooperative, and strongly opinionated
people to provide their respective points of view. Then the
group’s collective opinion often will be better than that of
any single member, even the most expert.
This basic message should resonate with forecasting
practitioners. So how does an organization leverage
Surowiecki’s findings? The author does not provide
straightforward, step-by-step answers. If you are looking
for the practical explanation of how to apply Surowiecki’s
ideas, you are left to your own creativity.
But I think we can infer certain messages for our own
organizations. For my organization, The Wisdom of Crowds
imparts the following advice:
• The individual forecaster needs to demonstrate
what Surowiecki calls “local knowledge” and must be
resolute in presenting his or her point of view;
• Managers must then focus their efforts on
capturing the local knowledge of practitioners through a
nonthreatening forum;
• Capturing this knowledge requires continuous
practitioner participation in the forecasting process.
Our internal forecasting process encompasses thousands
of forecasters, all of whom are required to participate
on a weekly basis. That this is no simple chore is a
point vividly made by my colleague, Simon Clarke, in
the previous issue of Foresight (Clarke, 2006). Local
knowledge must be captured as a routine, compulsory
activity. Then we need support tools to compile, digest,
and share that knowledge.
Sharing local knowledge does not have to take much time;
however, it does require an interactive environment. The
forecasting team might conduct face-to-face meetings or
conference calls; the important point is that strong facilitation
is employed and that feedback is not personalized. Assuring
depersonalization is critical and merely requires that
discussions are focused exclusively on the facts.
12 FORESIGHT Issue 5 Fall 2006
AND MEETINGS by Jamilya Kasymova and Catalin Vieru
Jamilya Kasymova is Manager of
Strategic Projects & Forecasting of
Global Reservation Sales and Customer
Care at Marriott International. She
holds a PhD in Applied Mathematics
from Novosibirsk State University,
Russia and an MBA from the University
of Nebraska at Omaha. She seems to
enjoy weather extremes, first from the
bitter cold of Siberia to the sweltering
days in Uzbekistan, and now the
extremes of Omaha – bitter winters, hot and humid summers
and tireless winds.
Catalin Vieru is Director of Information
Services of Global Reservation Sales
and Customer Care at Marriott
International. He has masters degrees
in Finance, Information Services, and
Economics, and currently works on
optimizing the handling of Marriott
customer contacts. Catalin moved from
Romania to Italy and from Italy to the
USA, ending up in Omaha. He says he
would like to meet and toast anyone
who, not being a native of the region, specifically wishes
to be in Omaha, Nebraska.
Many in our profession would agree that forecasters
are driven by two main objectives: accuracy and
usability. Based on a plethora of information
available, forecasters attempt to develop predictions that
are as accurate as possible. And they try to present their
forecasts in ways that are most useful to those responsible
for the organization’s planning and operation.
Scott Armstrong asks whether the fulfillment of these two goals
requires “traditional,” face-to-face forecasting meetings. Our
response is that the forecaster wears two hats, and traditional
meetings can be beneficial while wearing one of them.
The forecaster wears the statistician’s hat when she
analyzes historical data, chooses appropriate models, and
then uses these models to produce forecasts. In this role,
the forecaster doesn’t need meetings, and she will actually
seek to avoid them. After all, the forecaster has been hired
by the organization as an expert in the field.
But forecasting is not a pure science; it is also an art. There
are always elements of intuition and judgment incorporated
into the final forecast. The forecaster puts on a different hat
when she tries to gather all pertinent domain information.
This information is not always formalized in memos or
e-mail messages, and that is why meetings can be useful.
It is worth the forecaster’s time to attend meetings in
which (1) strategic planning is discussed, (2) marketing
plans are introduced, and (3) the pros and cons of new
Contact Info:
Joe Smith
Coca-Cola Enterprises Inc.
Surowiecki and Armstrong emphasize that transforming
local knowledge into the wisdom of the crowd requires
participant anonymity. In my experience, however, the
key is not anonymity so much as sharing information in
a cooperative environment. The participants in the crowd
need to learn from each other in order to improve their
forecasting capabilities. Like most large corporations, my
organization cannot afford to wait for anonymous answers.
The live interactive meeting is our catalyst. It would be
nice to leisurely await the natural evolution of a forecast
consensus, but waiting is not practical.
We employ any means possible to assure routine
live interaction with forecast participants, including
traditional face-to-face meetings, Web-based meetings,
and conference calls. Through these exchanges we
have established a compulsory forecasting discipline.
Our employees know that forecasting is not an optional
activity; it is part of the job.
Clarke, S. (2006). Managing the introduction of a structured
forecast process: Transformation lessons from Coca-Cola En-
terprises, Inc., Foresight: The International Journal of Applied
Forecasting, Issue 1, 21-25.
Editor’s Note: Readers from Omaha are invited to respond.
Fall 2006 Issue 5 FORESIGHT 13
product launches are presented. The traditional meeting
is the forum where forecasters can absorb other experts’
opinions and concerns, and where forecasters can refine
their understanding of the company’s operations. When the
forecaster attends a meeting to listen and learn, she is not
necessarily subjected to the litany of problems, including
intimidation, that Armstrong catalogs.
The forecaster needs managerial validation of the assumptions
that underpin her modeling, and meetings can foster cross-
business pollination. Meetings also allow the forecaster to
solicit feedback on her presentation of forecast outcomes;
for example, would management like to see optimistic,
pessimistic, and most-likely forecasts? Face-to-face meetings
are a practical, convenient way to share information.
The key participants in meetings request clarification,
question assumptions, and offer feedback on how the
forecast might affect their respective areas of operations.
The forecaster learns which projections the organizational
leadership needs—and which format is most useful to
management. By involving key players, the forecaster
is connected to the business’ synapses, and she gains a
realistic outlook, one that ensures that the forecast can be
applied to the firm’s operations.
To summarize, the forecaster as statistician is most effective
working alone. The forecaster as gatherer of key business
information will find value in face-to-face meetings. There
she will be able to perform the following tasks:
• Gathering nonformalized information,
• Validating the assumptions used for modeling,
• Ensuring that the forecasts are realistic and actionable and,
• Gaining credibility, earning trust, and enhancing
reputation and visibility within the organization.
Understandably, few organizations have the luxury of hiring
multiple, independent, diverse, and decentralized forecasters
and then averaging their projections into a “final” forecast.
In addition, the use of independent markets for designing
decisions and predicting business outcomes in organizational
settings is not feasible because the cost of maintaining such
markets would be prohibitive. In principle, if employees knew
that management would allow them to maintain some privacy
and to become independent agents, they might embrace such
a system. However, this approach would signal a major shift
in the business practices of most organizations.
Armstrong believes that forecasters’ abilities are hindered
by face-to-face meetings. This argument seems to presume
that forecasters will, after a traditional meeting, rush back
to their mathematical models and change their underlying
assumptions in order to satisfy the needs expressed at the
meeting. If that were true, the meeting should have taken
place before the model had been formalized. For example,
forecasting a particular financial situation without any
supporting assumptions from specific sales strategies can
be futile, as no one will give the forecast any credence. For
most forecasters, the credibility, confidence, and feedback
generated at meetings are all worthwhile. A forecaster cannot
sterilize statistical calculations from hard business facts.
Forecasting is a business function that provides a vital “public”
good to the organization. For this reason, the forecaster
cannot rely solely on private information. Meetings in which
forecasters play key roles can elevate forecasters from their
private world to the public function they need to serve.
Perhaps some meetings should be banned, but it is difficult
to determine which meetings are expendable. The forecaster
can decide whether or not to attend a particular meeting.
Let forecasters do what they do best, including shaping the
way they acquire and use information. They are hired to
produce accurate and usable forecasts. As icing on the cake,
forecasters are rewarded by being recognized as a source
of an organization’s foresight. The interactions in face-to-
face meetings can enable the evolution of the forecaster
from statistician-practitioner to forecaster-strategist.
Contact Info:
Jamilya Kasymova and Catalin Vieru
Mariott International
14 FORESIGHT Issue 5 Fall 2006
TO COMMENTARIES by J. Scott Armstrong
The commentaries reinforce my belief that research
evidence alone is not sufficient for organizations
to consider new methods. I suggest procedures to
facilitate the implementation of evidence-based findings.
Is there a need for more evidence?
Are there conditions under which face-to-face meetings
improve forecasting and decision making? The
commentators all provide opinions from their experiences,
but Marcus O’ Connor was the only one to offer evidence
from a research study. Ang and O’Connor (1991), in a
comparative study of 36 three-person student groups, found
that when the forecasting task was difficult, a procedure
whereby one person prepared forecasts prior to a face-to-
face meeting led to improved forecasts vs. a nominal group
process (in which face-to-face meetings did not occur).
Ang and O’Connor’s evidence conflicts with much of
the prior evidence. This makes it worthy of replications
or extensions that would allow for feedback within the
nominal group procedure and within Delphi, as well as
controls for reflection time. The prior evidence, much of
which has been around for decades, is not favorable to the
use of face-to-face meetings. For example, Van de Ven
and Delbecq (1974), based on a laboratory experiment,
concluded that nominal groups and Delphi were superior to
face-to-face meetings with respect to decision making. In
my judgment, the weight of the scientific evidence suggests
that face-to-face methods harm creativity, forecasting, and
decision making
Interestingly, research on persuasion suggests that
examples are more persuasive than scientific evidence
when people hold strong beliefs that are contrary to the
evidence. So here is my story. In developing a procedure
for forecasting the vote in political elections, I worked in a
virtual group consisting of myself and two others, neither
of whom I had met previously. For the first few months, all
communications were by e-mail, which provided a written
record of what was done by each of us. We eventually
had a conference call, mostly for social reasons. In my
experience, this is probably the most creative and efficient
group (defined as three or more people) with which I have
been involved. We produced a successful website, a near
perfect forecast of the U.S. Presidential election of 2004,
and received recognition in Foresight for forecasting
accuracy (Cuzán et al. 2005).
How can you implement research-based findings?
Some of the commentators reacted to my arguments by
giving their opinions on why their organizations’ current
procedures are optimal. How can one get around this
problem of resistance to research evidence?
Important changes in organizations should be under the
control of the decision makers who are affected. Thus, the
question might be framed, “What type of information (e.g.,
experimental, trial and error, or prior research) should we
obtain in order to decide when we can use alternatives to
face-to-face meetings?” Unfortunately, the commentaries
did not address this question.
There may be some value in using the “second solution”
technique, in which the decision makers are prohibited
from solving the problem as they currently do. Instead,
they must develop an alternative procedure. Once that is
done, the constraint is relaxed and they can compare the
new procedure with their original one. Maier and Hoffman
(1960), in a problem involving a change in employee work
procedures, found that solutions were of higher quality
when groups were instructed to find a second solution
after they had presumably solved the problem. The second
solutions were obtained in about two-thirds of the time
needed for the initial solutions, and the groups generally
preferred their second solutions to the first ones.
Restrict your consideration of alternative procedures to
those supported by comparative studies. Procedures that
have been tested fairly and found useful might be useful
for you as well. There are many such methods.
Joe Smith and Marcus O’Connor comment that prediction
markets – a major alternative to face-to-face meetings
recommended by Surowiecki and myself – are not feasible
for sales forecasting within organizations. However, Ray
(2006) mentions that Microsoft, Hewlett-Packard, Eli Lilly
Fall 2006 Issue 5 FORESIGHT 15
and other major firms use prediction markets; he also makes
suggestions on how to implement them. The proposal
to use nominal groups dates back at least three decades.
Detailed operational suggestions were provided in Delbecq
et al. (1975); the book also includes testimonials on the
successful application of the nominal group technique in a
health care organization and in a business, ARA Services. I
also believe that the Delphi technique is feasible – you can
go to Software on and obtain
freeware to guide you. My paper also provided operational
guidelines for conducting face-to-face meetings.
Jamilya Kasymova and Catalin Vieru comment that their
organization could not afford to adopt the procedures
recommended by Surowiecki and me. I believe this
to be contrary to the evidence. For example, based on
comparisons among 12-person groups, Gallupe et al.
(1992) found that electronic brainstorming groups produced
about three times as many unique ideas as did traditional
brainstorming groups. In any event, this issue can be easily
resolved; one has only to try alternative procedures and
monitor the costs.
Ang, S. & O’Connor, M. (1991). The effect of group interaction
processes on performance in time-series extrapolation, Interna-
tional Journal of Forecasting, 7, 141-149.
Cuzán, A., Armstrong, J. S. & Jones, R. J. (2005). How we
computed the Pollyvote, Foresight: The International Journal of
Applied Forecasting, Issue 1, 51-52.
Delbecq, A. L., Van de Ven, A. H. & Gustafson, D. H. (1975).
Group Techniques for Program Planning, Glenview, Ill.: Scott,
Gallupe, R. B., Dennis, A. R., Cooper, W. H., Valacich, J. S.,
Bastianutti, L. M. & Nunamaker, J. F., Jr. (1992). Electronic
brainstorming and group size, Academy of Management Journal,
35, 350-369.
Maier, N. R. F. & Hoffman, L. R. (1960). Quality of first and
second solutions in group problem solving, Journal of Applied
Psychology, 44, 278-283.
Ray, R. (2006). Prediction markets and the financial “Wisdom of
Crowds,” Journal of Behavioral Finance, 7, (1), 2-4.
Van de Ven, A. H. & Delbecq, A. L. (1974). The effectiveness
of nominal, Delphi, and interacting group decision making pro-
cesses, Academy of Management Journal, 17, 605-621.
for your
library subscription
rates on a
for the practicing
that won’t just
take up shelf space.
... The wisdom of the crowd has been observed in a wide range of settings, in estimating weights of objects (Galton 1907, Wagner andSuh 2014), in political forecasts (Sjöberg 2009, Murr 2015, climate-related events (Hueffer et al. 2013), physician diagnostics (Kurvers et al. 2016), and economic forecasts (Kelley and Tetlock 2013, Budescu and Chen 2014, Nofer and Hinz 2014. The principle has been applied by practitioners across fields (Surowiecki 2004) and used in electronic technologies and online platforms that collect and aggregate opinions, for example, for organizational decision making (Spann and Skiera 2003, Armstrong 2006, Cowgill and Zitzewitz 2015. ...
... 4273-4286, © 2020 do well to encourage independent expression. Our results could be interpreted in support of Armstrong's (2006) suggestion to avoid face-to-face meetings in order to achieve wise decisions in organizations. Our experiments suggest that organizations may be able to increase the likelihood of a correct majority verdict by reducing team members' stake in a correct majority decision. ...
Teams, juries, electorates, and committees must often select from various alternative courses of action what they judge to be the best option. The phenomenon that the central tendency of many independent estimates is often quite accurate—“the wisdom of the crowd”—suggests that group decisions based on plurality voting can be surprisingly wise. Recent experimental studies demonstrate that the wisdom of the crowd is further enhanced if individuals have the opportunity to revise their votes in response to the independent votes of others. We argue that this positive effect of social information turns negative if group members do not first contribute an independent vote but instead cast their votes sequentially such that early mistakes can cascade across strings of decision makers. Results from a laboratory experiment confirm that when subjects sequentially state which of two answers they deem correct, majorities are more often wrong when subjects can see how often the two answers have been chosen by previous subjects than when they cannot. As predicted by our theoretical model, this happens even though subjects’ use of social information improves the accuracy of their individual votes. A second experiment conducted over the internet involving larger groups indicates that although early mistakes on easy tasks are eventually corrected in long enough choice sequences, for difficult tasks wrong majorities perpetuate themselves, showing no tendency to self-correct. This paper was accepted by Yuval Rottenstreich, decision analysis.
... The gathering of experts in meetings carries the risk that the validity of the consensus opinion will be undermined by majority views and perceived pressure from peers and superiors (Armstrong, 2006). PMs though create a (monetary) incentive for individuals who have information to 'report' it and participate in the market. ...
Full-text available
Prediction markets (PMs) are virtual stock markets on which shares are traded taking advantage of the wisdom-of-crowds principle to access collective intelligence. It is claimed that the accumulation of information by groups leads to joint group decisions often better than individual participants’ approaches to solutions. A PM share represents a future event or a market condition (e.g. expected sales figures of a product for a specific month) and provides forecasts via its price which is interpreted as the probability of the event occurring. PMs can be used in competition with other forecasting tools; when applied for forecasting purposes within a company they are called corporate prediction markets (CPMs). Despite great praise in the (academic) literature for the use of PMs as an efficient instrument for bringing together scattered information and opinions, corporate usage and applications are limited. This research was directed towards an examination of this discrepancy by means of focusing on the barriers to adoption within enterprises. Literature and reality diverged and neglected the important aspect of corporate culture. Screening existing research and interviews with business executives and corporate planners revealed challenges of company hierarchy as an inhibitor to the acceptance of CPM outcomes. Findings from 55 interviews and a thematic analysis of the literature exposed that CPMs are useful but rarely used. Their lack of use arises from senior executives’ perception of the organisational hierarchy being taxed and fear of losing power as CPMs (can) include lower rungs of the corporate ladder in decision-making processes. If these challenges can be overcome the potential of CPMs can be released. It emerged – buttressed by ten additional interviews – that CPMs would be worthwhile for company forecasting, particularly supporting innovation management which would allow idea markets (as an embodiment of CPMs) to excel. A contribution of this research lies in its additions to the PM literature, explaining the lack of adoption of CPMs despite their apparent benefits and making a case for the incorporation of CPMs as a forecasting instrument to facilitate innovation management. Furthermore, a framework to understand decision-making in the adoption of strategic tools is provided. This framework permits tools to be accepted on a more rational base and curb the emotional and political influences which can act against the adoption of good and effective tools.
... Yet most studies of strategic decision-making do not allow individuals to talk. When talking is allowed, faceto-face interactions are typically prohibited as biasing decisions (Armstrong, 2006). Face-to-face interactions can influence decisions because of gender, ethnicity, apparel, tattoos, attractiveness, and other aspects that may activate stereotypes (Kurzban et al., 2001;Wright and Sladden, 2003). ...
Full-text available
Trust is risky. The mere perception of strategically deceptive behavior that disguises intent or conveys unreliable information can inhibit cooperation. As gregariously social creatures, human beings would have evolved physiologic mechanisms to identify likely defectors in cooperative tasks, though these mechanisms may not cross into conscious awareness. We examined trust and trustworthiness in an ecological valid manner by (i) studying working-age adults, (ii) who make decisions with meaningful stakes, and (iii) permitting participants to discuss their intentions face-to-face prior to making private decisions. In order to identify why people fulfill or renege on their commitments, we measured neurophysiologic responses in blood and with electrodermal activity while participants interacted. Participants (mean age 32) made decisions in a trust game in which they could earn up to $530. Nearly all interactions produced promises to cooperate, although first decision-makers in the trust game reneged on 30.7% of their promises while second decision-makers reneged on 28%. First decision-makers who reneged on a promise had elevated physiologic stress using two measures (the change in adrenocorticotropin hormone and the change in skin conductance levels) during pre-decision communication compared to those who fulfilled their promises and had increased negative affect after their decisions. Neurophysiologic reactivity predicted who would cooperate or defect with 86% accuracy. While self-serving behavior is not rare, those who exhibit it are stressed and unhappy.
... When using a jury of experts, the literature suggests avoiding face-to-face interactions (Armstrong 2006), as influential individuals may herd forecasts to a particular preference. A structured approach to overcome this is the Delphi method (for details see Rowe 2007). ...
The COVID-19 pandemic has had a devastating effect on many industries around the world including tourism and policy makers are interested in mapping out what the recovery path will look like. We propose a novel statistical methodology for generating scenario-based probabilistic forecasts based on a large survey of 443 tourism experts and stakeholders. The scenarios map out pessimistic, most-likely and optimistic paths to recovery. Taking advantage of the natural aggregation structure of tourism data due to geographic locations and purposes of travel, we propose combining forecast reconciliation and forecast combinations implemented to historical data to generate robust COVID-free counterfactual forecasts, to contrast against. Our empirical application focuses on Australia, analyzing international arrivals and domestic flows. Both sectors have been severely affected by travel restrictions in the form of international and interstate border closures and regional lockdowns. The two sets of forecasts, allow policy makers to map out the road to recovery and also estimate the expected effect of the pandemic.
... Moreover, group interactions often yield processes of herding, bullying, and social signaling (Kerr, MacCoun, & Kramer, 1996;Raafat, Chater, & Frith, 2009) that foster groupthink (Janis, 1982) and bias group judgments in aggregate. Some scholars have even suggested that, relative to aggregated independent answers, face-to-face discussion is antithetical to good group judgment (Armstrong, 2006). ...
‘Crowd wisdom’ refers to the surprising accuracy that can be attained by averaging judgments from independent individuals. However, independence is unusual; people often discuss and collaborate in groups. When does group interaction improve vs. degrade judgment accuracy relative to averaging the group's initial, independent answers? Two large laboratory studies explored the effects of 969 face-to-face discussions on the judgment accuracy of 211 teams facing a range of numeric estimation problems from geographic distances to historical dates to stock prices. Although participants nearly always expected discussions to make their answers more accurate, the actual effects of group interaction on judgment accuracy were decidedly mixed. Importantly, a novel, group-level measure of collective confidence calibration robustly predicted when discussion helped or hurt accuracy relative to the group's initial independent estimates. When groups were collectively calibrated prior to discussion, with more accurate members being more confident in their own judgment and less accurate members less confident, subsequent group interactions were likelier to yield increased accuracy. We argue that collective calibration predicts improvement because groups typically listen to their most confident members. When confidence and knowledge are positively associated across group members, the group's most knowledgeable members are more likely to influence the group's answers.
... It can be argued that to ensure the existing of diversity in a collective; its members should make own predictions independently of others in the collective (Armstrong, 2006;Kahneman, 2011;Lorenz, Rauhut, Schweitzer, & Helbing, 2011;Page, 2007;Sunstein, 2006;Surowiecki, 2005). Independence is helpful in the effective of crowdbased applications because of correlated mistake reduction. ...
Full-text available
In general, a common approach to enhance the diversity, an important criterion of intelligent collectives, is to enlarge the collective cardinality. The objective of this study was to analyse some issues with collective cardinality in the effectiveness of collective performance. For this aim, first, we investigate the impact of collective cardinality on collective performance. Subsequently, a comparison among collective performance measures is presented. The simulation results have qualified the positive impact of collective cardinality on the effectiveness of collective performance. Moreover, some issues on processing large collectives; some further challenges of the impact of diversity on collective prediction are discussed.
Managing rare risks is the Holy Grail of risk management. Those are extremely unlikely events with potentially catastrophic impact. This makes their prediction very difficult, yet supremely important as they often mean the difference between life and death of individuals, companies, and even whole systems. The chapter reviews traditional methods of analysis and then presents an integrated approach combining the strengths of expert judgment, statistical modeling, and Monte Carlo simulations to gain a better understanding of rare risks in the digital economy. The aim is to outline the limits of predictability of such unlikely events and yet create a meaningful framework that allows for the evaluation of their potential scope. Based on that, contingency management plans can be devised within the constraints of organizational resources and pre-defined risk appetite.
List of works referred to in Armstrong & Green (2022) The Scientific Method
Full-text available
What effect does having a group solve a problem the second time have on the quality of group solutions? 100 students were divided into 25 groups of 4 each to "role play the case of the Change of Work Procedures. The roles of the foreman and three workers were assigned randomly to the members of each group." When they finished, "they were asked to arrive at a second solution to the problem." Although the double-solution method seemed to require little more time, the "solution quality was generally increased without a loss of acceptance." (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
Many firms rely on external organizations to acquire knowledge that is useful for developing creative new products and reducing the time needed to bring these products to market. Cluster theory suggests that this knowledge is often obtained from organizations located in close geographic proximity. Specifically, proximity is assumed to fos-ter heightened face-to-face communication, strengthened relational ties, increased knowledge acquisition, and enhanced new product outcomes. The authors identify the limitations of these assumptions and offer an enriched model of the influence of geographic proximity on new product development, which they test using both a cross-sectional survey of 155 firms in the U.S. optics industry and a longitudinal follow-up survey of 73 of these firms. They find that firms located in close proximity engage in increased face-to-face communication, but this communi-cation has little effect on the acquisition of the types of knowledge that lead to enhanced new product outcomes. In contrast, they find that e-mail communication leads to both enhanced new product creativity and development speed. In addition, they find that relational ties moderate rather than mediate the path connecting geographic prox-imity and new product outcomes. These findings imply that the new product development outcomes typically ascribed to close geographic proximity may actually be attributed to strong relational ties.
Full-text available
Media richness theory argues that performance improves when team members use "richer" media for equivocal tasks. This experiment studied the effects of media richness on decision making in two-person teams using "new media" (i.e., computer-mediated and video communication). Media richness was varied based on multiplicity of cues and immediacy of feedback. Subjects perceived differences in richness due to both cues and feedback, but matching richness to task equivocality did not improve decision quality, decision time, consensus change, or communication satisfaction. Use of media providing fewer cues (i.e., computer mediated communication) led to slower decisions and more so for the less equivocal task. In short, the results found no support for the central proposition of media richness theory; matching media richness to task equivocality did not improve performance.
Full-text available
Brainstorming groups have consistently produced fewer ideas than have the equivalent number of individuals working by themselves. These results have been attributed to social loafing, evaluation apprehension, and production blocking in groups. In this study, a new brainstorming technique--electronic brainstorming--that may reduce both production blocking and evaluation apprehension was assessed. Electronic and nonelectronic groups and nominal and interacting groups were compared in a 2 x 2 factorial design. Electronic groups were more productive than nonelectronic groups, but the productivity of nominal and interacting groups did not differ. In contrast, interacting groups felt better about the idea-generation process than did nominal groups. Ways in which electronic brainstorming can reopen a long dormant area of research and application are discussed.
This paper examines a new genre of behavioral markets—"prediction" markets—and their remarkable ability to flush out and thereafter aggregate inside and expert information regarding interest rates, exchange rates, inflation rates, stock prices, commodity prices, and many other economic and financial variables. Comprehensive studies of these markets have found that these markets have "proven to be uncannily accurate in predicting all types of events." Existing in cyberspace and being unregulated, these markets are, arguably, the most efficient financial markets in history.
This study explores the ability of groups to forecast and establish judgmental confidence intervals in time series extrapolation. Thirty-six three-person groups were used to evaluate four different group interaction processes. In addition to staticized, nominal group technique and consensus processes, the study utilizes a modified consensus process, where a selected group member completes the task prior to group discussion and interaction. Using real life time series, subjects produced forecasts and related confidence intervals for six periods.Groups in the modified-consensus structuring process exhibited significantly greater forecast accuracy than all other experimental conditions (p < 0.001). The superiority was most pronounced for series of high forecast difficulty. These results are discussed in relation to the contribution of the initial estimates as an anchor on which the modified-consensus group can focus.
As might be expected, surveys show accuracy is the most important criterion in selecting a forecasting strategy. This may lead to the expectation that computer based software would be used to aid the forecasting effort. However, despite the wide availability of forecast software, management judgement appears to be the preferred method. This leads to the question: does management judgement provide the most accurate forecast estimates? This paper reports a field study of judgemental sales forecasting over thirteen manufacturing organisations to investigate whether these forecasts are accurate, unbiased and efficient, and whether better ex ante forecasts could have been developed using a computer based model. The study shows that the company forecasts were not uniformly more accurate than a simple, un-seasonally adjusted, naı̈ve forecast. Most of the source of error is due to both inefficiency (a serial correlation in the errors) and bias in the forecasts. These two factors seemed to mask any contribution of contextual information to accuracy. The results are also discussed in terms of forecasting objectives the organisations may have other than accuracy.
No one came closer to predicting the outcome of the 2004 U. S. presidential election than the team at, also called They tell us how they did it and whether they think they can do it again. Copyright International Institute of Forecasters, 2005
The effectiveness of nominal, Delphi, and interacting group decision making pro-cesses FORESIGHT for your library Unbeatable library subscription rates on a publication for the practicing forecaster that won’t just take up shelf space
  • Van
  • A H Ven
  • A L Delbecq
Van de Ven, A. H. & Delbecq, A. L. (1974). The effectiveness of nominal, Delphi, and interacting group decision making pro-cesses, Academy of Management Journal, 17, 605-621. FORESIGHT for your library. Unbeatable library subscription rates on a publication for the practicing forecaster that won’t just take up shelf space. $195 annually www. foresight Hard copy.