How to Make Better Forecasts and Decisions: Avoid Face-to-Face Meetings

Abstract

When financial columnist James Surowiecki wrote The Wisdom of Crowds, he wished to explain the successes and failures of markets (an example of a "crowd") and to understand why the average opinion of a crowd is frequently more accurate than the opinions of most of its individual members. In this expanded review of the book, Scott Armstrong asks a question of immediate relevance to forecasters: Are the traditional face-to-face meetings an effective way to elicit forecasts from forecast crowds (i.e. teams)? Armstrong doesn't believe so. Quite the contrary, he explains why he considers face-to-face meetings a detriment to good forecasting practice, and he proposes several alternatives that have been tried successfully.

Full text

Electronic copy available at: http://ssrn.com/abstract=988198Electronic copy available at: http://ssrn.com/abstract=988198
Fall 2006 Issue 5 FORESIGHT 3
HOW TO MAKE BETTER FORECASTS AND DECISIONS:
AVOID FACE-TO-FACE MEETINGS by J. Scott Armstrong
PREVIEW: When financial columnist James Surowiecki wrote The Wisdom of Crowds, he wished to explain the
successes and failures of markets (an example of a “crowd”) and to understand why the average opinion of a crowd is
frequently more accurate than the opinions of most of its individual members. In this expanded review of the book, Scott
Armstrong asks a question of immediate relevance to forecasters: Are the traditional face-to-face meetings an effective
way to elicit forecasts from forecast crowds (i.e. teams)? Armstrong doesn’t believe so. Quite the contrary, he explains
why he considers face-to-face meetings a detriment to good forecasting practice, and he proposes several alternatives
that have been tried successfully.
J. Scott Armstrong, Professor of Marketing at the Wharton School, University of Pennsylvania, was a
founder of the Journal of Forecasting, International Journal of Forecasting, and International
Symposium on Forecasting. He is the creator of forecastingprinciples.com, and editor of Principles
of Forecasting (Kluwer 2001), an evidence-based summary of knowledge on forecasting. In 1996, he
was selected as one of the first six Honorary Fellows by the International Institute of Forecasters. Along
with Philip Kotler and Gerald Zaltman, he was named the Society of Marketing Advances’ Distinguished
Marketing Scholar of 2000. For the past 13 years, he has been writing Persuasive Advertising: An
Evidence-Based Approach, which he forecasts will appear in 2007.
SHOULD THE FORECASTING
PROCESS ELIMINATE
FACE-TO-FACE
MEETINGS?
• In The Wisdom of Crowds, Surowiecki claims that the collective thinking of many individuals, acting alone,
contains wisdom. He concludes that traditional face-to-face meetings, which prevent forecasters from acting alone,
yield poor decisions and inaccurate forecasts.
• We do have guidelines on how to run meetings effectively, but it is rare to find group leaders who use them. Yet
traditional meetings persist partly because people falsely believe that they are necessary for aggregating opinions.
• There are better alternatives, including Markets, Nominal Groups, and Virtual Teams. I describe how these work and
report the evidence on their value.
• Some argue that face-to-face meetings are needed in certain contexts, such as when it is important to gain
commitment to decisions, but they have no evidence to back up these assertions.
SPECIAL FEATURE

Electronic copy available at: http://ssrn.com/abstract=988198Electronic copy available at: http://ssrn.com/abstract=988198
4 FORESIGHT Issue 5 Fall 2006
Introduction
Every week I hear people complain about meetings.
What would happen to your organization if it
became difficult to have face-to-face meetings?
To this end, some organizations hold meetings in rooms
without chairs. Some impose limits on the length of the
session or the number of people who attend.
But what if an organization went further and penalized
people for spending time in meetings? Or required that
the meeting have a clear-cut payoff? As part of assessing
the results, management could provide a visible taxi-
style meter that uses attendees’ billing rates to show the
meeting’s costs. Or what if management abolished face-
to-face meetings entirely?
The Wisdom of Crowds
I have been thinking about the need for face-to-face meetings
for some time now. Recently, I have been spurred on by
The Wisdom of Crowds (Surowiecki, 2004), a delightful yet
exasperating book. It is delightful because the writing is
so clever and contains descriptions of interesting research
studies, many of which were new to me; it is exasperating
because it is not well organized, but the writing is so clever
that one may not notice the gaps in logic. Nevertheless, the
book’s major conclusion is important:
Traditional meetings yield poor
decisions and inaccurate forecasts.
Dave Barry summarized this conclusion in fewer words:
“If you had to identify, in one word, the reason that the
human race has not achieved, and never will achieve, its
full potential, that word would be meetings.” Apparently
Barry’s quote hit a nerve; a Google search for his conclusion
turned up almost 600 relevant sites (out of 10,000 total
sites) in July 2006.
The term “crowds” in the title of Surowiecki’s Wisdom
of Crowds is unfortunate. He claims that the collective
thinking of many individuals, when acting alone, contains
wisdom. Crowds act together, and they do not have wisdom.
A more descriptive title would have been “The Superiority
of Combined Independent Anonymous Judgments.”
The book has been widely reviewed on Amazon, with
comments from over 200 readers who have provided
a bimodal ratings distribution. The negative reviewers
fell into two classes: those who were upset at the basic
conclusions and those who were upset at the gaps in logic.
The experts priced this book at $25, but the crowd’s price
for a new copy in May 2006 was $10. If you enjoy books
like Who Moved my Cheese? and Jack Welch’s Winning,
you are unlikely to enjoy The Wisdom of Crowds. But it will
make you reconsider your assumptions about meetings. At
least it had that strong effect on me.
Face-to-face Meetings Could Be Effective
We do have guidelines on how to run meetings effectively.
This was well summarized over four decades ago by
Norman R. F. Maier. His research showed how group
leaders could make effective use of people’s information.
His book (Maier, 1963) provides evidence-based principles
for running meetings. Exhibit 1 provides a summary of
guidelines that draws heavily on Maier’s research.
Unfortunately, it is rare to find group leaders who use
Maier’s advice. In my 46-year career, I can remember only
Exhibit 1.
GUIDELINES FOR PROBLEM-SOLVING MEETINGS
• Use time budgets. Allocate time to discuss various topics and provide ample slack time.
• Be problem centered. Keep your discussion focused on a problem. Avoid looking for excuses or seeking to blame others.
• Record suggestions. Keep track of all suggestions for solving a problem or making sense of an issue so that each
suggestion may be explored fully.
• Explore. Explore a number of suggestions for addressing an issue. Probing and evaluative questions can then be asked.
How would that strategy work out? Do I understand the issue, or do I need to search out more information? Am I
mistaken in my assumptions about the issue? What are the advantages or disadvantages of each proposal? Is there a
way to combine suggestions to generate a better solution?
• Protect people. Protect individuals from personal attacks and criticism, especially if they present minority or divergent
viewpoints. Avoid saying, “That’s a bad idea.”
• Understand and resolve differences. Once ideas have been generated, encourage dissent. Understand differences of
opinions within the group and attempt to resolve them.

Fall 2006 Issue 5 FORESIGHT 5
a handful of business students, academic administrators,
or business executives who have run meetings effectively.
Productive meetings are possible but rare.
Why do people persist in holding face-to-face meetings?
First, we are social animals; many of us enjoy the interaction
with others in a face-to-face setting. Second, managers like
the control that meetings give them over others; they can
see that others are coming together at their commands.
Third, people believe that meetings are effective; managers
believe that they are doing something useful when they
meet (although they often do not have the same opinion
about meetings among their blue-collar workers).
A fourth reason is that people falsely believe that by merely
aggregating opinions without a face-to-face meeting, one
would get a decision or forecast that is only average. The
scientist Sir Francis Galton dispelled such a belief in 1878.
He showed that by averaging portraits of women, the
resulting portrait was judged not average looking but rather
more beautiful than all the component portraits. Larrick
and Soll (2006), in a clever series of experiments, showed
that among highly intelligent subjects (MBA students at
INSEAD), most did not understand that the error of the
group-average judgment is almost always smaller than the
error of the average person in a group. More surprising to
them was that the group-average judgment is sometimes
better than the best judgment.
The Case Against Face-to-Face Meetings
Face-to-face meetings are expensive to schedule and run.
They might involve travel costs or come at inconvenient
times, when attendees are busy or tired. Time is wasted
when people come late, talk about irrelevant topics, or
leave early.
Meetings are also subject to many types of biases. How
loudly do people talk? How deep are their voices? What do
the people look like? How is the furniture arranged? How
are people dressed? What is each person’s body posture?
Who has the power? How does the group leader guide the
meeting? Does the group nurture dissent? Do people have
preconceived positions on the topic at hand?
Some attendees are so concerned about what they want to
say that they do not listen to what others are saying. Some
are so intent on listening that they have no time to think.
Some feel the need to restate their positions. Few people
take notes; therefore they soon forget what happened in the
meeting and are unable to develop useful action plans.
Not surprising then is the prevalence of studies showing that,
compared with other methods of aggregating opinions (such
as using the average of a set of independent judgments),
the simple act of meeting face-to-face harms forecasting
and decision making, although the people involved in these
experiments typically do not believe the results.
Interestingly, the findings for forecasting and decision
making are similar to those studies that involve groups
generating creative ideas. As shown in the research review
by Gallupe et al. (1991), individuals produce more creative
suggestions than groups do, even if the groups are well run.
There are two conditions under which independent
judgments should be combined. First, the experts must have
useful information about the topic of interest; combining
ignorance does not lead to wisdom. Second, participants
must represent diversity of knowledge. The key word is
“knowledge.” For example, it makes little sense to include
experts because of differences in looks, heights, weights,
religions, races, genders, and so on. In fact, Stewart’s (2006)
meta-analysis of 26 tests found a small negative relation
between team members’ demographic heterogeneity and
group performance.

6 FORESIGHT Issue 5 Fall 2006
Decision making and forecasting can be improved to the
extent that
• people state opinions independently, and
• opinions are aggregated objectively, using
a predetermined mechanical scheme.
The implication of the above research is that managers need to
be creative in finding ways to use the knowledge effectively
in a group while preventing members from meeting face-
to-face. This will improve forecasting and decision making.
It will also save time and money. Fortunately, modern
technology has provided useful alternatives.
Alternatives to Face-to-Face Meetings:
Markets, Nominal Groups, and Virtual Teams
There are a number of ways to implement alternatives
to face-to-face meetings. I will discuss three: markets,
nominal groups, and virtual teams.
Markets (Prediction Markets, Information Markets, or
Betting Markets)
Experts and nonexperts alike bet on outcomes. These
markets are common in finance and sporting events.
People receive feedback only through prices and volume
of trading.
In The Wisdom of Crowds, Surowiecki describes the use of
markets for prediction. Their superiority has been shown
by studies in financial markets since the 1920s. Although
investors do not meet, they observe the outcomes of
actions by others and draw on related information to
make their decisions.
Outside of finance and sports, there have been few
comparative studies on the value of prediction markets.
The future looks promising, however. Surowiecki reports
that some companies are using prediction markets for
new-product sales. Since predictions for such problems
are typically made in traditional meetings, I would expect
prediction markets to produce more accurate forecasts.
I hope that The Wisdom of Crowds will lead some companies
to consider the use of prediction markets. Technology
should not pose a barrier. Some organizations will delegate
a person to set up a betting market for sporting events.
Nominal Groups (Including Delphi)
In nominal groups, judgments are collected from a group
of experts and are summarized by a group facilitator.
Surowiecki relied on suggestive and interesting (but
indirect) evidence on the value of nominal groups. He
failed to effectively use the wisdom of the crowds of
forecasting researchers in his search for evidence on the
value of simply combining judgments. In Armstrong
(2001), I summarized 11 comparative empirical studies
on the value of combining judgmental forecasts, which, to
my knowledge, was an exhaustive listing of such studies.
The median error of the group average in these studies was
12.2% less than that of the average expert’s error.
The Delphi technique goes beyond nominal groups. It
involves an anonymous collection of expert judgments
by mail, Internet, or written responses. Feedback on the
responses is provided to the experts, who then repeat this
exercise for at least two rounds.
Rowe and Wright (2001) found that Delphi improved
accuracy over traditional groups in five of the studies,
harmed accuracy in one, and was inconclusive in two.
Using an alternative benchmark, they found that it was
more accurate than one-round expert surveys for 12 of 16
studies, with two ties and two cases in which Delphi was
less accurate. For these 24 comparisons, Delphi improved
accuracy in 71% of the cases and harmed it in 12%. I
was unable to find any published studies that compared
prediction markets with Delphi.
Freeware for Delphi is provided at forecastingprinciples.
com. Usage of this freeware has increased substantially in
recent years.
Virtual Teams
Virtual teams, enabled largely by the Internet, have several
advantages. They allow for a freer flow of information than
do markets or nominal groups. Members of virtual teams
can use mail, e-mail, and Web sites. Phone calls are used
only in emergencies, and conference calls are not used.
These procedures remove some biases (for example, body
language and modes of speech). They allow time for people
to think before responding, and they provide a record of
what was accomplished
Despite the growing popularity of virtual teams, I was
unable to find comparative studies on the value of these
groups. However, based on related research summarized
by Surowiecki, I expect that virtual teams would be much
more effective than face-to-face groups, but less effective
than prediction markets and Delphi. Consistent with this,

it is important to gain commitment to decisions. With
respect to the third condition, one must be concerned
not only with the quality of a decision but also with its
acceptability. Would the feeling of involvement in a
decision more likely lead to acceptance when the group
has made a forecast or decision?
Some papers have suggested that meetings are useful
when the situation is complex and the solutions are not
obvious. While this suggestion has some intuitive appeal,
tests of this concept have failed, according to Dennis and
Kinney (1998). I doubt that such meetings are effective,
given the evidence that (1) people can understand complex
material better and faster when it is written (Chaiken &
Eagly, 1976); (2) people in groups are poor at generating
creative approaches; (3) many participants have difficulty
performing complex analyses in the presence of others;
and (4) groups are not tolerant of creative solutions.
Although I have circulated my paper for comments from
e-mail lists and from other researchers, I have been
unable to obtain evidence to support the use of face-to-
face groups under these or any other conditions. Some
people have responded with their opinions that meetings
are useful or that many managers like meetings. There
was one paper that provided promising results for face-
to-face meetings, but the findings were not clear. Some
people responded that they could not think of evidence
favoring face-to-face meetings.
Such sessions may meet people’s needs for socializing.
Magne Jørgensen (personal communication) mentioned
one company that did away with face-to-face meetings
for their projects, replacing them with e-mail messages.
To satisfy people’s needs for meeting and talking, they
sponsored social events.
Action Steps
Perhaps the first step is damage control. Reduce the
number of meetings, the length of meetings, and the
number of people invited. Post a chart on the group’s
homepage to track the people-hours (and their associated
costs) that are consumed by the meetings. Ask the group
leader to use Maier’s guidelines for meetings. In addition,
ask attendees to summarize the actions they have taken
after each meeting.
If people in your organization do not know how to respond
without meetings, you can bring them together in a room
Fall 2006 Issue 5 FORESIGHT 7
Ganesan, Malter, and Rindfleisch (2005), in a study on
new-product development, found that e-mail was superior
to face-to-face meetings with respect to new-product
creativity and development speed.
A Prediction Case
Can you predict the results of the following experiment?
To solicit useful feedback on research studies, a group of
160 experts was provided with research papers, one paper
per expert. The experts were randomly divided into two
treatment groups. In group A, ten sets of eight experts
participated in 80-minute meetings, where authors of the
ten studies presented their papers and addressed questions.
(Each group heard only one study.) In group B, each subject
in the nominal groups of eight experts worked alone and
without interruption for 80 minutes on one of the ten papers.
These experts wrote comments in the margins of the papers.
In effect, the intent was to have equal amounts of time spent
on each paper. Which treatment, A or B, produced more
useful suggestions? In which treatment did the authors of
the study use the suggestions more effectively?
Unfortunately, there is little research to establish which of
the mechanical methods of combining expert judgments
are most creative, most accurate, least expensive, and most
acceptable. For example, I have found that no published
empirical comparisons have been made among prediction
markets, Delphi, and virtual teams. In fact, the above
study has not been conducted. Based on related research,
however, I assume that Treatment B (nominal groups)
would be superior to Treatment A (traditional groups) in
terms of producing useful, accurate, and creative ideas. I
also assume that, in Treatment B, the acceptance rate by the
authors of the papers would be much higher.
Are Face-to-Face Meetings
Useful Under Some Conditions?
The evidence against face-to-face meetings is extensive,
and I have made no attempt to provide a complete summary
here. I did, however, attempt to contact all authors whose
work I cite in order to ensure that I have referenced the
information properly. My primary concern is to find
evidence that favors face-to-face meetings.
Are there conditions under which meetings contribute
to forecasting or decision making? I speculate on three
possibilities. The first is when the experts cannot read.
The second is when very small groups, perhaps two
people, may be able to work effectively. The third is when

8 FORESIGHT Issue 5 Fall 2006
Dennis, A. R. & Kinney, S. T. (1998). Testing media richness
theory in new media: The effects of cues, feedback, and task
equivocality, Information Systems Research, 9(3), 256-274.
Gallupe, R. B., Bastianutti, L. M. & Cooper, W. H. (1991).
Unlocking brainstorms, Journal of Applied Psychology, 76(1),
137-142.
Ganesan, S., Malter, A. J. & Rindfleisch, A. (2005). Does
distance still matter? Geographic proximity and new product
development, Journal of Marketing, 69, 44-60.
Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining
opinions: Misappreciation of the averaging principle, Manage-
ment Science, 52, 111-127.
Maier, N. R. F. (1963). Problem Solving Discussions and Con-
ferences. New York: McGraw Hill. (Out of print, but used copies
are available.)
Rowe, G. & Wright, G. (2001). Expert opinions in forecast-
ing: The role of the Delphi technique, in J. S. Armstrong (Ed.),
Principles of Forecasting, Boston: Kluwer Academic Publishers,
125-144.
Stewart, G. L. (2006). A meta-analytic review of relationships
between team design features and team performance, Journal of
Management, 32, 29-54.
Surowiecki, J. (2004). The Wisdom of Crowds, New York:
Doubleday.
Valacich, J. S., Alan, R. D. & Connolly, T. (1994). Idea genera-
tion in computer-based groups: A new ending to an old story,
Organizational Behavior and Human Decision Processes, 57,
448-467.
Acknowledgements: Useful suggestions were provided by Mon-
ica Adya, Fred Collopy, Kesten Green, and Magne Jørgensen, as
well as by Foresight editors.
Contact Info:
J. Scott Armstrong
The Wharton School, University of Pennsylvania
Armstrong@wharton.upenn.edu
and then use structured procedures that simulate nominal
groups, as described by Aiken and Vanjani (2003). For
example, you could ask for a short “time-out” during
a meeting and ask everyone to write his or her ideas.
Software is available for conducting structured meetings,
and these products have proved useful (Valacich et al.,
1994) and have been gaining acceptance in organizations.
For example, Briggs et al. (1998) reported that electronic
brainwriting (individual idea generation) has been used by
several million people in over 1,500 organizations around
the world.
Individuals can also take action. My approach is to ask the
person who calls a meeting to describe the problem and
to inquire whether it would be useful to ask participants
to provide written suggestions rather than to attend the
meeting. The leader nearly always says yes and takes my
proposal in a positive way. This approach makes it easier
for people to absorb my suggestions and my reasoning
while it reduces their desire to argue against me (because
I am not there).
Conclusions
We rely heavily on face-to-face meetings, which are more
expensive than alternative approaches, even though it is
difficult to find evidence that supports their use. Although
evidence-based principles exist for running face-to-face
meetings effectively, they are used so rarely that we
must turn to more practical solutions. In fact, a pattern of
evidence suggests that prediction markets, nominal groups,
and virtual teams allow for a more effective use of a group’s
collective wisdom. Technology has enhanced the value of
these approaches.
References
Aiken, M. & Vanjani, M. B. (2003). Comment distribution in
electronic poolwriting and gallery writing meetings, Communi-
cations of the International Information Management Associa-
tion, 3(2), 17-36.
Armstrong, J. S. (2001). Combining forecasts, in J. S. Armstrong
(Ed.), Principles of Forecasting, Boston: Kluwer Academic
Publishers, 417-439.
Briggs, R.O., Nunamaker, J. F. Jr. & Sprague, R. H. Jr. (1998).
1001 Unanswered Research Questions in GSS, Journal of Man-
agement Information Systems, 14(3), 3-21.
Chaiken, S. & Eagly, A. H. (1976). Communication modality as
a determinant of message persuasiveness and message compre-
hensibility, Journal of Personality and Social Psychology, 34,
605-614.

Fall 2006 Issue 5 FORESIGHT 9
Introduction
I
always welcome Scott Armstrong’s contributions to the
discourse on forecasting. He has contributed more to our
understanding of the field than almost anyone else in
recent decades. In responding to James Surowiecki’s book The
Wisdom of Crowds, Scott considers the ubiquitous forecasting
meeting. He concludes that “the simple act of meeting face-
to-face harms forecasting and decision making.”
We have all attended unproductive meetings. But is it the
sessions themselves that are the problem or the inefficient
way they are run? Scott addresses this question by referring
to some guidelines from Norman Maier. Would meetings
still be useless if leaders followed Maier’s advice? Could
we make forecasting meetings more productive?
Drawing on my experience in attending many sales
forecasting meetings, I wish to suggest:
Forecasting meetings are really NOT about forecasting.
It might be termed a “forecasting meeting,” and its
identifiable output might be a final forecast, but the real
benefit is something quite different. I believe that meetings
have at least five purposes.
The Five Purposes of
Sales Forecasting Meetings
First, a meeting usually produces a final forecast that is
used by operations, finance, and procurement groups.
Second, a meeting produces a commitment to act. The
organization’s schedule for planning, procurement,
workforce numbers, and production are all based on this
commitment. Management must foster a sense of ownership
among members of the forecasting team.
Third, a meeting provides the important benefit of information
sharing. For example, a session of a major Mexican foods
producer considered the sales forecast for salsa. Based on
time-series information and the knowledge of existing
contracts, the product manager suggested a reasonable
estimate. The production manager then commented that
he was having difficulty in sourcing quality tomatoes, and
the purchasing manager questioned whether the new salsa
bottles would be available. On the basis of this input, the
final forecast was very different from the original.
Fourth, a meeting allows stakeholders the opportunity to be
heard. For example, a major global manufacturer of health
and hygiene products needed to revise a sales forecast.
The meeting was attended by the product manager, the
warehouse manager, the sales director, the production
manager, and the marketing director. The product manager
suggested a forecast for one of her major products. The
warehouse manager jumped up and exclaimed that he
refused to accept this forecast. His end-of-year bonus
was based on the value of the stock in his warehouse,
and he believed that the forecast would hurt his chances
of earning the bonus. The forecast was not adjusted, but
the contentious quantity of product was transferred out
of the warehouse. With an increasing emphasis on key
performance indicators (KPIs), the ability to be heard
becomes increasingly important.
Fifth, a meeting should result in a forecast that is linked to
the planning and budgeting cycle. Some might say that this
linkage takes us out of the realm of forecasting. But we
are essentially estimating, planning, and budgeting for the
same phenomena. This reconciliation process can be quite
tricky. To reflect plans and budgets, managers often adjust
forecasts during meetings. The March meeting of a major
cosmetics manufacturer dealt with the sudden realization
that to accept the next quarter’s forecast would mean that
the company would be underachieving its budget by some
20%. So the product manager was told to bring back an
“acceptable” forecast. Management also implied that the
revised forecast needed to be “close to the mark!”
COMMENTARY: FORECASTING MEETINGS ARE REALLY NOT
ABOUT FORECASTING by Marcus O’Connor
Marcus O’Connor is a Professor
of Business Information Systems
at the University of Sydney.
His interest in forecasting was
stimulated while engaging in his
passion for surfing. Why could
other surfers pick the right wave
to ride when he couldn’t? He has
since spent many years exploring
the ability to engage with the future. Chiefly, that has involved
sitting in as an observer in sales forecasting meetings. He
enjoyed that task immensely and believes he now understands
some of the main drivers of those meetings. But, sadly, his
ability to pick the right wave has never improved.

10 FORESIGHT Issue 5 Fall 2006
Despite these five functions, some claim that the forecasting
process can be implemented through e-mail. My contention is
that a meeting gives us a better opportunity to develop ideas
sequentially, to share information, and to be creative. This is
an empirical proposition that deserves attention and research.
What Does the Research Show?
Forecasting meetings might be seen as a waste of time.
They may not produce more accurate forecasts. In fact,
there is some evidence that meetings compromise forecast
accuracy (Lawrence et al., 2000). However, a close look at
some of this research suggests that the forecasts were better
characterized as targets. In these cases, there was a strong
incentive to skew the forecasting numbers based on the cost
function appropriate to the business.
Scott Armstrong contends that “the research findings (both
direct and indirect) support the conclusion that people should
not meet face-to-face.” He cites some of the research on the
benefits of averaging and bootstrapping, and he touts the lure
of prediction markets. (Incidentally, how would a prediction
market be used in a sales-forecasting environment?) I do not
wish to address this research, but I will respond to Scott’s
challenge that we nominate contrarian views and research. I
would like to mention two articles of interest. First, in Ang &
O’Connor (1991), we compared four consensus forecasting
techniques used in meetings:
• a forecasting meeting in which a consensus forecast
was accepted,
• the same meeting, but a participant had produced his or
her own forecast prior to the session,
• a simple average of individual estimates (which Scott
seems to recommend), and
• a nominal group technique (another Scott-
recommended method).
One technique seemed superior: the forecasting meeting in
which one attendee had produced his or her own forecast
prior to the meeting. This was especially true for high-
difficulty time series. Moreover, we were able to isolate
the reason for the superior forecasts. The improvement
came from the discussion in meetings! This technique was
actually borrowed from an international organization that
produces household consumer products.
The second piece of research is relatively recent.
Rockenbach et al. (2006) were concerned with the
optimality of team decisions in an investment setting (a
guided meeting, in their case). These authors compared
team decisions with individual decisions. The results from
the meetings were more consistent with portfolio selection
theory, and the team decisions accumulated surprisingly
greater value at a significantly lower risk. Perhaps some
meetings are useful.
Scott cites and extends the work of Norman Maier,
suggesting six principles for the conduct of effective
meetings: (1) use time budgets; (2) be problem centered;
(3) record suggestions; (4) explore; (5) protect people;
and (6) understand and resolve differences. The last two
principles address why forecasting meetings are needed.
The real problem is the way that meetings are conducted.
Ineffective meetings lack definable objectives, have
ambiguous outcomes, or lack definable KPIs. Sales
forecasting meetings have a number of key advantages.
They have well-defined objectives. Simple validation
KPIs can be applied to the output. These meetings are
held regularly, so continual improvements can be made
to the structuring process. But I recognize the problems
associated with the lack of a good meeting chairperson.
Also, there is a tendency to get bogged down while
producing forecasts for products for which a statistical
method would have sufficed.
I welcome the debate on the efficacy of sales forecasting
meetings. To my mind, these meetings have important
organizational and social consequences. Could these live
sessions be replaced by e-mail messages or virtual groups?
Perhaps. But e-mail has become a major burden for many
organizations. Some firms are even configuring their
e-mail systems to restrict employee access.
References
Ang, S. & O’Connor, M. (1991). The effect of group interaction
processes on performance in time series extrapolation, Interna-
tional Journal of Forecasting, 7, 141-149.
Lawrence, M., O’Connor, M. & Edmundson, R. (2000). A field
study of sales forecasting accuracy and processes, European
Journal of Operations Research, 122, 151-160.
Rockenbach, B, Sadrich, A. & Mathauschek, B. (2006). Teams
take better risks, Journal of Economic Behavior and Organiza-
tion, at press.
Contact Info:
Marcus O’Connor
University of Sydney
m.oconnor@econ.usyd.edu.au

Fall 2006 Issue 5 FORESIGHT 11
COMMENTARY: A DEPERSONALIZED INTERACTIVE PROCESS
IS THE KEY by Joe Smith
Joe Smith is the Director of Forecasting
and Revenue Management Solutions
fo r Coca-C ola Ent erpri ses Inc .
North American Business Unit.
He has successfully facilitated the
design and implementation of CCE’s
standard, internally collaborative
forecasting system. Joe’s career at
CCE spans over seventeen years
with experience in strategic planning,
category management, promotion control and sales
analysis. Joe graduated from Eastern Michigan University
in 1987 with a BBA in Finance with a concentration in
investment analysis. His passion for applying mathematics,
planning processes and consensus building has led him
to his most recent pursuits in building enterprise-level
planning support systems.
When I was approached by the editor of
Foresight to write a book review, I was more
than happy to participate. I was told that the
book was The Wisdom of Crowds, by James Surowiecki.
I logged onto Amazon and ordered it the same day. As a
practicing business professional serving as the director of
forecasting and revenue management solutions for a $20
billion corporation, and as one who is an active reader of
business books, I thought this book would be a really good
fit for me. I am constantly challenging my team to convert
academic ideas into pragmatic policies for the company.
Most companies have little tolerance for new theories until
they are successfully applied. So I set out not only to write
a review but also to flush out the implications of the book
for our company processes.
My organization, Coca-Cola Enterprises, has been actively
transforming its approach to forecasting and business
planning. In our own way, we have been leveraging “the
wisdom of crowds” by creating an internally collaborative
forecasting process. We request input from over two
thousand employees every week – key account managers,
distribution center managers, and business unit staff.
We centrally aggregate this input into official forecasts.
We continually strive for greater levels of participant
cooperation, although I must confess that obtaining such
cooperation is always a challenge!
The Wisdom of Crowds brings to life the well-understood
precept that “two heads are better than one” – or that
many heads are better than one. Of course these sayings
are true if you can figure out how to get the heads to
communicate. Surowiecki recognizes the significance of
this challenge, and he provides dozens of examples of how
organizations have improved internal communication. The
common thread of the solution is getting a diverse group
of knowledgeable, cooperative, and strongly opinionated
people to provide their respective points of view. Then the
group’s collective opinion often will be better than that of
any single member, even the most expert.
This basic message should resonate with forecasting
practitioners. So how does an organization leverage
Surowiecki’s findings? The author does not provide
straightforward, step-by-step answers. If you are looking
for the practical explanation of how to apply Surowiecki’s
ideas, you are left to your own creativity.
But I think we can infer certain messages for our own
organizations. For my organization, The Wisdom of Crowds
imparts the following advice:
• The individual forecaster needs to demonstrate
what Surowiecki calls “local knowledge” and must be
resolute in presenting his or her point of view;
• Managers must then focus their efforts on
capturing the local knowledge of practitioners through a
nonthreatening forum;
• Capturing this knowledge requires continuous
practitioner participation in the forecasting process.
Our internal forecasting process encompasses thousands
of forecasters, all of whom are required to participate
on a weekly basis. That this is no simple chore is a
point vividly made by my colleague, Simon Clarke, in
the previous issue of Foresight (Clarke, 2006). Local
knowledge must be captured as a routine, compulsory
activity. Then we need support tools to compile, digest,
and share that knowledge.
Sharing local knowledge does not have to take much time;
however, it does require an interactive environment. The
forecasting team might conduct face-to-face meetings or
conference calls; the important point is that strong facilitation
is employed and that feedback is not personalized. Assuring
depersonalization is critical and merely requires that
discussions are focused exclusively on the facts.

12 FORESIGHT Issue 5 Fall 2006
COMMENTARY: BUSINESS OBJECTIVES, FORECASTERS
AND MEETINGS by Jamilya Kasymova and Catalin Vieru
Jamilya Kasymova is Manager of
Strategic Projects & Forecasting of
Global Reservation Sales and Customer
Care at Marriott International. She
holds a PhD in Applied Mathematics
from Novosibirsk State University,
Russia and an MBA from the University
of Nebraska at Omaha. She seems to
enjoy weather extremes, first from the
bitter cold of Siberia to the sweltering
days in Uzbekistan, and now the
extremes of Omaha – bitter winters, hot and humid summers
and tireless winds.
Catalin Vieru is Director of Information
Services of Global Reservation Sales
and Customer Care at Marriott
International. He has masters degrees
in Finance, Information Services, and
Economics, and currently works on
optimizing the handling of Marriott
customer contacts. Catalin moved from
Romania to Italy and from Italy to the
USA, ending up in Omaha. He says he
would like to meet and toast anyone
who, not being a native of the region, specifically wishes
to be in Omaha, Nebraska.
Many in our profession would agree that forecasters
are driven by two main objectives: accuracy and
usability. Based on a plethora of information
available, forecasters attempt to develop predictions that
are as accurate as possible. And they try to present their
forecasts in ways that are most useful to those responsible
for the organization’s planning and operation.
Scott Armstrong asks whether the fulfillment of these two goals
requires “traditional,” face-to-face forecasting meetings. Our
response is that the forecaster wears two hats, and traditional
meetings can be beneficial while wearing one of them.
The forecaster wears the statistician’s hat when she
analyzes historical data, chooses appropriate models, and
then uses these models to produce forecasts. In this role,
the forecaster doesn’t need meetings, and she will actually
seek to avoid them. After all, the forecaster has been hired
by the organization as an expert in the field.
But forecasting is not a pure science; it is also an art. There
are always elements of intuition and judgment incorporated
into the final forecast. The forecaster puts on a different hat
when she tries to gather all pertinent domain information.
This information is not always formalized in memos or
e-mail messages, and that is why meetings can be useful.
It is worth the forecaster’s time to attend meetings in
which (1) strategic planning is discussed, (2) marketing
plans are introduced, and (3) the pros and cons of new
Contact Info:
Joe Smith
Coca-Cola Enterprises Inc.
joesmith@na.cokecce.com
Surowiecki and Armstrong emphasize that transforming
local knowledge into the wisdom of the crowd requires
participant anonymity. In my experience, however, the
key is not anonymity so much as sharing information in
a cooperative environment. The participants in the crowd
need to learn from each other in order to improve their
forecasting capabilities. Like most large corporations, my
organization cannot afford to wait for anonymous answers.
The live interactive meeting is our catalyst. It would be
nice to leisurely await the natural evolution of a forecast
consensus, but waiting is not practical.
We employ any means possible to assure routine
live interaction with forecast participants, including
traditional face-to-face meetings, Web-based meetings,
and conference calls. Through these exchanges we
have established a compulsory forecasting discipline.
Our employees know that forecasting is not an optional
activity; it is part of the job.
Reference
Clarke, S. (2006). Managing the introduction of a structured
forecast process: Transformation lessons from Coca-Cola En-
terprises, Inc., Foresight: The International Journal of Applied
Forecasting, Issue 1, 21-25.
Editor’s Note: Readers from Omaha are invited to respond.

Fall 2006 Issue 5 FORESIGHT 13
product launches are presented. The traditional meeting
is the forum where forecasters can absorb other experts’
opinions and concerns, and where forecasters can refine
their understanding of the company’s operations. When the
forecaster attends a meeting to listen and learn, she is not
necessarily subjected to the litany of problems, including
intimidation, that Armstrong catalogs.
The forecaster needs managerial validation of the assumptions
that underpin her modeling, and meetings can foster cross-
business pollination. Meetings also allow the forecaster to
solicit feedback on her presentation of forecast outcomes;
for example, would management like to see optimistic,
pessimistic, and most-likely forecasts? Face-to-face meetings
are a practical, convenient way to share information.
The key participants in meetings request clarification,
question assumptions, and offer feedback on how the
forecast might affect their respective areas of operations.
The forecaster learns which projections the organizational
leadership needs—and which format is most useful to
management. By involving key players, the forecaster
is connected to the business’ synapses, and she gains a
realistic outlook, one that ensures that the forecast can be
applied to the firm’s operations.
To summarize, the forecaster as statistician is most effective
working alone. The forecaster as gatherer of key business
information will find value in face-to-face meetings. There
she will be able to perform the following tasks:
• Gathering nonformalized information,
• Validating the assumptions used for modeling,
• Ensuring that the forecasts are realistic and actionable and,
• Gaining credibility, earning trust, and enhancing
reputation and visibility within the organization.
Understandably, few organizations have the luxury of hiring
multiple, independent, diverse, and decentralized forecasters
and then averaging their projections into a “final” forecast.
In addition, the use of independent markets for designing
decisions and predicting business outcomes in organizational
settings is not feasible because the cost of maintaining such
markets would be prohibitive. In principle, if employees knew
that management would allow them to maintain some privacy
and to become independent agents, they might embrace such
a system. However, this approach would signal a major shift
in the business practices of most organizations.
Armstrong believes that forecasters’ abilities are hindered
by face-to-face meetings. This argument seems to presume
that forecasters will, after a traditional meeting, rush back
to their mathematical models and change their underlying
assumptions in order to satisfy the needs expressed at the
meeting. If that were true, the meeting should have taken
place before the model had been formalized. For example,
forecasting a particular financial situation without any
supporting assumptions from specific sales strategies can
be futile, as no one will give the forecast any credence. For
most forecasters, the credibility, confidence, and feedback
generated at meetings are all worthwhile. A forecaster cannot
sterilize statistical calculations from hard business facts.
Forecasting is a business function that provides a vital “public”
good to the organization. For this reason, the forecaster
cannot rely solely on private information. Meetings in which
forecasters play key roles can elevate forecasters from their
private world to the public function they need to serve.
Perhaps some meetings should be banned, but it is difficult
to determine which meetings are expendable. The forecaster
can decide whether or not to attend a particular meeting.
Let forecasters do what they do best, including shaping the
way they acquire and use information. They are hired to
produce accurate and usable forecasts. As icing on the cake,
forecasters are rewarded by being recognized as a source
of an organization’s foresight. The interactions in face-to-
face meetings can enable the evolution of the forecaster
from statistician-practitioner to forecaster-strategist.
Contact Info:
Jamilya Kasymova and Catalin Vieru
Mariott International
Jamilya.Kasymova@marriott.com

14 FORESIGHT Issue 5 Fall 2006
HOW PRACTITIONERS CAN USE EVIDENCE-BASED FINDINGS: REPLY
TO COMMENTARIES by J. Scott Armstrong
The commentaries reinforce my belief that research
evidence alone is not sufficient for organizations
to consider new methods. I suggest procedures to
facilitate the implementation of evidence-based findings.
Is there a need for more evidence?
Are there conditions under which face-to-face meetings
improve forecasting and decision making? The
commentators all provide opinions from their experiences,
but Marcus O’ Connor was the only one to offer evidence
from a research study. Ang and O’Connor (1991), in a
comparative study of 36 three-person student groups, found
that when the forecasting task was difficult, a procedure
whereby one person prepared forecasts prior to a face-to-
face meeting led to improved forecasts vs. a nominal group
process (in which face-to-face meetings did not occur).
Ang and O’Connor’s evidence conflicts with much of
the prior evidence. This makes it worthy of replications
or extensions that would allow for feedback within the
nominal group procedure and within Delphi, as well as
controls for reflection time. The prior evidence, much of
which has been around for decades, is not favorable to the
use of face-to-face meetings. For example, Van de Ven
and Delbecq (1974), based on a laboratory experiment,
concluded that nominal groups and Delphi were superior to
face-to-face meetings with respect to decision making. In
my judgment, the weight of the scientific evidence suggests
that face-to-face methods harm creativity, forecasting, and
decision making
Interestingly, research on persuasion suggests that
examples are more persuasive than scientific evidence
when people hold strong beliefs that are contrary to the
evidence. So here is my story. In developing a procedure
for forecasting the vote in political elections, I worked in a
virtual group consisting of myself and two others, neither
of whom I had met previously. For the first few months, all
communications were by e-mail, which provided a written
record of what was done by each of us. We eventually
had a conference call, mostly for social reasons. In my
experience, this is probably the most creative and efficient
group (defined as three or more people) with which I have
been involved. We produced a successful website, a near
perfect forecast of the U.S. Presidential election of 2004,
and received recognition in Foresight for forecasting
accuracy (Cuzán et al. 2005).
How can you implement research-based findings?
Some of the commentators reacted to my arguments by
giving their opinions on why their organizations’ current
procedures are optimal. How can one get around this
problem of resistance to research evidence?
Important changes in organizations should be under the
control of the decision makers who are affected. Thus, the
question might be framed, “What type of information (e.g.,
experimental, trial and error, or prior research) should we
obtain in order to decide when we can use alternatives to
face-to-face meetings?” Unfortunately, the commentaries
did not address this question.
There may be some value in using the “second solution”
technique, in which the decision makers are prohibited
from solving the problem as they currently do. Instead,
they must develop an alternative procedure. Once that is
done, the constraint is relaxed and they can compare the
new procedure with their original one. Maier and Hoffman
(1960), in a problem involving a change in employee work
procedures, found that solutions were of higher quality
when groups were instructed to find a second solution
after they had presumably solved the problem. The second
solutions were obtained in about two-thirds of the time
needed for the initial solutions, and the groups generally
preferred their second solutions to the first ones.
Restrict your consideration of alternative procedures to
those supported by comparative studies. Procedures that
have been tested fairly and found useful might be useful
for you as well. There are many such methods.
Joe Smith and Marcus O’Connor comment that prediction
markets – a major alternative to face-to-face meetings
recommended by Surowiecki and myself – are not feasible
for sales forecasting within organizations. However, Ray
(2006) mentions that Microsoft, Hewlett-Packard, Eli Lilly

Fall 2006 Issue 5 FORESIGHT 15
and other major firms use prediction markets; he also makes
suggestions on how to implement them. The proposal
to use nominal groups dates back at least three decades.
Detailed operational suggestions were provided in Delbecq
et al. (1975); the book also includes testimonials on the
successful application of the nominal group technique in a
health care organization and in a business, ARA Services. I
also believe that the Delphi technique is feasible – you can
go to Software on forecastingprinciples.com and obtain
freeware to guide you. My paper also provided operational
guidelines for conducting face-to-face meetings.
Jamilya Kasymova and Catalin Vieru comment that their
organization could not afford to adopt the procedures
recommended by Surowiecki and me. I believe this
to be contrary to the evidence. For example, based on
comparisons among 12-person groups, Gallupe et al.
(1992) found that electronic brainstorming groups produced
about three times as many unique ideas as did traditional
brainstorming groups. In any event, this issue can be easily
resolved; one has only to try alternative procedures and
monitor the costs.
References
Ang, S. & O’Connor, M. (1991). The effect of group interaction
processes on performance in time-series extrapolation, Interna-
tional Journal of Forecasting, 7, 141-149.
Cuzán, A., Armstrong, J. S. & Jones, R. J. (2005). How we
computed the Pollyvote, Foresight: The International Journal of
Applied Forecasting, Issue 1, 51-52.
Delbecq, A. L., Van de Ven, A. H. & Gustafson, D. H. (1975).
Group Techniques for Program Planning, Glenview, Ill.: Scott,
Foresman.
Gallupe, R. B., Dennis, A. R., Cooper, W. H., Valacich, J. S.,
Bastianutti, L. M. & Nunamaker, J. F., Jr. (1992). Electronic
brainstorming and group size, Academy of Management Journal,
35, 350-369.
Maier, N. R. F. & Hoffman, L. R. (1960). Quality of first and
second solutions in group problem solving, Journal of Applied
Psychology, 44, 278-283.
Ray, R. (2006). Prediction markets and the financial “Wisdom of
Crowds,” Journal of Behavioral Finance, 7, (1), 2-4.
Van de Ven, A. H. & Delbecq, A. L. (1974). The effectiveness
of nominal, Delphi, and interacting group decision making pro-
cesses, Academy of Management Journal, 17, 605-621.
FORESIGHT
for your
library.
Unbeatable
library subscription
rates on a
publication
for the practicing
forecaster
that won’t just
take up shelf space.
$195
annually
www.
forecasters.org/
foresight
Hard
copy.