ArticlePDF Available

Abstract and Figures

Simple surveys that ask people who they expect to win are among the most accurate methods for forecasting U.S. presidential elections. The majority of respondents correctly predicted the election winner in 193 (89%) of 217 surveys conducted from 1932 to 2012. Across the last 100 days prior to the seven elections from 1988 to 2012, vote expectation surveys provided more accurate forecasts of election winners and vote shares than four established methods (vote intention polls, prediction markets, econometric models, and expert judgment). Gains in accuracy were particularly large compared to polls. On average, the error of expectation-based vote-share forecasts was 51% lower than the error of polls published the same day. Compared to prediction markets, vote expectation forecasts reduced the error on average by 6%. Vote expectation surveys are inexpensive, easy to conduct, and the results are easy to understand. They provide accurate and stable forecasts and thus make it difficult to frame elections as horse races. Vote expectation surveys should be more strongly utilized in the coverage of election campaigns.
Content may be subject to copyright.
Accuracy of Vot e E xp ecta ti on S urve ys in Fo re cas ti ng E le ctio ns
Forthcoming (subject to changes) in Public Opinion Quarterly
Andreas Graefe
Center for Advanced Studies
Institute of Communication Studies and Media Research
LMU Munich, Germany
a.graefe@lmu.de
Abstract. Simple surveys that ask people who they expect to win are among the most
accurate methods for forecasting U.S. presidential elections. The majority of respondents
correctly predicted the election winner in 193 (89%) of 217 surveys conducted from 1932 to
2012. Across the last 100 days prior to the seven elections from 1988 to 2012, vote expectation
surveys provided more accurate forecasts of election winners and vote shares than four
established methods (vote intention polls, prediction markets, econometric models, and expert
judgment). Gains in accuracy were particularly large compared to polls. On average, the error of
expectation-based vote-share forecasts was 51% lower than the error of polls published the same
day. Compared to prediction markets, vote expectation forecasts reduced the error on average by
6%. Vote expectation surveys are inexpensive, easy to conduct, and the results are easy to
understand. They provide accurate and stable forecasts and thus make it difficult to frame
elections as horse races. Vote expectation surveys should be more strongly utilized in the
coverage of election campaigns.
Keywords: betting markets, citizen forecasts, combining forecasts, Condorcet’s jury
theorem, econometric models, election forecasting, expertise, FiveThirtyEight, forecast accuracy,
IEM, Iowa Electronic Markets, Occam’s Razor, political economy models, wishful thinking
Acknowledgments: J. Scott Armstrong, Alfred Cuzán, Michael Lewis-Beck, Andreas
Murr, and Christopher Wlezien provided helpful comments. I also received valuable suggestions
when presenting the manuscript at the 2013 Annual Conference of the International
Communication Association in London, the 2013 International Symposium on Forecasting in
Seoul, and the 2013 APSA Annual Meeting in Chicago. Bettina Zerwes helped with collecting
data. Jamie Graefe did editorial work.
“Who do you think will win the U.S. presidential election?”
Pollsters have regularly asked variations of this question since the 1940s. Yet, little is
known about the relative accuracy of such vote expectation surveys compared to alternative
methods for forecasting elections. Although researchers have long demonstrated that responses
to the vote expectation question, also known as “citizen forecasts”, provide accurate predictions
of election outcomes (Lewis-Beck and Skalaban 1989, Lewis-Beck and Tien 1999, Lewis-Beck
and Stegmaier 2011, Murr 2011), few studies compare their accuracy to established benchmarks.
The present study provides empirical evidence on the relative accuracy of vote
expectation surveys for forecasting U.S. presidential elections by comparing their results to
predictions from four established methods: traditional polls, prediction markets, expert judgment,
and quantitative models.
Established methods for forecasting U.S. presidential elections
As long as there have been elections, people have tried to predict their results. Currently,
the most common methods for election forecasting are polls, prediction markets, expert
judgment, and quantitative models.
Expert judgment
Judgment of political insiders and experienced election observers were used to forecast
elections long before the emergence of scientific polling (Kernell 2000); and they still are. The
common assumption is that experts have experience in reading and interpreting polls, assessing
their significance during campaigns, and estimating the effects of recent or expected events on
the aggregate vote. Given their omnipresence, surprisingly little is known about the relative
accuracy of experts’ judgment for election forecasting.
One study found that experts, in this case politicians, provided more accurate predictions
of the outcomes of two controversial ballot measures than voters who had just left the voting
booths (Lemert 1986). Another study compared the accuracy of experts (political scientists,
journalists, and editors) and non-experts in predicting vote-shares in the 2006 Swedish
Parliamentary election. At the individual level, the experts provided more accurate forecasts than
the non-experts. Interestingly, however, the results were reversed when calculating average (and
median) forecasts within each group: the combined forecasts from the non-experts were more
accurate than the combined forecasts from the experts (Sjöberg 2009).
Polls
Traditional polls ask respondents for whom they intend to vote if the election were held
today. That is, polls do not provide predictions; they provide snapshots of public opinion at a
certain point in time. However, this is not how the media commonly treat polls. Polling results
are routinely interpreted as forecasts of what will happen on Election Day. This can result in
poor predictions, in particular if the election is still far away, because public opinion can be
difficult to measure and fragile over the course of a campaign (Campbell 1996). However,
researchers found ways to deal with these problems and to increase the accuracy of poll-based
predictions by combining polls and projecting the results to Election Day.
Combining polls to reduce measurement error
There is often high variance in the results of polls by different survey organizations, even
if these polls were conducted at around the same time. Such variance can be caused by sampling
problems, non-responses, and faulty processing (Erikson and Wlezien 1999). Therefore, one
should not look at the results from single polls. Rather, one should combine polls that were
conducted at around the same time. The reason is that the systematic (and random) errors that are
associated with individual polls tend to cancel out in the aggregate (Graefe et al. 2014). The good
news is that combining has impacted how people nowadays consume polls and online polling
aggregators such as realclearpolitics.com and pollster.com have become increasingly popular.
Projecting poll results to Election Day
Polls conducted by the same survey organization, and by the polling industry as a whole,
can fluctuate widely across the course of a campaign. The reason is that a large share of the
electorate has not spent much time thinking about the important issues and the candidates’
positions if the election is still far away. As a result, people’s response behavior in early polls is
influenced by campaign events such as conventions (Campbell, Cherry, and Wink 1992) and
debates (Benoit, Hansen, and Verser 2003).
The effects of such events on the outcome of high-visibility elections such as U.S.
presidential elections are limited, however. As the election nears, people are less influenced by
the latest campaign events and have formed stable vote intentions based on a combination of
information gleaned during the campaign, such as the state of the economy, and their basic
predispositions, such as ideology and party identification (Gelman and King 1993). Therefore, it
is not until shortly before Election Day that polls provide accurate forecasts.
However, researchers found ways to utilize early polls for use in forecasting by
calculating poll projections, as they are termed hereafter. Poll projections take into account the
historical record of polls in order to make a forecast. For example, assume that the incumbent
leads the polls by fifteen points in August. In analyzing historical polls conducted around the
same time along with the respective election outcomes, one can derive a formula for translating
August polling figures into an estimate of the incumbent’s final vote share in November. This is
commonly done by regressing the incumbent’s share of the vote on his polling results during
certain time periods before the election. Prior research found that such poll projections are much
more accurate than treating raw polls as forecasts (e.g., Erikson and Wlezien 2008, Campbell
1996).
Calculating combined poll projections
One can also combine both strategies (i.e., combining polls and calculating poll
projections) to generate poll-based forecasts. One study first calculated rolling averages of all
polls that were published in a one-week period and then used these results to calculate poll
projections. This procedure resulted in large gains in accuracy. Across the last 100 days prior to
each of the six elections from 1992 to 2012, such combined poll projections reduced the error of
a randomly picked poll that was published the same day by 39% (Graefe et al. 2014).
Prediction markets
Prediction (or betting) markets allow people to bet on the election outcome. The resulting
betting odds can then be interpreted as forecasts. Such markets were already popular in the late
19th and early 20th century, when newspapers such as the New York Times regularly reported the
latest predictions. However, around the time of World War II, prediction markets began to
disappear, likely due to a combination of factors such as the rise of the polling industry in the
1930s, the introduction of laws to eliminate organized election betting, and the emergence of
alternative betting opportunities such as horse-racing (Rhode and Strumpf 2004). It took almost
half a century, and the rise of the Internet, for the method to be rediscovered. In 1988,
researchers at the University of Iowa launched the online Iowa Electronic Markets (IEM) to
predict the U.S. presidential elections held in the same year. Since then, interest in prediction
markets resurged.
Studies of prediction market accuracy for election forecasting commonly compare the
daily market forecasts to results from polls published the same day. These studies generally find
that prediction markets yield more accurate forecasts than single polls. For example, Berg,
Nelson, and Rietz (2008) compared the accuracy of IEM forecasts to results from nearly 1,000
polls across the five U.S. presidential elections from 1988 to 2004. The IEM forecasts were more
accurate than single polls 74% of the time. However, as outlined above, single polls provide poor
predictions and thus only serve as a weak benchmark. Erikson and Wlezien (2008) accounted for
this problem and compared the IEM forecasts to poll projections, using the same data and time
period as analyzed by Berg, Nelson, and Rietz (2008). The authors found that poll projections
were more accurate than the IEM. I extended these analyses and compared the IEM forecasts to
combined poll projections, using the approach suggested by Graefe et al. (2014). Across the last
100 days prior to each of the seven elections from 1988 to 2012, the IEM forecasts yielded an
average error of 1.7 percentage points, which is 17% below the corresponding error of combined
poll projections of 2.0 percentage points (cf. Table 3).
Quantitative models
A popular theory of electoral behavior is that elections are referenda on the incumbent’s
performance. That is, voters are expected to reward the government for good performance and
punish the incumbent party otherwise. Since the late 1970s, economists and political scientists
tested this theory by developing quantitative models to predict election results. Most models are
based on multiple regression analysis of two to five predictor variables, which typically capture
economic conditions, the incumbent’s popularity, and how long the President (or his party) have
controlled the White House. The development and testing of these models has become a well-
established sub-discipline of political science and the models’ forecasts are regularly published
about two months prior to Election Day in scientific journals.1
These models predict the correct election winner most of the time. Across the six
elections from 1992 to 2012, 34 of 39 forecasts of seven well-known models correctly predicted
the winner. However, the models’ performance in predicting vote shares is mixed. Their mean
1 Since the 1992 election, forecasts from established models were published prior to the respective elections in
special symposiums in Political Methodologist 5(2), American Politics Research 24(4) and PS: Political Science
and Politics 34(1), 37(4), 41(4), and 45(4).
absolute error (MAE) was three percentage points, and ranged from zero to ten points (cf.
Appendix I).
Vote expectation surveys
Why not simply ask voters whom they expect to win and then use the aggregate result as
forecast? A common concern with using expectations as forecasts is that people’s expectations
are influenced by their preferences. In the case of elections, this means that people tend to predict
their preferred candidate to win. This bias, which is known as wishful thinking, is long known to
be common in the context of elections. Hayes (1936) reported results from a 1932 pre-election
survey of an unrepresentative sample of 8,419 men and women and finds that 72% of Hoover
supporters predicted Hoover to win, whereas 91% of Roosevelt supporters predicted Roosevelt
to win; Roosevelt won by a landslide. In their seminal study of voting behavior, Lazarsfeld,
Berelson, and Gaudet (1948) reported a strong relationship between people’s vote intentions and
their expectations of who will win. Granberg and Brent (1983) studied wishful thinking across
the eight U.S. presidential elections from 1952 to 1980. Their analysis revealed a strong positive
relationship between expectations and candidate preferences (r=.8). Since these early studies,
evidence has accumulated. Wishful thinking occurs in all types of elections, from local referenda
to national elections, and across various countries. See Miller et al. (2012) for an overview of
recent research.
Although people are subject to wishful thinking, most of them can still correctly predict
the outcome of elections. In their pioneering study of “citizen forecasts”, Lewis-Beck and
Skalaban (1989) found that, across the eight U.S. presidential elections from 1956 to 1984, 69%
of respondents to the ANES vote expectation question correctly predicted the election winner. I
updated their results by analyzing data from all ANES pre-election surveys for the sixteen
elections from 1952 to 2012. I found that, on average, 70% of 30,573 respondents correctly
predicted the election winner. In addition, I analyzed the accuracy of predictions of which
candidate would win the election in the respondent’s state.2 For this task, 69% of 23,301
2 In 11 of the 16 ANES surveys from 1952 to 2012, respondents were asked to predict the election winner in their
state. The state-level question was not asked prior to the four elections from 1956 to 1968 as well as prior to the
2000 election.
responses predicted the correct winner.3 Others obtained similar results for multi-party elections
in the UK. Across thirteen elections conducted between 1951 and 2005, 60% of survey
respondents correctly predicted which party will win the governing majority of seats (Lewis-
Beck and Stegmaier 2011). Another study analyzed data from the 2010 British Election Study
Internet survey, in which respondents were asked to predict which party would win the election
in their local constituency. Of the 13,334 respondents that provided a valid answer to the
expectation question, 69% correctly predicted the winner (Murr 2011). See Table 1 for an
overview of these results.
The accuracy of individual vote expectations depends on the characteristics of the
forecaster and the electoral context. In their micro-level analysis of citizen forecasts of the eleven
ANES surveys from 1956 to 1996, Lewis-Beck and Tien (1999) showed that more educated
individuals make more accurate predictions and that political involvement harms accuracy.
Furthermore, individual forecasts are less accurate for tight races and if the election is further
away.
The available evidence reveals that the chance for a typical respondent to correctly
predict the election winner is usually above 50%, and thus more accurate than guessing. In other
words, individuals are more likely to get the election winner right than wrong. Under such
conditions, pooling the individual estimates into a group estimate will inevitably yield a correct
decision. The mathematical proof for this relationship was provided in the 18th century by
Marquis the Condorcet and has become known as Condorcet’s jury theorem (Condorcet 1994).
The theorem allows for calculating the probability that a given group of individuals arrives at a
correct decision. For example, the pooled estimate of five individuals who are each correct 60%
of the time is expected to be correct 68% of the time. If the accuracy level of the five individuals
is at 70%, the group estimate should get it right about 84% of the time. In general, given that
individual accuracy is better than chance, Condorcet’s jury theorem implies that the probability
for a group to arrive at a correct decision rapidly increases towards 100% as the number of
individual estimates increases.
3 Analyses of the ANES surveys in the present paper consider only responses that predicted one of the major two
parties’ candidates to win.
Murr (2011) was the first to identify the powerful theoretical link between Condorcet’s
jury theorem and the benefits of aggregating vote expectations. In his study of the 2010 British
election, the majority vote of respondents from the same constituency correctly predicted the
winner in 537 (86%) of 627 constituencies. Thus, compared to the individual forecasts, which
were correct 69% of the time, pooling estimates increased the likelihood of a correct forecast by
17 percentage points. Murr (2011) further analyzed the conditions under which pooling
expectations is most beneficial. In general, the accuracy of aggregated expectations increases
with the number of individual expectations and for races with large winning margins. In addition,
aggregated vote expectations tended to be somewhat more accurate if variation in interview dates
was higher.
Table 1 provides further evidence from other data sets on the gains from pooling
expectations. For example, Lewis-Beck and Stegmaier (2011) provided similar results for
forecasts of British election outcomes at the national level. While, on average, individuals were
correct 60% of the time, the majority of vote expectations correctly predicted the election result
in ten of thirteen elections, a hit rate of 77%. For U.S. presidential elections, I found that the
aggregated responses to the ANES vote expectation question predicted the correct election
winner in thirteen of the sixteen elections from 1952 to 2012, a hit rate of 81%. That is,
compared to the average individual, who was correct 70% of the time, pooling estimates
increased the chance of a correct forecast by 11 percentage points. At the state level, pooled
expectations correctly predicted the winner in 329 (82%) of 399 state races. In comparison, the
average individual was correct 69% of the time.4
These results suggest that vote expectations are useful for election forecasting. However,
little is known about how accurate such vote expectation surveys are relative to other methods, as
few studies compare their accuracy to established benchmarks.
4 As noted above for the case of combining polls, the phenomenon that aggregating individual estimates yields more
accurate forecasts also holds for quantitative estimates. Hogarth (1978) shows analytically that combining
judgments is most useful if individual judges possess diverse information. Adding more forecasts increases
accuracy, although at a diminishing rate of improvement. The power of combining for generating accurate
predictions is one of the major findings from forecasting research conducted since the 1970s, which impacted many
fields such as weather forecasting, economic forecasting, and political forecasting (Graefe et al. 2014).
Vote expectation surveys v. polls
One exception is a study by Rothschild and Wolfers (2012), who analyzed the relative
accuracy of the vote expectation question and the vote intention question when both are asked in
the same survey. Based on an analysis of ANES data from the fifteen U.S. presidential elections
from 1952 to 2008, the authors found that expectations were more accurate than intentions when
predicting election winners, vote shares, and probabilities of victory.
A possible explanation for this result is that expectations capture more information than
intentions. First, vote intention polls ignore information from respondents who are undecided,
who do not want to reveal for whom they are going to vote, and who do not intend to vote at all.
However, these people may also have valuable expectations about the election outcome. Second,
expectations likely incorporate not only information about one’s own vote intention but also
about the intentions of others, as well as information from other sources. For example, people
might form expectations of the election outcome from following general media coverage of the
campaign, reading the latest polls and expert analyses, and talking to peers.
In fact, as shown by Rothschild and Wolfers (2012), each person’s expectation is
equivalent to a multi-person intention poll. The authors estimated that one vote expectation
contains about as much information as twenty vote intentions. This is a major advantage of
expectation surveys since sample size and composition are less critical. Rothschild and Wolfers
(2012) demonstrated this by calculating forecasts based on expectations from biased subsamples
(i.e., only Democrats and only Republicans). In both cases, the expectation-based forecasts of the
biased subsamples provided more accurate forecasts than the complete sample of vote intentions.
These results suggest that responses to the vote expectation question provide more
accurate forecasts of what will happen on Election Day than the vote intention question.
However, as outlined earlier, single polls are a poor benchmark of forecast accuracy. The present
study thus compares the relative accuracy of vote expectations and sophisticated poll-based
forecasts such as combined poll projections.
Vote expectation surveys v. prediction markets and experts
In eliciting expectations, forecasts from experts and prediction markets are closely related
to vote expectation surveys. One major difference between the three approaches lies in the
composition of the sample. While vote expectation surveys sample respondents randomly, the
other two approaches rely on selected (as in the case of expert surveys) or self-selected (as in the
case of prediction markets) experts.5 Given the similarities of these methods, it is surprising that
few researchers studied their relative accuracy.
One study compared responses to the ANES vote expectation question to forecasts from
the IEM vote-share markets during the past two weeks prior to each of the five U.S. presidential
elections from 1988 to 2004. The relationship between the respective forecasts and the final vote
was slightly higher for the vote expectations compared to the IEM forecasts (Holbrook 2010).
Another study found that an online expectation survey of more than 19,000 respondents was
more accurate than the Intrade prediction market when forecasting winning probabilities for the
2008 U.S. presidential election (Miller et al. 2012). Finally, a study of the 2006 Swedish
parliament elections found that combined vote-share forecasts of non-experts (members of the
public) were more accurate than combined forecasts of experts (Sjöberg 2009).
These results may surprise. How is it possible that the combined predictions of regular
citizens perform as well as – or even better than – combined predictions of (self-)selected
experts? In particular, since at the individual level, Sjöberg (2009) found what one would expect:
experts were more accurate than the less interested, less informed, and less educated non-experts.
A possible explanation is variance in the heterogeneity of the groups. In Sjöberg’s study,
the non-experts varied in demographics, and their party preferences were in line with the general
public (i.e., the final election result). In contrast, the experts (three groups consisting of political
scientists, journalists, and editors) were mostly male and well educated. In addition, the experts
showed a particularly low preference for the Conservative party (the second strongest party in
the polls) and high preferences for parties that were less popular among the general public (e.g.,
the Liberals and the Center Party). As a result, the less diverse expert group was likely biased in
the same direction. In such a situation, combining judgments is of limited value, since the
individual estimates are highly correlated and biases do not cancel out in the aggregate (Graefe et
al. 2014, Hogarth 1978).
5 In addition, prediction markets differ from regular surveys in how they aggregate information. Prediction market
participants buy and sell shares, whose prices reflect the combined expectations of all participants. That is,
participants can win and lose money depending on their performance and thus have an incentive to be accurate.
Since participants should only become active if they think they know better than the market as a whole, they are
often referred to as “self-selected experts”.
The same appears to be true for prediction markets. One study analyzed trading behavior
in two markets operated by the IEM: the 1988 U.S. presidential election vote-share market and
the 1993 Canadian House of Commons Market. In both markets, participants exhibited
substantial biases. Compared to the average trader, participants bought more shares of candidates
that they favored and sold more shares of candidates that they did not support (Forsythe, Rietz,
and Ross 1999). In other words, even market participants, who have an incentive to be accurate,
depart from rational behavior and exhibit wishful thinking. This would be less concerning if
participants formed a heterogeneous group, as individual errors would likely cancel out in the
aggregate. However, as shown in a study on the 1988 IEM, this might often not be the case.
Participants were predominantly white, male, well educated, and belonged to the middle and
upper income categories. In addition, participants tended to be more Republican and less
independent in their partisan leanings, and were more politically active than the general public
(Forsythe et al. 1992).
Given these findings, there is reason to believe that vote expectation surveys can provide
forecasts that are competitive with expert surveys and prediction markets.
Vote expectation surveys v. quantitative models
I was unable to find prior empirical evidence on the relative accuracy of vote
expectations and quantitative models. The advantage of models is that they follow a structured
approach to forecasting and include much information about historical elections, such as the
influence of the state of the economy, the popularity of the incumbent, and the time the
incumbent was in the White House. While the accuracy of single models can vary widely across
elections, one usually gets accurate forecasts when combining forecasts from different models
(Graefe et al. 2014). However, a disadvantage of quantitative models is their limited ability to
incorporate information about the specific context of a particular election such as an economic
crisis, threat of terrorism, or some scandal.
Accuracy of vote expectation surveys in forecasting U.S. presidential elections
The following analysis provides empirical evidence on the relative accuracy of vote
expectation surveys compared to polls, prediction markets, expert judgment and quantitative
models for the task of forecasting U.S. presidential elections.
Method
All data and calculations are publicly available (Graefe 2013).
Time horizon and error measures
The methods’ forecast accuracy is analyzed across the last 100 days prior to the seven
elections from 1988 to 2012. The hit rate and the absolute error were used as measures of
accuracy. The hit rate is the percentage of forecasts that correctly predict the winner. For
methods that provide forecasts of two-party popular vote shares, the candidate with a vote share
of more than 50% is predicted to win the election. In the case that each candidate is predicted to
gain 50% of the popular vote, a tie is recorded. Ties score as half of a correct prediction. The
absolute error is the absolute deviation of the predicted and the actual two-party popular vote for
the incumbent party’s candidate.
Data and forecast calculations
To allow for fair comparisons, all forecasts are calculated as if they were made ex ante. That
is, when calculating the forecasts I used only data that would have been available at the time of
the election. In addition, comparisons include only forecasts made around the same time.
Vote expectation surveys
A total of 217 vote expectation surveys were collected across twenty elections from 1932 to
2012. For example: „Regardless of whom you support, and trying to be objective as possible,
who do you think will win the presidential election in November (2008)--Barack Obama or John
McCain?“ (Gallup Poll, October 23-26, 2008). All surveys obtained were conducted within 150
days prior to Election Day. The data set includes sixteen ANES surveys, one for each of the
elections from 1952 to 2012, and the 1932 survey reported by Hayes (1936). The remaining 200
surveys were derived from the iPoll Databank of the Roper Center for Public Opinion Research.
For the 1936 election, no survey was found.
Vote expectation surveys provide direct forecasts of who will win; the candidate that the
majority of respondents expect to win is predicted as the election winner. Table 2 provides an
overview of the accuracy of the collected surveys for this task. The vote expectation surveys
correctly predicted the winner in 193 (89%) of the 217 surveys. Only 18 (8%) of the surveys
predicted the wrong winner, half of which were conducted during the very close 2000 election.
The remaining six surveys predicted a tie.
For each survey that was published during the past 100 days prior to Election Day, the two-
party percentage of respondents that expected the incumbent party candidate to win was
recorded.6 This figure was kept constant on days without any published surveys and was
replaced with the results from a more recent survey once available. If more than one survey was
published on the same day, the results of all surveys from that date were averaged.7
While the results of vote expectation surveys allow for making quick forecasts of who
will win an election, they cannot be directly interpreted as vote share forecasts. For example, a
survey that reveals that 60% of respondents expect the incumbent candidate to win does not
mean that the incumbent can be expected to gain 60% of the vote. In order to translate vote
expectation survey results into vote share forecasts it is thus necessary to use data from historical
surveys. That is, one estimates how a hypothetical incumbent lead of 60-40 in a vote expectation
survey translated to the incumbent’s final vote share in past elections. A simple approach for
estimating this relationship is linear regression analysis (Lewis-Beck and Tien 1999). Thereby,
the incumbent party’s actual two-party popular vote share is regressed on the results from the
vote expectation surveys. When using all 217 vote expectation surveys from 1932 to 2012,
regression analysis yields the following vote equation:
V = 41.0 + 17.1 E + e
(20.4) (78.5)
R2 = .66; SEE = 2.2;
where V is the actual two-party popular vote share of the incumbent party, E is the two-party
percentage of survey respondents that expect the incumbent party candidate to win, and e is the
error term. The figures in parentheses show the t-values.
The results show that the vote expectation survey results explain about two thirds of the
variance in the incumbent’s popular vote share. If the share of respondents that expect the
incumbent to win increases by 10%, the incumbent’s vote share increases by 1.7 percentage
points. The model also reveals the strong degree of partisanship among U.S. voters. As noted by
6 If no survey was published on day 100 prior to Election Day, the most recent available survey was used as the
starting point. This is the reason why surveys were collected up to 150 days prior to Election Day.
7 Instead of a survey’s publication date, the iPoll databank reports the last day a survey was in the field. For the
present analysis, I therefore assumed that the surveys were published two days later. I also tested whether using
different values for the publication delay (i.e., from zero to five days) would affect the results; they did not.
Campbell (1996, 423), “no matter how bad the campaign goes for a party, it can count on
receiving about 40% of the two-party vote; no matter how well a campaign goes for a party, it
will receive no more than about 60% of the two-party vote.” The vote expectation model is
consistent with this view. The model predicts that the incumbent would receive at least 41% of
the vote, even if nobody expected him to win the election (E=0). Vice versa, if all survey
respondents expected the incumbent to win (E=1), the model would predict the incumbent to
receive at maximum 58.1% of the vote.8
Since this vote equation is based on an in-sample estimation using all available data, it
cannot be used to evaluate the predictive accuracy of vote expectation surveys for past elections.
For this, it is necessary to calculate forecasts by using only data that would have been available
at the time of making a forecast. Such ex ante forecasts were calculated by successive updating.
For example, to estimate the equation for predicting the 1988 election, only the 40 surveys
available from 1932 to 1984 were used, while the 2004 equation is based on the 124 surveys
available through 2000, and so on.9 With this approach, the estimated vote equation for a
particular year does not change over the course of a campaign.10 Thus, the vote equation
estimated above would be used to generate forecasts of the upcoming 2016 election. The
resulting equations were then used to translate the results of the most recent vote expectation
survey available at any day prior to the election under observation into a vote share forecast.11
Polls
Polls that were conducted within 100 days prior to each of the sixteen elections from 1952
to 2012 were obtained from Graefe (2014). For each poll, the two-party percentage of
8 Lewis-Beck and Tien (1999) ran the same regression analysis using data from the eleven ANES surveys from 1956
to 1996. With an intercept of 39.5 and an estimated coefficient of 21.0, the results are comparable, despite their
small sample. This provides additional support for the robustness of the estimated relationship.
9 The number of available surveys for estimating the vote equation for a particular election can be derived from
Table 2.
10 Ideally, one would follow an approach similar to the calculation of poll projections. That is, one would estimate
different vote equations depending on the time to Election Day. Such an approach would account for the uncertainty
that occurs over the course of a campaign, which is reflected in the decreasing accuracy of people’s expectations for
long time horizons (Lewis-Beck and Skalaban 1989, Lewis-Beck and Tien 1999, Miller et al. 2012, Murr 2011).
Unfortunately, limited historical data currently preclude such an analysis, as only few surveys are available for early
elections (cf. Table 2).
11 Data from the ANES surveys are not available until months after the election. Therefore, the ANES surveys could
not be used to calculate ex ante forecasts of the election outcome. However, the ANES data were used to estimate
the vote equations for forecasting succeeding elections.
respondents that intended to vote for the candidate of the incumbent party was recorded. If more
than one poll ended on the same day, the results of all polls ending that date were averaged. On
days without any polls ending, the most recent poll from preceding days was used.
Three different poll-based forecasts were used as benchmarks: (1) single polls, (2)
combined polls, and (3) combined poll projections. The single polls benchmark simply interprets
the result of a single poll published on a particular day as forecast of the election outcome. The
combined polls benchmark calculates rolling averages of all polls released over a 7-day period.
The third benchmark, combined poll projections, was adopted from Graefe et al. (2014). That is,
for each of the 100 days prior to an election, starting with 1952, the incumbent’s actual two-party
share of the popular vote was regressed on the combined polls value for that day. This process
produced 100 vote equations (and thus poll projections) per election year. Again, successive
updating was used to calculate ex ante poll projections.
Prediction markets
Daily prediction market data from the IEM vote-share markets were obtained from the IEM
website (http://tippie.uiowa.edu/iem). On these markets, people buy and sell future contracts
according to their own expectations of the candidates’ final vote shares. The market price
represents the combined forecast of all market participants. To put the prediction market
forecasts on equal footing with vote expectation surveys and polls, two-party forecasts were
calculated by ignoring any third-party options. The last traded prices per day were used as the
market forecasts.
Experts
One expert survey was available for each of the elections in 1992 and 2000, four surveys
were available for each of the elections in 2004 and 2008, and five surveys were available for the
2012 election.12 The average number of experts per survey ranged from ten to fifteen. In each
survey, experts were asked to provide forecasts of the popular vote shares. Individual and
average expert forecasts were compared to the average of the daily vote expectation forecasts
during the seven days prior to the publication of the expert survey.
12 The 1992 survey was published in The Washington Post. Pundits’ brew: How it looks; Who’ll win? Our fearless
oracles speak, November 1, 1992, p. C1, by David S. Broder. The 2000 survey was published in The Hotline.
Predictions: Potpourri of picks from pundits to professors, November 6, 2000. The thirteen surveys conducted for
the three elections from 2004 to 2012 were derived from Graefe (2014).
Quantitative models
The present study uses forecasts from seven established quantitative models.13 These
models, along with their forecasts, were published in Political Methodologist 5(2), American
Politics Research 24(4) and PS: Political Science and Politics 34(1), 37(4), 41(4), and 45(4).
Most model forecasts were published about two to three months prior to the election. Therefore,
the model forecasts were compared to the average of the daily vote expectation forecasts
calculated across 90 to 60 days prior to Election Day, which is about the month of August in the
election year.
Results
The following analysis provides empirical evidence on the relative accuracy of vote
expectation surveys in predicting election winners and vote shares compared to each of the four
established methods discussed above. The analysis is based on forecasts that were made during
the last 100 days prior to the seven elections from 1988 to 2012.
Relative accuracy compared to polls and prediction markets
The accuracy of daily forecasts from vote expectations surveys, polls, and the IEM is
analyzed across the full 100-day period prior to Election Day. Table 3 shows the methods’ hit
rate and mean absolute error (MAE) across and for each of the seven elections from 1988 to
2012.
On average, vote expectation surveys were most accurate in predicting election winners
and vote shares. If one had simply relied on the most recent vote expectation survey available on
a particular day, one would have achieved an average hit rate of 92%.14 That is, one would have
correctly predicted the winner more than nine out of ten times. In comparison, if one had relied
on the most recent single poll at the same day, one would have predicted the correct winner only
13 These models were developed by Abramowitz, Campbell, Erikson & Wlezien, Holbook, Lewis-Beck & Tien,
Lockerbie, and Norpoth (cf. Appendix I).
14 Due to the disagreement of the popular and the electoral vote in the 2000 election, the calculation of hit rates for
that particular election requires special consideration. For each of the benchmark methods, a forecast of more than
50% for Gore was coded as correct, because these methods predict popular vote shares. In contrast, the vote
expectation question asks respondents who will be elected president. Thus, responses that expected Bush to win
were coded as correct. Readers who disagree with this coding could invert the vote expectation hit rates for the 2000
election. In that case, for example, the hit rate for the 2000 election reported in Table 3 would equal 40%, and the
average hit rate across the seven elections would equal 89%. The results for the vote share forecasts are not affected
by the special case of the 2000 election.
79% of the time. As expected, combining polls (86% correct predictions) and calculating
combined poll projections (88%) increased upon the accuracy of single polls. Surprisingly, with
a hit rate of 79%, the IEM vote-share markets were no more accurate than single polls.15
In addition, vote expectation surveys provided highly accurate vote-share forecasts. The
vote expectation forecasts missed the final election results on average by 1.6 percentage points
and were thus more accurate than each of its competitors. Compared to single polls, which
missed the final election result on average by 3.2 percentage points, vote expectation surveys
reduced the error by 51%.16 In other words, if one had relied on the most recent vote expectation
forecast rather than on the results from the most recent single poll, one would have cut the
forecast error by more than half. Compared to combined polls and combined poll projections,
error reduction was 46% and 21%, respectively. Gains in accuracy compared to the IEM vote-
share markets were smaller (6%).
Relative accuracy compared to experts
Table 4 compares the accuracy of vote expectation surveys and experts’ judgment made
around the same time. In every single comparison, the vote expectation surveys predicted the
correct winner. In comparison, the combined experts predicted the correct winner 70% of the
time and were thus slightly more accurate than the typical expert (66% correct).17
In terms of vote share forecasts, the MAE of vote expectation surveys (1.2 percentage
points) was 32% lower than the MAE of the typical and 14% lower than the MAE of the
combined expert.
Relative accuracy compared to quantitative models
Table 5 shows the hit rates and MAE of vote expectation surveys and the typical and
average forecast of seven quantitative models for the six elections from 1992 to 2012. The vote
expectation surveys as well as the average forecast of all available models correctly predicted the
winner in each election. The track record of the individual models is not perfect with two models
15 I also analyzed the accuracy compared to the IEM winner-takes-all markets, which were first launched in 1992.
The winner-take-all markets were specifically designed to predict popular vote winners and thus provide the better
benchmark for this type of task. These markets achieved a hit rate of 88% across the six elections from 1992 to
2012. The corresponding hit rate of vote expectation surveys for the same time period was 93%.
16 The error reduction in % is calculated as 1 - MAE(vote expectations) / MAE (single polls) = 1 - 1.57 / 3.23 = 0.51.
17 The performance of the typical expert is the performance that one would achieve if one would randomly pick an
expert.
missing the winner in 2012, and one each in 1992, 2004, and 2008. This results in an average hit
rate of 86% for the typical model. In other words, if one had randomly picked a model in each
election, one would have correctly picked the winner in six of the seven elections.
In terms of vote share forecasts, the vote expectation surveys yielded a MAE of 1.5
percentage points and thus reduced the corresponding error of the typical and combined model
forecast by 50% and 36%, respectively.18
Discussion
There has been much progress in our ability to forecast elections over the last three
decades. Combining polls and projecting their results to Election Day yielded substantial
improvements in accuracy compared to single polls. Researchers have developed econometric
models that can quite accurately predict election outcomes from structural information that is
available months before Election Day. Finally, prediction markets reappeared as a powerful tool.
One simple method, which has existed at least since the advent of scientific polling, has
been largely overlooked in this development: surveying people on who they expect to win.
Across the past seven U.S. presidential elections, vote expectation surveys provided more
accurate forecasts of election winners and vote shares than any other established method. Gains
in accuracy were particularly large compared to the method that is still standard practice for the
coverage of election campaigns: single polls. The error of vote-share forecasts derived from vote
expectation surveys was 51% lower than the corresponding error of a single poll. Nevertheless,
the widespread belief that polls provide accurate forecasts remains.
Relative performance of polls and vote expectation surveys in the 2012 election
The National Council on Public Polls (NCPP) analyzed 25 national polls that were
conducted within the final week of the 2012 campaign. The NCPP concluded that, with an
average error of 1.46 percentage points, the polls “came close to the election outcome”.19
However, the NCPP failed to compare the polls’ performance to a benchmark. I calculated the
18 The careful reader might note an apparent discrepancy between the expectation surveys’ perfect hit rate and the
rather large error in their vote share forecasts for the 2000 election. The reason is that, at the time the forecasts are
compared, which is around three months prior to Election Day, a large portion of respondents expected Bush to
defeat Gore. While this led to accurate predictions of the election winner (i.e., a perfect hit rate), the vote share
forecasts underestimated Gore’s two-party vote (see also footnote 9 for further details).
19 The NCPP’s “Analysis of Final 2012 Pre-Election Polls” is available at www.ncpp.org.
corresponding error of forecasts from vote expectation surveys and the IEM for the same time
period. Both benchmarks provided much more accurate predictions than the NCPP poll sample.
The vote expectation surveys yielded an error of 0.71 and thus reduced the error of polls by more
than 50%. With an error of 0.99 percentage points, the IEM were 30% more accurate than the
polls. Note that these gains in accuracy were obtained for the last week prior to the election, a
time in which polls usually provide very accurate forecasts.20
Figure 1 extends this analysis and shows the relative accuracy of vote share predictions of
110 polls and 20 vote expectation surveys that were published during the last 100 days prior to
the 2012 election. The vertical axis shows Obama’s predicted lead in the two-party popular vote
and the dotted grey line depicts the final election outcome; Obama won the election with a four-
point advantage over Romney. Single polls varied wildly and predicted anything from a three-
point lead for Romney to an eighteen-point lead for Obama (standard deviation: 4.3). In
comparison, the vote expectation survey forecasts ranged from a one-point to a six-point lead for
Obama (standard deviation: 1.3). That is, vote expectation survey forecasts were much more
stable, less extreme, and closer to the election result than individual polls throughout the 100-day
time horizon. The elections from 1988 to 2008 show a similar pattern (cf., Appendix III).
Relative performance of FiveThirtyEight.com and vote expectation surveys in the 2012
election
One cannot discuss forecasts of the 2012 election without mentioning Nate Silver’s
FiveThirtyEight.com, a polling aggregation website that was launched in 2008 and has become
part of the The New York Times online in 2010. Silver uses sophisticated statistical analyses to
analyze the type and extent of biases of single pollsters, and to demonstrate the value of polling
aggregation for forecasting. His forecast model aggregates information from state-level polls by
accounting for the relative performance of different pollsters and considering relationships
between states. In addition, the model incorporates an index of economic indicators, whose
weight decreases as the election nears. Simply put, Silver’s model is an enhanced and much
20 I also compared the accuracy of the final vote expectation forecasts and the final Gallup pre-election poll for the
past seven elections from 1988 to 2012. The vote expectation survey forecasts were more accurate in four of the
seven surveys, with an average error of 1.5 percentage points. In comparison, the error of the final Gallup poll was
27% higher (1.9 percentage points). The numbers are provided in Appendix II.
more sophisticated version of the traditional quantitative models, some of which also combine
polls and economic fundamentals.
FiveThirtyEight has become extremely popular. In the week prior to the 2012 election,
almost three out of four politics visits at the New York Times website included a stop at
FiveThirtyEight. The day before the election, one in five nytimes.com visitors looked at Silver’s
site (Tracy 2012). FiveThirtyEight had become a synonym for election forecasting, which
becomes evident when looking at the relative volumes of Google searches for variants of
“Fivethirtyeight” and “election forecast” (cf., Appendix IV).
I compared the accuracy of vote expectation surveys to Silver’s popular vote forecast.
Figure 2 reports the error of both approaches for the last 100 days prior to the election. Any point
on the lines in the chart shows the average error for the remaining days in the forecast horizon.
For example, if one had relied on the FiveThirtyEight forecast on each of the 100 days prior to
Election Day (i.e., starting from July 29th), one would have achieved a MAE of 0.65 percentage
points. If one had relied on FiveThirtyEight on each day from October 11th, one would have
achieved a MAE of 1.2 percentage points, and so on. The corresponding values for the MAE of
the vote expectation surveys are 0.40 (from July 29th) and 0.76 (from October 11th). As shown in
Figure 2, at any point in the campaign, one would have fared better by relying on the most recent
vote expectation survey instead of relying on the forecasts at FiveThirtyEight.com. Across the
full 100-day forecast horizon, vote expectation surveys reduced the error of FiveThirtyEight on
average by 38%.21
Barriers to the adoption of vote expectation surveys
The accuracy of vote expectation surveys in forecasting elections may surprise given that
the results of vote expectation surveys are rarely reported, not to say ignored, by the media.
21 One advantage of FiveThirtyEight is that the model also provides forecasts at the state level. For the 2012
election, the model’s final forecast correctly predicted the winner in all 51 states. I analyzed data from the 2012
ANES pre-election survey to analyze the performance of vote expectation surveys for the same task. Although the
number of responses for some states was limited and data collection started in early September, two months before
the election, the vote expectation survey performed well. In 49 of 51 states, the majority of respondents correctly
predicted the election winner in that state; the two exceptions were North Carolina and New Hampshire. The results
from this single survey suggest that vote expectation surveys can also provide highly accurate forecasts at the state
level.
Instead, the trial-heat question, which asks respondents for whom they intend to vote, dominates
the media coverage of election campaigns.
One likely reason for the disregard of vote expectation surveys is that people simply do
not know about their accuracy. The present study aims at addressing this gap by comparing the
method to established benchmarks. But additional barriers are likely to hinder the adoption of
vote expectation surveys in practice: people have no faith in simple methods and journalists are
interested in newsworthiness rather than accuracy.
Complexity persuades
Occam’s Razor advises researchers to prefer simple models unless simplicity is offset by
more explanatory power. Since Occam, many famous researchers advocated the use of simple
models. Albert Einstein is reputed to have said that “everything should be made as simple as
possible but not simpler”. Zellner (2004), who coined the phrase “keep it sophisticatedly
simple”, named several Nobel laureates as proponents of simplicity. Vote expectation surveys
adhere to Occam’s Razor; they are easy to conduct, the results are easy to understand, and they
provide forecasts that are at least as accurate as more complex methods.
Unfortunately, simple models often face resistance, because people tend to wrongly
believe that complex solutions are necessary to solve complex problems. Hogarth (2012)
reported results from four studies, which showed that simple models often perform better than
more complex ones. In each case, however, people resisted the findings regarding the
performance of simple models. The same appears to be true for election forecasting. People are
impressed by sophistication and complexity (e.g., FiveThirtyEight.com) and overlook obvious
approaches, such as simply asking people whom they expect to win.
Newsworthiness beats accuracy
Journalists and political commentators need to meet the demands of the news cycle and
constantly look for interesting stories and analyses. In this endeavor, they often select
newsworthiness over accuracy and relevance. In particular, journalists increasingly generate
news by focusing on who is ahead in the polls or linking the latest poll results to campaign
events. As shown in Figure 1, polls from different survey organizations often vary wildly, even if
they are conducted at around the same time. In such a situation, journalists can cherry-pick on
polls that support their story. Thereby, they might be little concerned about the accuracy of a
poll, in particular if the election is still some time away. Such horse race coverage negatively
affects the quality of campaign coverage as it comes at the expense of informing the public about
candidates and their proposed policies (Rosenstiel 2005, Patterson 2005).
In contrast, vote expectation surveys are much more robust and less extreme. Since their
forecast rarely changes, vote expectation surveys are less suited for generating news. It would be
desirable if journalists would focus more on vote expectation surveys. First, voters would be
much better informed about who is really ahead. Second, the stability of vote expectation surveys
makes it difficult to frame the election as a horse race. Rather, journalists could concentrate on
providing explanations for the relative performance of candidates and their proposed policies.
Conclusion
The general election observer is probably most interested in who will win. When it comes
to U.S. presidential elections, vote expectation surveys are likely to provide the best answer to
this question. In addition, the results of such surveys can be translated into highly accurate vote
share forecasts.
Vote expectation surveys are inexpensive and easy to conduct, and the results are easy to
understand. Because vote expectations are much more stable than vote intentions, they are not
suited to frame the election as a horse race. Thus, an increased focus on vote expectation surveys
is likely to positively affect the quality of election campaign coverage.
References
Benoit, William L., Glenn J. Hansen, and Rebecca M. Verser. 2003. "A meta-analysis of the effects of viewing US
presidential debates." Communication Monographs 70 (4):335-350.
Berg, Joyce E., Forrest D. Nelson, and Thomas A. Rietz. 2008. "Prediction market accuracy in the long run."
International Journal of Forecasting 24 (2):285-300.
Campbell, James E. 1996. "Polls and Votes The Trial-Heat Presidential Election Forecasting Model, Certainty, and
Political Campaigns." American Politics Research 24 (4):408-433.
Campbell, James E., Lynna L. Cherry, and Kenneth A. Wink. 1992. "The convention bump." American Politics
Research 20 (3):287-307.
Condorcet, M. J. A. N. de Caritat, Marquis de. 1994. "Essay on the application of probability analyses to decisions
returned by a plurality of people." In Condorcet: Foundations of Social Choice and Political Theory, edited
by Iain McLean and Fiona Hewitt, 11-36. Brookfield, VT: Edward Elgar Publishing (Original work
published 1785).
Erikson, Robert S., and Christopher Wlezien. 1999. "Presidential polls as a time series: the case of 1996." Public
Opinion Quarterly 63 (2):163-177.
Erikson, Robert S., and Christopher Wlezien. 2008. "Are political markets really superior to polls as election
predictors?" Public Opinion Quarterly 72 (2):190-215.
Forsythe, Robert, Forrest Nelson, George R. Neumann, and Jack Wright. 1992. "Anatomy of an experimental
political stock market." The American Economic Review 82 (5):1142-1161.
Forsythe, Robert, Thomas A. Rietz, and Thomas W. Ross. 1999. "Wishes, expectations and actions: a survey on
price formation in election stock markets." Journal of Economic Behavior & Organization 39 (1):83-110.
Gelman, Andrew, and Gary King. 1993. "Why are American presidential election campaign polls so variable when
votes are so predictable?" British Journal of Political Science 23 (4):409-451.
Graefe, Andreas. 2013. "Replication data for: Accuracy of vote expectation surveys in forecasting elections."
Harvard Dataverse Network, doi:10.7910/DVN/VOTEEXPECTATIONSURVEYS.
Graefe, Andreas. 2014. "Replication data for: Combining forecasts: An application to elections." Harvard Dataverse
Network, dx.doi.org/10.7910/DVN/23184.
Graefe, Andreas, J. Scott Armstrong, Randall J. Jones Jr., and Alfred G. Cuzán. 2014. "Combining forecasts: An
application to elections." International Journal of Forecasting 30 (1):43-54.
Granberg, Donald, and Edward Brent. 1983. "When prophecy bends: The preference-expectation link in US
presidential elections, 19521980." Journal of Personality and Social Psychology 45 (3):477-491.
Hayes, Samuel P. Jr. 1936. "The predictive ability of voters." Journal of Social Psychology 7 (2):183-191.
Hogarth, Robin M. 1978. "A note on aggregating opinions." Organizational Behavior and Human Performance 21
(1):40-46.
Hogarth, Robin M. 2012. "When simple is hard to accept." In Ecological Rationality: Intelligence in the World,
edited by Peter M. Todd, Gerd Gigerenzer and ABC Research Group, 61-79. Oxford: Oxford University
Press.
Holbrook, Thomas M. 2010. "Forecasting US presidential elections." In The Oxford Handbook of American
Elections and Political Behavior, edited by Jan E. Leighley, 346-371. Oxford: Oxford University Press.
Kernell, Samuel. 2000. "Life before polls: Ohio politicians predict the 1828 presidential vote." PS: Political Science
and Politics 33 (3):569-574.
Lazarsfeld, Paul F., Bernard Berelson, and Hazel Gaudet. 1948. The People's Choice. How the Voter Makes up His
Mind in a Presidential Campaign. 2nd ed. New York: Columbia University Press.
Lemert, James B. 1986. "Picking the winners: Politician vs. voter predictions of two controversial ballot measures."
Public Opinion Quarterly 50 (2):208-221.
Lewis-Beck, Michael S., and Andrew Skalaban. 1989. "Citizen forecasting: can voters see into the future?" British
Journal of Political Science 19 (1):146-153.
Lewis-Beck, Michael S., and Mary Stegmaier. 2011. "Citizen forecasting: Can UK voters see the future?" Electoral
Studies 30 (2):264-268.
Lewis-Beck, Michael S., and Charles Tien. 1999. "Voters as forecasters: a micromodel of election prediction."
International Journal of Forecasting 15 (2):175-184.
Miller, Michael K, Guanchun Wang, Sanjeev R Kulkarni, H Vincent Poor, and Daniel N Osherson. 2012. "Citizen
Forecasts of the 2008 US Presidential Election." Politics & Policy 40 (6):1019-1052.
Murr, Andreas E. 2011. "“Wisdom of crowds”? A decentralised election forecasting model that uses citizens’ local
expectations." Electoral Studies 30 (4):771-783.
Patterson, Thomas E. 2005. "Of polls, mountains." Public Opinion Quarterly 69 (5):716-724.
Rhode, Paul W., and Koleman S. Strumpf. 2004. "Historical presidential betting markets." Journal of Economic
Perspectives 18 (2):127-141.
Rosenstiel, Tom. 2005. "Political polling and the new media culture: A case of more being less." Public Opinion
Quarterly 69 (5):698-715.
Rothschild, David, and Justin Wolfers. 2012. "Forecasting elections: voter intentions versus expectations." Working
paper:Available at: ssrn.com/abstract=1884644.
Sjöberg, Lennart. 2009. "Are all crowds equally wise? A comparison of political election forecasts by experts and
the public." Journal of Forecasting 28 (1):1-18.
Tracy, Marc. 2012. "Nate Silver is a one-man traffic machine for the Times." New Republic November 6.
Zellner, Arnold. 2004. "Keep it sophisticatedly simple." In Simplicity, Inference and Modelling: Keeping It
Sophisticatedly Simple, edited by Arnold Zellner, Hugo A. Keuzenkamp and Michael McAleer, 242-262.
Cambridge: Cambridge University Press.
Table 1
Accuracy of individual and pooled vote expectations in British and U.S. elections
Election
British general elections
U.S. presidential elections
Source
Murr (2011)
Lewis-Beck and
Stegmaier (2011)
ANES data, own
calculations
Number of elections
1 (in 2010)
13 (between 1951
and 2005)
16 (from 1952 to
2012)
Prediction task
Predict election
winner in local
constituency
Predict party that
wins the governing
majority of seats
Predict national
election winner
Individual expectations
No. of observations
13,334
N/A
30,573
% correct
69%
60%
70%
Pooled expectations
No. of observations
627
13
16
Correct
537
10
13
% correct
86%
77%
81%
Table 2
Accuracy of vote expectation surveys in predicting the winner (1932-2012)
Predicted winner
Election
No. of
surveys
Correct
Wrong
Tie
1932
1
1
0
0
1940
3
3
0
0
1944
6
6
0
0
1948
1
0
1
0
1952
3
3
0
0
1956
5
5
0
0
1960
1
0
1
0
1964
1
1
0
0
1968
1
1
0
0
1972
3
3
0
0
1976
3
3
0
0
1980
3
1
2
0
1984
9
9
0
0
1988
17
16
1
0
1992
23
20
3
0
1996
20
20
0
0
2000
24
13
9
2
2004
43
39
1
3
2008
23
22
0
1
2012
27
27
0
0
Total
217
193
18
6
Table 3
Hit rate and mean absolute error of vote expectation surveys, polls, and the IEM vote share prediction
markets across the last 100 days prior to Election Day (1988-2012)
Mean
1988
1992
1996
2000
2004
2008
2012
Hit rate (in %)
Vote expectation surveys
92
85
100
100
60
98
99
100
Single polls
79
74
100
100
48
68
90
76
Combined polls
86
78
100
100
56
69
96
100
Combined poll projections
88
84
99
100
97
100
39
100
IEM vote share market
79
70
67
100
30
88
100
100
Mean absolute error (in %-points)
Vote expectation surveys
1.6
3.0
1.4
1.1
2.2
1.1
1.8
0.4
Single polls
3.2
4.0
4.9
5.0
3.1
1.9
1.9
1.8
Combined polls
2.9
3.9
4.7
4.7
2.7
1.5
1.7
1.3
Combined poll projections
2.0
1.4
1.9
2.4
1.8
1.1
4.1
1.2
IEM vote share market
1.7
3.0
2.8
1.0
1.4
0.7
1.2
1.5
The analysis of the 1988 election covers only the last 91 days prior to Election Day, since no vote expectation survey was
available before that day.
Table 4
Hit rate and mean absolute error of vote expectation surveys and experts (1992, 2000-2012)
Mean
1992
2000
2004
2008
2012
Hit rate (in %)
Vote expectations surveys
100
100
100
100
100
100
Typical expert forecast
66
87
20
47
94
82
Average expert forecast
70
100
0
50
100
100
Mean absolute error (in %-points)
Vote expectations surveys
1.2
0.3
2.4
1.1
1.8
0.5
Typical expert forecast
1.8
1.8
2.4
1.8
1.7
1.3
Average expert forecast
1.4
0.7
2.3
1.4
1.5
1.3
The expert forecasts are compared to the average of the daily vote expectation forecasts during the
seven days prior to the publication of the expert survey.
Table 5
Hit rate and mean absolute error of vote expectation surveys and seven established quantitative models
(1992-2012)
Mean
1992
1996
2000
2004
2008
2012
Hit rate (in %)
Voter expectations
100
100
100
100
100
100
100
Typical model forecast
86
75
100
100
86
86
71
Average model forecast
100
100
100
100
100
100
100
Mean absolute error (in %-points)
Voter expectations
1.5
1.8
0.4
3.9
0.8
1.7
0.2
Typical model forecast
2.9
1.7
2.1
5.7
2.8
3.2
1.8
Average model forecast
2.3
1.4
2.1
5.7
2.5
1.1
0.8
Only four model forecasts were available for the 1992 election. See Appendix I for the yearly forecasts of each
model. The model forecasts are compared to the average of the daily vote expectation forecasts calculated
across 90 to 60 days prior to Election Day.
Figure 1
2012 U.S. presidential election vote-share forecasts
of 20 vote expectation surveys and 110 polls (last 100 days prior to Election Day)
-4
0
4
8
12
16
0102030405060708090100
Obama lead in percentage points (two-party
popular vote)
Days to Election Day
Individual polls Expectation surveys Election result
Figure 2
Mean absolute error of popular vote share forecasts from vote expectation surveys and
FiveThirtyEight.com for the 2012 U.S. presidential election
0.00
0.20
0.40
0.60
0.80
1.00
1.20
MAE across remaining days to Election Day
FiveThirtyEight.com Vote expectation surveys
Appendix I
Quantitative model forecasts of the U.S. presidential elections from 1992 to 2012
Election
1992
1996
2000
2004
2008
2012
Election result
46.4
54.7
50.3
51.2
46.3
52.0
FC
AE
FC
AE
FC
AE
FC
AE
FC
AE
FC
AE
Abramowitz
46.7
0.3
56.8
2.1
53.2
2.9
53.7
2.5
45.7
0.6
50.6
1.4
Campbell
47.1
0.7
58.1
3.4
52.8
2.5
53.8
2.6
52.7
6.4
52.0
0.0
Holbrook
57.2
2.5
60.0
9.7
54.5
3.3
44.3
2.0
47.9
4.1
Lewis-Beck & Tien
51.5
5.1
54.8
0.1
55.4
5.1
49.9
1.3
49.9
3.6
48.2
3.8
Lockerbie
57.6
2.8
60.3
10.0
57.6
6.4
41.8
4.5
53.8
1.8
Norpoth
57.1
2.4
55.0
4.7
54.7
3.5
49.9
3.6
53.2
1.2
Wlezien & Erikson
45.9
0.5
56.0
1.3
55.2
4.9
51.7
0.5
47.8
1.5
52.6
0.6
Forecasts were published in Political Methodologist 5(2), American Politics Research 24(4) and PS: Political Science and
Politics 34(1), 37(4), 41(4), and 45(4).
Forecasts and election results refer to the two-party share of the popular vote received by the candidate of the incumbent
party.
Appendix II
Accuracy of Election Eve forecasts for the seven elections from 1988 to 2012:
Vote expectation surveys vs. final Gallup survey
Forecast
Absolute error
Election
Election
result
Vote
expectations
Final
Gallup poll*
Vote
expectations
Final
Gallup poll
1988
53.8
56.9
56.0
3.1
2.2
1992
46.4
47.3
43.0
0.9
3.4
1996
54.7
56.7
55.9
2.0
1.2
2000
50.3
47.2
48.9
3.1
1.3
2004
51.2
51.8
50.0
0.5
1.2
2008
46.3
45.3
44.4
1.0
1.9
2012
52.0
51.8
49.5
0.1
2.5
Average
1.5
1.9
* Gallup poll numbers obtained from http://www.gallup.com/poll/9442/election-polls-accuracy-record-presidential-
elections.aspx.
Forecasts and election results refer to the two-party share of the popular vote received by the candidate of the
incumbent party.
Appendix III
Error of vote-share forecasts from vote expectation surveys and individual polls (1988 to 2008)
Horizontal axis: Days to Election Day; Vertical axis: Percentage error in predicting the incumbent’s popular two-party vote (positive
values mean that the forecast was higher than the actual result, negative values mean that the forecast was lower);
Black markers: Error of expectation surveys; Grey markers: Error of individual polls
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
2008
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
2004
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
2000
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
1996
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
1992
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
1988
Appendix IV
Google searches for FiveThirtyEight and “election forecast” prior to the 2012 election
0
10
20
30
40
50
60
70
80
90
100
Google search volume
"Fivethirtyeight OR 538"
"election forecast OR election prediction OR forecast election OR prediction election"
... (Graefe, 2014;Lewis-Beck & Tien, 1999;Miller et al., 2012;Rothschild & Wolfers, 2013), and general elections in the United Kingdom (UK) (Lewis- Beck & Stegmaier, 2011;Murr, 2011Murr, , 2016. These studies suggest the existence of a "wisdom of the crowd," meaning that citizens possess a type of knowledge of everyday politics that is embedded in social networks and can hardly be captured by traditional polling methods. ...
... A growing number of studies has investigated whether citizens' perceptions of politics can yield accurate election forecasts in different types of elections (Ganser & Riordan, 2015;Graefe, 2014Graefe, , 2015Lewis-Beck & Skalaban, 1989;Lewis-Beck & Tien, 1999;Murr, 2016;Rothschild & Wolfers, 2013). While these studies are primarily concerned with whether citizen forecasts outperform other forecast methods, we focus instead on the relatively unexplored question of what factors influence citizens' predictions of electoral results. ...
... In particular, we find that politically sophisticated, in addition to older people and wealthier voters, predict the referendum result more closely than their relative counterparts. These findings contribute to the emergent literature on citizen forecasting (Graefe, 2014;Irwin & van Holsteyn, 2002;Murr, 2011Murr, , 2016 by showing that some voters are able to forecast election results more accurately than others. Further research should explore additional individual factors that might explain forecasting accuracy, such as interpersonal discussion, political involvement, media consumption, or numeracy, using a number of measures that were not available in our study. ...
Article
Full-text available
The emergent literature on citizen forecasting suggests that the public, in the aggregate, can often accurately predict the outcomes of elections. However, it is not clear how citizens form judgments about election results or what factors influence individual predictions. Drawing on an original survey experiment conducted during the campaign for the United Kingdom’s Brexit referendum, we provide novel evidence of what influences citizen forecasts in a so-far unexplored context of direct democracy. Specifically, we investigate the effect of voting preferences and political sophistication, in addition to three “exogenous factors” that we manipulate experimentally—i.e., social cues, elite cues and campaign arguments. Our findings indicate that citizens are reasonably accurate in their predictions, with the average forecast being close to the actual result of the referendum. However, important individual heterogeneity exists, with politically sophisticated voters being more accurate in their predictions and less prone to wishful thinking than non-sophisticated voters. Experimental findings show that partisan voters adjust their predictions in response to cues provided by their favorite party’s elites and partly in response to campaign arguments, and the effects are larger for low-sophisticated voters. We discuss the mechanisms accounting for the experimental effects, in addition to the implications of our findings for public opinion research and the literature on citizen forecasting.
... The advantage of vote-expectation polls is that people are invited to think about the public's vote intention; therefore, in addition to their own intention, they may report that of their family and friends (Leiter et al. 2018). Indeed, several studies have shown that vote-expectation polls are more accurate than vote-intention polls (e.g., Graefe 2014;Murr, Stegmaier, and Lewis-Beck 2021). That said, a disadvantage of our approach is that we asked the likelihood for each candidate and therefore needed survey questions for each candidate instead of a single vote-intention question. ...
... In the eight surveys that they examined since 1956, voters correctly predicted the winner 69% of the time (Lewis-Beck and Skalaban 1989, 148). This benchmark result was repeated in subsequent US studies (Graefe 2014;Lewis-Beck and Tien 1999;Murr 2015). Moreover, regarding Europe, primarily the United Kingdom, the finding also has been sustained (Lewis-Beck and Stegmaier 2011; Murr 2011Murr , 2016. ...
... Another aspect of perceptions of surveys is survey reliability, which refers to the perceived quality and credibility of surveys (Kuru, Pasek, and Traugott 2017;Loosveldt and Storms 2008;Stocké 2014). These perceptions are largely shaped by the degree to which respondents put in the necessary effort to provide meaningful answers (Loosveldt and Storms 2008), whether surveys are conducted consistent with the standard in the field (Kuru et al. 2017(Kuru et al. , 2020, and whether surveys can accurately display the current situation and predict future events such as election outcomes (Graefe 2014). ...
Article
With response rates of large‐scale surveys having decreased significantly over the years and rebounds seeming unlikely, many studies now examine how response rates vary with methodological design and incentives. This investigation delves into how individual‐level factors shape survey participation. Specifically, we examine the influence of individuals’ democratic engagement and their trust in institutions on intent to participate in surveys, both directly and indirectly through their perceptions of surveys. We collected survey data from a probability sample of adults (N = 1343) in Mannheim, Germany, from November 2019 to March 2020. Structural equation models were estimated to test the hypothesized relationships. The analyses support most, but not all, hypothesized relationships. Democratic engagement bolstered intent to participate, directly as well as indirectly through perceptions of surveys. Institutional trust, on the other hand, only influenced the outcome measure indirectly. Perceptions of surveys had a strong bearing overall effect on intent to participate. The study's results suggest that the response rates and larger issues related to the perceived legitimacy of public opinion and survey research might be intertwined with orientations related to people's civic and political life. The article discusses potential ways survey researchers can counteract distrust in surveys.
... For example, analyzing T a bl e 1 Murr, Stegmaier, and Lewis-Beck (2021) found that citizen forecasts were more accurate than polls in predicting both election winners and seat shares. Graefe (2014) found that across the last 100 days prior to the seven presidential elections from 1988 to 2012, citizen forecasts were more accurate than four alternative methods: polls, betting markets, models, and expert forecasts. Despite the predictive power of citizen forecasts, publicopinion surveys rarely include vote-expectation questions and only ask for people's vote intention. ...
... There are also issues with survey design. For example, some pollsters recommend asking the participant to predict the outcome of the election rather than ask who they plan to vote for (Graefe 2014;cf. Gelman & Azari 2017). ...
Article
Numerous philosophers of science have argued that the incentive structure of science is a major contributor to the replication crisis. In this paper I review FiveThirtyEight’s election forecasting model and show that it confronts similar problems in political polling and successfully mitigates them, not by changing the incentives, but by weighting a pollster’s effect on the model by their past reliability. I argue that a similar weighting procedure holds a promising approach to addressing the replication crisis in science.
... Citizen forecasting relies on the aggregation of voters' expectations about election outcomes. This approach has proven quite successful in the US and the UK (see, e.g., Boon 2012;Graefe 2014;Lewis-Beck and Stegmaier 2011;Murr 2011Murr , 2015Murr , 2016Murr et al. 2021) and it will be formally put to the test for the 2022 French presidential election (see Dufresne et al. 2022). Most studies explain the quality of citizens' forecasts by the "miracle of aggregation" theorem, which states that errors in individuals' judgments tend to cancel out in the aggregate. ...
Article
Full-text available
In the last 40 years or so, scholars have proposed a vast array of models and approaches to predict election outcomes in a variety of democracies. Election forecasting has garnered increasing attention and has been the subject of multiple symposia and special issues in political science journals. This article reviews the forecasting efforts that have been deployed in the case of France since pioneering work in the late 1970s and early 1980s and discusses the peculiarities of the French political system and their consequences as well as the challenges they create for election forecasting.
Article
Partisan preferences usually stand out as the major driving force behind voters' expectations about election outcomes. Apart from partisan preferences however, purely individual‐level factors appear to be only weakly associated with forecasting ability. Some studies argue that we need to move from the strictly personal sphere to the interpersonal one to better understand the underpinnings of individuals' forecasting ability. This paper leverages data from 77 elections at the district, municipal, regional, and/or national levels in ten different countries to assess the impact of social networks and social interactions on the accuracy of citizens' electoral expectations. The results cast doubt on the capacity of social interactions to influence citizens' forecasting skills. This article is protected by copyright. All rights reserved
Article
I produce and compare election forecasts for the period 2010–19. The dominant methods of polls, odds and economic models have mixed success, each predicting three out of four federal elections. Consistent with prior research, polls then prediction markets are found to produce the most volatile forecasts, while economic models produce the least. Prediction markets are found to efficiently price available information. Economic models performed well at the 2019 election, however, they were limited by outdated dummy variables for ‘honeymoon effects’. I conclude by discussing some alternative and emerging forecasting methods, including demographic explanations of voting and online sentiment analysis.
Article
Full-text available
Forecasting has always been at the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The large number of forecasting applications calls for a diverse set of forecasting methods to tackle real-life challenges. This article provides a non-systematic review of the theory and the practice of forecasting. We provide an overview of a wide range of theoretical, state-of-the-art models, methods, principles, and approaches to prepare, produce, organise, and evaluate forecasts. We then demonstrate how such theoretical concepts are applied in a variety of real-life contexts. We do not claim that this review is an exhaustive list of methods and applications. However, we wish that our encyclopedic presentation will offer a point of reference for the rich work that has been undertaken over the last decades, with some key insights for the future of forecasting theory and practice. Given its encyclopedic nature, the intended mode of reading is non-linear. We offer cross-references to allow the readers to navigate through the various topics. We complement the theoretical concepts and applications covered by large lists of free or open-source software implementations and publicly-available databases.
Article
While combining forecasts is well-known to reduce error, the question of how to best combine forecasts remains. Prior research suggests that combining is most beneficial when relying on diverse forecasts that incorporate different information. Here, I provide evidence in support of this hypothesis by analyzing data from the PollyVote project, which has published combined forecasts of the popular vote in U.S. presidential elections since 2004. Prior to the 2020 election, the PollyVote revised its original method of combining forecasts by, first, restructuring individual forecasts based on their underlying information and, second, adding naïve forecasts as a new component method. On average across the last 100 days prior to the five elections from 2004 to 2020, the revised PollyVote reduced the error of the original specification by eight percent and, with a mean absolute error (MAE) of 0.8 percentage points, was more accurate than any of its component forecasts. The results suggest that, when deciding about which forecasts to include in the combination, forecasters should be more concerned about the component forecasts’ diversity than their historical accuracy.
Article
Full-text available
This article offers an assessment of the current state of US presidential election forecasting models. It pays attention to presidential forecasting models from the last three election cycles. It starts by exploring 'under the hood' and describes the specifics of the most widely known models from the 2004 election. In addition, the predictions made by these models are addressed and the determinants of forecasting accuracy from 1996 to 2004 are identified. Moreover, the article explores the lessons learned from the 2000 campaign and the alternatives to the dominant aggregate-national forecasting models: electronic markets, citizen forecasts, and state-level forecasting models. From a forecasting perspective, the 2008 election outcome was business as usual. Some models were more accurate than others, as is always the case, but the average error was somewhat lower than in the past two elections cycles.
Article
Full-text available
The Incumbency and National Conditions (presidential approval and aggregated personal finances) Model predicted President Obama would garner 47.9% of the two-party vote, whereas he ended up with 51.8% (based on available information on December 3, 2012). The error in this forecast (3.9 points) is somewhat higher than the average out-of-sample error from 1952 to 2008 (2.4 points). Although the forecast was off the mark, the addition of the 2012 result to the data set does little to change the slope estimates, and the overall fit of the model is only slightly worse (table 1).
Data
Full-text available
We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert judgment, quantitative models, and the Iowa Electronic Markets. Across the six elections, the resulting combined forecasts were more accurate than any individual component method, on average. The gains in accuracy from combining increased with the numbers of forecasts used, especially when these forecasts were based on different methods and different data, and in situations involving high levels of uncertainty. Such combining yielded error reductions of between 16% and 59%, compared to the average errors of the individual forecasts. This improvement is substantially greater than the 12% reduction in error that had been reported previously for combining forecasts.
Article
The leading approaches to election forecasting have been statistical models, prediction markets, or vote intention polls. This paper explores a different, little used approach - citizen forecasting - a method never yet tried for the case of the United Kingdom. Citizen forecasting simply asks voters in a scientific pre-election survey who they think will win. This aggregated (averaged) result is then assessed for its accuracy in forecasting the overall election outcome. This assessment is carried out on UK general elections from 1951 to 2005, and then applied to 2010. We find, among other things, that generally speaking the majority of the electorate can foretell the winner in advance.
Article
We analyze individual probabilistic predictions of state outcomes in the 2008 U.S. presidential election. Employing an original survey of more than 19,000 respondents, we find that partisans gave higher probabilities to their favored candidates, but this bias was reduced by education, numerical sophistication, and the level of Obama support in their home states. In aggregate, we show that individual biases balance out, and the group's predictions were highly accurate, outperforming both Intrade (a prediction market) and fivethirtyeight.com (a poll-based forecast). The implication is that electoral forecasters can often do better asking individuals who they think will win rather than who they want to win.
Article
The 2012 presidential campaign takes place at a time of deep political division in the United States. Democrats and Republicans differ sharply over Barack Obama's performance in office as well as a wide range of issues ranging from government spending and health care to immigration and gay marriage. These divisions are shaping the strategies of the candidates and the outlook for November. Overwhelming majorities of Democrats and Republicans, including overwhelming majorities of independents who lean toward a party, can be expected to support their own party's nominee. As a result, the outcome will depend on which party does a better job of mobilizing its supporters and appealing to a small group of swing voters in 10 or 12 battleground states.
Article
Do our models of political behavior bear any resemblance to reality? Forecasting elections is one opportunity to assess whether our models of voting behavior are accurate. Over the past few decades, political scientists have been willing to put themselves out there to forecast elections. Explaining a past event allows us the ability to retrofit our models before we make them available to the broader community. In short, forecasting elections provides us the opportunity to develop humility. The forecasting community has done a reasonable job over the past few elections. Aside from 2000, forecasters have been largely accurate. Even in 2000, the forecasting community can claim a modest victory. The community was right about the popular vote winner; it just happened that the popular vote winner lost the election that counts—the Electoral College.
Article
This article presents forecasts of the 2012 presidential and US House of Representatives elections. The presidential forecasts are of the national two-party presidential vote percentage for the in-party candidate and are based on the trial-heat and economy forecasting equation and its companion convention-bump equation. The House election forecast is of the net seat change for the Democratic Party from 2010 to 2012. This forecast is produced from two versions of a seats-in-trouble forecasting equation.