ArticlePDF Available

Figures

Content may be subject to copyright.
Accuracy of Vot e E xp ecta ti on S urve ys in Fo re cas ti ng E le ctio ns
Forthcoming (subject to changes) in Public Opinion Quarterly
Andreas Graefe
Center for Advanced Studies
Institute of Communication Studies and Media Research
LMU Munich, Germany
a.graefe@lmu.de
Abstract. Simple surveys that ask people who they expect to win are among the most
accurate methods for forecasting U.S. presidential elections. The majority of respondents
correctly predicted the election winner in 193 (89%) of 217 surveys conducted from 1932 to
2012. Across the last 100 days prior to the seven elections from 1988 to 2012, vote expectation
surveys provided more accurate forecasts of election winners and vote shares than four
established methods (vote intention polls, prediction markets, econometric models, and expert
judgment). Gains in accuracy were particularly large compared to polls. On average, the error of
expectation-based vote-share forecasts was 51% lower than the error of polls published the same
day. Compared to prediction markets, vote expectation forecasts reduced the error on average by
6%. Vote expectation surveys are inexpensive, easy to conduct, and the results are easy to
understand. They provide accurate and stable forecasts and thus make it difficult to frame
elections as horse races. Vote expectation surveys should be more strongly utilized in the
coverage of election campaigns.
Keywords: betting markets, citizen forecasts, combining forecasts, Condorcet’s jury
theorem, econometric models, election forecasting, expertise, FiveThirtyEight, forecast accuracy,
IEM, Iowa Electronic Markets, Occam’s Razor, political economy models, wishful thinking
Acknowledgments: J. Scott Armstrong, Alfred Cuzán, Michael Lewis-Beck, Andreas
Murr, and Christopher Wlezien provided helpful comments. I also received valuable suggestions
when presenting the manuscript at the 2013 Annual Conference of the International
Communication Association in London, the 2013 International Symposium on Forecasting in
Seoul, and the 2013 APSA Annual Meeting in Chicago. Bettina Zerwes helped with collecting
data. Jamie Graefe did editorial work.
“Who do you think will win the U.S. presidential election?”
Pollsters have regularly asked variations of this question since the 1940s. Yet, little is
known about the relative accuracy of such vote expectation surveys compared to alternative
methods for forecasting elections. Although researchers have long demonstrated that responses
to the vote expectation question, also known as “citizen forecasts”, provide accurate predictions
of election outcomes (Lewis-Beck and Skalaban 1989, Lewis-Beck and Tien 1999, Lewis-Beck
and Stegmaier 2011, Murr 2011), few studies compare their accuracy to established benchmarks.
The present study provides empirical evidence on the relative accuracy of vote
expectation surveys for forecasting U.S. presidential elections by comparing their results to
predictions from four established methods: traditional polls, prediction markets, expert judgment,
and quantitative models.
Established methods for forecasting U.S. presidential elections
As long as there have been elections, people have tried to predict their results. Currently,
the most common methods for election forecasting are polls, prediction markets, expert
judgment, and quantitative models.
Expert judgment
Judgment of political insiders and experienced election observers were used to forecast
elections long before the emergence of scientific polling (Kernell 2000); and they still are. The
common assumption is that experts have experience in reading and interpreting polls, assessing
their significance during campaigns, and estimating the effects of recent or expected events on
the aggregate vote. Given their omnipresence, surprisingly little is known about the relative
accuracy of experts’ judgment for election forecasting.
One study found that experts, in this case politicians, provided more accurate predictions
of the outcomes of two controversial ballot measures than voters who had just left the voting
booths (Lemert 1986). Another study compared the accuracy of experts (political scientists,
journalists, and editors) and non-experts in predicting vote-shares in the 2006 Swedish
Parliamentary election. At the individual level, the experts provided more accurate forecasts than
the non-experts. Interestingly, however, the results were reversed when calculating average (and
median) forecasts within each group: the combined forecasts from the non-experts were more
accurate than the combined forecasts from the experts (Sjöberg 2009).
Polls
Traditional polls ask respondents for whom they intend to vote if the election were held
today. That is, polls do not provide predictions; they provide snapshots of public opinion at a
certain point in time. However, this is not how the media commonly treat polls. Polling results
are routinely interpreted as forecasts of what will happen on Election Day. This can result in
poor predictions, in particular if the election is still far away, because public opinion can be
difficult to measure and fragile over the course of a campaign (Campbell 1996). However,
researchers found ways to deal with these problems and to increase the accuracy of poll-based
predictions by combining polls and projecting the results to Election Day.
Combining polls to reduce measurement error
There is often high variance in the results of polls by different survey organizations, even
if these polls were conducted at around the same time. Such variance can be caused by sampling
problems, non-responses, and faulty processing (Erikson and Wlezien 1999). Therefore, one
should not look at the results from single polls. Rather, one should combine polls that were
conducted at around the same time. The reason is that the systematic (and random) errors that are
associated with individual polls tend to cancel out in the aggregate (Graefe et al. 2014). The good
news is that combining has impacted how people nowadays consume polls and online polling
aggregators such as realclearpolitics.com and pollster.com have become increasingly popular.
Projecting poll results to Election Day
Polls conducted by the same survey organization, and by the polling industry as a whole,
can fluctuate widely across the course of a campaign. The reason is that a large share of the
electorate has not spent much time thinking about the important issues and the candidates’
positions if the election is still far away. As a result, people’s response behavior in early polls is
influenced by campaign events such as conventions (Campbell, Cherry, and Wink 1992) and
debates (Benoit, Hansen, and Verser 2003).
The effects of such events on the outcome of high-visibility elections such as U.S.
presidential elections are limited, however. As the election nears, people are less influenced by
the latest campaign events and have formed stable vote intentions based on a combination of
information gleaned during the campaign, such as the state of the economy, and their basic
predispositions, such as ideology and party identification (Gelman and King 1993). Therefore, it
is not until shortly before Election Day that polls provide accurate forecasts.
However, researchers found ways to utilize early polls for use in forecasting by
calculating poll projections, as they are termed hereafter. Poll projections take into account the
historical record of polls in order to make a forecast. For example, assume that the incumbent
leads the polls by fifteen points in August. In analyzing historical polls conducted around the
same time along with the respective election outcomes, one can derive a formula for translating
August polling figures into an estimate of the incumbent’s final vote share in November. This is
commonly done by regressing the incumbent’s share of the vote on his polling results during
certain time periods before the election. Prior research found that such poll projections are much
more accurate than treating raw polls as forecasts (e.g., Erikson and Wlezien 2008, Campbell
1996).
Calculating combined poll projections
One can also combine both strategies (i.e., combining polls and calculating poll
projections) to generate poll-based forecasts. One study first calculated rolling averages of all
polls that were published in a one-week period and then used these results to calculate poll
projections. This procedure resulted in large gains in accuracy. Across the last 100 days prior to
each of the six elections from 1992 to 2012, such combined poll projections reduced the error of
a randomly picked poll that was published the same day by 39% (Graefe et al. 2014).
Prediction markets
Prediction (or betting) markets allow people to bet on the election outcome. The resulting
betting odds can then be interpreted as forecasts. Such markets were already popular in the late
19th and early 20th century, when newspapers such as the New York Times regularly reported the
latest predictions. However, around the time of World War II, prediction markets began to
disappear, likely due to a combination of factors such as the rise of the polling industry in the
1930s, the introduction of laws to eliminate organized election betting, and the emergence of
alternative betting opportunities such as horse-racing (Rhode and Strumpf 2004). It took almost
half a century, and the rise of the Internet, for the method to be rediscovered. In 1988,
researchers at the University of Iowa launched the online Iowa Electronic Markets (IEM) to
predict the U.S. presidential elections held in the same year. Since then, interest in prediction
markets resurged.
Studies of prediction market accuracy for election forecasting commonly compare the
daily market forecasts to results from polls published the same day. These studies generally find
that prediction markets yield more accurate forecasts than single polls. For example, Berg,
Nelson, and Rietz (2008) compared the accuracy of IEM forecasts to results from nearly 1,000
polls across the five U.S. presidential elections from 1988 to 2004. The IEM forecasts were more
accurate than single polls 74% of the time. However, as outlined above, single polls provide poor
predictions and thus only serve as a weak benchmark. Erikson and Wlezien (2008) accounted for
this problem and compared the IEM forecasts to poll projections, using the same data and time
period as analyzed by Berg, Nelson, and Rietz (2008). The authors found that poll projections
were more accurate than the IEM. I extended these analyses and compared the IEM forecasts to
combined poll projections, using the approach suggested by Graefe et al. (2014). Across the last
100 days prior to each of the seven elections from 1988 to 2012, the IEM forecasts yielded an
average error of 1.7 percentage points, which is 17% below the corresponding error of combined
poll projections of 2.0 percentage points (cf. Table 3).
Quantitative models
A popular theory of electoral behavior is that elections are referenda on the incumbent’s
performance. That is, voters are expected to reward the government for good performance and
punish the incumbent party otherwise. Since the late 1970s, economists and political scientists
tested this theory by developing quantitative models to predict election results. Most models are
based on multiple regression analysis of two to five predictor variables, which typically capture
economic conditions, the incumbent’s popularity, and how long the President (or his party) have
controlled the White House. The development and testing of these models has become a well-
established sub-discipline of political science and the models’ forecasts are regularly published
about two months prior to Election Day in scientific journals.1
These models predict the correct election winner most of the time. Across the six
elections from 1992 to 2012, 34 of 39 forecasts of seven well-known models correctly predicted
the winner. However, the models’ performance in predicting vote shares is mixed. Their mean
1 Since the 1992 election, forecasts from established models were published prior to the respective elections in
special symposiums in Political Methodologist 5(2), American Politics Research 24(4) and PS: Political Science
and Politics 34(1), 37(4), 41(4), and 45(4).
absolute error (MAE) was three percentage points, and ranged from zero to ten points (cf.
Appendix I).
Vote expectation surveys
Why not simply ask voters whom they expect to win and then use the aggregate result as
forecast? A common concern with using expectations as forecasts is that people’s expectations
are influenced by their preferences. In the case of elections, this means that people tend to predict
their preferred candidate to win. This bias, which is known as wishful thinking, is long known to
be common in the context of elections. Hayes (1936) reported results from a 1932 pre-election
survey of an unrepresentative sample of 8,419 men and women and finds that 72% of Hoover
supporters predicted Hoover to win, whereas 91% of Roosevelt supporters predicted Roosevelt
to win; Roosevelt won by a landslide. In their seminal study of voting behavior, Lazarsfeld,
Berelson, and Gaudet (1948) reported a strong relationship between people’s vote intentions and
their expectations of who will win. Granberg and Brent (1983) studied wishful thinking across
the eight U.S. presidential elections from 1952 to 1980. Their analysis revealed a strong positive
relationship between expectations and candidate preferences (r=.8). Since these early studies,
evidence has accumulated. Wishful thinking occurs in all types of elections, from local referenda
to national elections, and across various countries. See Miller et al. (2012) for an overview of
recent research.
Although people are subject to wishful thinking, most of them can still correctly predict
the outcome of elections. In their pioneering study of “citizen forecasts”, Lewis-Beck and
Skalaban (1989) found that, across the eight U.S. presidential elections from 1956 to 1984, 69%
of respondents to the ANES vote expectation question correctly predicted the election winner. I
updated their results by analyzing data from all ANES pre-election surveys for the sixteen
elections from 1952 to 2012. I found that, on average, 70% of 30,573 respondents correctly
predicted the election winner. In addition, I analyzed the accuracy of predictions of which
candidate would win the election in the respondent’s state.2 For this task, 69% of 23,301
2 In 11 of the 16 ANES surveys from 1952 to 2012, respondents were asked to predict the election winner in their
state. The state-level question was not asked prior to the four elections from 1956 to 1968 as well as prior to the
2000 election.
responses predicted the correct winner.3 Others obtained similar results for multi-party elections
in the UK. Across thirteen elections conducted between 1951 and 2005, 60% of survey
respondents correctly predicted which party will win the governing majority of seats (Lewis-
Beck and Stegmaier 2011). Another study analyzed data from the 2010 British Election Study
Internet survey, in which respondents were asked to predict which party would win the election
in their local constituency. Of the 13,334 respondents that provided a valid answer to the
expectation question, 69% correctly predicted the winner (Murr 2011). See Table 1 for an
overview of these results.
The accuracy of individual vote expectations depends on the characteristics of the
forecaster and the electoral context. In their micro-level analysis of citizen forecasts of the eleven
ANES surveys from 1956 to 1996, Lewis-Beck and Tien (1999) showed that more educated
individuals make more accurate predictions and that political involvement harms accuracy.
Furthermore, individual forecasts are less accurate for tight races and if the election is further
away.
The available evidence reveals that the chance for a typical respondent to correctly
predict the election winner is usually above 50%, and thus more accurate than guessing. In other
words, individuals are more likely to get the election winner right than wrong. Under such
conditions, pooling the individual estimates into a group estimate will inevitably yield a correct
decision. The mathematical proof for this relationship was provided in the 18th century by
Marquis the Condorcet and has become known as Condorcet’s jury theorem (Condorcet 1994).
The theorem allows for calculating the probability that a given group of individuals arrives at a
correct decision. For example, the pooled estimate of five individuals who are each correct 60%
of the time is expected to be correct 68% of the time. If the accuracy level of the five individuals
is at 70%, the group estimate should get it right about 84% of the time. In general, given that
individual accuracy is better than chance, Condorcet’s jury theorem implies that the probability
for a group to arrive at a correct decision rapidly increases towards 100% as the number of
individual estimates increases.
3 Analyses of the ANES surveys in the present paper consider only responses that predicted one of the major two
parties’ candidates to win.
Murr (2011) was the first to identify the powerful theoretical link between Condorcet’s
jury theorem and the benefits of aggregating vote expectations. In his study of the 2010 British
election, the majority vote of respondents from the same constituency correctly predicted the
winner in 537 (86%) of 627 constituencies. Thus, compared to the individual forecasts, which
were correct 69% of the time, pooling estimates increased the likelihood of a correct forecast by
17 percentage points. Murr (2011) further analyzed the conditions under which pooling
expectations is most beneficial. In general, the accuracy of aggregated expectations increases
with the number of individual expectations and for races with large winning margins. In addition,
aggregated vote expectations tended to be somewhat more accurate if variation in interview dates
was higher.
Table 1 provides further evidence from other data sets on the gains from pooling
expectations. For example, Lewis-Beck and Stegmaier (2011) provided similar results for
forecasts of British election outcomes at the national level. While, on average, individuals were
correct 60% of the time, the majority of vote expectations correctly predicted the election result
in ten of thirteen elections, a hit rate of 77%. For U.S. presidential elections, I found that the
aggregated responses to the ANES vote expectation question predicted the correct election
winner in thirteen of the sixteen elections from 1952 to 2012, a hit rate of 81%. That is,
compared to the average individual, who was correct 70% of the time, pooling estimates
increased the chance of a correct forecast by 11 percentage points. At the state level, pooled
expectations correctly predicted the winner in 329 (82%) of 399 state races. In comparison, the
average individual was correct 69% of the time.4
These results suggest that vote expectations are useful for election forecasting. However,
little is known about how accurate such vote expectation surveys are relative to other methods, as
few studies compare their accuracy to established benchmarks.
4 As noted above for the case of combining polls, the phenomenon that aggregating individual estimates yields more
accurate forecasts also holds for quantitative estimates. Hogarth (1978) shows analytically that combining
judgments is most useful if individual judges possess diverse information. Adding more forecasts increases
accuracy, although at a diminishing rate of improvement. The power of combining for generating accurate
predictions is one of the major findings from forecasting research conducted since the 1970s, which impacted many
fields such as weather forecasting, economic forecasting, and political forecasting (Graefe et al. 2014).
Vote expectation surveys v. polls
One exception is a study by Rothschild and Wolfers (2012), who analyzed the relative
accuracy of the vote expectation question and the vote intention question when both are asked in
the same survey. Based on an analysis of ANES data from the fifteen U.S. presidential elections
from 1952 to 2008, the authors found that expectations were more accurate than intentions when
predicting election winners, vote shares, and probabilities of victory.
A possible explanation for this result is that expectations capture more information than
intentions. First, vote intention polls ignore information from respondents who are undecided,
who do not want to reveal for whom they are going to vote, and who do not intend to vote at all.
However, these people may also have valuable expectations about the election outcome. Second,
expectations likely incorporate not only information about one’s own vote intention but also
about the intentions of others, as well as information from other sources. For example, people
might form expectations of the election outcome from following general media coverage of the
campaign, reading the latest polls and expert analyses, and talking to peers.
In fact, as shown by Rothschild and Wolfers (2012), each person’s expectation is
equivalent to a multi-person intention poll. The authors estimated that one vote expectation
contains about as much information as twenty vote intentions. This is a major advantage of
expectation surveys since sample size and composition are less critical. Rothschild and Wolfers
(2012) demonstrated this by calculating forecasts based on expectations from biased subsamples
(i.e., only Democrats and only Republicans). In both cases, the expectation-based forecasts of the
biased subsamples provided more accurate forecasts than the complete sample of vote intentions.
These results suggest that responses to the vote expectation question provide more
accurate forecasts of what will happen on Election Day than the vote intention question.
However, as outlined earlier, single polls are a poor benchmark of forecast accuracy. The present
study thus compares the relative accuracy of vote expectations and sophisticated poll-based
forecasts such as combined poll projections.
Vote expectation surveys v. prediction markets and experts
In eliciting expectations, forecasts from experts and prediction markets are closely related
to vote expectation surveys. One major difference between the three approaches lies in the
composition of the sample. While vote expectation surveys sample respondents randomly, the
other two approaches rely on selected (as in the case of expert surveys) or self-selected (as in the
case of prediction markets) experts.5 Given the similarities of these methods, it is surprising that
few researchers studied their relative accuracy.
One study compared responses to the ANES vote expectation question to forecasts from
the IEM vote-share markets during the past two weeks prior to each of the five U.S. presidential
elections from 1988 to 2004. The relationship between the respective forecasts and the final vote
was slightly higher for the vote expectations compared to the IEM forecasts (Holbrook 2010).
Another study found that an online expectation survey of more than 19,000 respondents was
more accurate than the Intrade prediction market when forecasting winning probabilities for the
2008 U.S. presidential election (Miller et al. 2012). Finally, a study of the 2006 Swedish
parliament elections found that combined vote-share forecasts of non-experts (members of the
public) were more accurate than combined forecasts of experts (Sjöberg 2009).
These results may surprise. How is it possible that the combined predictions of regular
citizens perform as well as – or even better than – combined predictions of (self-)selected
experts? In particular, since at the individual level, Sjöberg (2009) found what one would expect:
experts were more accurate than the less interested, less informed, and less educated non-experts.
A possible explanation is variance in the heterogeneity of the groups. In Sjöberg’s study,
the non-experts varied in demographics, and their party preferences were in line with the general
public (i.e., the final election result). In contrast, the experts (three groups consisting of political
scientists, journalists, and editors) were mostly male and well educated. In addition, the experts
showed a particularly low preference for the Conservative party (the second strongest party in
the polls) and high preferences for parties that were less popular among the general public (e.g.,
the Liberals and the Center Party). As a result, the less diverse expert group was likely biased in
the same direction. In such a situation, combining judgments is of limited value, since the
individual estimates are highly correlated and biases do not cancel out in the aggregate (Graefe et
al. 2014, Hogarth 1978).
5 In addition, prediction markets differ from regular surveys in how they aggregate information. Prediction market
participants buy and sell shares, whose prices reflect the combined expectations of all participants. That is,
participants can win and lose money depending on their performance and thus have an incentive to be accurate.
Since participants should only become active if they think they know better than the market as a whole, they are
often referred to as “self-selected experts”.
The same appears to be true for prediction markets. One study analyzed trading behavior
in two markets operated by the IEM: the 1988 U.S. presidential election vote-share market and
the 1993 Canadian House of Commons Market. In both markets, participants exhibited
substantial biases. Compared to the average trader, participants bought more shares of candidates
that they favored and sold more shares of candidates that they did not support (Forsythe, Rietz,
and Ross 1999). In other words, even market participants, who have an incentive to be accurate,
depart from rational behavior and exhibit wishful thinking. This would be less concerning if
participants formed a heterogeneous group, as individual errors would likely cancel out in the
aggregate. However, as shown in a study on the 1988 IEM, this might often not be the case.
Participants were predominantly white, male, well educated, and belonged to the middle and
upper income categories. In addition, participants tended to be more Republican and less
independent in their partisan leanings, and were more politically active than the general public
(Forsythe et al. 1992).
Given these findings, there is reason to believe that vote expectation surveys can provide
forecasts that are competitive with expert surveys and prediction markets.
Vote expectation surveys v. quantitative models
I was unable to find prior empirical evidence on the relative accuracy of vote
expectations and quantitative models. The advantage of models is that they follow a structured
approach to forecasting and include much information about historical elections, such as the
influence of the state of the economy, the popularity of the incumbent, and the time the
incumbent was in the White House. While the accuracy of single models can vary widely across
elections, one usually gets accurate forecasts when combining forecasts from different models
(Graefe et al. 2014). However, a disadvantage of quantitative models is their limited ability to
incorporate information about the specific context of a particular election such as an economic
crisis, threat of terrorism, or some scandal.
Accuracy of vote expectation surveys in forecasting U.S. presidential elections
The following analysis provides empirical evidence on the relative accuracy of vote
expectation surveys compared to polls, prediction markets, expert judgment and quantitative
models for the task of forecasting U.S. presidential elections.
Method
All data and calculations are publicly available (Graefe 2013).
Time horizon and error measures
The methods’ forecast accuracy is analyzed across the last 100 days prior to the seven
elections from 1988 to 2012. The hit rate and the absolute error were used as measures of
accuracy. The hit rate is the percentage of forecasts that correctly predict the winner. For
methods that provide forecasts of two-party popular vote shares, the candidate with a vote share
of more than 50% is predicted to win the election. In the case that each candidate is predicted to
gain 50% of the popular vote, a tie is recorded. Ties score as half of a correct prediction. The
absolute error is the absolute deviation of the predicted and the actual two-party popular vote for
the incumbent party’s candidate.
Data and forecast calculations
To allow for fair comparisons, all forecasts are calculated as if they were made ex ante. That
is, when calculating the forecasts I used only data that would have been available at the time of
the election. In addition, comparisons include only forecasts made around the same time.
Vote expectation surveys
A total of 217 vote expectation surveys were collected across twenty elections from 1932 to
2012. For example: „Regardless of whom you support, and trying to be objective as possible,
who do you think will win the presidential election in November (2008)--Barack Obama or John
McCain?“ (Gallup Poll, October 23-26, 2008). All surveys obtained were conducted within 150
days prior to Election Day. The data set includes sixteen ANES surveys, one for each of the
elections from 1952 to 2012, and the 1932 survey reported by Hayes (1936). The remaining 200
surveys were derived from the iPoll Databank of the Roper Center for Public Opinion Research.
For the 1936 election, no survey was found.
Vote expectation surveys provide direct forecasts of who will win; the candidate that the
majority of respondents expect to win is predicted as the election winner. Table 2 provides an
overview of the accuracy of the collected surveys for this task. The vote expectation surveys
correctly predicted the winner in 193 (89%) of the 217 surveys. Only 18 (8%) of the surveys
predicted the wrong winner, half of which were conducted during the very close 2000 election.
The remaining six surveys predicted a tie.
For each survey that was published during the past 100 days prior to Election Day, the two-
party percentage of respondents that expected the incumbent party candidate to win was
recorded.6 This figure was kept constant on days without any published surveys and was
replaced with the results from a more recent survey once available. If more than one survey was
published on the same day, the results of all surveys from that date were averaged.7
While the results of vote expectation surveys allow for making quick forecasts of who
will win an election, they cannot be directly interpreted as vote share forecasts. For example, a
survey that reveals that 60% of respondents expect the incumbent candidate to win does not
mean that the incumbent can be expected to gain 60% of the vote. In order to translate vote
expectation survey results into vote share forecasts it is thus necessary to use data from historical
surveys. That is, one estimates how a hypothetical incumbent lead of 60-40 in a vote expectation
survey translated to the incumbent’s final vote share in past elections. A simple approach for
estimating this relationship is linear regression analysis (Lewis-Beck and Tien 1999). Thereby,
the incumbent party’s actual two-party popular vote share is regressed on the results from the
vote expectation surveys. When using all 217 vote expectation surveys from 1932 to 2012,
regression analysis yields the following vote equation:
V = 41.0 + 17.1 E + e
(20.4) (78.5)
R2 = .66; SEE = 2.2;
where V is the actual two-party popular vote share of the incumbent party, E is the two-party
percentage of survey respondents that expect the incumbent party candidate to win, and e is the
error term. The figures in parentheses show the t-values.
The results show that the vote expectation survey results explain about two thirds of the
variance in the incumbent’s popular vote share. If the share of respondents that expect the
incumbent to win increases by 10%, the incumbent’s vote share increases by 1.7 percentage
points. The model also reveals the strong degree of partisanship among U.S. voters. As noted by
6 If no survey was published on day 100 prior to Election Day, the most recent available survey was used as the
starting point. This is the reason why surveys were collected up to 150 days prior to Election Day.
7 Instead of a survey’s publication date, the iPoll databank reports the last day a survey was in the field. For the
present analysis, I therefore assumed that the surveys were published two days later. I also tested whether using
different values for the publication delay (i.e., from zero to five days) would affect the results; they did not.
Campbell (1996, 423), “no matter how bad the campaign goes for a party, it can count on
receiving about 40% of the two-party vote; no matter how well a campaign goes for a party, it
will receive no more than about 60% of the two-party vote.” The vote expectation model is
consistent with this view. The model predicts that the incumbent would receive at least 41% of
the vote, even if nobody expected him to win the election (E=0). Vice versa, if all survey
respondents expected the incumbent to win (E=1), the model would predict the incumbent to
receive at maximum 58.1% of the vote.8
Since this vote equation is based on an in-sample estimation using all available data, it
cannot be used to evaluate the predictive accuracy of vote expectation surveys for past elections.
For this, it is necessary to calculate forecasts by using only data that would have been available
at the time of making a forecast. Such ex ante forecasts were calculated by successive updating.
For example, to estimate the equation for predicting the 1988 election, only the 40 surveys
available from 1932 to 1984 were used, while the 2004 equation is based on the 124 surveys
available through 2000, and so on.9 With this approach, the estimated vote equation for a
particular year does not change over the course of a campaign.10 Thus, the vote equation
estimated above would be used to generate forecasts of the upcoming 2016 election. The
resulting equations were then used to translate the results of the most recent vote expectation
survey available at any day prior to the election under observation into a vote share forecast.11
Polls
Polls that were conducted within 100 days prior to each of the sixteen elections from 1952
to 2012 were obtained from Graefe (2014). For each poll, the two-party percentage of
8 Lewis-Beck and Tien (1999) ran the same regression analysis using data from the eleven ANES surveys from 1956
to 1996. With an intercept of 39.5 and an estimated coefficient of 21.0, the results are comparable, despite their
small sample. This provides additional support for the robustness of the estimated relationship.
9 The number of available surveys for estimating the vote equation for a particular election can be derived from
Table 2.
10 Ideally, one would follow an approach similar to the calculation of poll projections. That is, one would estimate
different vote equations depending on the time to Election Day. Such an approach would account for the uncertainty
that occurs over the course of a campaign, which is reflected in the decreasing accuracy of people’s expectations for
long time horizons (Lewis-Beck and Skalaban 1989, Lewis-Beck and Tien 1999, Miller et al. 2012, Murr 2011).
Unfortunately, limited historical data currently preclude such an analysis, as only few surveys are available for early
elections (cf. Table 2).
11 Data from the ANES surveys are not available until months after the election. Therefore, the ANES surveys could
not be used to calculate ex ante forecasts of the election outcome. However, the ANES data were used to estimate
the vote equations for forecasting succeeding elections.
respondents that intended to vote for the candidate of the incumbent party was recorded. If more
than one poll ended on the same day, the results of all polls ending that date were averaged. On
days without any polls ending, the most recent poll from preceding days was used.
Three different poll-based forecasts were used as benchmarks: (1) single polls, (2)
combined polls, and (3) combined poll projections. The single polls benchmark simply interprets
the result of a single poll published on a particular day as forecast of the election outcome. The
combined polls benchmark calculates rolling averages of all polls released over a 7-day period.
The third benchmark, combined poll projections, was adopted from Graefe et al. (2014). That is,
for each of the 100 days prior to an election, starting with 1952, the incumbent’s actual two-party
share of the popular vote was regressed on the combined polls value for that day. This process
produced 100 vote equations (and thus poll projections) per election year. Again, successive
updating was used to calculate ex ante poll projections.
Prediction markets
Daily prediction market data from the IEM vote-share markets were obtained from the IEM
website (http://tippie.uiowa.edu/iem). On these markets, people buy and sell future contracts
according to their own expectations of the candidates’ final vote shares. The market price
represents the combined forecast of all market participants. To put the prediction market
forecasts on equal footing with vote expectation surveys and polls, two-party forecasts were
calculated by ignoring any third-party options. The last traded prices per day were used as the
market forecasts.
Experts
One expert survey was available for each of the elections in 1992 and 2000, four surveys
were available for each of the elections in 2004 and 2008, and five surveys were available for the
2012 election.12 The average number of experts per survey ranged from ten to fifteen. In each
survey, experts were asked to provide forecasts of the popular vote shares. Individual and
average expert forecasts were compared to the average of the daily vote expectation forecasts
during the seven days prior to the publication of the expert survey.
12 The 1992 survey was published in The Washington Post. Pundits’ brew: How it looks; Who’ll win? Our fearless
oracles speak, November 1, 1992, p. C1, by David S. Broder. The 2000 survey was published in The Hotline.
Predictions: Potpourri of picks from pundits to professors, November 6, 2000. The thirteen surveys conducted for
the three elections from 2004 to 2012 were derived from Graefe (2014).
Quantitative models
The present study uses forecasts from seven established quantitative models.13 These
models, along with their forecasts, were published in Political Methodologist 5(2), American
Politics Research 24(4) and PS: Political Science and Politics 34(1), 37(4), 41(4), and 45(4).
Most model forecasts were published about two to three months prior to the election. Therefore,
the model forecasts were compared to the average of the daily vote expectation forecasts
calculated across 90 to 60 days prior to Election Day, which is about the month of August in the
election year.
Results
The following analysis provides empirical evidence on the relative accuracy of vote
expectation surveys in predicting election winners and vote shares compared to each of the four
established methods discussed above. The analysis is based on forecasts that were made during
the last 100 days prior to the seven elections from 1988 to 2012.
Relative accuracy compared to polls and prediction markets
The accuracy of daily forecasts from vote expectations surveys, polls, and the IEM is
analyzed across the full 100-day period prior to Election Day. Table 3 shows the methods’ hit
rate and mean absolute error (MAE) across and for each of the seven elections from 1988 to
2012.
On average, vote expectation surveys were most accurate in predicting election winners
and vote shares. If one had simply relied on the most recent vote expectation survey available on
a particular day, one would have achieved an average hit rate of 92%.14 That is, one would have
correctly predicted the winner more than nine out of ten times. In comparison, if one had relied
on the most recent single poll at the same day, one would have predicted the correct winner only
13 These models were developed by Abramowitz, Campbell, Erikson & Wlezien, Holbook, Lewis-Beck & Tien,
Lockerbie, and Norpoth (cf. Appendix I).
14 Due to the disagreement of the popular and the electoral vote in the 2000 election, the calculation of hit rates for
that particular election requires special consideration. For each of the benchmark methods, a forecast of more than
50% for Gore was coded as correct, because these methods predict popular vote shares. In contrast, the vote
expectation question asks respondents who will be elected president. Thus, responses that expected Bush to win
were coded as correct. Readers who disagree with this coding could invert the vote expectation hit rates for the 2000
election. In that case, for example, the hit rate for the 2000 election reported in Table 3 would equal 40%, and the
average hit rate across the seven elections would equal 89%. The results for the vote share forecasts are not affected
by the special case of the 2000 election.
79% of the time. As expected, combining polls (86% correct predictions) and calculating
combined poll projections (88%) increased upon the accuracy of single polls. Surprisingly, with
a hit rate of 79%, the IEM vote-share markets were no more accurate than single polls.15
In addition, vote expectation surveys provided highly accurate vote-share forecasts. The
vote expectation forecasts missed the final election results on average by 1.6 percentage points
and were thus more accurate than each of its competitors. Compared to single polls, which
missed the final election result on average by 3.2 percentage points, vote expectation surveys
reduced the error by 51%.16 In other words, if one had relied on the most recent vote expectation
forecast rather than on the results from the most recent single poll, one would have cut the
forecast error by more than half. Compared to combined polls and combined poll projections,
error reduction was 46% and 21%, respectively. Gains in accuracy compared to the IEM vote-
share markets were smaller (6%).
Relative accuracy compared to experts
Table 4 compares the accuracy of vote expectation surveys and experts’ judgment made
around the same time. In every single comparison, the vote expectation surveys predicted the
correct winner. In comparison, the combined experts predicted the correct winner 70% of the
time and were thus slightly more accurate than the typical expert (66% correct).17
In terms of vote share forecasts, the MAE of vote expectation surveys (1.2 percentage
points) was 32% lower than the MAE of the typical and 14% lower than the MAE of the
combined expert.
Relative accuracy compared to quantitative models
Table 5 shows the hit rates and MAE of vote expectation surveys and the typical and
average forecast of seven quantitative models for the six elections from 1992 to 2012. The vote
expectation surveys as well as the average forecast of all available models correctly predicted the
winner in each election. The track record of the individual models is not perfect with two models
15 I also analyzed the accuracy compared to the IEM winner-takes-all markets, which were first launched in 1992.
The winner-take-all markets were specifically designed to predict popular vote winners and thus provide the better
benchmark for this type of task. These markets achieved a hit rate of 88% across the six elections from 1992 to
2012. The corresponding hit rate of vote expectation surveys for the same time period was 93%.
16 The error reduction in % is calculated as 1 - MAE(vote expectations) / MAE (single polls) = 1 - 1.57 / 3.23 = 0.51.
17 The performance of the typical expert is the performance that one would achieve if one would randomly pick an
expert.
missing the winner in 2012, and one each in 1992, 2004, and 2008. This results in an average hit
rate of 86% for the typical model. In other words, if one had randomly picked a model in each
election, one would have correctly picked the winner in six of the seven elections.
In terms of vote share forecasts, the vote expectation surveys yielded a MAE of 1.5
percentage points and thus reduced the corresponding error of the typical and combined model
forecast by 50% and 36%, respectively.18
Discussion
There has been much progress in our ability to forecast elections over the last three
decades. Combining polls and projecting their results to Election Day yielded substantial
improvements in accuracy compared to single polls. Researchers have developed econometric
models that can quite accurately predict election outcomes from structural information that is
available months before Election Day. Finally, prediction markets reappeared as a powerful tool.
One simple method, which has existed at least since the advent of scientific polling, has
been largely overlooked in this development: surveying people on who they expect to win.
Across the past seven U.S. presidential elections, vote expectation surveys provided more
accurate forecasts of election winners and vote shares than any other established method. Gains
in accuracy were particularly large compared to the method that is still standard practice for the
coverage of election campaigns: single polls. The error of vote-share forecasts derived from vote
expectation surveys was 51% lower than the corresponding error of a single poll. Nevertheless,
the widespread belief that polls provide accurate forecasts remains.
Relative performance of polls and vote expectation surveys in the 2012 election
The National Council on Public Polls (NCPP) analyzed 25 national polls that were
conducted within the final week of the 2012 campaign. The NCPP concluded that, with an
average error of 1.46 percentage points, the polls “came close to the election outcome”.19
However, the NCPP failed to compare the polls’ performance to a benchmark. I calculated the
18 The careful reader might note an apparent discrepancy between the expectation surveys’ perfect hit rate and the
rather large error in their vote share forecasts for the 2000 election. The reason is that, at the time the forecasts are
compared, which is around three months prior to Election Day, a large portion of respondents expected Bush to
defeat Gore. While this led to accurate predictions of the election winner (i.e., a perfect hit rate), the vote share
forecasts underestimated Gore’s two-party vote (see also footnote 9 for further details).
19 The NCPP’s “Analysis of Final 2012 Pre-Election Polls” is available at www.ncpp.org.
corresponding error of forecasts from vote expectation surveys and the IEM for the same time
period. Both benchmarks provided much more accurate predictions than the NCPP poll sample.
The vote expectation surveys yielded an error of 0.71 and thus reduced the error of polls by more
than 50%. With an error of 0.99 percentage points, the IEM were 30% more accurate than the
polls. Note that these gains in accuracy were obtained for the last week prior to the election, a
time in which polls usually provide very accurate forecasts.20
Figure 1 extends this analysis and shows the relative accuracy of vote share predictions of
110 polls and 20 vote expectation surveys that were published during the last 100 days prior to
the 2012 election. The vertical axis shows Obama’s predicted lead in the two-party popular vote
and the dotted grey line depicts the final election outcome; Obama won the election with a four-
point advantage over Romney. Single polls varied wildly and predicted anything from a three-
point lead for Romney to an eighteen-point lead for Obama (standard deviation: 4.3). In
comparison, the vote expectation survey forecasts ranged from a one-point to a six-point lead for
Obama (standard deviation: 1.3). That is, vote expectation survey forecasts were much more
stable, less extreme, and closer to the election result than individual polls throughout the 100-day
time horizon. The elections from 1988 to 2008 show a similar pattern (cf., Appendix III).
Relative performance of FiveThirtyEight.com and vote expectation surveys in the 2012
election
One cannot discuss forecasts of the 2012 election without mentioning Nate Silver’s
FiveThirtyEight.com, a polling aggregation website that was launched in 2008 and has become
part of the The New York Times online in 2010. Silver uses sophisticated statistical analyses to
analyze the type and extent of biases of single pollsters, and to demonstrate the value of polling
aggregation for forecasting. His forecast model aggregates information from state-level polls by
accounting for the relative performance of different pollsters and considering relationships
between states. In addition, the model incorporates an index of economic indicators, whose
weight decreases as the election nears. Simply put, Silver’s model is an enhanced and much
20 I also compared the accuracy of the final vote expectation forecasts and the final Gallup pre-election poll for the
past seven elections from 1988 to 2012. The vote expectation survey forecasts were more accurate in four of the
seven surveys, with an average error of 1.5 percentage points. In comparison, the error of the final Gallup poll was
27% higher (1.9 percentage points). The numbers are provided in Appendix II.
more sophisticated version of the traditional quantitative models, some of which also combine
polls and economic fundamentals.
FiveThirtyEight has become extremely popular. In the week prior to the 2012 election,
almost three out of four politics visits at the New York Times website included a stop at
FiveThirtyEight. The day before the election, one in five nytimes.com visitors looked at Silver’s
site (Tracy 2012). FiveThirtyEight had become a synonym for election forecasting, which
becomes evident when looking at the relative volumes of Google searches for variants of
“Fivethirtyeight” and “election forecast” (cf., Appendix IV).
I compared the accuracy of vote expectation surveys to Silver’s popular vote forecast.
Figure 2 reports the error of both approaches for the last 100 days prior to the election. Any point
on the lines in the chart shows the average error for the remaining days in the forecast horizon.
For example, if one had relied on the FiveThirtyEight forecast on each of the 100 days prior to
Election Day (i.e., starting from July 29th), one would have achieved a MAE of 0.65 percentage
points. If one had relied on FiveThirtyEight on each day from October 11th, one would have
achieved a MAE of 1.2 percentage points, and so on. The corresponding values for the MAE of
the vote expectation surveys are 0.40 (from July 29th) and 0.76 (from October 11th). As shown in
Figure 2, at any point in the campaign, one would have fared better by relying on the most recent
vote expectation survey instead of relying on the forecasts at FiveThirtyEight.com. Across the
full 100-day forecast horizon, vote expectation surveys reduced the error of FiveThirtyEight on
average by 38%.21
Barriers to the adoption of vote expectation surveys
The accuracy of vote expectation surveys in forecasting elections may surprise given that
the results of vote expectation surveys are rarely reported, not to say ignored, by the media.
21 One advantage of FiveThirtyEight is that the model also provides forecasts at the state level. For the 2012
election, the model’s final forecast correctly predicted the winner in all 51 states. I analyzed data from the 2012
ANES pre-election survey to analyze the performance of vote expectation surveys for the same task. Although the
number of responses for some states was limited and data collection started in early September, two months before
the election, the vote expectation survey performed well. In 49 of 51 states, the majority of respondents correctly
predicted the election winner in that state; the two exceptions were North Carolina and New Hampshire. The results
from this single survey suggest that vote expectation surveys can also provide highly accurate forecasts at the state
level.
Instead, the trial-heat question, which asks respondents for whom they intend to vote, dominates
the media coverage of election campaigns.
One likely reason for the disregard of vote expectation surveys is that people simply do
not know about their accuracy. The present study aims at addressing this gap by comparing the
method to established benchmarks. But additional barriers are likely to hinder the adoption of
vote expectation surveys in practice: people have no faith in simple methods and journalists are
interested in newsworthiness rather than accuracy.
Complexity persuades
Occam’s Razor advises researchers to prefer simple models unless simplicity is offset by
more explanatory power. Since Occam, many famous researchers advocated the use of simple
models. Albert Einstein is reputed to have said that “everything should be made as simple as
possible but not simpler”. Zellner (2004), who coined the phrase “keep it sophisticatedly
simple”, named several Nobel laureates as proponents of simplicity. Vote expectation surveys
adhere to Occam’s Razor; they are easy to conduct, the results are easy to understand, and they
provide forecasts that are at least as accurate as more complex methods.
Unfortunately, simple models often face resistance, because people tend to wrongly
believe that complex solutions are necessary to solve complex problems. Hogarth (2012)
reported results from four studies, which showed that simple models often perform better than
more complex ones. In each case, however, people resisted the findings regarding the
performance of simple models. The same appears to be true for election forecasting. People are
impressed by sophistication and complexity (e.g., FiveThirtyEight.com) and overlook obvious
approaches, such as simply asking people whom they expect to win.
Newsworthiness beats accuracy
Journalists and political commentators need to meet the demands of the news cycle and
constantly look for interesting stories and analyses. In this endeavor, they often select
newsworthiness over accuracy and relevance. In particular, journalists increasingly generate
news by focusing on who is ahead in the polls or linking the latest poll results to campaign
events. As shown in Figure 1, polls from different survey organizations often vary wildly, even if
they are conducted at around the same time. In such a situation, journalists can cherry-pick on
polls that support their story. Thereby, they might be little concerned about the accuracy of a
poll, in particular if the election is still some time away. Such horse race coverage negatively
affects the quality of campaign coverage as it comes at the expense of informing the public about
candidates and their proposed policies (Rosenstiel 2005, Patterson 2005).
In contrast, vote expectation surveys are much more robust and less extreme. Since their
forecast rarely changes, vote expectation surveys are less suited for generating news. It would be
desirable if journalists would focus more on vote expectation surveys. First, voters would be
much better informed about who is really ahead. Second, the stability of vote expectation surveys
makes it difficult to frame the election as a horse race. Rather, journalists could concentrate on
providing explanations for the relative performance of candidates and their proposed policies.
Conclusion
The general election observer is probably most interested in who will win. When it comes
to U.S. presidential elections, vote expectation surveys are likely to provide the best answer to
this question. In addition, the results of such surveys can be translated into highly accurate vote
share forecasts.
Vote expectation surveys are inexpensive and easy to conduct, and the results are easy to
understand. Because vote expectations are much more stable than vote intentions, they are not
suited to frame the election as a horse race. Thus, an increased focus on vote expectation surveys
is likely to positively affect the quality of election campaign coverage.
References
Benoit, William L., Glenn J. Hansen, and Rebecca M. Verser. 2003. "A meta-analysis of the effects of viewing US
presidential debates." Communication Monographs 70 (4):335-350.
Berg, Joyce E., Forrest D. Nelson, and Thomas A. Rietz. 2008. "Prediction market accuracy in the long run."
International Journal of Forecasting 24 (2):285-300.
Campbell, James E. 1996. "Polls and Votes The Trial-Heat Presidential Election Forecasting Model, Certainty, and
Political Campaigns." American Politics Research 24 (4):408-433.
Campbell, James E., Lynna L. Cherry, and Kenneth A. Wink. 1992. "The convention bump." American Politics
Research 20 (3):287-307.
Condorcet, M. J. A. N. de Caritat, Marquis de. 1994. "Essay on the application of probability analyses to decisions
returned by a plurality of people." In Condorcet: Foundations of Social Choice and Political Theory, edited
by Iain McLean and Fiona Hewitt, 11-36. Brookfield, VT: Edward Elgar Publishing (Original work
published 1785).
Erikson, Robert S., and Christopher Wlezien. 1999. "Presidential polls as a time series: the case of 1996." Public
Opinion Quarterly 63 (2):163-177.
Erikson, Robert S., and Christopher Wlezien. 2008. "Are political markets really superior to polls as election
predictors?" Public Opinion Quarterly 72 (2):190-215.
Forsythe, Robert, Forrest Nelson, George R. Neumann, and Jack Wright. 1992. "Anatomy of an experimental
political stock market." The American Economic Review 82 (5):1142-1161.
Forsythe, Robert, Thomas A. Rietz, and Thomas W. Ross. 1999. "Wishes, expectations and actions: a survey on
price formation in election stock markets." Journal of Economic Behavior & Organization 39 (1):83-110.
Gelman, Andrew, and Gary King. 1993. "Why are American presidential election campaign polls so variable when
votes are so predictable?" British Journal of Political Science 23 (4):409-451.
Graefe, Andreas. 2013. "Replication data for: Accuracy of vote expectation surveys in forecasting elections."
Harvard Dataverse Network, doi:10.7910/DVN/VOTEEXPECTATIONSURVEYS.
Graefe, Andreas. 2014. "Replication data for: Combining forecasts: An application to elections." Harvard Dataverse
Network, dx.doi.org/10.7910/DVN/23184.
Graefe, Andreas, J. Scott Armstrong, Randall J. Jones Jr., and Alfred G. Cuzán. 2014. "Combining forecasts: An
application to elections." International Journal of Forecasting 30 (1):43-54.
Granberg, Donald, and Edward Brent. 1983. "When prophecy bends: The preference-expectation link in US
presidential elections, 19521980." Journal of Personality and Social Psychology 45 (3):477-491.
Hayes, Samuel P. Jr. 1936. "The predictive ability of voters." Journal of Social Psychology 7 (2):183-191.
Hogarth, Robin M. 1978. "A note on aggregating opinions." Organizational Behavior and Human Performance 21
(1):40-46.
Hogarth, Robin M. 2012. "When simple is hard to accept." In Ecological Rationality: Intelligence in the World,
edited by Peter M. Todd, Gerd Gigerenzer and ABC Research Group, 61-79. Oxford: Oxford University
Press.
Holbrook, Thomas M. 2010. "Forecasting US presidential elections." In The Oxford Handbook of American
Elections and Political Behavior, edited by Jan E. Leighley, 346-371. Oxford: Oxford University Press.
Kernell, Samuel. 2000. "Life before polls: Ohio politicians predict the 1828 presidential vote." PS: Political Science
and Politics 33 (3):569-574.
Lazarsfeld, Paul F., Bernard Berelson, and Hazel Gaudet. 1948. The People's Choice. How the Voter Makes up His
Mind in a Presidential Campaign. 2nd ed. New York: Columbia University Press.
Lemert, James B. 1986. "Picking the winners: Politician vs. voter predictions of two controversial ballot measures."
Public Opinion Quarterly 50 (2):208-221.
Lewis-Beck, Michael S., and Andrew Skalaban. 1989. "Citizen forecasting: can voters see into the future?" British
Journal of Political Science 19 (1):146-153.
Lewis-Beck, Michael S., and Mary Stegmaier. 2011. "Citizen forecasting: Can UK voters see the future?" Electoral
Studies 30 (2):264-268.
Lewis-Beck, Michael S., and Charles Tien. 1999. "Voters as forecasters: a micromodel of election prediction."
International Journal of Forecasting 15 (2):175-184.
Miller, Michael K, Guanchun Wang, Sanjeev R Kulkarni, H Vincent Poor, and Daniel N Osherson. 2012. "Citizen
Forecasts of the 2008 US Presidential Election." Politics & Policy 40 (6):1019-1052.
Murr, Andreas E. 2011. "“Wisdom of crowds”? A decentralised election forecasting model that uses citizens’ local
expectations." Electoral Studies 30 (4):771-783.
Patterson, Thomas E. 2005. "Of polls, mountains." Public Opinion Quarterly 69 (5):716-724.
Rhode, Paul W., and Koleman S. Strumpf. 2004. "Historical presidential betting markets." Journal of Economic
Perspectives 18 (2):127-141.
Rosenstiel, Tom. 2005. "Political polling and the new media culture: A case of more being less." Public Opinion
Quarterly 69 (5):698-715.
Rothschild, David, and Justin Wolfers. 2012. "Forecasting elections: voter intentions versus expectations." Working
paper:Available at: ssrn.com/abstract=1884644.
Sjöberg, Lennart. 2009. "Are all crowds equally wise? A comparison of political election forecasts by experts and
the public." Journal of Forecasting 28 (1):1-18.
Tracy, Marc. 2012. "Nate Silver is a one-man traffic machine for the Times." New Republic November 6.
Zellner, Arnold. 2004. "Keep it sophisticatedly simple." In Simplicity, Inference and Modelling: Keeping It
Sophisticatedly Simple, edited by Arnold Zellner, Hugo A. Keuzenkamp and Michael McAleer, 242-262.
Cambridge: Cambridge University Press.
Table 1
Accuracy of individual and pooled vote expectations in British and U.S. elections
Election
British general elections
U.S. presidential elections
Source
Murr (2011)
Lewis-Beck and
Stegmaier (2011)
ANES data, own
calculations
Number of elections
1 (in 2010)
13 (between 1951
and 2005)
16 (from 1952 to
2012)
Prediction task
Predict election
winner in local
constituency
Predict party that
wins the governing
majority of seats
Predict national
election winner
Individual expectations
No. of observations
13,334
N/A
30,573
% correct
69%
60%
70%
Pooled expectations
No. of observations
627
13
16
Correct
537
10
13
% correct
86%
77%
81%
Table 2
Accuracy of vote expectation surveys in predicting the winner (1932-2012)
Predicted winner
Election
No. of
surveys
Correct
Wrong
Tie
1932
1
1
0
0
1940
3
3
0
0
1944
6
6
0
0
1948
1
0
1
0
1952
3
3
0
0
1956
5
5
0
0
1960
1
0
1
0
1964
1
1
0
0
1968
1
1
0
0
1972
3
3
0
0
1976
3
3
0
0
1980
3
1
2
0
1984
9
9
0
0
1988
17
16
1
0
1992
23
20
3
0
1996
20
20
0
0
2000
24
13
9
2
2004
43
39
1
3
2008
23
22
0
1
2012
27
27
0
0
Total
217
193
18
6
Table 3
Hit rate and mean absolute error of vote expectation surveys, polls, and the IEM vote share prediction
markets across the last 100 days prior to Election Day (1988-2012)
Mean
1988
1992
1996
2000
2004
2008
2012
Hit rate (in %)
Vote expectation surveys
92
85
100
100
60
98
99
100
Single polls
79
74
100
100
48
68
90
76
Combined polls
86
78
100
100
56
69
96
100
Combined poll projections
88
84
99
100
97
100
39
100
IEM vote share market
79
70
67
100
30
88
100
100
Mean absolute error (in %-points)
Vote expectation surveys
1.6
3.0
1.4
1.1
2.2
1.1
1.8
0.4
Single polls
3.2
4.0
4.9
5.0
3.1
1.9
1.9
1.8
Combined polls
2.9
3.9
4.7
4.7
2.7
1.5
1.7
1.3
Combined poll projections
2.0
1.4
1.9
2.4
1.8
1.1
4.1
1.2
IEM vote share market
1.7
3.0
2.8
1.0
1.4
0.7
1.2
1.5
The analysis of the 1988 election covers only the last 91 days prior to Election Day, since no vote expectation survey was
available before that day.
Table 4
Hit rate and mean absolute error of vote expectation surveys and experts (1992, 2000-2012)
Mean
1992
2000
2004
2008
2012
Hit rate (in %)
Vote expectations surveys
100
100
100
100
100
100
Typical expert forecast
66
87
20
47
94
82
Average expert forecast
70
100
0
50
100
100
Mean absolute error (in %-points)
Vote expectations surveys
1.2
0.3
2.4
1.1
1.8
0.5
Typical expert forecast
1.8
1.8
2.4
1.8
1.7
1.3
Average expert forecast
1.4
0.7
2.3
1.4
1.5
1.3
The expert forecasts are compared to the average of the daily vote expectation forecasts during the
seven days prior to the publication of the expert survey.
Table 5
Hit rate and mean absolute error of vote expectation surveys and seven established quantitative models
(1992-2012)
Mean
1992
1996
2000
2004
2008
2012
Hit rate (in %)
Voter expectations
100
100
100
100
100
100
100
Typical model forecast
86
75
100
100
86
86
71
Average model forecast
100
100
100
100
100
100
100
Mean absolute error (in %-points)
Voter expectations
1.5
1.8
0.4
3.9
0.8
1.7
0.2
Typical model forecast
2.9
1.7
2.1
5.7
2.8
3.2
1.8
Average model forecast
2.3
1.4
2.1
5.7
2.5
1.1
0.8
Only four model forecasts were available for the 1992 election. See Appendix I for the yearly forecasts of each
model. The model forecasts are compared to the average of the daily vote expectation forecasts calculated
across 90 to 60 days prior to Election Day.
Figure 1
2012 U.S. presidential election vote-share forecasts
of 20 vote expectation surveys and 110 polls (last 100 days prior to Election Day)
-4
0
4
8
12
16
0102030405060708090100
Obama lead in percentage points (two-party
popular vote)
Days to Election Day
Individual polls Expectation surveys Election result
Figure 2
Mean absolute error of popular vote share forecasts from vote expectation surveys and
FiveThirtyEight.com for the 2012 U.S. presidential election
0.00
0.20
0.40
0.60
0.80
1.00
1.20
MAE across remaining days to Election Day
FiveThirtyEight.com Vote expectation surveys
Appendix I
Quantitative model forecasts of the U.S. presidential elections from 1992 to 2012
Election
1992
1996
2000
2004
2008
2012
Election result
46.4
54.7
50.3
51.2
46.3
52.0
FC
AE
FC
AE
FC
AE
FC
AE
FC
AE
FC
AE
Abramowitz
46.7
0.3
56.8
2.1
53.2
2.9
53.7
2.5
45.7
0.6
50.6
1.4
Campbell
47.1
0.7
58.1
3.4
52.8
2.5
53.8
2.6
52.7
6.4
52.0
0.0
Holbrook
57.2
2.5
60.0
9.7
54.5
3.3
44.3
2.0
47.9
4.1
Lewis-Beck & Tien
51.5
5.1
54.8
0.1
55.4
5.1
49.9
1.3
49.9
3.6
48.2
3.8
Lockerbie
57.6
2.8
60.3
10.0
57.6
6.4
41.8
4.5
53.8
1.8
Norpoth
57.1
2.4
55.0
4.7
54.7
3.5
49.9
3.6
53.2
1.2
Wlezien & Erikson
45.9
0.5
56.0
1.3
55.2
4.9
51.7
0.5
47.8
1.5
52.6
0.6
Forecasts were published in Political Methodologist 5(2), American Politics Research 24(4) and PS: Political Science and
Politics 34(1), 37(4), 41(4), and 45(4).
Forecasts and election results refer to the two-party share of the popular vote received by the candidate of the incumbent
party.
Appendix II
Accuracy of Election Eve forecasts for the seven elections from 1988 to 2012:
Vote expectation surveys vs. final Gallup survey
Forecast
Absolute error
Election
Election
result
Vote
expectations
Final
Gallup poll*
Vote
expectations
Final
Gallup poll
1988
53.8
56.9
56.0
3.1
2.2
1992
46.4
47.3
43.0
0.9
3.4
1996
54.7
56.7
55.9
2.0
1.2
2000
50.3
47.2
48.9
3.1
1.3
2004
51.2
51.8
50.0
0.5
1.2
2008
46.3
45.3
44.4
1.0
1.9
2012
52.0
51.8
49.5
0.1
2.5
Average
1.5
1.9
* Gallup poll numbers obtained from http://www.gallup.com/poll/9442/election-polls-accuracy-record-presidential-
elections.aspx.
Forecasts and election results refer to the two-party share of the popular vote received by the candidate of the
incumbent party.
Appendix III
Error of vote-share forecasts from vote expectation surveys and individual polls (1988 to 2008)
Horizontal axis: Days to Election Day; Vertical axis: Percentage error in predicting the incumbent’s popular two-party vote (positive
values mean that the forecast was higher than the actual result, negative values mean that the forecast was lower);
Black markers: Error of expectation surveys; Grey markers: Error of individual polls
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
2008
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
2004
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
2000
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
1996
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
1992
-15
-10
-5
0
5
10
15
0 10 20 30 40 50 60 70 80 90 100
1988
Appendix IV
Google searches for FiveThirtyEight and “election forecast” prior to the 2012 election
0
10
20
30
40
50
60
70
80
90
100
Google search volume
"Fivethirtyeight OR 538"
"election forecast OR election prediction OR forecast election OR prediction election"
... In addition, the goal is to continuously track advances in forecasting research and apply them to election forecasting. This has led to the development of index models, which are particularly well suited to aiding decisions by campaign strategists, and to validating previous work on citizen forecasts, an old method that has been widely overlooked despite its accuracy (Graefe 2014). ...
... Though often overlooked, these citizen forecasts are highly accurate predictors of election outcomes (Graefe 2014). In 89% of 217 surveys administered between 1932 and 2012, a majority of respondents correctly predicted the winner. ...
... Across the last 100 days prior to the 2016 election, we collected 39 surveys that asked people who they think will win the election, plus daily data starting on August 8 from the Reuters tracking poll. We translated the results of each individual vote expectation survey into a two-party vote share prediction using the vote equation estimated by Graefe (2014). We In addition to other indicators, the PollyVote combined forecasts from two types of models: econometric models and index models. ...
Chapter
Full-text available
The PollyVote uses evidence-based techniques for forecasting the popular vote in presidential elections. The forecasts are derived by averaging existing forecasts generated by six different forecasting methods. In 2016, the PollyVote correctly predicted that Hillary Clinton would win the popular vote. The 1.9 percentage-point error across the last 100 days before the election was lower than the average error for the six component forecasts from which it was calculated (2.3 percentage points). The gains in forecast accuracy from combining are best demonstrated by comparing the error of PollyVote forecasts with the average error of the component methods across the seven elections from 1992 to 2016. The average errors for last 100 days prior to the election were: public opinion polls (2.6 percentage points), econometric models (2.4), betting markets (1.8), and citizens’ expectations (1.2); for expert opinions (1.6) and index models (1.8), data were only available since 2004 and 2008, respectively. The average error for PollyVote forecasts was 1.1 percentage points, lower than the error for even the most accurate component method.
... Increasingly, modern public opinion surveys also have to cope with challenges from declining response rates (Keeter, et al. 2000, Tourangeau and Plewes 2013, PRC 2012) combined with difficulties in achieving complete coverage of the population (Jacobs and Shapiro 2005, Traugott 2005, Leigh and Wolfers 2006, Erikson and Wlezien 2008). Robust evidence has demonstrated that the role of statistical uncertainty in the opinion polls has not been adequately understood (Martin, et al. 2005, Sturgis, et al. 2016, Hillygus 2011, Graefe 2014) and a failure to fully reflect this uncertainty leads to an over-statement of confidence level in predictions from survey results (Erikson and Wlezien 2008, Rothschild 2015, Lock and Gelman 2010). ...
Article
Full-text available
Accounting for undecided and uncertain voters is a challenging issue for predicting election results from public opinion polls. Undecided voters typify the uncertainty of swing voters in polls but are often ignored or allocated to each candidate in a simplistic manner. Historically this has been adequate because first, the undecided tend to settle on a candidate as the election day draws closer, and second, they are comparatively small enough to assume that the undecided voters do not affect the relative proportions of the decided voters. These assumptions are used by poll authors and meta-poll analysts, but in the presence of high numbers of undecided voters these static rules may bias election predictions. In this paper, we examine the effect of undecided voters in the 2016 US presidential election. This election was unique in that a) there was a relatively high number of undecided voters and b) the major party candidates had high unfavorability ratings. We draw on psychological theories of decision making such as decision field theory and prospect theory to explain the link between candidate unfavorability and voter indecisiveness, and to describe how these factors likely contributed to a systematic bias in polling. We then show that the allocation of undecided voters in the 2016 election biased polls and meta-polls in a manner consistent with these theories. These findings imply that, given the increasing number of undecided voters in recent elections, it will be important to take into account the underlying psychology of voting when making predictions about elections.
Working Paper
Full-text available
Problem: How to help practitioners, academics, and decision makers use experimental research findings to substantially reduce forecast errors for all types of forecasting problems. Methods: Findings from our review of forecasting experiments were used to identify methods and principles that lead to accurate forecasts. Cited authors were contacted to verify that summaries of their research were correct. Checklists to help forecasters and their clients practice and commission studies that adhere to principles and use valid methods were developed. Leading researchers were asked to identify errors of omission or commission in the analyses and summaries of research findings. Findings: Forecast accuracy can be improved by using one of 15 relatively simple evidence-based forecasting methods. One of those methods, knowledge models, provides substantial improvements in accuracy when causal knowledge is good. On the other hand, data models—developed using multiple regression, data mining, neural nets, and “big data analytics”—are unsuited for forecasting. Originality: Three new checklists for choosing validated methods, developing knowledge models, and assessing uncertainty are presented. A fourth checklist, based on the Golden Rule of Forecasting, was improved. Usefulness: Combining forecasts within individual methods and across different methods can reduce forecast errors by as much as 50%. Forecasts errors from currently used methods can be reduced by increasing their compliance with the principles of conservatism (Golden Rule of Forecasting) and simplicity (Occam’s Razor). Clients and other interested parties can use the checklists to determine whether forecasts were derived using evidence-based procedures and can, therefore, be trusted for making decisions. Scientists can use the checklists to devise tests of the predictive validity of their findings. Key words: combining forecasts, data models, decomposition, equalizing, expectations, extrapolation, knowledge models, intentions, Occam’s razor, prediction intervals, predictive validity, regression analysis, uncertainty
Conference Paper
Are ordinary citizens better at predicting election results than conventional voter intention polls? We address this question by comparing predictive models for British elections: one based on voters' expectations of who will win and others based on who voters themselves intend to vote for (including " cube rule " and uniform national swing models). The data come from Gallup polls and the British Election Study and Essex Continuous Monitoring Surveys, 1950–2015, yielding 446 months with both expectation and intention polls. The large sample size allows us to compare the models' prediction accuracy not just in the months prior to the election, but over the years leading up to it. In predicting both the winning party and par-ties' seat shares, we find that voter expectations outperform voter intention models. Voter expectations thus appear an excellent tool for predicting the winning party and its seat share.
Chapter
Full-text available
Most citizens correctly forecast which party will win the election, and groups of citizens forecast even better than individual citizens. Using Condorcet's jury theorem , I explain why individual citizens are better than chance at forecasting, and why in turn groups of citizens forecast better than individual citizen. The so-called 'wisdom of crowds' in election forecasting is a robust finding across countries, as the research reviewed here demonstrates. Asking citizens who they think will win results in accurate forecasts of which party will win, and of what seat or vote share a party will receive. Some of the studies reviewed here even suggest that asking citizens who they think will win is the most accurate approach to election forecasting. Similarly, asking citizens what vote share or seat number they think a party will receive accurately forecasts its actual vote share or seat number. But the analyses conducted here suggests that asking citizens what vote share or seat number they think a party will win is not the most accurate approach to election forecasting. Accordingly, when forecasting elections, academics, journalists, and pollsters should conduct more surveys asking citizens who they think will win.
Article
Full-text available
This article offers an assessment of the current state of US presidential election forecasting models. It pays attention to presidential forecasting models from the last three election cycles. It starts by exploring 'under the hood' and describes the specifics of the most widely known models from the 2004 election. In addition, the predictions made by these models are addressed and the determinants of forecasting accuracy from 1996 to 2004 are identified. Moreover, the article explores the lessons learned from the 2000 campaign and the alternatives to the dominant aggregate-national forecasting models: electronic markets, citizen forecasts, and state-level forecasting models. From a forecasting perspective, the 2008 election outcome was business as usual. Some models were more accurate than others, as is always the case, but the average error was somewhat lower than in the past two elections cycles.
Data
Full-text available
We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert judgment, quantitative models, and the Iowa Electronic Markets. Across the six elections, the resulting combined forecasts were more accurate than any individual component method, on average. The gains in accuracy from combining increased with the numbers of forecasts used, especially when these forecasts were based on different methods and different data, and in situations involving high levels of uncertainty. Such combining yielded error reductions of between 16% and 59%, compared to the average errors of the individual forecasts. This improvement is substantially greater than the 12% reduction in error that had been reported previously for combining forecasts.
Article
Full-text available
This article revises, updates, and examines the background for a highly accurate model for forecasting the national two-party popular vote in presidential elections. The model provides a vote prediction in early September based on Gallup trial-heat or presidential preference polls and the (nonannualized) rate of economic growth in the second quarter of the election year. It is estimated over the 12 presidential elections from 1948 to 1992. The mean absolute error of the model's out-of-sample postdictions is less than 1 1/3 percentage point, and its actual error in predicting the 1992 vote was about half a percentage point. The article also assesses the reasons for confidence in the model, as well as an approach to gauging uncertainty in any specific forecast. The reasons presidential elections can be forecast at or before the beginning of the general election campaign also are explored. Finally, the forecasting model is applied to the 1996 presidential campaign between Clinton and Dole.
Article
Full-text available
Do the national conventions of the Democratic and Republican parties affect the poll standings of the presidential candidates they nominate? This study investigates whether these poll stand ings are bumped upwards following the party conventions. The convention bump is examined with Gallup and Harris time series data of presidential trial-heats throughout the course of seven campaigns from 1964 to 1988. We find that (1) with few exceptions, there is a convention bump; (2) the bump typically adds about 5 to 7 percentage points to the nominee's postoonvention poll standing; (3) the effects of conventions carry well into the general election campaign; (4) the first convention in the campaign sequence, held by the out-party, generates an additional but temporary increase in the nominee's support; and (5) convention bumps may be greater for harmonious conventions following divisive nomination contests.
Article
The leading approaches to election forecasting have been statistical models, prediction markets, or vote intention polls. This paper explores a different, little used approach - citizen forecasting - a method never yet tried for the case of the United Kingdom. Citizen forecasting simply asks voters in a scientific pre-election survey who they think will win. This aggregated (averaged) result is then assessed for its accuracy in forecasting the overall election outcome. This assessment is carried out on UK general elections from 1951 to 2005, and then applied to 2010. We find, among other things, that generally speaking the majority of the electorate can foretell the winner in advance.
Article
We analyze individual probabilistic predictions of state outcomes in the 2008 U.S. presidential election. Employing an original survey of more than 19,000 respondents, we find that partisans gave higher probabilities to their favored candidates, but this bias was reduced by education, numerical sophistication, and the level of Obama support in their home states. In aggregate, we show that individual biases balance out, and the group's predictions were highly accurate, outperforming both Intrade (a prediction market) and fivethirtyeight.com (a poll-based forecast). The implication is that electoral forecasters can often do better asking individuals who they think will win rather than who they want to win.
Article
Most pollsters base their election projections off questions of voter intentions, which ask “If the election were held today, who would you vote for?” By contrast, we probe the value of questions probing voters’ expectations, which typically ask: “Regardless of who you plan to vote for, who do you think will win the upcoming election?” We demonstrate that polls of voter expectations yield consistently more accurate forecasts than polls of voter intentions. A small-scale structural model reveals that this is because we are polling from a broader information set, and voters respond as if they had polled ten of their friends. This model also provides a rational interpretation for why respondents’ forecasts are correlated with their expectations. We use our structural model to extract accurate election forecasts from non-random samples.
Article
Many studies report the “wonders of aggregation” and that groups (often) yield better decisions than individuals. Can this “wisdom of crowds”-effect be used to forecast elections? Forecasting models in first-past-the-post systems need to translate vote shares into seat shares by some formula; however, the seat–vote ratio alters from election to election. To circumvent this problem, this paper proposes citizen forecasting, which aggregates citizens’ local expectations to directly forecast constituencies. Using data from the 2010 British Election Study, this paper finds (1) that groups are better forecasters than individuals, (2) that citizen forecasting correctly predicts a hung parliament, and (3) that marginality and group size are important predictors for “getting it right”.