Article

Campaign trial heats as election forecasts: Measurement error and bias in 2004 presidential campaign polls

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

If late-campaign polls are to be used as forecasts, it is important to ask, how well do the polls do and why are some polls better forecasts than others? We analytically compare alternative methods for estimating the systematic bias in the election trial heat polls of the individual polling houses and of the polling industry as a whole. We put each technique to the test using data from the 2004 US Presidential election. From the collection of evidence we are able to identify the approach that produces the most efficient unbiased estimates and answer the question of how the polls did in 2004. A third of the houses exhibited large and significant biases, but the industry as a whole converged on the truth.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Some scholarsdmost notably Green et al. (1999)dhave advocated a variant of the Kalman filter that takes into account expected sampling error variance. Other scholars have adopted this approach in their estimates of survey house effects (Jackman, 2005;Pickup and Johnston, 2008). 2 In so doing, they provide estimates of real underlying movement. ...
... This clearly complicates the assessment of change over time. A good amount of work has been done (and is underway) in estimating and correcting for the differences across organizations, thus better enabling us to combine data from different sources (Erikson and Wlezien, 1999;Jackman, 2005;Wlezien and Erikson, 2007;Pickup and Johnston, 2008 to count the same respondents on multiple days, and it is very easy to remove this overlap, e.g., for survey house reporting 3-day moving averages, we only use poll results for every third day. We then calculate the Bush share of the two-party vote intention, ignoring all other candidates, e.g., Nader. ...
... Even where we do have full information about sampling error, we want to know how reliable filtering is. We will be examining the variant of the Kalman filter typically applied to political science public opinion data (Beck, 1989;1991Green et al., 1999Jackman, 2005;and Pickup and Johnston, 2008). The traditional Kalman filter produces the optimal prediction of a time series at time t, only considering the observations previous to time t and produces the optimal filtered values considering information previous to and including time t. ...
Article
The Kalman filter is a popular tool in engineering and economics. It is becoming popular in political science, touted for its abilities to reduce measurement error and produce more precise estimates of true public opinion. Its application to survey measures of public opinion varies in important ways compared to the traditionally understood Kalman filter. It makes a priori assumptions about the variance of the sampling error that would not usually be made and does so in a way that violates an important property of the Kalman filter. Consequently, the behavior of the filter modified for public opinion measures is less well-known. Through simulations we assess whether and to what extent filtering: reliably detects the characteristics of time series; does so across series with different rates of autoregressive decay; and does so when the variance of the sampling error is unknown. We also examine whether the filtered data represents the level of true underlying variance and the extent to which filtering assists or hinders our ability to detect exogenous shocks. We learn a numbers of things. Most importantly, taking into account sampling error variance when filtering data can work well, though its performance does vary. First, filtering works best identifying time series characteristics when assuming a stationary process, even if the underlying process contains a unit root. Second, the performance of filtering drops off when we incorrectly specify the variance of the sampling error, and especially when we overestimate it. Third, when estimating exogenous shocks it is better to make no a priori assumptions regarding a measurement error variance unless we are absolutely certain we know what it is. In fact, applying the filter without specifying the measurement error variance is more often than not the best choice.
... Polling data provide an imperfect measure of true public opinion: each poll is potentially subject to random measurement error, design effects and systematic bias. By pooling together estimates from the different polls, we can produce a more accurate estimate, which limits the impact of both random variation between polling samples and systematic effects generated by the methods chosen by the polling houses -commonly termed 'house effects' or 'house bias' (Erikson and Wlezien, 1999;Jackman, 2005;Pickup and Johnston, 2008). In doing so, we are also able to estimate the magnitudes of the systematic errors in polls. ...
... To identify the patterns of bias in British polling and produce a more accurate estimate of public opinion, we build upon techniques developed by Pickup andJohnston (2007, 2008), Jackman (2005) and by Erikson and Wlezien (1999). We employ a state-space model of the kind described in Pickup (2009). ...
Article
Full-text available
Pollsters once again found themselves in the firing line in the aftermath of the 2010 British general election. Many critics noted that nearly all pollsters in 2010 expected a substantial surge for the Liberal Democrats that did not materialize. Basing conclusions regarding the relative merits of pollsters or benefits of methodological design features on inspection of just the final poll from each pollster is inherently problematic, because each poll is subject to sampling error. This paper uses a state‐space model of polls from across the course of the 2010 election campaign which allows us to assess the extent to which particular pollsters systematically over‐ or under‐estimate each main party’s share of the vote, while allowing for both the usual margins of error for each poll and changes in public opinion from day‐to‐day. Thus, we can assess the evidence for systematic differences between pollsters’ results according to the use of particular methodologies, and estimate how much of the discrepancy between the final polls and the election outcome is due to methodological differences that are associated with systematic error in the polls. We find robust evidence of an over‐estimation in Liberal Democrat support, but do not find evidence to support the hypothesis that the polls erred due to a late swing away from the party, nor that any of the methodological choices made by pollsters were significantly associated with this over‐estimation.
... Such naı¨ve endogenous models would fail to accurately predict the variation in the support for the FN across the whole period (MAEs of 4.56 and 4.47, respectively). Similarly, our model can be contrasted with trial-heat polls that are conducted before the election (Campbell, 1996;Pickup and Johnston, 2008). Comparing the model's ex-post adjusted predicted values (out-of-sample) with the average of voting intention polls released for the 2002 and 2007 presidential elections showed that the model estimates were closer to the actual election outcome on both occasions (Evans and Ivaldi, 2010, pp. ...
Article
Full-text available
Forecasting the Extreme Right vote in French elections is one of the few ‘third-party’ forecasts that has attracted attention in a forecasting literature focusing generally on incumbent performance and winners. Despite being a ‘hard case’ because of third-party status, unstable polling estimates and relatively few data points, previous models have provided relatively strong forecasts of the performance of the Front National (FN) and its erstwhile leader, Jean-Marie Le Pen. The recent succession of Le Pen by his daughter, Marine, and her apparent popularity pose a significant challenge to these models, however. In this article, we consider our previous model's prediction of her likely score in the first round of the presidentials, comparing this to standard forecasting benchmarks, and look at possible adjustments to account for the speculated ‘Marine effect’. We then compare this with other vote indicators including the results of an experimental expert judgment survey, finding that there is currently little evidence for a likely runaway success for the new FN leader in April 2012.
... Thus, it may be that we actually have a fairly reasonable portrait of electoral preferences, particularly when we combine polls from different survey organizations, where errors may to a large extent cancel out 15 . Unfortunately, we do not have much information about the practices of different survey organizations, particularly going back in time, which precludes an analysis of the effects of 15,17,18 . We can still examine the performance of poll aggregations. ...
Article
Full-text available
Are election polling misses becoming more prevalent? Are they more likely in some contexts than others? In this paper we undertake an over-time and cross-national assessment of prediction errors in pre-election polls. Our analysis draws on more than 26,000 polls from 338 elections in 45 countries over the period between 1942 and 2013, as well as data on more recent elections from 2014 to 2016. We proceed in the following way. First, building on previous studies, we demonstrate how poll errors evolve in a structured way over the election timeline. Second, we then focus on errors in polls in the final week of the campaign to examine poll performance across election years. Third, we use the historical performance of polls to benchmark recent polling “misses” in the UK, US and elsewhere. Fourth, we undertake a pooled analysis of polling errors – controlling for a number of institutional and party features – which enables us to test whether poll errors have increased or decreased over time. We find that, contrary to conventional wisdom, recent performance of polls has not been outside the ordinary. The performance of polls does vary across political contexts, however, in understandable ways.
... The Kalman filter uses a state-space model of the time series of interest. This approach has been applied to a variety of political science variables (Pickup and Johnston, 2008;Jackman, 2005;Martin and Quinn, 2002;McAvoy, 1998;Brandt and Williams, 2001;Kellstedt, McAvoy and Stimson, 1996). It has been applied to presidential approval and vote intention polls in a series of publications, dating back to Beck (1989). ...
... Moreover, besides the change in vote intention from day to day within one polling organization, there is the change in vote intention across polling firms. These latter changes are sometimes referred to as "house effects" (Pickup and Johnston 2008). By now, there are many firms, in addition to Gallup, competing in the world of political polling, such as Roper, Harris, Pew, the New York Times, and news network polls, to name only a few. ...
Chapter
Full-text available
In democratic nations, elections are premier political events. They require citizens to choose their leaders and leaders to be held accountable. The winners of an electoral contest are allowed to wield power, sometimes great power. Therefore, citizens follow campaigns with interest and are often eager to know who will win. In other words, they would like a forecast of the outcome. This impulse to election forecasting has been around for a long time. To forecast an election means to declare its outcome before it happens. For example, say in advance which candidate will win the race. There are many methods of forecasting elec-tions, and they can be divided into two groups: prescientific and scientific. While this distinction may seem clear, the differences are not always easy to spot. Below, we emphasize the scientific approach to election forecasting, tracing its his-torical development in the study of United States
... Similar models have been adopted by Jackman (2005) and Hanretty (2013) as well as by Pickup and Johnston (2007) and Pickup and Johnston (2008) (see section 1). However, a major difference between those models and our proposal is that we do not need to consider the true vote share of each party (i.e., their actual election results), choice that requires additional and partly unjustified model assumptions. ...
Article
Full-text available
It is widely known that pre-electoral polls often suffer from non- sampling errors that pollsters try to compensate for in final estimates by means of diverse ad hoc adjustments, thus leading to well-known house effects. We propose a Bayesian hierarchical model to investigate the role of house effects on the total variability of predictions. To illustrate the model, data from pre-electoral polls in Italy in 2006, 2008 and 2013 are considered. Unlike alternative techniques or models, our proposal leads: (i) to correctly decompose the different sources of variability; (ii) to recognize the role of house effects; (iii) to evaluate its dynamics, showing that variability of house effects across pollsters diminishes as the date of election approaches; (iv) to investigate the relationship between house effects and overall prediction errors.
... Much of the variation is likely to be an artefact of the polling process rather than a true reflection of public opinion volatility. By pooling together results from the different polls, we aim to produce a more accurate estimate that reduces the impact of both random variation between polling samples and systematic effects generated by the methods chosen by the polling houses, which commonly are termed 'house effects' or 'house bias' (Erikson and Wlezien, 1999;Jackman, 2005;Pickup and Johnston, 2008). ...
Article
This paper outlines and discusses techniques for three stages in forecasting parliamentary seats from British opinion polls: adjusting and aggregating published vote-intention figures from across different polls; forecasting how public opinion might change before election day; and predicting the seat totals from the forecasted election-day vote shares. Specifically, we consider a state-space model for opinion polls which correct for house effects and other sources of survey error, the estimation of the historical relationship between polls and the election-day share of the vote, and a probabilistic approach to predicting the winner in each constituency.
... The solid lines in Figure 3 depict an aggregation of the polls, which amounts to daily samples well over onethousand, using a variety of polling methodologies. It is widely accepted that an aggregation of polls is very likely to be very close to the true levels of support on a given day (Pickup and Johnston, 2008). Indeed the polls converged very close to the popular vote results. ...
Article
Full-text available
Election studies must optimize on sample size, cost and data quality. The 2015 Canadian Election Study was the first CES to employ a full mixed-mode design, aiming to take advantage of the opportunities of each mode while preserving enough commonality to compare them. This paper examines the phone interviews conducted by ISR-York and the online questionnaires from panellists purchased from a sample provider. We compare data quality and representativeness. We conduct a comprehensive comparison of the distributions of responses across modes and a comparative analysis of inferences about voting. We find that the cost/power advantages of the online mode will likely make it the mode of choice for subsequent election studies.
... There is an important study that indicates that one of the reasons behind the failure of the surveys in predicting accurate results is that many of the centers conducting the surveys does not declare with transparency the measures they adopted in carrying out the survey and may not be sincere in describing the steps they have taken particularly in the absence of two important elements: the control and benchmarking [14]. The fact is that there is no accountability or control or any kind on the actions of the public opinion centers in spite that its works is not less important than that of the mass media. ...
Article
This study involves the controversial role that public opinion surveys played during the American presidential elections 2016 that has acquired exclamation marks due to the evident conflict between reports of mass media, particularly the private ones, that placed the democratic candidate, Clinton, ahead in the race while results comes directly the opposite. The study applied “Time for Change” approach and includes three sections; first, the methodology, questions, and objectives. Secondly, the theoretical part and mechanisms of the survey centers, reasons leading to failure in achieving sound results, and the political and economic effects of the newspapers’ agendas in guiding the survey results Thirdly, the last section demonstrates the findings that include: the major reasons that enabled Trump to overcoming the intentional mass media disregard and finally the reasons that made his winning an unexpected shock.
... 8 Public opinion poll data are drawn from the lists available from www.electionalmanac. com. 9 Certainly there are documented issues with polls in Alberta, Quebec and BC to suggest that these data may not accurately reflect whether a party's support is genuinely in decline (see Grénier, 2012a, b;; see also Pickup and Johnston, 2008;Pickup et al., 2011). 10 Marois sought the leadership of the Parti Québécois (PQ) three times. ...
Article
The majority of Canada's women premiers were selected to that office while their parties held government. This is uncommon, both in the comparative literature and among premiers who are men. What explains this gendered selection pattern to Canada's provincial premiers’ offices? This paper explores the most common explanation found in the comparative literature for women's emergence as leaders of electorally competitive parties and as chief political executives: women are more likely to be selected when that party is in crisis or decline. Using the population of women provincial premiers in Canada as case studies, evidence suggests three of eight women premiers were selected to lead parties in government that were in crisis or decline; a fourth was selected to lead a small, left-leaning party as predicted by the literature. However, for half of the women premiers, evidence of their party's decline is partial or inconclusive. As a result of this exploration, more research is required to draw generalizations about the gendered opportunity structures that shape how women enter (and exit) the premier's office in Canada.
... Esta información de última hora permite corregir las estimaciones incrementando su fiabilidad y precisión. Según varios estudios, las encuestas próximas a las elecciones (como media tres meses antes de ellas) facilitan los datos más fiables para una predicción electoral (Traugott, 2001(Traugott, , 2005Pickup y Johnston, 2008;Panagopoulos, 2009c;Alaminos, 1994). Cuanto más antiguos son los datos, la precisión disminuye (Campbell y Wink, 1990;Gelman y King, 1993;Campbell, 1996), siendo útiles para evaluar las tendencias observadas, pero contribuyendo en menor grado al estado actual. ...
Article
Full-text available
Tras introducir brevemente algunos conceptos explicativos del voto desde el punto de vista de su influencia a largo, medio y corto plazo, se efectúa un análisis desde el enfoque del modelo de voto retrospectivo. Se analizan cuatro series del banco de datos del Centro de Investigaciones Sociológicas, con la valoración de la gestión/actuación del Partido Popular y del Partido Socialista Obrero Español, así como las series de intención de voto a los dos partidos. Del análisis se concluye la estrecha relación existente entre la valoración de la gestión/actuación de ambos partidos con su intención de voto. Las tendencias de declive observadas en la confianza en la gestión/actuación de los dos partidos anticipaban el escenario de crisis del bipartidismo de 2015, sugiriendo su incorporación como elemento de su «imagen de partido». Así mismo, considerando las dinámicas para los dos partidos, se observa la anomalía que, dentro de la pauta general, representan las elecciones generales de 2004 y 2011. Como estrategia de control de la falacia ecológica de los agregados, se ha efectuado un control en cinco años (1997, 2002, 2007, 2012 y 2017) analizando los microdatos para establecer la relación entre valoración de la gestión/actuación del partido y su intención de voto en la unidad de análisis individual.After briefly introducing some of the explanatory concepts of voting from the point of view of its influence in the long, medium and short term, an analysis is made from the retrospective voting model approach. For the Spanish case, four series of the Centre for Sociological Research database are analysed, with the assessment of the management/performance of the Popular Party and the Spanish Socialist Workers’ Party, as well as the series of voting intention to the two political parties. The analysis concludes the close relationship between the assessment of the management/performance of both political parties with their voting intention. The trends of decline observed in the trust in management/ performance of the two parties anticipated the bipartisan crisis scenario of 2015. Furthermore, considering the dynamics for the two parties, we observe the anomaly that, within the general pattern, represents the general elections of 2004 and 2011. As a strategy to control the ecological fallacy of the aggregates, an analysis has been carried out in five years (1997, 2002, 2007, 2012 and 2017) studying the microdata to establish the relationship between management/performance valuation of the party and the voting intention at an individual level.
... On the one hand, there are the technical characteristics and survey design. However, the difference between electoral predictions and results cannot simply be put down to a statistical error, but derive from sample design, question formulation, weighting, and screening [Pickup and Johnston, (2008)]. Second, there is the behavior of the respondents themselves. ...
Article
Full-text available
The main purpose of this paper is to use the Spanish case, through an econometric analysis of 226 electoral polls, to explain why polls are making more mistakes in times of great socioeconomic slumps, political instability and the emergence of new political parties. In this context, it is the very instrument with which society tries to reduce the reigning uncertainty that, paradoxically, can ultimately drive uncertainty up. Our results show that the prediction error for the new emerging parties is significantly higher than for the traditional parties and this error is not sensitive to solutions for increasing the reliability of surveys, such as increasing sample size, transparency constantly conducting periodical surveys, the closeness of the approaching election or the survey mode that is used. It can be observed that pollsters do not want to make predictions that vary greatly from the average of the other polls. Finally, editorial bias appears to play a significant role, especially in the case of traditional parties.
... We also looked at the media mentions for the two weeks preceding Super Tuesday, but a candidate's share did not change much as the correlation between the one-week and two-week measures was 0.99. 2 Analytically, including polling poses an interesting issue for our models. Not all polls are a valid measurement of where a race stands (Pickup and Johnston 2008) as polls can be inaccurate and/or biased (Martin, Traugott, and Kennedy 2005). Second, polling is often correlated with other measures of primary success such as media attention and public attention (e.g. ...
Article
Full-text available
In presidential primaries, Super Tuesday elections play a significant role in winnowing candidate fields and establishing nomination frontrunners. Despite their importance, scholars know little about why and how candidates win or lose the states comprising these events. This study explores which factors help explain candidate performance in Super Tuesday primaries between 2008 and 2016. Using pooled cross-sectional time-series analysis, the results indicate three key drivers of Super Tuesday success: candidate viability, public attention, and media attention. These findings imply that presidential campaigns continue to be complex electoral events beyond the early primary states and suggest that underdog candidates can still win states under the right conditions. Future research should explore the interrelatedness of these three critical factors.
Article
Full-text available
Although “horserace journalism” is thought to be central to contemporary election news coverage and has generated a great deal of criticism, there is no general model of the nature and dynamics of horserace journalism or “poll reporting.” This paper proposes and empirically evaluates such a model. The model builds on and extends John Zaller's “theory of media politics” to consider specifically what citizens demand from polls and what journalists supply. Aside from the generic motivations of politicians, citizens and journalists, the model emphasizes the unique features of polls as objects of news coverage. The paper finds considerable support for the model in an analysis of newspaper coverage of horserace polls (that is, vote intention polls) in the Canadian general election of 2006. Our findings from this one case have potentially broad implications for our understanding of the relationship between polls and electoral democracy both empirically and normatively. Résumé. Même si le journalisme de course (“horserace journalism”) est vu comme étant une composante centrale de la couverture électorale et qu'il a généré sa part de critiques, il n'existe pas de modèle général de la nature et de la dynamique de ce type de journalisme. Cet article propose, et évalue empiriquement, un tel modèle. Prenant comme point de départ la « Theory of Media Politics » de John Zaller, ce modèle considère plus spécifiquement ce que les citoyens demandent des sondages et ce que les journalistes leurs procurent. Au-delà des motivations génériques des politiciens, citoyens et journalistes, le modèle met l'accent sur les caractéristiques uniques des sondages en tant qu'objet de couverture journalistique. L'article présente des résultats supportant considérablement le modèle à travers une analyse de la couverture des sondages par les journaux (c'est-à-dire des sondages sur les intentions de vote) durant l'élection générale canadienne de 2006. Nos résultats émanant de ce cas ont potentiellement des implications beaucoup plus grandes pour notre compréhension de la relation entre les sondages et la démocratie électorale, à la fois sur le plan empirique et sur le plan normatif.
Article
The underlying motivation of this collection is the concern that the failure to understand the relationship between our measure of a concept and the concept itself has real consequences. It may cause us to overlook model specification problems, to unnecessarily and inappropriately change our theories, and to miss out on information that could be usefully applied to the specification and estimation of our model. In order to take modelling seriously, we first must take measurement seriously. This is most effectively done when multiple measures of the underlying concept are utilised, allowing for the application of more appropriate models with more precise estimates. The contributions to this symposium demonstrate a wide range of such modelling approaches, each of which is tailored to match the theoretical expectations for the statistical characteristics of the data being used. By way of introduction, I highlight the measurement problems that these approaches are designed to tackle.
Conference Paper
After an election campaign, it is important to identify events that marked change points in voter support. Pre-election polls provide a measure of the state of voter support at points in time during the election campaign. However, polling data is difficult to analyze because it is sparse and comes from multiple sources, which can be individually biased. We propose a change point model for polling data that increases confidence by combining polls and identifying change points simultaneously. We demonstrate the utility of our model on polling data from the 2008 U.S. presidential election.
Article
I present a dynamic Bayesian forecasting model that enables early and accurate prediction of U.S. presidential election outcomes at the state level. The method systematically combines information from historical forecasting models in real time with results from the large number of state-level opinion surveys that are released publicly during the campaign. The result is a set of forecasts that are initially as good as the historical model, and then gradually increase in accuracy as Election Day nears. I employ a hierarchical specification to overcome the limitation that not every state is polled on every day, allowing the model to borrow strength both across states and, through the use of random-walk priors, across time. The model also filters away day-to-day variation in the polls due to sampling error and national campaign effects, which enables daily tracking of voter preferences toward the presidential candidates at the state and national levels. Simulation techniques are used to estimate the candidates’ probability of winning each state and, consequently, a majority of votes in the Electoral College. I apply the model to preelection polls from the 2008 presidential campaign and demonstrate that the victory of Barack Obama was never realistically in doubt.
Article
Are campaigns epiphenomena? The dominant answer in the US literature seems to be Yes. Outcomes are driven by ‘fundamental’ considerations in place before the campaign begins, such that dynamics within the campaign reflect the elimination of ‘accidental’ considerations. Multi-country comparisons, where they exist, tend to be fragmentary. Where they are not fragmentary, they are taken from so high an altitude that critical interpretive details are lost. Also fragmentary, surprisingly, is documentation of the US case itself. This paper identifies two overlapping but distinct claims: about frontrunners and about incumbents. We also identify an insufficiently recognized fact, that the very idea of a ‘fundamental’ is contested, and that this broadens the scope for potential within-campaign dynamics. We employ rolling cross-section survey data from the US, Germany, Italy, New Zealand, and Canada.
Article
Full-text available
Surveys have long been critical tools for understanding elections and forecasting their results. As the number of election surveys has increased in prevalence, researchers, journalists, and standalone political bloggers have sought to learn from the wealth of information released. This paper explores three central strategies for pooling surveys and other information to better understand both the state of an election and what can be expected when voters head to the polls. Aggregation, predictive modeling, and hybrid models are assessed as ways of improving on the results of individual surveys. For each method, central questions, key choices, applications, and considerations for use are discussed. Trade-offs in choices between pooling strategies are considered, and the accuracies of each set of strategies for forecasting results in 2012 are compared. Although hybrid models have the potential to most accurately pool election information and make predictions, the simplicity of aggregations and the theory-testing capacity of predictive models can sometimes prove more valuable.
Conference Paper
After an election campaign, it is important to identify events that marked change points in voter support. Pre-election polls provide a measure of the state of voter support at points in time during the election campaign. However, polling data is difficult to analyze because it is sparse and comes from multiple sources, which can be individually biased. In this paper, we propose a change point model for polling data that increases confidence by combining polls and identifying change points simultaneously. We demonstrate the utility of our model on polling data from the 2008 U.S. presidential election.
Article
Many types of time series data in political science, including polling data and events data, exhibit important features—such as irregular spacing, noninstantaneous observation, overlapping observation times, and sampling or other measurement error—that are ignored in most statistical analyses because of model limitations. Ignoring these properties can lead not only to biased coefficients but also to incorrect inference about the direction of causality. This article develops a continuous-time model to overcome these limitations. This new model treats observations as noisy samples collected over an interval of time and can be viewed as a generalization of the vector autoregressive model. Monte Carlo simulations and two empirical examples demonstrate the importance of modeling these features of the data.
Article
Full-text available
This paper develops a three-stage method to forecast parliamentary election results from vote preferences in British opinion polls: (1) adjusting and aggregating vote-intentions from different polling organizations; (2) forecasting how public support for parties will change in the period before election day; and (3) translating, through simulations, the forecast of election day vote shares into seat totals while incorporating constituency-level information, including local vote-intention polls. Overall, this approach seeks to combine relevant national, regional and local information, and uncertainty about that information, to better reflect the fragmentation and diversity of political contexts found in the new era of five/six-party British politics.
Chapter
Opinion polls are central to the study of electoral politics. With modern election polling dating back to the 1936 US presidential election, and proto-straw polls going back as far as the 1824 presidential election, polls have long been employed to gauge the popularity of different political competitors and, for as long as they have been available, researchers have used them to make predictions about future election results (Smith, 1990; Bean, 1948). Research on the links between opinion polls and election outcomes took off in the 1970s and early 1980s, as a cluster of (mostly) American researchers analysed these relationships, primarily in the context of US presidential elections (Campbell, 2008). The field has evolved rapidly since then, as polls have proliferated and analysis tools have grown ever more sophisticated. Opinion poll-based election analysis and forecasting is now a global enterprise, and often one with an unusually high public profile – poll analysis and election forecasts by academics and data journalists are now widely reported and discussed in election campaigns. This chapter reviews research on the relationships between opinion polls and election outcomes, and some of the election forecasting techniques built on this relationship.
Article
Full-text available
On 23 June 2016 the UK held a referendum so to decide whether to stay or leave the European Union. The uncertainty surrounding the outcome of this referendum had major consequences in terms of public policy, investment decisions, and currency markets. We discuss some subtleties entailed in smoothing and disentangling poll data at the light of the problem of tracking the dynamics of the intention to Brexit, and propose a multivariate singular spectrum analysis method that produces trendlines on the unit simplex. The trendline yield via multivariate singular spectrum analysis is shown to bear a resemblance with that of local polynomial smoothing, and singular spectrum analysis presents the nice feature of disentangling directly the dynamics into components that can be interpreted as changes in public opinion or sampling error. Merits and disadvantages of some different approaches to obtain smooth trendlines on the unit simplex are contrasted, both in terms of local polynomial smoothing and of multivariate singular spectrum analysis.
Article
Full-text available
On 23 June 2016 the UK held a referendum so to decide whether to stay or leave the European Union. The uncertainty surrounding the outcome of this referendum had major consequences in terms of public policy, investment decisions, and currency markets. We discuss some subtleties entailed in smoothing and disentangling poll data at the light of the problem of tracking the dynamics of the intention to Brexit, and propose a multivariate singular spectrum analysis method that produces trendlines on the unit simplex. The trendline yield via multivariate singular spectrum analysis is shown to bear a resemblance with that of local polynomial smoothing, and singular spectrum analysis presents the nice feature of disentangling directly the dynamics into components that can be interpreted as changes in public opinion or sampling error. Merits and disadvantages of some different approaches to obtain smooth trendlines on the unit simplex are contrasted, both in terms of local polynomial smoothing and of multivariate singular spectrum analysis.
Article
Radical Right Parties (RRPs) have traditionally been seen as ‘hard cases’ to forecast, with unstable voter bases affected by short-term influences. Building upon our previous work on forecasting the French Front National’s vote across time, we construct a comparable model for three other European countries–Austria, Denmark and Norway–with significant RRPs, using economic, cultural and political predictors. We find that the model performs surprisingly well, with the partial exception of Norway, and provides an accurate forecast of RRP electoral performance which improves upon naive endogenous models and, significantly, upon polling estimates. Moreover, the model is firmly rooted in existing explanations of RRP success, allowing a robust explanation not only of variation in these parties’ votes, but also of less successful estimates in a small number of country-specific contexts. Overall, we find that standard approaches to electoral forecasting in fact offer a useful tool in the analysis of RRPs.
Article
Full-text available
Before every presidential election, journalists, pollsters, and politicians commission dozens of public opinion polls. Although the primary function of these surveys is to forecast the election winners, they also generate a wealth of political data valuable even after the election.'
Article
Full-text available
Only in recent years has the "likely voter" technology been extended to polls well in advance of an election. In the case of the 2000 U.S. presidential election, CNN/USA Today/Gallup tracking polls indi- cated considerable fluctuations in likely voter preferences, greater than among the larger pool of registered voters surveyed. This article explores how Gallup's likely voter model exaggerates the reported volatility of voter preferences during the campaign. Much of the reported variation in candidate preference reported by Gallup in that election is not due to actual voter shifts in preference but ra ther to changes in the composition of Gallup's likely voter pool. The findings highlight dangers of relying on samples of likely voters when polling well before Election Day.
Article
Full-text available
The study of voters and elections has shed considerable light on people's vote choices and election outcomes. Yet little is known about the evolution of electoral sentiment over the campaign cycle. This article takes a small step toward addressing this issue by examining polls for a single election in a single year-the U.S, presidential race in 1996. The volume of poll data for 1996 allows us to observe the dynamics of voter preferences in far greater depth than is possible in previous years. Our analysis indicates that most of the variation in the polls during the 1996 presidential campaign represents survey error. What remains is mostly concentrated in the run-up to the fall campaign, not the fall campaign itself. During the fall, when political activity and media attention were at their peaks, aggregate presidential preferences remained largely unchanged. To the extent that campaign events influenced the underlying division of preferences, the effects were small and short-lived. Thus, our findings are consistent with the interpretation that the electoral verdict is already in place before the general election campaign begins.
Article
Full-text available
The 2004 presidential election campaign provided a venue for a wide variety of polling, and it was not without its controversies. In the end, the final estimates of the preelection polls, the bread and butter of the polling industry, were very good at suggesting it would be a close race, with Bush the likely winner. In historical perspective, the overall performance was above average for the period since 1956. Issues raised in the media leading up to the end of the campaign and the final estimates, however, created some controversy, especially about the likely voter methodology used by different organizations. There were also some anomalies at the end of the campaign as some firms and collaborators ended up producing different estimates of the outcome depending on likely voter definitions or the mode of data collection.
Article
Full-text available
Direct measurements of public opinion about national affairs appear with increasing frequency in all of the mass media. While such survey results are often with statements as to expected error margins, discrepancies between multiple surveys in the news at the same time on what seem to be the same topics may convince casual consumers that such error margins must be considerably understated. A brief review of the several sources of variability and fixed bias in such surveys provides a clearer frame of reference for the evaluation of such data.
Article
Campaigns suddenly seem to matter, as do questions about the electoral process in the aftermath of the 2000 presidential election. The authors examine the U.S. electoral process as an integrated event spanning a full year, drawing upon the Annenberg 2000 Election Study. The scale of their fieldwork is such that they have been able to isolate key turning points and that dynamics can be studied within certain segments. Johnston, Hagen and Jamieson have also utilized candidate appearances, news coverage, and campaign advertising to provide this integrated account of a U.S. campaign. Richard Johnston is Professor and Head of Political Science at the University of British Columbia and an Associate Member of Nuffield College, Oxford. He is co-author of Letting the People Decide (Stanford University Press, 1992) and The Challenge of Direct Democracy (McGill-Queen's University Press, 1996). Michael G. Hagen is Associate Research Professor and Director of the Center for Public Interest Polling at the Eagleton Institute of Politics at Rutgers University. He is co-author of Race and Inequality: A Study in American Values (Chatham House, 1986) and a contributor to Reasoning and Choice: Explorations in Political Psychology (Cambridge, 2003). Kathleen Hall Jamieson is Ware Professor of Communication and Director of the Annenberg Public Policy Center at the University of Pennsylvania. She is author or co-author of twelve books on politics and media including Packaging the Presidency (Oxford University Press, 1988). © Richard Johnston, Michael G. Hagen, and Kathleen Hall Jamieson 2004.
Article
The record of British election polls was good--until the general election of 1992. The Market Research Society's inquiry into the performance of the polls in 1992 found inadequacies in the implementation of the sampling system used, evidence of a late swing and some reluctance of Conservative supporters to reveal their loyalty; but it generally endorsed the principle of well-conducted quota polling and found that variations in methodological detail had nil effect on the results. The evidence is presented and some possible future developments to counter the `spiral of silence' are discussed.
Article
The authors examine the results of 48 national public opinion polls measuring support for George Bush and Michael Dukakis throughout the 1988 presidential campaign conducted or reported by five major media polling organizations. Polling trends are discussed, and the consistency of estimates across polls are assessed, across seven distinct time periods defined by key events during the 1988 election year, while accuracy is assessed by comparing final pre-election polls against election results. Time series transfer function methods are employed to assess the short-term and long-term effects of the two major national party conventions, the two Bush-Dukakis debates, and the Ouayle-Bentsen debate on candidate support. Statistically significant positive effects on Bush support are obtained for the Republican convention and for the second presidential debate, while support for Dukakis was affected significantly, and in opposite directions, by the two party conventions. The results are discussed in the context of recent research conducted by Crespi and others.
Article
This article proposes a new measure of the predictive accuracy (A) of election polls that permits examination of both accuracy and bias, and it applies the new measure to summarize the results of a number of preelection polls. We first briefly review past measures of accuracy, then introduce the new measure. After the new measure is described, the general strategy is to apply it to three presidential elec- tions (1948, 1996, and 2000) and to compare the results derived from it to the results obtained with the Mosteller measures. Then, the new mea- sure is applied to the results of 548 state polls from gubernatorial and senatorial races in the 2002 elections to illustrate its application to a large body of preelection polls conducted in "off-year" races with dif- ferent outcomes. We believe that this new measure will be useful as a summary measure of accuracy in election forecasts. It is easily com- puted and summarized, and it can be used as a dependent variable in multivariate statistical analyses of the nature and extent of biases that affect election forecasts and to identify their potential sources. It is com- parable across elections with different outcomes and among polls that vary in their treatment or numbers of undecided voters.
Article
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
Article
New tools for measuring changes in public opinion can be derived from the theory of the spiral of silence. Measures of individual assessment of the climate of opinion and of confidence about showing one's own opinion document the processes by which the losing side falls increasingly silent and the winning side is therefore overrated.
Article
The Kalman filter is a popular tool in engineering and economics. It is becoming popular in political science, touted for its abilities to reduce measurement error and produce more precise estimates of true public opinion. Its application to survey measures of public opinion varies in important ways compared to the traditionally understood Kalman filter. It makes a priori assumptions about the variance of the sampling error that would not usually be made and does so in a way that violates an important property of the Kalman filter. Consequently, the behavior of the filter modified for public opinion measures is less well-known. Through simulations we assess whether and to what extent filtering: reliably detects the characteristics of time series; does so across series with different rates of autoregressive decay; and does so when the variance of the sampling error is unknown. We also examine whether the filtered data represents the level of true underlying variance and the extent to which filtering assists or hinders our ability to detect exogenous shocks. We learn a numbers of things. Most importantly, taking into account sampling error variance when filtering data can work well, though its performance does vary. First, filtering works best identifying time series characteristics when assuming a stationary process, even if the underlying process contains a unit root. Second, the performance of filtering drops off when we incorrectly specify the variance of the sampling error, and especially when we overestimate it. Third, when estimating exogenous shocks it is better to make no a priori assumptions regarding a measurement error variance unless we are absolutely certain we know what it is. In fact, applying the filter without specifying the measurement error variance is more often than not the best choice.
Article
The 2004 and 2006 Canadian elections saw unprecedented numbers of trial heat polls, including the first daily tracking polls in the country's history. This paper attempts to assess the competing claims regarding the accuracy of these polls by setting them in proper context and providing a systematic accounting of what might reasonably be expected from published results. We build on and utilise a method for disentangling sampling error, “house” effects, and true change. Most critically we estimate a value for bias in the polling industry as a whole.Strikingly, systematic variation across firms within the industry is less notable than bias in the industry as a whole. There was less bias in 2004 than in 2006, but only marginally so. In each election, the industry underestimated the Liberal share and overestimated the NDP one. We speculate on sources of bias, and then ask if the discrepancy between polls and the electorate is methodological at the core or the product of last-minute strategic decision-making.
Article
This article describes some solutions to common problems in pre-election surveys drawing upon Gallup Poll experience. It touches upon problems in sampling, estimation, response validity, the undecided, measuring likelihood to vote, and measuring late trends in voter preference. In conclusion it cites demonstrable gains in accuracy that have followed application of the solutions described.Paul Perry is Vice Chairman, The Gallup Organization, Inc.
Article
This article undertakes an initial evaluation of the performance of the opinion polls in the 1997 Election. Although widely regarded as a successful election for the industry, in sharp contrast to 1992, there were important differences between the figures produced by the various companies. These differences provide us with important clues as to the relative effectiveness of the various methods that were used. They suggest that many of the more radical changes that were made to polling methods in response to the problems encountered in 1992 were successful and that the industry will need to continue to experiment and innovate before the next election.
Article
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
Article
There is little systematic research on the multitude of factors that influence the accuracy of poll results. This article examines six methodological factors directly under the survey researcher's control and two exogenous factors concerning the nature of public opinion as sources of survey error. Data for this study come from 56 ''trial heat'' polls conducted during the last month of the 1992 presidential election. The most important variables influencing survey accuracy were the number of days a poll is in the field, which increased total accuracy one-half of a percentage point per day; conducting interviews only on weekdays (and thus only during evening hours), which reduced overall accuracy rates by more than 1 percentage point; and conducting a ''tracking'' poll, which increased accuracy by about 1.5 points. Sample size was not related to accuracy rates. Results also indicated that sampling frames of ''likely voters'' (relative to ''registered voters'') tended to overestimate support for George Bush and underestimate support for Ross Perot, that interviewing only on weekdays led to overestimates of support for Bush, and that strict methods of defining a respondent as ''supporting'' a candidate hurt the two newcomers, Perot and Bill Clinton, more than Bush. In light of these data it is recommended that the common practice of reporting ''margins of error'' based solely on sample sizes be abandoned as misleading and replaced by a more empirically justifiable measure based more on response rates.
Article
Examination of 155 poll forecasts in 68 national elections since 1949 shows that errors average nearly twice what statistical theory would indicate. Polls predict the division of vote between major parties better than individual party percentages, leading to 85 percent success in picking the winner. The worst failures occurred in a few elections where most polls went wrong. Liberal party votes are correctly forecast, conservatives slightly underestimated. Improved polling methods have not led to better forecasts.
Article
Poll results vary over the course of a campaign election and across polling organisations, making it difficult to track genuine changes in voter support. I present a statistical model that tracks changes in voter support over time by pooling the polls, and corrects for variation across polling organisations due to biases known as ‘house effects’. The result is a less biased and more precise estimate of vote intentions than is possible from any one poll alone. I use five series of polls fielded over the 2004 Australian federal election campaign (ACNielsen, the ANU/ninemsn online poll, Galaxy, Newspoll, and Roy Morgan) to generate daily estimates of the Coalition's share of two-party preferred (2PP) and first preference vote intentions. Over the course of the campaign there is about a 4 percentage point swing to the Coalition in first preference vote share (and a smaller swing in 2PP terms), that begins prior to the formal announcement of the election, but is complete shortly after the leader debates. The ANU/ninemsn online poll and Morgan are found to have large and statistically significant biases, while, generally, the three phone polls have small and/or statistically insignificant biases, with ACNielsen and (in particular) Galaxy performing quite well in 2004.* An earlier version of this paper was prepared for the annual meeting of the Australasian Political Studies Association, University of Adelaide, 29 September–1 October 2004.
Article
Little is known about the evolution of electoral sentiment over the campaign cycle. How does the outcome come into focus as the election cycle evolves? Do voters’ preferences evolve in a patterned and understandable way? What role does the election campaign play? In this article, we address these issues. We translate general arguments about the role of campaigns into a set of formal, statistical expectations. Then, we outline an empirical analysis and examine poll results for the 15 U.S. presidential elections between 1944 and 2000. Our analysis focuses specifically on two questions. First, to what extent does the observable variation in aggregate poll results represent real movement in electoral preferences (if the election were held the day of the poll) as opposed to mere survey error? Second, to the extent polls register true movement of preferences owing to the shocks of campaign events, do the effects last or do they decay? Answers to these questions tell us whether and the extent to which campaign events have effects on preferences and, if so, whether these effects persist until Election Day. The answers thus inform us about what we ultimately want to know: Do campaigns have any real impact on the election outcome? The results of our analysis suggest that they do.
Article
Much research shows that voters behave in understandable ways on election day and that election outcomes themselves are quite predictable. Even to the extent we can predict what voters do at the very end of the campaign, we know relatively little about how electoral preferences evolve over time. The U. S. presidential race in 2000 offers a fairly unique opportunity. The volume of available poll data for this particular election allows us to examine the dynamics of voter preferences in great detail for much of the election cycle. Analysis of the poll results in 2000 reveals that underlying electoral preferences changed quite meaningfully during the course of the campaign. The analysis also provides evidence that a significant portion of these changes in preferences actually persisted over time to affect the outcome on election day. Based on these results, it appears that the 2000 presidential election campaign mattered quite a lot.
Assessing poll performance in the 2000 campaign A review: preelection survey methodology: Details from eight polling organizations
  • M W Traugott
Traugott, M. W. (2005). Assessing poll performance in the 2000 campaign. Public Opinion Quarterly, 65, 389−419. Voss, D. S., Gelman, A., & King, G. (1995). A review: preelection survey methodology: Details from eight polling organizations, 1988 and 1992. Public Opinion Quarterly, 59, 98−132.
Election predictions: an empirical assessment Assessing the accuracy of polls and surveys Pre-election polling: Sources of accuracy and error The opinion polls: the election they got (almost) right So how well did they do? The polls in the 1997 election
  • W Buchanan
  • −227 Converse
  • P E Traugott
  • −1099 Crespi
Buchanan, W. (1986). Election predictions: an empirical assessment. Public Opinion Quarterly, 40, 220−227. Converse, P. E., & Traugott, M. W. (1986). Assessing the accuracy of polls and surveys. Science, 234, 1094−1099. Crespi, I. (1988). Pre-election polling: Sources of accuracy and error. New York: Russell Sage Foundation. Crewe, I. (2005). The opinion polls: the election they got (almost) right. Parliamentary Affairs, 58, 684−698. Curtice, J. (1997). So how well did they do? The polls in the 1997 election. Journal of the Market Research Society, 39, 449−462.