No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Campaign trial heats as election forecasts: Measurement error and bias in 2004 presidential campaign polls
Abstract
If late-campaign polls are to be used as forecasts, it is important to ask, how well do the polls do and why are some polls better forecasts than others? We analytically compare alternative methods for estimating the systematic bias in the election trial heat polls of the individual polling houses and of the polling industry as a whole. We put each technique to the test using data from the 2004 US Presidential election. From the collection of evidence we are able to identify the approach that produces the most efficient unbiased estimates and answer the question of how the polls did in 2004. A third of the houses exhibited large and significant biases, but the industry as a whole converged on the truth.
... Detecting house effects in polls is challenging, and we highlight four perspectives on doing this. First, it is appealing to define house effects using the mechanisms that we think are responsible for them [74]. However, the specific choices about methodology that pollsters make are often proprietary [74]. ...
... First, it is appealing to define house effects using the mechanisms that we think are responsible for them [74]. However, the specific choices about methodology that pollsters make are often proprietary [74]. As a second option, Jackman [62] used the election results to back out the house effects active in polls for that election, but this does not make sense for real-time forecasting. ...
... As a second option, Jackman [62] used the election results to back out the house effects active in polls for that election, but this does not make sense for real-time forecasting. A third option is to assume that the net partisan lean across all of the polls is zero or some other value for the election in question [58,74,83], and then use this to anchor estimates of poll bias from a Bayesian perspective. This approach can suffer if industry bias causes all of the polls to collectively miss the result [74]. ...
In the months leading up to political elections in the United States, forecasts are widespread and take on multiple forms, including projections of what party will win the popular vote, state ratings, and predictions of vote margins at the state level. It can be challenging to evaluate how accuracy changes in the lead up to Election Day or to put probabilistic forecasts into historical context. Moreover, forecasts differ between analysts, highlighting the many choices in the forecasting process. With this as motivation, here we take a more comprehensive view and begin to unpack some of the choices involved in election forecasting. Building on a prior compartmental model of election dynamics, we present the forecasts of this model across months, years, and types of race. By gathering together monthly forecasts of presidential, senatorial, and gubernatorial races from 2004--2022, we provide a larger-scale perspective and discuss how treating polling data in different ways affects forecast accuracy. We conclude with our 2024 election forecasts (upcoming at the time of writing).
... Second, authors generally focus on average estimates, whether at the end of the campaign (Clinton et al., 2021;Panagopoulos, 2021;Silver, 2021) or for the entire main campaign (Enns & Rothschild, 2021), without considering possible movements in support during the campaign (Pickup & Johnston, 2008;Durand & Johnson, 2021). They act as if polling error was stable throughout the campaign (Clinton et al., 2021) without verifying that it is the case and that different modes trace the same trends in voting intentions (Pickup & Johnston, 2008;Durand & Johnson, 2021). ...
... Second, authors generally focus on average estimates, whether at the end of the campaign (Clinton et al., 2021;Panagopoulos, 2021;Silver, 2021) or for the entire main campaign (Enns & Rothschild, 2021), without considering possible movements in support during the campaign (Pickup & Johnston, 2008;Durand & Johnson, 2021). They act as if polling error was stable throughout the campaign (Clinton et al., 2021) without verifying that it is the case and that different modes trace the same trends in voting intentions (Pickup & Johnston, 2008;Durand & Johnson, 2021). ...
... The methodological characteristics introduced are the ones suggested by the authors (e.g., Pickup and Johnston, 2008;Chen, Garnett, & Montgomery, 2022) as having a possible impact. At the poll level, the most important predictor is the period when each poll was conducted. ...
The poll performance of the 2020 US election has stood out as the worst in US presidential elections since 1996. National election polls also presented the largest difference in estimates between modes of administration in US elections since 2008. What happened? This article proposes a new perspective on the polls published at the national level during the campaign. It examines the 222 polls conducted from September 1st to November 2nd 2020 to assess whether the polls differed according to mode of administration and sampling source in the portrait they trace of what happened during the campaign and in their capacity to forecast the election results. The polls are grouped into three categories that combine mode of administration and sampling source: a) the mixed-mode polls (16% of the polls), which use multiple modes and sampling sources, b) the one-mode quasi-random polls (25%), which use one mode of administration but resort to random or quasi-random sampling sources, and c) the web polls that use exclusively opt-in panels (59%). Two types of regression analysis are performed. Local regression gives an estimate of the trends in voting intention, and multilevel analysis provides a statistical validation of these trends, all else being equal, after controlling for the polls’ dependency on pollsters and their methodological characteristics. These analyses show that the various mode-source combinations trace different portraits of what happened during the campaign. The polls that used random or quasi-random sampling sources, whether mixed-mode or one-mode, estimate that vote intention for Joe Biden increased until mid-campaign and declined afterward. On the contrary, the Web Opt-in polls estimate that support for Joe Biden was stable throughout the campaign. In addition, the mixed-mode polls lead to a perfect vote forecast. A descriptive analysis of the polls conducted during the last ten days of the campaign illustrates and validates these results. Almost all the mixed-mode polls and the one-mode quasi-random polls produced estimates within their margin of error or credibility interval, whereas only 54% of the web opt-in polls did. Other methodological practices, like resorting to multiple opt-in panels or weighting using propensity scores, are also associated with better estimates. We hypothesize that modes and sampling sources reach different voters, but none reach them all. Hence the necessity to resort to mix modes or to vary the sampling sources. If we consider that the perfect forecasting produced by mixed-mode polls is an indication of their overall reliability, we conclude that what likely happened during the campaign is an increase and subsequent decrease in vote intention for Joe Biden. American citizens did not get this information because the trend traced by web opt-in polls dominated the averages. Finally, academics, media, and pollsters in the US and elsewhere need to monitor the new methods that emerged in the US 2020 election and draw lessons from their performance.
Keywords: Electoral polls; Survey methods; Election Forecasting; US 2020 presidential election; Local regression; Multilevel analysis.
... Some scholarsdmost notably Green et al. (1999)dhave advocated a variant of the Kalman filter that takes into account expected sampling error variance. Other scholars have adopted this approach in their estimates of survey house effects (Jackman, 2005;Pickup and Johnston, 2008). 2 In so doing, they provide estimates of real underlying movement. ...
... This clearly complicates the assessment of change over time. A good amount of work has been done (and is underway) in estimating and correcting for the differences across organizations, thus better enabling us to combine data from different sources (Erikson and Wlezien, 1999;Jackman, 2005;Wlezien and Erikson, 2007;Pickup and Johnston, 2008 to count the same respondents on multiple days, and it is very easy to remove this overlap, e.g., for survey house reporting 3-day moving averages, we only use poll results for every third day. We then calculate the Bush share of the two-party vote intention, ignoring all other candidates, e.g., Nader. ...
... Even where we do have full information about sampling error, we want to know how reliable filtering is. We will be examining the variant of the Kalman filter typically applied to political science public opinion data (Beck, 1989;1991Green et al., 1999Jackman, 2005;and Pickup and Johnston, 2008). The traditional Kalman filter produces the optimal prediction of a time series at time t, only considering the observations previous to time t and produces the optimal filtered values considering information previous to and including time t. ...
The Kalman filter is a popular tool in engineering and economics. It is becoming popular in political science, touted for its abilities to reduce measurement error and produce more precise estimates of true public opinion. Its application to survey measures of public opinion varies in important ways compared to the traditionally understood Kalman filter. It makes a priori assumptions about the variance of the sampling error that would not usually be made and does so in a way that violates an important property of the Kalman filter. Consequently, the behavior of the filter modified for public opinion measures is less well-known. Through simulations we assess whether and to what extent filtering: reliably detects the characteristics of time series; does so across series with different rates of autoregressive decay; and does so when the variance of the sampling error is unknown. We also examine whether the filtered data represents the level of true underlying variance and the extent to which filtering assists or hinders our ability to detect exogenous shocks. We learn a numbers of things. Most importantly, taking into account sampling error variance when filtering data can work well, though its performance does vary. First, filtering works best identifying time series characteristics when assuming a stationary process, even if the underlying process contains a unit root. Second, the performance of filtering drops off when we incorrectly specify the variance of the sampling error, and especially when we overestimate it. Third, when estimating exogenous shocks it is better to make no a priori assumptions regarding a measurement error variance unless we are absolutely certain we know what it is. In fact, applying the filter without specifying the measurement error variance is more often than not the best choice.
... Or, la part combinée des deux principaux candidats varie considérablement entre les sondages en raison de diverses caractéristiques méthodologiques, y compris la formulation des questions (Clinton et al., 2021). Deuxièmement, les chercheurs se concentrent généralement sur les estimations moyennes, que ce soit à la fin de la campagne (Clinton et al., 2021;Panagopoulos, 2021;Silver, 2021) ou pour l'ensemble de la campagne (Enns & Rothschild, 2021), sans prendre en compte les éventuels changements dans les appuis pendant la campagne, comme recommandé (Pickup & Johnston, 2008;Durand & Johnson, 2021). Troisièmement, la dépendance des sondages envers les instituts qui les produisent doit impérativement être prise en compte pour éviter d'attribuer aux sondages individuels ce qui relève des pratiques des instituts. ...
... incluons a) la marge d'erreur (ME)l'intervalle de crédibilité (IC) 4 dans la plupart des casqui prend en compte l'impact attendu de la taille de l'échantillon, b) le nombre de jours sur le terrain, comme indicateur du taux de réponse et de la qualité du sondage et c) l'utilisation d'un modèle de probabilité de vote (MPV) par opposition à une base d'électeurs inscrits, une pratique possiblement liée au biais d'estimation(Pickup & Johnston, 2008;Sturgis et al., 2018;Kennedy et al., 2017;Enns & Rothschild, 2021).Au niveau des instituts de sondage, le nombre de sondages menés depuis le 1er septembre est une approximation de l'expérience de l'institut. Nous postulons que les instituts de sondage plus actifs ont plus d'expérience dans la conduite de sondages électoraux 5 . ...
Auteur Claire Durand est professeur au département de sociologie de l'Université de Montréal. Ses recherches portent sur la méthodologie des sondages-plus particulièrement celle des sondages électoraux et sur les méthodes quantitatives avancées. Remerciements L'auteur tient à remercier Luis Patricio Peña Ibarra, qui a identifié et collecté les données pour cette recherche et Timothy P. Johnson, qui a apporté son soutien à la première phase de cette recherche. L'auteur tient également à remercier les nombreux instituts qui ont collaboré avec nous pour compléter l'information sur les méthodes qu'ils utilisent. Accès aux données: Les données utilisées dans cette publication sont disponibles sur le dépôt Dataverse à l'adresse: https://doi.org/10.5683/SP3/ORS1DH Conflit d'intérêt: L'auteur déclare n'avoir aucune affiliation ni collaboration avec une organisation quelconque qui pourrait avoir des intérêts financiers dans la question discutée et les conclusions de ce manuscrit. 1 Résumé : Les performances des sondages de l'élection présidentielle américaine de 2020 se sont avérées les pires depuis l'élection présidentielle de 1996. Les sondages nationaux ont également présenté la plus grande différence d'estimations entre les modes d'administration dans les élections américaines depuis 2008. Que s'est-il passé? Cet article propose une nouvelle perspective sur les sondages publiés au niveau national pendant la campagne. Il analyse les 222 sondages menés par 51 instituts de sondage du 1er septembre au 2 novembre 2020 pour évaluer s'ils différent selon le mode d'administration et les bases de sondage ou sources d'échantillons utilisés. L'analyse porte sur le portrait que les sondages tracent de ce qui s'est passé pendant la campagne et sur leur capacité à prédire les résultats de l'élection. Les sondages sont regroupés en trois catégories qui combinent mode d'administration et source d'échantillonnage : a) les sondages à modes mixtes (16% des sondages), qui utilisent plusieurs modes et sources d'échantillonnage; b) les sondages à mode unique quasi-aléatoire (25%), qui utilisent un seul mode d'administration mais font appel à des sources d'échantillonnage aléatoires ou quasi-aléatoires; et c) les sondages web qui utilisent exclusivement des access panels (59%). Deux types d'analyses de régression sont effectuées. La régression locale génère une estimation de l'évolution des intentions de vote en fonction des combinaisons de mode et de source. L'analyse multiniveaux fournit une validation statistique de cette évolution, toutes choses égales par ailleurs, après avoir contrôlé pour la dépendance des sondages aux instituts et à leurs caractéristiques méthodologiques. Ces analyses montrent que les combinaisons mode-source diffèrent dans le portrait qu'elles tracent de ce qui s'est passé pendant la campagne. Les sondages qui ont utilisé des sources d'échantillonnage aléatoires ou quasi-aléatoires, qu'ils soient à modes mixtes ou à mode unique, estiment que l'intention de vote pour Joe Biden a augmenté jusqu'à mi-campagne et a diminué par la suite. Au contraire, les sondages web utilisant des access panels estiment que l'appui à Joe Biden est resté stable tout au long de la campagne. En outre, les sondages à modes mixtes mènent à une prédiction parfaite du vote. Une analyse descriptive des sondages menés pendant les dix derniers jours de la campagne valide ces résultats. Presque tous les sondages à modes mixtes et les sondages à mode unique quasi-aléatoire ont produit des estimations à l'intérieur de leur marge d'erreur ou intervalle de crédibilité comparé à seulement 54% des sondages utilisant des access panels. D'autres pratiques méthodologiques, comme le recours à plusieurs sources d'échantillon ou la pondération utilisant des scores de propension, sont également associées à de meilleures estimations. Nous émettons l'hypothèse que les modes et les sources d'échantillonnage atteignent différents électeurs, mais aucun mode ou source n'atteint tout le monde, d'où la nécessité de recourir à des modes mixtes ou de varier les sources d'échantillonnage. Si nous considérons que la prévision parfaite produite par les sondages à modes mixtes est une indication de leur fiabilité globale, nous concluons que ce qui s'est fort probablement passé pendant la campagne est une augmentation jusqu'à mi-campagne de l'intention de vote pour Joe Biden et une diminution subséquente jusqu'à l'élection. Les citoyens américains n'ont pas reçu cette information parce que la tendance tracée par les sondages web utilisant des access panels (59% des sondages) ont dominé. Les universitaires, les médias et les sondeurs aux États-Unis et ailleurs doivent suivre les nouvelles méthodes qui ont émergé lors de l'élection américaine de 2020 et tirer des leçons de leur performance. 2
... Jackman (2005) outlines a very similar model that is more focused on pooling polls to estimate current preferences and house effects rather than making explicit election forecasts. Pickup andJohnston (2007, 2008) expanded upon Jackman's work to estimate house effects and industry-wide bias in the 2004 and 2006 Canadian and 2004 US Presidential elections. Our method, rather than estimating pollster-specific errors from multiple polls of the same race, will estimate aggregate errors across multiple elections. ...
... Jackman (2005) outlines a very similar model that is more focused on pooling polls to estimate current preferences and house effects rather than making explicit election forecasts. Pickup andJohnston (2007, 2008) expanded upon Jackman's work to estimate house effects and industry-wide bias in the 2004 and 2006 Canadian and 2004 US Presidential elections. Our method, rather than estimating pollster-specific errors from multiple polls of the same race, will estimate aggregate errors across multiple elections. ...
With historic misses in the 2016 and 2020 US Presidential elections, interest in measuring polling errors has increased. The most common method for measuring directional errors and non-sampling excess variability during a postmortem for an election is by assessing the difference between the poll result and election result for polls conducted within a few days of the day of the election. Analyzing such polling error data is notoriously difficult with typical models being extremely sensitive to the time between the poll and the election. We leverage hidden Markov models traditionally used for election forecasting to flexibly capture time-varying preferences and treat the election result as a peak at the typically hidden Markovian process. Our results are much less sensitive to the choice of time window, avoid conflating shifting preferences with polling error, and are more interpretable despite a highly flexible model. We demonstrate these results with data on polls from the 2004 through 2020 US Presidential elections, concluding that previously reported estimates of pro-Democratic bias in 2016 and 2020 were too small, while excess variability estimates were too large.
... Polling data provide an imperfect measure of true public opinion: each poll is potentially subject to random measurement error, design effects and systematic bias. By pooling together estimates from the different polls, we can produce a more accurate estimate, which limits the impact of both random variation between polling samples and systematic effects generated by the methods chosen by the polling houses -commonly termed 'house effects' or 'house bias' (Erikson and Wlezien, 1999;Jackman, 2005;Pickup and Johnston, 2008). In doing so, we are also able to estimate the magnitudes of the systematic errors in polls. ...
... To identify the patterns of bias in British polling and produce a more accurate estimate of public opinion, we build upon techniques developed by Pickup andJohnston (2007, 2008), Jackman (2005) and by Erikson and Wlezien (1999). We employ a state-space model of the kind described in Pickup (2009). ...
Pollsters once again found themselves in the firing line in the aftermath of the 2010 British general election. Many critics noted that nearly all pollsters in 2010 expected a substantial surge for the Liberal Democrats that did not materialize. Basing conclusions regarding the relative merits of pollsters or benefits of methodological design features on inspection of just the final poll from each pollster is inherently problematic, because each poll is subject to sampling error. This paper uses a state‐space model of polls from across the course of the 2010 election campaign which allows us to assess the extent to which particular pollsters systematically over‐ or under‐estimate each main party’s share of the vote, while allowing for both the usual margins of error for each poll and changes in public opinion from day‐to‐day. Thus, we can assess the evidence for systematic differences between pollsters’ results according to the use of particular methodologies, and estimate how much of the discrepancy between the final polls and the election outcome is due to methodological differences that are associated with systematic error in the polls. We find robust evidence of an over‐estimation in Liberal Democrat support, but do not find evidence to support the hypothesis that the polls erred due to a late swing away from the party, nor that any of the methodological choices made by pollsters were significantly associated with this over‐estimation.
... Similar models have been adopted by Jackman (2005) and Hanretty (2013) as well as by Pickup and Johnston (2007) and Pickup and Johnston (2008) (see section 1). However, a major difference between those models and our proposal is that we do not need to consider the true vote share of each party (i.e., their actual election results), choice that requires additional and partly unjustified model assumptions. ...
It is widely known that pre-electoral polls often suffer from non-
sampling errors that pollsters try to compensate for in final estimates
by means of diverse ad hoc adjustments, thus leading to well-known
house effects. We propose a Bayesian hierarchical model to investigate the role of house effects on the total variability of predictions. To illustrate the model, data from pre-electoral polls in Italy in 2006, 2008 and 2013 are considered. Unlike alternative techniques or models, our proposal leads: (i) to correctly decompose the different sources of variability; (ii) to recognize the role of house effects; (iii) to evaluate its dynamics, showing that variability of house effects across pollsters diminishes as the date of election approaches; (iv) to investigate the relationship between house effects and overall prediction errors.
... We also looked at the media mentions for the two weeks preceding Super Tuesday, but a candidate's share did not change much as the correlation between the one-week and two-week measures was 0.99. 2 Analytically, including polling poses an interesting issue for our models. Not all polls are a valid measurement of where a race stands (Pickup and Johnston 2008) as polls can be inaccurate and/or biased (Martin, Traugott, and Kennedy 2005). Second, polling is often correlated with other measures of primary success such as media attention and public attention (e.g. ...
In presidential primaries, Super Tuesday elections play a significant role in winnowing candidate fields and establishing nomination frontrunners. Despite their importance, scholars know little about why and how candidates win or lose the states comprising these events. This study explores which factors help explain candidate performance in Super Tuesday primaries between 2008 and 2016. Using pooled cross-sectional time-series analysis, the results indicate three key drivers of Super Tuesday success: candidate viability, public attention, and media attention. These findings imply that presidential campaigns continue to be complex electoral events beyond the early primary states and suggest that underdog candidates can still win states under the right conditions. Future research should explore the interrelatedness of these three critical factors.
... On the one hand, there are the technical characteristics and survey design. However, the difference between electoral predictions and results cannot simply be put down to a statistical error, but derive from sample design, question formulation, weighting, and screening [Pickup and Johnston, (2008)]. Second, there is the behavior of the respondents themselves. ...
The main purpose of this paper is to use the Spanish case, through an econometric analysis of 226 electoral polls, to explain why polls are making more mistakes in times of great socioeconomic slumps, political instability and the emergence of new political parties. In this context, it is the very instrument with which society tries to reduce the reigning uncertainty that, paradoxically, can ultimately drive uncertainty up. Our results show that the prediction error for the new emerging parties is significantly higher than for the traditional parties and this error is not sensitive to solutions for increasing the reliability of surveys, such as increasing sample size, transparency constantly conducting periodical surveys, the closeness of the approaching election or the survey mode that is used. It can be observed that pollsters do not want to make predictions that vary greatly from the average of the other polls. Finally, editorial bias appears to play a significant role, especially in the case of traditional parties.
... Esta información de última hora permite corregir las estimaciones incrementando su fiabilidad y precisión. Según varios estudios, las encuestas próximas a las elecciones (como media tres meses antes de ellas) facilitan los datos más fiables para una predicción electoral (Traugott, 2001(Traugott, , 2005Pickup y Johnston, 2008;Panagopoulos, 2009c;Alaminos, 1994). Cuanto más antiguos son los datos, la precisión disminuye (Campbell y Wink, 1990;Gelman y King, 1993;Campbell, 1996), siendo útiles para evaluar las tendencias observadas, pero contribuyendo en menor grado al estado actual. ...
Tras introducir brevemente algunos conceptos explicativos del voto desde el punto de vista de su influencia a largo, medio y corto plazo, se efectúa un análisis desde el enfoque del modelo de voto retrospectivo. Se analizan cuatro series del banco de datos del Centro de Investigaciones Sociológicas, con la valoración de la gestión/actuación del Partido Popular y del Partido Socialista Obrero Español, así como las series de intención de voto a los dos partidos. Del análisis se concluye la estrecha relación existente entre la valoración de la gestión/actuación de ambos partidos con su intención de voto. Las tendencias de declive observadas en la confianza en la gestión/actuación de los dos partidos anticipaban el escenario de crisis del bipartidismo de 2015, sugiriendo su incorporación como elemento de su «imagen de partido». Así mismo, considerando las dinámicas para los dos partidos, se observa la anomalía que, dentro de la pauta general, representan las elecciones generales de 2004 y 2011. Como estrategia de control de la falacia ecológica de los agregados, se ha efectuado un control en cinco años (1997, 2002, 2007, 2012 y 2017) analizando los microdatos para establecer la relación entre valoración de la gestión/actuación del partido y su intención de voto en la unidad de análisis individual.After briefly introducing some of the explanatory concepts of voting from the point of view of its influence in the long, medium and short term, an analysis is made from the retrospective voting model approach. For the Spanish case, four series of the Centre for Sociological Research database are analysed, with the assessment of the management/performance of the Popular Party and the Spanish Socialist Workers’ Party, as well as the series of voting intention to the two political parties. The analysis concludes the close relationship between the assessment of the management/performance of both political parties with their voting intention. The trends of decline observed in the trust in management/ performance of the two parties anticipated the bipartisan crisis scenario of 2015. Furthermore, considering the dynamics for the two parties, we observe the anomaly that, within the general pattern, represents the general elections of 2004 and 2011. As a strategy to control the ecological fallacy of the aggregates, an analysis has been carried out in five years (1997, 2002, 2007, 2012 and 2017) studying the microdata to establish the relationship between management/performance valuation of the party and the voting intention at an individual level.
... 8 Public opinion poll data are drawn from the lists available from www.electionalmanac. com. 9 Certainly there are documented issues with polls in Alberta, Quebec and BC to suggest that these data may not accurately reflect whether a party's support is genuinely in decline (see Grénier, 2012a, b;; see also Pickup and Johnston, 2008;Pickup et al., 2011). 10 Marois sought the leadership of the Parti Québécois (PQ) three times. ...
The majority of Canada's women premiers were selected to that office while their parties held government. This is uncommon, both in the comparative literature and among premiers who are men. What explains this gendered selection pattern to Canada's provincial premiers’ offices? This paper explores the most common explanation found in the comparative literature for women's emergence as leaders of electorally competitive parties and as chief political executives: women are more likely to be selected when that party is in crisis or decline. Using the population of women provincial premiers in Canada as case studies, evidence suggests three of eight women premiers were selected to lead parties in government that were in crisis or decline; a fourth was selected to lead a small, left-leaning party as predicted by the literature. However, for half of the women premiers, evidence of their party's decline is partial or inconclusive. As a result of this exploration, more research is required to draw generalizations about the gendered opportunity structures that shape how women enter (and exit) the premier's office in Canada.
... There is an important study that indicates that one of the reasons behind the failure of the surveys in predicting accurate results is that many of the centers conducting the surveys does not declare with transparency the measures they adopted in carrying out the survey and may not be sincere in describing the steps they have taken particularly in the absence of two important elements: the control and benchmarking [14]. The fact is that there is no accountability or control or any kind on the actions of the public opinion centers in spite that its works is not less important than that of the mass media. ...
This study involves the controversial role that public opinion surveys played during the American presidential elections 2016 that has acquired exclamation marks due to the evident conflict between reports of mass media, particularly the private ones, that placed the democratic candidate, Clinton, ahead in the race while results comes directly the opposite. The study applied “Time for Change” approach and includes three sections; first, the methodology, questions, and objectives. Secondly, the theoretical part and mechanisms of the survey centers, reasons leading to failure in achieving sound results, and the political and economic effects of the newspapers’ agendas in guiding the survey results Thirdly, the last section demonstrates the findings that include: the major reasons that enabled Trump to overcoming the intentional mass media disregard and finally the reasons that made his winning an unexpected shock.
... Thus, it may be that we actually have a fairly reasonable portrait of electoral preferences, particularly when we combine polls from different survey organizations, where errors may to a large extent cancel out 15 . Unfortunately, we do not have much information about the practices of different survey organizations, particularly going back in time, which precludes an analysis of the effects of 15,17,18 . We can still examine the performance of poll aggregations. ...
Are election polling misses becoming more prevalent? Are they more likely in some contexts than others? In this paper we undertake an over-time and cross-national assessment of prediction errors in pre-election polls. Our analysis draws on more than 26,000 polls from 338 elections in 45 countries over the period between 1942 and 2013, as well as data on more recent elections from 2014 to 2016. We proceed in the following way. First, building on previous studies, we demonstrate how poll errors evolve in a structured way over the election timeline. Second, we then focus on errors in polls in the final week of the campaign to examine poll performance across election years. Third, we use the historical performance of polls to benchmark recent polling “misses” in the UK, US and elsewhere. Fourth, we undertake a pooled analysis of polling errors – controlling for a number of institutional and party features – which enables us to test whether poll errors have increased or decreased over time. We find that, contrary to conventional wisdom, recent performance of polls has not been outside the ordinary. The performance of polls does vary across political contexts, however, in understandable ways.
... The solid lines in Figure 3 depict an aggregation of the polls, which amounts to daily samples well over onethousand, using a variety of polling methodologies. It is widely accepted that an aggregation of polls is very likely to be very close to the true levels of support on a given day (Pickup and Johnston, 2008). Indeed the polls converged very close to the popular vote results. ...
Election studies must optimize on sample size, cost and data quality. The 2015 Canadian Election Study was the first CES to employ a full mixed-mode design, aiming to take advantage of the opportunities of each mode while preserving enough commonality to compare them. This paper examines the phone interviews conducted by ISR-York and the online questionnaires from panellists purchased from a sample provider. We compare data quality and representativeness. We conduct a comprehensive comparison of the distributions of responses across modes and a comparative analysis of inferences about voting. We find that the cost/power advantages of the online mode will likely make it the mode of choice for subsequent election studies.
... Moreover, besides the change in vote intention from day to day within one polling organization, there is the change in vote intention across polling firms. These latter changes are sometimes referred to as "house effects" (Pickup and Johnston 2008). By now, there are many firms, in addition to Gallup, competing in the world of political polling, such as Roper, Harris, Pew, the New York Times, and news network polls, to name only a few. ...
... Moreover, besides the change in vote intention from day to day within one polling organization, there is the change in vote intention across polling firms. These latter changes are sometimes referred to as "house effects" (Pickup and Johnston 2008). By now, there are many firms, in addition to Gallup, competing in the world of political polling, such as Roper, Harris, Pew, the New York Times, and news network polls, to name only a few. ...
In democratic nations, elections are premier political events. They require citizens to choose their leaders and leaders to be held accountable. The winners of an electoral contest are allowed to wield power, sometimes great power. Therefore, citizens follow campaigns with interest and are often eager to know who will win. In other words, they would like a forecast of the outcome. This impulse to election forecasting has been around for a long time. To forecast an election means to declare its outcome before it happens. For example, say in advance which candidate will win the race. There are many methods of forecasting elec-tions, and they can be divided into two groups: prescientific and scientific. While this distinction may seem clear, the differences are not always easy to spot. Below, we emphasize the scientific approach to election forecasting, tracing its his-torical development in the study of United States
... The Kalman filter uses a state-space model of the time series of interest. This approach has been applied to a variety of political science variables (Pickup and Johnston, 2008;Jackman, 2005;Martin and Quinn, 2002;McAvoy, 1998;Brandt and Williams, 2001;Kellstedt, McAvoy and Stimson, 1996). It has been applied to presidential approval and vote intention polls in a series of publications, dating back to Beck (1989). ...
... Such naı¨ve endogenous models would fail to accurately predict the variation in the support for the FN across the whole period (MAEs of 4.56 and 4.47, respectively). Similarly, our model can be contrasted with trial-heat polls that are conducted before the election (Campbell, 1996;Pickup and Johnston, 2008). Comparing the model's ex-post adjusted predicted values (out-of-sample) with the average of voting intention polls released for the 2002 and 2007 presidential elections showed that the model estimates were closer to the actual election outcome on both occasions (Evans and Ivaldi, 2010, pp. ...
Forecasting the Extreme Right vote in French elections is one of the few ‘third-party’ forecasts that has attracted attention in a forecasting literature focusing generally on incumbent performance and winners. Despite being a ‘hard case’ because of third-party status, unstable polling estimates and relatively few data points, previous models have provided relatively strong forecasts of the performance of the Front National (FN) and its erstwhile leader, Jean-Marie Le Pen. The recent succession of Le Pen by his daughter, Marine, and her apparent popularity pose a significant challenge to these models, however. In this article, we consider our previous model's prediction of her likely score in the first round of the presidentials, comparing this to standard forecasting benchmarks, and look at possible adjustments to account for the speculated ‘Marine effect’. We then compare this with other vote indicators including the results of an experimental expert judgment survey, finding that there is currently little evidence for a likely runaway success for the new FN leader in April 2012.
... Much of the variation is likely to be an artefact of the polling process rather than a true reflection of public opinion volatility. By pooling together results from the different polls, we aim to produce a more accurate estimate that reduces the impact of both random variation between polling samples and systematic effects generated by the methods chosen by the polling houses, which commonly are termed 'house effects' or 'house bias' (Erikson and Wlezien, 1999;Jackman, 2005;Pickup and Johnston, 2008). ...
This paper outlines and discusses techniques for three stages in forecasting parliamentary seats from British opinion polls: adjusting and aggregating published vote-intention figures from across different polls; forecasting how public opinion might change before election day; and predicting the seat totals from the forecasted election-day vote shares. Specifically, we consider a state-space model for opinion polls which correct for house effects and other sources of survey error, the estimation of the historical relationship between polls and the election-day share of the vote, and a probabilistic approach to predicting the winner in each constituency.
Résumé
À l’aube de l’élection présidentielle américaine de 2024, il est essentiel de se pencher sur ce qui s’est passé en 2020. Cinquante-et-un instituts ou organisations ont conduit 222 sondages indépendants du 1er septembre jusqu’à la veille du vote, le 3 novembre 2020. Ces sondages ont présenté les plus grands écarts avec le vote depuis 1996 et la plus importante disparité entre les différents modes d'administration depuis 2008. Nous avons classé les sondages en trois catégories en fonction de leur mode d’administration et de leur base de sondage : mixtes, combinant plus d’un mode d’administration (16%), à mode unique utilisant une base aléatoire ou quasi-aléatoire (25%) et web utilisant un access panel (59%). L'analyse utilise des régressions locales pour visualiser l’évolution des appuis durant la campagne, puis des analyses multiniveaux pour valider statistiquement comment les diverses combinaisons de mode et de base de sondage influencent l’estimation de l’évolution des intentions de vote durant la campagne et du vote final. Les analyses révèlent que les sondages à sources d'échantillonnage aléatoires ou quasi-aléatoires (modes mixtes ou uniques) indiquaient une hausse initiale puis une baisse des intentions de vote pour Joe Biden alors que les panels web estimaient que l’intention de vote était stable. Par ailleurs, les sondages mixtes ont mené à une prédiction exacte du vote.
Les analyses des dix derniers jours de campagne confirment globalement ces résultats. Les sondages mixtes et à mode unique quasi-aléatoire sont les plus précis. De plus, l'utilisation de multiples sources d'échantillonnage ou la pondération par scores de propension amènent également de meilleures estimations. Cette étude suggère que différents modes et sources d'échantillonnage rejoignent des segments variés de la population, et que leur combinaison mène à une meilleure estimation de la dynamique électorale. Les sondages web sur invitation, dominants dans l’espace médiatique, ont pu fausser la perception des électeurs. L'article appelle à une surveillance des méthodologies de sondage émergentes pour mieux comprendre leur impact et informer adéquatement la population de l’état de l’opinion publique.
With historic misses in the 2016 and 2020 US Presidential elections, interest in measuring polling errors has increased. The most common method for measuring directional errors and non-sampling excess variability during a postmortem for an election is by assessing the difference between the poll result and election result for polls conducted within a few days of the day of the election. Analysing such polling error data is notoriously difficult with typical models being extremely sensitive to the time between the poll and the election. We leverage hidden Markov models traditionally used for election forecasting to flexibly capture time-varying preferences and treat the election result as a peek at the typically hidden Markovian process. Our results are much less sensitive to the choice of time window, avoid conflating shifting preferences with polling error, and are more interpretable despite a highly flexible model. We demonstrate these results with data on polls from the 2004 through 2020 US Presidential elections and 1992 through 2020 US Senate elections, concluding that previously reported estimates of bias in Presidential elections were too extreme by 10%, estimated bias in Senatorial elections was too extreme by 25%, and excess variability estimates were also too large.
Esta publicación es una edición revisada y ampliada del estudio efectuado en 2017. Ha sido ampliado con el análisis de covariaciones de patrones motivacionales, así como una nueva exploración de las motivaciones de no participación, incrementando con ello el análisis y diagnóstico del reconocimiento de motivaciones por parte de los electorados de acuerdo con el modelo heptafactorial. En el análisis desde el modelo heptafactorial se consideran cinco factores como motivos explicativos del voto {líder, partido, gestión, ideología, clima} y dos como factores explicativos de la abstención {estructural y coyuntural}. Teniendo en cuenta las variables disponibles, el estudio de los factores de abstención no ofrece tipologías significativas en base a los datos, estableciéndose por ello una tipología funcional en dos categorías: abstencionistas crónicos y desmovilizados coyunturales. En este documento de trabajo se va a analizar la influencia que han tenido las cinco motivaciones de voto en la elección de cada partido, según la opinión de las personas entrevistadas, así como se exploran las motivaciones de abstención en las elecciones generales del 26 de junio de 2016.
Para ello se emplea el estudio postelectoral E2920, efectuado por el CIS tras las elecciones generales de 2016. Este documento de trabajo da continuidad a los ya efectuados para las elecciones generales de 1993, 1996, 2000, 2004, 2008, 2011 y 2015.
Are campaigns epiphenomena? The dominant answer in the US literature seems to be Yes. Outcomes are driven by ‘fundamental’ considerations in place before the campaign begins, such that dynamics within the campaign reflect the elimination of ‘accidental’ considerations. Multi-country comparisons, where they exist, tend to be fragmentary. Where they are not fragmentary, they are taken from so high an altitude that critical interpretive details are lost. Also fragmentary, surprisingly, is documentation of the US case itself. This paper identifies two overlapping but distinct claims: about frontrunners and about incumbents. We also identify an insufficiently recognized fact, that the very idea of a ‘fundamental’ is contested, and that this broadens the scope for potential within-campaign dynamics. We employ rolling cross-section survey data from the US, Germany, Italy, New Zealand, and Canada.
On 23 June 2016 the UK held a referendum so to decide whether to stay or leave the European Union. The uncertainty surrounding the outcome of this referendum had major consequences in terms of public policy, investment decisions, and currency markets. We discuss some subtleties entailed in smoothing and disentangling poll data at the light of the problem of tracking the dynamics of the intention to Brexit, and propose a multivariate singular spectrum analysis method that produces trendlines on the unit simplex. The trendline yield via multivariate singular spectrum analysis is shown to bear a resemblance with that of local polynomial smoothing, and singular spectrum analysis presents the nice feature of disentangling directly the dynamics into components that can be interpreted as changes in public opinion or sampling error. Merits and disadvantages of some different approaches to obtain smooth trendlines on the unit simplex are contrasted, both in terms of local polynomial smoothing and of multivariate singular spectrum analysis.
On 23 June 2016 the UK held a referendum so to decide whether to stay or leave the European Union. The uncertainty surrounding the outcome of this referendum had major consequences in terms of public policy, investment decisions, and currency markets. We discuss some subtleties entailed in smoothing and disentangling poll data at the light of the problem of tracking the dynamics of the intention to Brexit, and propose a multivariate singular spectrum analysis method that produces trendlines on the unit simplex. The trendline yield via multivariate singular spectrum analysis is shown to bear a resemblance with that of local polynomial smoothing, and singular spectrum analysis presents the nice feature of disentangling directly the dynamics into components that can be interpreted as changes in public opinion or sampling error. Merits and disadvantages of some different approaches to obtain smooth trendlines on the unit simplex are contrasted, both in terms of local polynomial smoothing and of multivariate singular spectrum analysis.
Opinion polls are central to the study of electoral politics. With modern election polling dating back to the 1936 US presidential election, and proto-straw polls going back as far as the 1824 presidential election, polls have long been employed to gauge the popularity of different political competitors and, for as long as they have been available, researchers have used them to make predictions about future election results (Smith, 1990; Bean, 1948). Research on the links between opinion polls and election outcomes took off in the 1970s and early 1980s, as a cluster of (mostly) American researchers analysed these relationships, primarily in the context of US presidential elections (Campbell, 2008). The field has evolved rapidly since then, as polls have proliferated and analysis tools have grown ever more sophisticated. Opinion poll-based election analysis and forecasting is now a global enterprise, and often one with an unusually high public profile – poll analysis and election forecasts by academics and data journalists are now widely reported and discussed in election campaigns. This chapter reviews research on the relationships between opinion polls and election outcomes, and some of the election forecasting techniques built on this relationship.
This paper develops a three-stage method to forecast parliamentary election results from vote preferences in British opinion polls: (1) adjusting and aggregating vote-intentions from different polling organizations; (2) forecasting how public support for parties will change in the period before election day; and (3) translating, through simulations, the forecast of election day vote shares into seat totals while incorporating constituency-level information, including local vote-intention polls. Overall, this approach seeks to combine relevant national, regional and local information, and uncertainty about that information, to better reflect the fragmentation and diversity of political contexts found in the new era of five/six-party British politics.
Surveys have long been critical tools for understanding elections and forecasting their results. As the number of election
surveys has increased in prevalence, researchers, journalists, and standalone political bloggers have sought to learn from
the wealth of information released. This paper explores three central strategies for pooling surveys and other information
to better understand both the state of an election and what can be expected when voters head to the polls. Aggregation, predictive
modeling, and hybrid models are assessed as ways of improving on the results of individual surveys. For each method, central
questions, key choices, applications, and considerations for use are discussed. Trade-offs in choices between pooling strategies
are considered, and the accuracies of each set of strategies for forecasting results in 2012 are compared. Although hybrid
models have the potential to most accurately pool election information and make predictions, the simplicity of aggregations
and the theory-testing capacity of predictive models can sometimes prove more valuable.
Many types of time series data in political science, including polling data and events data, exhibit important features—such
as irregular spacing, noninstantaneous observation, overlapping observation times, and sampling or other measurement error—that
are ignored in most statistical analyses because of model limitations. Ignoring these properties can lead not only to biased
coefficients but also to incorrect inference about the direction of causality. This article develops a continuous-time model
to overcome these limitations. This new model treats observations as noisy samples collected over an interval of time and
can be viewed as a generalization of the vector autoregressive model. Monte Carlo simulations and two empirical examples demonstrate
the importance of modeling these features of the data.
After an election campaign, it is important to identify events that marked change points in voter support. Pre-election polls provide a measure of the state of voter support at points in time during the election campaign. However, polling data is difficult to analyze because it is sparse and comes from multiple sources, which can be individually biased. In this paper, we propose a change point model for polling data that increases confidence by combining polls and identifying change points simultaneously. We demonstrate the utility of our model on polling data from the 2008 U.S. presidential election.
Although “horserace journalism” is thought to be central to contemporary election news coverage and has generated a great deal of criticism, there is no general model of the nature and dynamics of horserace journalism or “poll reporting.” This paper proposes and empirically evaluates such a model. The model builds on and extends John Zaller's “theory of media politics” to consider specifically what citizens demand from polls and what journalists supply. Aside from the generic motivations of politicians, citizens and journalists, the model emphasizes the unique features of polls as objects of news coverage. The paper finds considerable support for the model in an analysis of newspaper coverage of horserace polls (that is, vote intention polls) in the Canadian general election of 2006. Our findings from this one case have potentially broad implications for our understanding of the relationship between polls and electoral democracy both empirically and normatively.
Résumé. Même si le journalisme de course (“horserace journalism”) est vu comme étant une composante centrale de la couverture électorale et qu'il a généré sa part de critiques, il n'existe pas de modèle général de la nature et de la dynamique de ce type de journalisme. Cet article propose, et évalue empiriquement, un tel modèle. Prenant comme point de départ la « Theory of Media Politics » de John Zaller, ce modèle considère plus spécifiquement ce que les citoyens demandent des sondages et ce que les journalistes leurs procurent. Au-delà des motivations génériques des politiciens, citoyens et journalistes, le modèle met l'accent sur les caractéristiques uniques des sondages en tant qu'objet de couverture journalistique. L'article présente des résultats supportant considérablement le modèle à travers une analyse de la couverture des sondages par les journaux (c'est-à-dire des sondages sur les intentions de vote) durant l'élection générale canadienne de 2006. Nos résultats émanant de ce cas ont potentiellement des implications beaucoup plus grandes pour notre compréhension de la relation entre les sondages et la démocratie électorale, à la fois sur le plan empirique et sur le plan normatif.
I present a dynamic Bayesian forecasting model that enables early and accurate prediction of U.S. presidential election outcomes at the state level. The method systematically combines information from historical forecasting models in real time with results from the large number of state-level opinion surveys that are released publicly during the campaign. The result is a set of forecasts that are initially as good as the historical model, and then gradually increase in accuracy as Election Day nears. I employ a hierarchical specification to overcome the limitation that not every state is polled on every day, allowing the model to borrow strength both across states and, through the use of random-walk priors, across time. The model also filters away day-to-day variation in the polls due to sampling error and national campaign effects, which enables daily tracking of voter preferences toward the presidential candidates at the state and national levels. Simulation techniques are used to estimate the candidates’ probability of winning each state and, consequently, a majority of votes in the Electoral College. I apply the model to preelection polls from the 2008 presidential campaign and demonstrate that the victory of Barack Obama was never realistically in doubt.
The underlying motivation of this collection is the concern that the failure to understand the relationship between our measure of a concept and the concept itself has real consequences. It may cause us to overlook model specification problems, to unnecessarily and inappropriately change our theories, and to miss out on information that could be usefully applied to the specification and estimation of our model. In order to take modelling seriously, we first must take measurement seriously. This is most effectively done when multiple measures of the underlying concept are utilised, allowing for the application of more appropriate models with more precise estimates. The contributions to this symposium demonstrate a wide range of such modelling approaches, each of which is tailored to match the theoretical expectations for the statistical characteristics of the data being used. By way of introduction, I highlight the measurement problems that these approaches are designed to tackle.
After an election campaign, it is important to identify events that marked change points in voter support. Pre-election polls provide a measure of the state of voter support at points in time during the election campaign. However, polling data is difficult to analyze because it is sparse and comes from multiple sources, which can be individually biased. We propose a change point model for polling data that increases confidence by combining polls and identifying change points simultaneously. We demonstrate the utility of our model on polling data from the 2008 U.S. presidential election.
Radical Right Parties (RRPs) have traditionally been seen as ‘hard cases’ to forecast, with unstable voter bases affected by short-term influences. Building upon our previous work on forecasting the French Front National’s vote across time, we construct a comparable model for three other European countries–Austria, Denmark and Norway–with significant RRPs, using economic, cultural and political predictors. We find that the model performs surprisingly well, with the partial exception of Norway, and provides an accurate forecast of RRP electoral performance which improves upon naive endogenous models and, significantly, upon polling estimates. Moreover, the model is firmly rooted in existing explanations of RRP success, allowing a robust explanation not only of variation in these parties’ votes, but also of less successful estimates in a small number of country-specific contexts. Overall, we find that standard approaches to electoral forecasting in fact offer a useful tool in the analysis of RRPs.
Before every presidential election, journalists, pollsters, and politicians commission dozens of public opinion polls. Although the primary function of these surveys is to forecast the election winners, they also generate a wealth of political data valuable even after the election.'
Only in recent years has the "likely voter" technology been extended to polls well in advance of an election. In the case of the 2000 U.S. presidential election, CNN/USA Today/Gallup tracking polls indi- cated considerable fluctuations in likely voter preferences, greater than among the larger pool of registered voters surveyed. This article explores how Gallup's likely voter model exaggerates the reported volatility of voter preferences during the campaign. Much of the reported variation in candidate preference reported by Gallup in that election is not due to actual voter shifts in preference but ra ther to changes in the composition of Gallup's likely voter pool. The findings highlight dangers of relying on samples of likely voters when polling well before Election Day.
The study of voters and elections has shed considerable light on people's vote choices and election outcomes. Yet little is known about the evolution of electoral sentiment over the campaign cycle. This article takes a small step toward addressing this issue by examining polls for a single election in a single year-the U.S, presidential race in 1996. The volume of poll data for 1996 allows us to observe the dynamics of voter preferences in far greater depth than is possible in previous years. Our analysis indicates that most of the variation in the polls during the 1996 presidential campaign represents survey error. What remains is mostly concentrated in the run-up to the fall campaign, not the fall campaign itself. During the fall, when political activity and media attention were at their peaks, aggregate presidential preferences remained largely unchanged. To the extent that campaign events influenced the underlying division of preferences, the effects were small and short-lived. Thus, our findings are consistent with the interpretation that the electoral verdict is already in place before the general election campaign begins.
The 2004 presidential election campaign provided a venue for a wide variety of polling, and it was not without its controversies. In the end, the final estimates of the preelection polls, the bread and butter of the polling industry, were very good at suggesting it would be a close race, with Bush the likely winner. In historical perspective, the overall performance was above average for the period since 1956. Issues raised in the media leading up to the end of the campaign and the final estimates, however, created some controversy, especially about the likely voter methodology used by different organizations. There were also some anomalies at the end of the campaign as some firms and collaborators ended up producing different estimates of the outcome depending on likely voter definitions or the mode of data collection.
Direct measurements of public opinion about national affairs appear with increasing frequency in all of the mass media. While such survey results are often with statements as to expected error margins, discrepancies between multiple surveys in the news at the same time on what seem to be the same topics may convince casual consumers that such error margins must be considerably understated. A brief review of the several sources of variability and fixed bias in such surveys provides a clearer frame of reference for the evaluation of such data.
Campaigns suddenly seem to matter, as do questions about the electoral process in the aftermath of the 2000 presidential election. The authors examine the U.S. electoral process as an integrated event spanning a full year, drawing upon the Annenberg 2000 Election Study. The scale of their fieldwork is such that they have been able to isolate key turning points and that dynamics can be studied within certain segments. Johnston, Hagen and Jamieson have also utilized candidate appearances, news coverage, and campaign advertising to provide this integrated account of a U.S. campaign. Richard Johnston is Professor and Head of Political Science at the University of British Columbia and an Associate Member of Nuffield College, Oxford. He is co-author of Letting the People Decide (Stanford University Press, 1992) and The Challenge of Direct Democracy (McGill-Queen's University Press, 1996). Michael G. Hagen is Associate Research Professor and Director of the Center for Public Interest Polling at the Eagleton Institute of Politics at Rutgers University. He is co-author of Race and Inequality: A Study in American Values (Chatham House, 1986) and a contributor to Reasoning and Choice: Explorations in Political Psychology (Cambridge, 2003). Kathleen Hall Jamieson is Ware Professor of Communication and Director of the Annenberg Public Policy Center at the University of Pennsylvania. She is author or co-author of twelve books on politics and media including Packaging the Presidency (Oxford University Press, 1988). © Richard Johnston, Michael G. Hagen, and Kathleen Hall Jamieson 2004.
The record of British election polls was good--until the general election of 1992. The Market Research Society's inquiry into the performance of the polls in 1992 found inadequacies in the implementation of the sampling system used, evidence of a late swing and some reluctance of Conservative supporters to reveal their loyalty; but it generally endorsed the principle of well-conducted quota polling and found that variations in methodological detail had nil effect on the results. The evidence is presented and some possible future developments to counter the `spiral of silence' are discussed.
The authors examine the results of 48 national public opinion polls measuring support for George Bush and Michael Dukakis throughout the 1988 presidential campaign conducted or reported by five major media polling organizations. Polling trends are discussed, and the consistency of estimates across polls are assessed, across seven distinct time periods defined by key events during the 1988 election year, while accuracy is assessed by comparing final pre-election polls against election results. Time series transfer function methods are employed to assess the short-term and long-term effects of the two major national party conventions, the two Bush-Dukakis debates, and the Ouayle-Bentsen debate on candidate support. Statistically significant positive effects on Bush support are obtained for the Republican convention and for the second presidential debate, while support for Dukakis was affected significantly, and in opposite directions, by the two party conventions. The results are discussed in the context of recent research conducted by Crespi and others.
This article proposes a new measure of the predictive accuracy (A) of election polls that permits examination of both accuracy and bias, and it applies the new measure to summarize the results of a number of preelection polls. We first briefly review past measures of accuracy, then introduce the new measure. After the new measure is described, the general strategy is to apply it to three presidential elec- tions (1948, 1996, and 2000) and to compare the results derived from it to the results obtained with the Mosteller measures. Then, the new mea- sure is applied to the results of 548 state polls from gubernatorial and senatorial races in the 2002 elections to illustrate its application to a large body of preelection polls conducted in "off-year" races with dif- ferent outcomes. We believe that this new measure will be useful as a summary measure of accuracy in election forecasts. It is easily com- puted and summarized, and it can be used as a dependent variable in multivariate statistical analyses of the nature and extent of biases that affect election forecasts and to identify their potential sources. It is com- parable across elections with different outcomes and among polls that vary in their treatment or numbers of undecided voters.
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
New tools for measuring changes in public opinion can be derived from the theory of the spiral of silence. Measures of individual assessment of the climate of opinion and of confidence about showing one's own opinion document the processes by which the losing side falls increasingly silent and the winning side is therefore overrated.
The Kalman filter is a popular tool in engineering and economics. It is becoming popular in political science, touted for its abilities to reduce measurement error and produce more precise estimates of true public opinion. Its application to survey measures of public opinion varies in important ways compared to the traditionally understood Kalman filter. It makes a priori assumptions about the variance of the sampling error that would not usually be made and does so in a way that violates an important property of the Kalman filter. Consequently, the behavior of the filter modified for public opinion measures is less well-known. Through simulations we assess whether and to what extent filtering: reliably detects the characteristics of time series; does so across series with different rates of autoregressive decay; and does so when the variance of the sampling error is unknown. We also examine whether the filtered data represents the level of true underlying variance and the extent to which filtering assists or hinders our ability to detect exogenous shocks. We learn a numbers of things. Most importantly, taking into account sampling error variance when filtering data can work well, though its performance does vary. First, filtering works best identifying time series characteristics when assuming a stationary process, even if the underlying process contains a unit root. Second, the performance of filtering drops off when we incorrectly specify the variance of the sampling error, and especially when we overestimate it. Third, when estimating exogenous shocks it is better to make no a priori assumptions regarding a measurement error variance unless we are absolutely certain we know what it is. In fact, applying the filter without specifying the measurement error variance is more often than not the best choice.
The 2004 and 2006 Canadian elections saw unprecedented numbers of trial heat polls, including the first daily tracking polls in the country's history. This paper attempts to assess the competing claims regarding the accuracy of these polls by setting them in proper context and providing a systematic accounting of what might reasonably be expected from published results. We build on and utilise a method for disentangling sampling error, “house” effects, and true change. Most critically we estimate a value for bias in the polling industry as a whole.Strikingly, systematic variation across firms within the industry is less notable than bias in the industry as a whole. There was less bias in 2004 than in 2006, but only marginally so. In each election, the industry underestimated the Liberal share and overestimated the NDP one. We speculate on sources of bias, and then ask if the discrepancy between polls and the electorate is methodological at the core or the product of last-minute strategic decision-making.
This article describes some solutions to common problems in pre-election surveys drawing upon Gallup Poll experience. It touches upon problems in sampling, estimation, response validity, the undecided, measuring likelihood to vote, and measuring late trends in voter preference. In conclusion it cites demonstrable gains in accuracy that have followed application of the solutions described.Paul Perry is Vice Chairman, The Gallup Organization, Inc.
This article undertakes an initial evaluation of the performance of the opinion polls in the 1997 Election. Although widely regarded as a successful election for the industry, in sharp contrast to 1992, there were important differences between the figures produced by the various companies. These differences provide us with important clues as to the relative effectiveness of the various methods that were used. They suggest that many of the more radical changes that were made to polling methods in response to the problems encountered in 1992 were successful and that the industry will need to continue to experiment and innovate before the next election.
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
There is little systematic research on the multitude of factors that influence the accuracy of poll results. This article examines six methodological factors directly under the survey researcher's control and two exogenous factors concerning the nature of public opinion as sources of survey error. Data for this study come from 56 ''trial heat'' polls conducted during the last month of the 1992 presidential election. The most important variables influencing survey accuracy were the number of days a poll is in the field, which increased total accuracy one-half of a percentage point per day; conducting interviews only on weekdays (and thus only during evening hours), which reduced overall accuracy rates by more than 1 percentage point; and conducting a ''tracking'' poll, which increased accuracy by about 1.5 points. Sample size was not related to accuracy rates. Results also indicated that sampling frames of ''likely voters'' (relative to ''registered voters'') tended to overestimate support for George Bush and underestimate support for Ross Perot, that interviewing only on weekdays led to overestimates of support for Bush, and that strict methods of defining a respondent as ''supporting'' a candidate hurt the two newcomers, Perot and Bill Clinton, more than Bush. In light of these data it is recommended that the common practice of reporting ''margins of error'' based solely on sample sizes be abandoned as misleading and replaced by a more empirically justifiable measure based more on response rates.
Examination of 155 poll forecasts in 68 national elections since 1949 shows that errors average nearly twice what statistical
theory would indicate. Polls predict the division of vote between major parties better than individual party percentages,
leading to 85 percent success in picking the winner. The worst failures occurred in a few elections where most polls went
wrong. Liberal party votes are correctly forecast, conservatives slightly underestimated. Improved polling methods have not
led to better forecasts.
Poll results vary over the course of a campaign election and across polling organisations, making it difficult to track genuine changes in voter support. I present a statistical model that tracks changes in voter support over time by pooling the polls, and corrects for variation across polling organisations due to biases known as ‘house effects’. The result is a less biased and more precise estimate of vote intentions than is possible from any one poll alone. I use five series of polls fielded over the 2004 Australian federal election campaign (ACNielsen, the ANU/ninemsn online poll, Galaxy, Newspoll, and Roy Morgan) to generate daily estimates of the Coalition's share of two-party preferred (2PP) and first preference vote intentions. Over the course of the campaign there is about a 4 percentage point swing to the Coalition in first preference vote share (and a smaller swing in 2PP terms), that begins prior to the formal announcement of the election, but is complete shortly after the leader debates. The ANU/ninemsn online poll and Morgan are found to have large and statistically significant biases, while, generally, the three phone polls have small and/or statistically insignificant biases, with ACNielsen and (in particular) Galaxy performing quite well in 2004.* An earlier version of this paper was prepared for the annual meeting of the Australasian Political Studies Association, University of Adelaide, 29 September–1 October 2004.
Little is known about the evolution of electoral sentiment over the campaign cycle. How does the outcome come into focus as the election cycle evolves? Do voters’ preferences evolve in a patterned and understandable way? What role does the election campaign play? In this article, we address these issues. We translate general arguments about the role of campaigns into a set of formal, statistical expectations. Then, we outline an empirical analysis and examine poll results for the 15 U.S. presidential elections between 1944 and 2000. Our analysis focuses specifically on two questions. First, to what extent does the observable variation in aggregate poll results represent real movement in electoral preferences (if the election were held the day of the poll) as opposed to mere survey error? Second, to the extent polls register true movement of preferences owing to the shocks of campaign events, do the effects last or do they decay? Answers to these questions tell us whether and the extent to which campaign events have effects on preferences and, if so, whether these effects persist until Election Day. The answers thus inform us about what we ultimately want to know: Do campaigns have any real impact on the election outcome? The results of our analysis suggest that they do.
Much research shows that voters behave in understandable ways on election day and that election outcomes themselves are quite predictable. Even to the extent we can predict what voters do at the very end of the campaign, we know relatively little about how electoral preferences evolve over time. The U. S. presidential race in 2000 offers a fairly unique opportunity. The volume of available poll data for this particular election allows us to examine the dynamics of voter preferences in great detail for much of the election cycle. Analysis of the poll results in 2000 reveals that underlying electoral preferences changed quite meaningfully during the course of the campaign. The analysis also provides evidence that a significant portion of these changes in preferences actually persisted over time to affect the outcome on election day. Based on these results, it appears that the 2000 presidential election campaign mattered quite a lot.
Assessing poll performance in the 2000 campaign A review: preelection survey methodology: Details from eight polling organizations
- M W Traugott
Traugott, M. W. (2005). Assessing poll performance in the 2000 campaign. Public Opinion Quarterly, 65, 389−419. Voss, D. S., Gelman, A., & King, G. (1995). A review: preelection survey methodology: Details from eight polling organizations, 1988 and 1992. Public Opinion Quarterly, 59, 98−132.
Election predictions: an empirical assessment Assessing the accuracy of polls and surveys Pre-election polling: Sources of accuracy and error The opinion polls: the election they got (almost) right So how well did they do? The polls in the 1997 election
- W Buchanan
- −227 Converse
- P E Traugott
- −1099 Crespi
Buchanan, W. (1986). Election predictions: an empirical assessment. Public Opinion Quarterly, 40, 220−227. Converse, P. E., & Traugott, M. W. (1986). Assessing the accuracy of polls and surveys. Science, 234, 1094−1099. Crespi, I. (1988). Pre-election polling: Sources of accuracy and error. New York: Russell Sage Foundation. Crewe, I. (2005). The opinion polls: the election they got (almost) right. Parliamentary Affairs, 58, 684−698. Curtice, J. (1997). So how well did they do? The polls in the 1997 election. Journal of the Market Research Society, 39, 449−462.