Election polling still works

Study finds polling is still the best predictor of election results.

President Donald Trump's victory in the 2016 election, a surprise to many, called the effectiveness of polling into question. But a new study in Science finds that polling is still the best predictor of election outcomes. Ryan Kennedy, a political scientist at the University of Houston, and his colleagues looked at 4,331 polls from 146 election rounds across the world and found the polls could predict election results with about 90 percent accuracy overall. We spoke to Kennedy about the work.

ResearchGate: Can you give us a brief insight into what motivated you to look into polling?

Ryan Kennedy: We were asked as part of a larger project to look into predicting elections in Latin America, through polling and other means. What we found was that there were only a small number of structural factors that produced consistent success across countries, including how democratic the country was and whether the current office holder was running for another term. We also found that polling data was not as rare as we might have initially anticipated and that it produced substantial leverage for prediction across states. We broadened our analysis to include all global direct executive elections, so we would have more cases with which to study the utility of structural factors and of polling data as predictors, and, as they say, the rest is history.

RG: What did you find?

Kennedy: We found that the use of polling for elections has spread substantially around the world and that many of these polls were quite accurate in predicting the outcome. Moreover, with some corrections for polling-house bias and some structural characteristics of the country, we could produce quite accurate predictions for the election outcomes, even where there was limited polling data. This is quite interesting, since experts in the field had been debating about how well the techniques used to predict US elections could travel to other places around the globe. In addition, we suspect that there is a lot of polling data that we could not access. Foreign offices of governments around the world, as well as a number of NGOs, regularly conduct political polls that they either do not release publicly or only advertise them and then take the results down. Our attempts to gain access to these through FOIA requests and other techniques produced mixed results.

RG: How did you conduct your study?

Kennedy: We produced two datasets. The first built on a project that has been ongoing for several years at Yale. This dataset covered all elections in which the incumbent party could lose the election from 1945 to 2012. We used a machine learning technique to produce in-sample and out-of-sample predictions from this data. This model proved to be about 80 percent accurate in predicting election outcomes.

For the second dataset, we tried to collect as much polling data as possible from around the world. In the end, we found 4,331 different polls from 146 election rounds. We estimated the "house bias" of the polling firms and used a technique with information from all polls to produce a "smoothed" estimate of the eventual victory (or loss) margin of the incumbent party's candidate. Doing this, we had about 90 percent predictive accuracy.

RG: Why do you have to prove that polling is effective? Does it need to be improved?

Kennedy: Initially, our question dealt with the effectiveness of various indicators in predicting elections, especially outside of the relatively data-rich US context. After this last US presidential election, we found that polling itself has come under fire. We put several quotes in the conclusions to illustrate how the idea of quantitative election prediction is now being questioned in a way not seen since Truman defeated Dewey in 1948. These results suggest that polling is generally effective, both inside and outside the US. It is certainly not perfect, and there are going to be misses, but it seems to be our best current technology for this purpose.

As far as whether it needs to be improved, absolutely. Polling today faces a number of obstacles, even in the US: low response rates, models for likely voters, how to assign weights to particular respondents, etc. I don't think there is a perfect system—even in our polling model, we miss one out of ten times. But we can't respond to these challenges, as many pundits have, by dismissing the entire enterprise. That is a mistake.

RG: Does the US present pollsters with any unique challenges compared to other countries? If so, why?

Kennedy: This is probably a better question for someone who works at a polling firm. The US case has experienced major drops in participation in opinion polls, which has made analysts rely more and more on weighting their samples to make them look like the US population. This can cause issues. There is also the issue of cell phones—as more and more people start solely using cell phones, it can make a particular portion of the population more difficult to reach. These are just a couple of challenges, there are others, but these are at least somewhat particular to US laws and experience.

RG: Did Trump’s victory affect the public’s opinion of polling?

Kennedy: Yes, absolutely. There were plenty of reactions saying "Tonight data died"; "We should all feel bamboozled"; and "The pollsters...won't be believed on anything soon." President Trump has even picked up on this himself, calling polls showing high disapproval ratings fake and using the election predictions to substantiate his contention. It has certainly tarnished the reputation of Nate Silver and others, who achieved minor celebrity status for predicting the past election.

RG: Can the public’s faith in polling be restored?

Kennedy: Luckily, the public generally has a pretty short memory. Polling data and election forecasts have generally proved accurate, certainly more accurate than pundits who utilize their gut feelings to predict elections. There will certainly be many changes in how polling is conducted and many experiments with new techniques. But I would guess that quantitative election prediction will still prove to be more accurate than gut feelings, and the public will eventually recognize this.

Featured image courtesy of wikimedia.