Article

Forecasting Accuracy: Comparing Prediction Markets and Surveys - An Experimental Study

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Prediction markets are viewed as the most accurate instrument for collective forecasts. However, empirical studies, mostly based on political elections, deliver mixed results. An experimental study was conducted to avoid certain biases and problems and to better control conditions of eliciting information from individuals. One typical problem is for example comparing prediction markets that focus on judging the public opinion in the future with polls asking for individual election preferences at a certain point of time. Therefore, our study compared forecast accuracy between prediction markets and a simple survey for the same forecasting item. The results showed roughly the same accuracy for all employed methods with the survey delivering slightly better results at lower costs, which was surprising. The experiments demonstrated also that it is possible to gain highly accurate forecasts with a relatively small number of participants (6-17) taking part continuously.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... A few studies have compared the performance of prediction markets and polls, though there is no conclusive answer [8,16,1]. Both Goel et al. [8] and Rieg and Schoder [16] find no significant differences between these two methods. ...
... A few studies have compared the performance of prediction markets and polls, though there is no conclusive answer [8,16,1]. Both Goel et al. [8] and Rieg and Schoder [16] find no significant differences between these two methods. Atanasov et al. [1] find that the aggregation rules in prediction polls affect its accuracy level. ...
Preprint
Information flow measures, over the duration of a game, the audience's belief of who will win, and thus can reflect the amount of surprise in a game. To quantify the relationship between information flow and audiences' perceived quality, we conduct a case study where subjects watch one of the world's biggest esports events, LOL S10. In addition to eliciting information flow, we also ask subjects to report their rating for each game. We find that the amount of surprise in the end of the game plays a dominant role in predicting the rating. This suggests the importance of incorporating when the surprise occurs, in addition to the amount of surprise, in perceived quality models. For content providers, it implies that everything else being equal, it is better for twists to be more likely to happen toward the end of a show rather than uniformly throughout.
... Prediction markets were more accurate, but not by a significant margin. Rieg and Schoder (2010) found no differences in accuracy when comparing markets and opinion polls in small-scale experiments, where each of 6 to 17 participants made a single prediction in either a market or a poll. In another small-group experiment, Graefe and Armstrong (2011) compared the accuracy of answers to general knowledge questions using the Delphi method, the Nominal Groups technique, and a prediction market with a logarithmic market scoring rule (LMSR) (Hanson 2003). ...
Article
We report the results of the first large-scale, long-term, experimental test between two crowdsourcing methods: prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a geopolitical prediction tournament. Forecasters were randomly assigned to either prediction markets (continuous double auction markets) in which they were ranked based on earnings, or prediction polls in which they submitted probability judgments, independently or in teams, and were ranked based on Brier scores. In both seasons of the tournament, prices from the prediction market were more accurate than the simple mean of forecasts from prediction polls. However, team prediction polls outperformed prediction markets when forecasts were statistically aggregated using temporal decay, differential weighting based on past performance, and recalibration. The biggest advantage of prediction polls was atthe beginning of long-duration questions. Results suggest that prediction polls with proper scoring feedback, collaboration features, and statistical aggregation are an attractive alternative to prediction markets for distilling the wisdom of crowds.
... Prediction markets were more accurate, but not by a significant margin. Rieg and Schoder (2010) found no differences in accuracy when comparing markets and opinion polls in small-scale experiments, where each of 6 to 17 participants made a single prediction in either a market or a poll. In another small-group experiment, Graefe and Armstrong (2011) compared the accuracy of answers to general knowledge questions using the Delphi method, the Nominal Groups technique, and a prediction market with a logarithmic market scoring rule (LMSR) (Hanson 2003). ...
Article
Full-text available
We report the results of the first large-scale, long-term, experimental test between two crowd sourcing methods – prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a geopolitical prediction tournament. Some forecasters traded in a continuous double auction market and were ranked based on earnings. Others submitted probability judgments, independently or in teams, and were ranked based on Brier scores. In both seasons of the tournament, last day prices from the prediction market were more accurate than the simple mean of forecasts from prediction polls. However, team prediction polls outperformed prediction markets when poll forecasts were aggregated with algorithms using temporal decay, performance weighting and recalibration. The biggest advantage of prediction polls occurred at the start of long-duration questions. Prediction polls with proper scoring, algorithmic aggregation and teaming offer an attractive alternative to prediction markets for distilling the wisdom of crowds.
... A detailed description of this experiment can be found in(Rieg & Schoder, 2010). ...
Article
Full-text available
Prediction markets are seen as a powerful instrument to retrieve collective forecasts. Often their forecast performance is measured against election polls. Several of such studies about elections found a higher forecasting accuracy than polls. However, other studies questioned that. In total it seems to date not clear if and when prediction markets are performing better than polls or surveys. We conducted several experiments which compared surveys and prediction markets in regard of their accuracy. We found both methods delivered accurate forecasts even with small numbers of participants/traders. However prediction markets did not per se outperform surveys.
Chapter
Full-text available
Aus dem Wettverhalten, so die Erfahrung bei Sportwetten und Präsidentschaftswahlen, lassen sich relative gute Vorhersagen für den Ausgang von Sportereignissen und Wahlen ableiten. In diesem Beitrag wird das Konzept und Erfahrungen mit einer Live-Wette beschrieben, bei der Expertinnen und Experten auf kurzfristige Entwicklungen im Bereich der Lern- und Lehrmaterialien wetten. Der Beitrag schließt mit Empfehlungen für Nachahmer/innen.
Article
Full-text available
In recent years, Prediction Markets gained growing interest as a forecasting tool among researchers as well as practitioners, which resulted in an increasing number of publications. In order to track the latest development of research, comprising the extent and focus of research, this article provides a comprehensive review and classification of the literature related to the topic of Prediction Markets. Overall, 316 relevant articles, published in the timeframe from 2007 through 2013, were identified and assigned to a herein presented classification scheme, differentiating between descriptive works, articles of theoretical nature, application-oriented studies and articles dealing with the topic of law and policy. The analysis of the research results reveals that more than half of the literature pool deals with the application and actual function tests of Prediction Markets. The results are further compared to two previous works published by Zhao, Wagner and Chen (2008) and Tziralis and Tatsiopoulos (2007a). The article concludes with an extended bibliography section and may therefore serve as a guidance and basis for further research.
Article
Prediction markets are becoming an ever more popular method for aggregating disparate information. Companies use prediction markets to forecast sales, release dates, and other internal activities. Recently, several well-known economists have argued that government ought to eliminate impediments to unleashing the power of prediction markets broadly. This survey paper looks at the laboratory experiments which provide part of the scientific justification for using prediction markets. These systems work because people who have private knowledge can use this “insider information” in the market, but doing so causes the price to reflect these data. Starting with early experiments that show markets can, under certain circumstances, aggregate information, we discuss the robustness of prediction markets to various design features. We also discuss places where performance does not match rhetoric and briefly look at experimental comparisons of alternative information aggregating institutions. Another issue consistently raised regarding the implementation of prediction markets is the degree to which traders can manipulate prices and thus the information gleaned. By following a series of experiments on manipulation, we present the circumstances that are and are not sufficient for manipulation attempts to be successful. Finally, we offer suggestions on where we believe future experiments are needed in this area.
Article
Full-text available
The accuracy of prediction markets has been documented both for markets based on real money and those based on play money. To test how much extra accuracy can be obtained by using real money versus play money, we set up a real-world online experiment pitting the predictions of TradeSports.com (real money) against those of NewsFutures.com (play money) regarding American Football outcomes during the 2003-2004 NFL season. As expected, both types of markets exhibited significant predictive powers, and remarkable performance compared to individual humans. But, perhaps surprisingly, the play-money markets performed as well as the real-money markets. We speculate that this result reflects two opposing forces: real-money markets may better motivate information discovery while play-money markets may yield more efficient information aggregation.
Article
Full-text available
Election markets have been praised for their ability to forecast election outcomes, and to forecast better than trial-heat polls. This paper challenges that optimistic assessment of election markets, based on an analysis of Iowa Electronic Market (IEM) data from presidential elections between 1988 and 2004. We argue that it is inappropriate to naively compare market forecasts of an election outcome with exact poll results on the day prices are recorded, that is, market prices reflect forecasts of what will happen on Election Day whereas trial-heat polls register preferences on the day of the poll. We then show that when poll leads are properly discounted, poll-based forecasts outperform vote-share market prices. Moreover, we show that win projections based on the polls dominate prices from winner-take-all markets. Traders in these markets generally see more uncertainty ahead in the campaign than the polling numbers warrant--in effect, they overestimate the role of election campaigns. Reasons for the performance of the IEM election markets are considered in concluding sections.
Article
Full-text available
In 1998, University of Iowa faculty members created their own futures markets. These experimental markets, designed to provide insights into the behavior of traders and naturally occurring markets, are still going strong. Their clever design gives them another practical use: They can be used to predict future events such as election outcomes and Federal Open Market Committee voting.
Article
What is the best way to predict Australian federal election results? This article analyses three forecasting tools: opinion polls, economic models, and betting odds. Historically, we � nd that opinion polls taken close to the election are quite accurate, while economic models provide better medium-run forecasts. The November 2001 federal election largely follows this pattern, although the economic models provided more accurate projections than recorded through the 1990s. Against these, we compare betting odds, analysing a rich data source from one of Australia’s largest bookmakers, Centrebet. The betting market not only correctly forecast the election outcome, but also provided very precise estimates of outcomes in a host of individual electorates. Betting � uctuations present an intriguing quantitative record of the shifting fortunes of the campaign. Particularly in marginal seats, the press may have better served its readers by reporting betting odds than by conducting polls. We conclude that the results of these three models can help determine how important the events of August and September 2001 were in deciding the outcome of the election.
Article
The dominant methodology for short-term forecasting of electoral outcomes uses trial-heat polls, where respondents report their current electoral preferences (not their election-day predictions). Election markets, where self-selected participants trade shares of candidates at prices predictive of election-day results, provide an alternative method that often produces more accurate forecasts. Consequently, increasing attention is being paid to this methodology. However, it is poorly understood and lacks theoretical justification. Surprisingly, the rationale for forecasting using trial-heat polls has not been completely developed either. We develop the justification for using both election markets and public opinion polls to forecast electoral outcomes, giving conditions under which each method performs ideally. For the ideal case, we prove (under the reasonable assumption that market participants are aware of the poll results) that the mean square prediction error for the market forecast is smaller than that of any forecast based on one or more polls. The case in which the assumptions supporting each method fail is also considered. It is often reasonable to expect that the best case results hold approximately, in which case the market forecast should also beat any poll-based forecast. We also compare the bias and variance of market and poll-based forecasts; our results suggest the utility of using the series of market prices to study the course of campaigns.
Article
This is the fourth of a series of Tutorials in Clinical Research (1-3). The objectives of this article are to heighten reader awareness of biases and of methods to reduce their impact and to provide an easy reference document for the reader during future journal reading. Tutorial. The authors met weekly for 4 months discussing clinical research articles and biases for which they might be at risk. Liberal use of reference texts and specific articles on bias were reviewed. Like the example by Sackett, biases were catalogued to create an easily understood reference. Articles were chosen to demonstrate how understanding bias might facilitate assessment of the validity of medical publications. The article is organized into three main sections. The first section introduces specific biases. Two tables serve as rapid reference tools. The second section describes the most common biases linked to specific research approaches and reviews techniques to minimize them. The last section demonstrates the application of the information to an article in a manner that can be applied to any article. Assessing the validity of a medical publication requires an awareness of bias for which the research is inherently at risk. A review of the publication to determine what steps the authors did or did not undertake to minimize the impact of biases on their results and conclusions helps establish the validity. This article should be of assistance in this critical review task.
Article
This paper reports on a comprehensive study of the distributions of summary measures of error for a large collection of quarterly multiperiod predictions of six variables representing inflation, real qrowth, unemployment,and percentage changes in nominal GNP and two of its more volatile components.The data come from surveys conducted since 1968 by the National Bureau of Economic Research and the American Statistical Association and cover more than 70 individuals professionally engaged in forecasting the course of the U. S.economy (mostly economists, analysts, and executives from the world of corporate business and finance). There is considerable differentiation among these forecasts, across the individuals, variables, and predictive horizons covered. Combining corresponding predictions from different sources can result insignificant gains; thus the group mean forecasts are on the average over timemore accurate than most of the corresponding sets of individual forecasts. But there is also a moderate deqree of consistency in the relative performance of a sufficient number of the survey members, as evidenced in positive rank correlations among ratios of the individual to group root mean square errors.
Article
Introduction and description of election futures markets The Iowa Electronic Markets are small-scale, real-money futures markets conducted by the University of Iowa College of Business. In this review we focus on the best known of these markets, The Iowa Political Markets. Contracts in these markets are designed so that prices should predict election outcomes. The data set contains the results of 49 markets covering 41 elections in 13 countries. The Iowa Markets operate 24-hours a day, using a continuous, double-auction trading mechanism. Traders invest their own funds, make their own trades, and conduct their own information search. The markets occupy a niche between the stylized, tightly controlled markets conducted in the laboratory and the information-rich environments of naturally occurring markets. By virtue of this design, the Iowa Markets provide data to researchers that is not otherwise available. Investments are typically limited to a $500 maximum per trader and general