Figure - available from: PLOS One
This content is subject to copyright.
Source publication
Problem
Do conservative econometric models that comply with the Golden Rule of Forecasting provide more accurate forecasts?
Methods
To test the effects of forecast accuracy, we applied three evidence-based guidelines to 19 published regression models used for forecasting 154 elections in Australia, Canada, Italy, Japan, Netherlands, Portugal, Spai...
Similar publications
In the present article, the main principles of the circular economy are outlined in contradistinction with the existing and traditional linear economic model. An econometric model describing the influence of the linear economy on the environment is presented. The environment is characterized by seven key processes: change in global temperature; emi...
To achieve sustainable development, companies are aware that they need to move from the ‘take-make-consume-throw away’ pattern toward an economic model based on sharing, re-use, repair, refurbishing, and recycling. Implementing the circular economy concept into a company’s business model is not an easy process, given that some eco-innovations have...
Citations
... • Findings that conservative alternatives to 19 MRA models provide forecasts that are more accurate (Graefe, Green, & Armstrong 2019 (82) MRA-eq (55) MRA-eq (82) MLAD (55) MLAD (64) MLAD-eq'zd (55) MLAD-eq'zd (64) MLAD (91) MLAD (88) MLAD-eq'zd (91) MLAD-eq'zd (52) MLAD (52) MLAD-eq'zd (52) 1 UMBRAE of out-of-sample forecasts < 1.0 ...
Findings from tests of the predictive validity of causal models estimated using ordinary-least-squares (OLS) multiple regression were presented. Out-of-sample forecasts from 99 published models were compared with those from models estimated using nine simple and conservative alternative methods and one naive model. Forecast errors from models estimated using one of the alternative methods (multiple least absolute deviation regression, or median, regression) were smaller than those from the published OLS models on average and for most individual models. The findings have implications for model building practice and the quantitative estimation of causal relationships from empirical non-experimental data, and for AI (machine learning), which is based on OLS regression analysis.
... Their ordinary least squares (OLS) multiple regression (MRA) model estimated that the homicide victimisation rate of those countries increased by 3.6 per cent for every one percentage point increase in the immigrant population after allowing for year effects and for country specific influences by using dummy variables . 2 This Research Note asks, does Lott and Varney's MRA model have predictive validity? The answer to that question will contribute to my ongoing research on the predictive validity of multiple regression analysis compared to alternative, simpler and more conservative, methods for estimating forecasting models Green, Graefe, and Armstrong, 2018;Armstrong, Green, and Graefe, 2015;and Graefe, Green, and Armstrong, 2019). ...
Lott and Varney (2022) estimated a model of the effect of immigrant population numbers, as a proportion of the total population, on the homicide victim rate for 31 European countries for which data were available for some or all years between 2010 to 2020 amounting to 315 observations in total. 1 Their ordinary least squares (OLS) multiple regression (MRA) model estimated that the homicide victimisation rate of those countries increased by 3.6 per cent for every one percentage point increase in the immigrant population after allowing for year effects and for country specific influences by using dummy variables. 2 This Research Note asks, does Lott and Varney's MRA model have predictive validity? The answer to that question will contribute to my ongoing research on the predictive validity of multiple regression analysis compared to alternative, simpler and more conservative, methods for estimating forecasting models (Green and Armstrong, 2015; Green, Graefe, and Armstrong, 2018; Armstrong, Green, and Graefe, 2015; and Graefe, Green, and Armstrong, 2019). Method I compared the out-of-sample forecast errors-the 315 errors from forecasting each observation using a model estimated from the other n-1 (314) observations-from the Lott and Varney specified MRA model with the out-of-sample forecast errors from five alternative models. Three of the five were estimated using one of two alternative estimation methods, and one of those alternatives differed from the Lott and Varney model only in the estimation method.
... Many econometric and machine learning methods, which can deal with the curse of dimensionality, totally ignore the geography present in electoral data, such as spatial boundaries, clustering effects, and distance measures [3,1]. Consequently, they treat data separated into regions as independent and identically distributed. ...
... The vast literature on voting behavior varies from standard econometric techniques [3] to regression analysis [13] and machine learning models [1,7]. Econo-metrics and regression analysis studies usually focus on national-level estimators using surveys and economic features to understand election results. ...
... Econo-metrics and regression analysis studies usually focus on national-level estimators using surveys and economic features to understand election results. Although these methods are well established [3], applying them to thousands of features in several locations is challenging. Conversely, machine learning models can deal with the curse of dimensionality more naturally. ...
Voting behavior analysis involves understanding factors influencing an election to identify possible trends, new features, and extrapolations. A growing body of research has joined efforts to automate this process from high-dimensional spatial data. Although some studies have investigated machine learning methods, the capability of this artificial intelligence subarea has not been fully explored due to the challenges posed by the spatial autocorrelation structure prevalent in the data. This paper advances the current literature by proposing a geographic context-based stacking learning approach for predicting election outcomes from census data. Our proposal models data in spatial contexts of different dimensions and operates on them at two levels. First, it captures local patterns extracted from spatial contexts. Then, at the meta-level, it globally captures information from the K contexts nearest to a region we want to predict. We introduce a spatial cross-validation-driven experimental setup to assess and compare the stacking approach with state-of-the-art methods fairly. This validation mechanism aims to diminish spatial dependence's influence and avoid overoptimistic results. We estimated a considerable multi-criteria performance of our proposal concerning baseline and reference models taking data from the second round of the 2018 Brazilian presidential elections into account. The stacking approach presented the best overall performance, being able to generalize better than the compared ones. It also provided intelligible and coherent predictions in challenging regions, emphasizing its interpretability. These results evidence the potential use of our proposal to support social research.
... Analytically, the later is advantageous over the former in that it is able to give information about the future values of observed time series, depict forecaster in real time, and strongly discourage model overfitting ( [1], [2], [3], [4], [5]). Apart from predicting for future observations, forecasting is also a proven and valid tool for selecting a model with best predictive values ( [6], [7], [8], [9], [10]). However, in a quest for best predictive performance, model instability has become a subject of discourse and tends to pose a serious threat to time series forecasting accuracy. ...
The study is aimed at investigating the robustness of forecast performance of a hybridized (ARIMA-GARCH-type) model over each single component using different periods of horizon to display consistency over time. Daily closing share prices were explored from the Nigerian Stock Exchange for First City Monument Bank and Wema Bank Plc, spanning from January 3, 2006 to December 30, 2016, with a total of 2,713 observations. ARIMA model, GARCH-type, and hybridized ARIMA-GARCH-type were considered. The hybridized ARIMA-GARCH-type was found to produce the best forecast performance in terms of robustness over each single component model and the robustness was found to be consistent over different time horizons for the datasets. The implication is that, it provides an essential remedy to the problem associated with model instability when forecasting a discrete-time stochastic series.
... It combines data from published pre-election public opinion polls with information from fundamentalsbased forecasting models. Graefe [2] test the effects of forecasting accuracy, we applied three evidencebased guidelines to 19 published regression models used for forecasting 154 elections in Australia, Canada, Italy, Japan, Netherlands, Portugal, Spain, Turkey, U.K., and the U.S. Huber [3], forecasts the results of the German State Elections based on polls data from different institutes. To predict the participation of a single vote in multiparty elections, the range of methods varies from basic methods such as averaging over methods based on nonparametric regression to dynamic linear models. ...
In the field of data analysis, it is common not to distinguish clearly between prediction and forecast. Although the results of both processes may tend to converge, the mechanisms used in each case tend to be completely different. Prediction has to do with statistical extrapolation and estimation and forecasting can consider expert judgments on the subject. A methodology is proposed to carry out this latter task, with a mechanism that uses both historical and current data with the judgement of an expert. The methodology is applied to the case study of the Spanish general elections of April 2019.
... Evidence to date suggests that knowledge models are likely to produce forecasts that are more accurate than those from data models in situations where many causal variables are important. One knowledge model found error reductions of 10% to 43% compared to established regression models for forecasting elections in the U.S. and Australia (Graefe, Green, and Armstrong, 2019). ...
Purpose: Commentary on M4-Competition and findings to assess the contribution of data models--such as from machine learning methods--to improving forecast accuracy.
Methods: (1) Use prior knowledge on the relative accuracy of forecasts from validated forecasting methods to assess the M4 findings. (2) Use prior knowledge on forecasting principles and the scientific method to assess whether data models can be expected to improve accuracy relative to forecasts from previously validated methods under any conditions.
Findings: Prior knowledge from experimental research is supported by the M4 findings that simple validated methods provided forecasts that are: (1) typically more accurate than those from complex and costly methods; (2) considerably more accurate than those from data models.
Limitations: Conclusions were limited by incomplete hypotheses from prior knowledge such as would have permitted experimental tests of which methods, and which individual models, would be most accurate under which conditions.
Implications: Data models should not be used for forecasting under any conditions. Forecasters interested in situations where much relevant data are available should use knowledge models.
... Evidence to date suggests that knowledge models are likely to produce forecasts that are more accurate than those from data models in situations where many causal variables are important. One knowledge model found error reductions of 10% to 43% compared to established regression models for forecasting elections in the U.S. and Australia (Graefe, Green, and Armstrong, 2019). ...
Purpose: Commentary on M4-Competition and findings to assess the contribution of data models—such as from machine learning methods—to improving forecast accuracy. Methods: (1) Use prior knowledge on the relative accuracy of forecasts from validated forecasting methods to assess the M4 findings. (2) Use prior knowledge on forecasting principles and the scientific method to assess whether data models can be expected to improve accuracy relative to forecasts from previously validated methods under any conditions. Findings: Prior knowledge from experimental research is supported by the M4 findings that simple validated methods provided forecasts that are: (1) typically more accurate than those from complex and costly methods; (2) considerably more accurate than those from data models. Limitations: Conclusions were limited by incomplete hypotheses from prior knowledge such as would have permitted experimental tests of which methods, and which individual models, would be most accurate under which conditions. Implications: Data models should not be used for forecasting under any conditions. Forecasters interested in situations where much relevant data are available should use knowledge models.
... Evidence to date suggests that knowledge models are likely to produce forecasts that are more accurate than those from data models in situations where many causal variables are important. One study found error reductions of 10% to 43% compared to established regression models for forecasting elections in the US and Australia (Graefe, Green, and Armstrong, 2019). ...
In the mid-1900s, there were two streams of thought about forecasting methods. One stream-led by econometricians-was concerned with developing causal models by using prior knowledge and evidence from experiments. The other was led by statisticians, who were concerned with identifying idealized "data generating processes" and with developing models from statistical relationships in data, both in the expectation that the resulting models would provide accurate forecasts. At that time, regression analysis was a costly process. In more recent times, regression analysis and related techniques have become simple and inexpensive to use. That development led to automated procedures such as stepwise regression, which selects "predictor variables" on the basis of statistical significance. An early response to the development was titled, "Alchemy in the behavioral sciences" (Einhorn, 1972). We refer to the product of data-driven approaches to forecasting as "data models." The M4-Competition (Makridakis, Spiliotis, Assimakopoulos, 2018) has provided extensive tests of whether data models-which they refer to as "ML methods"-can provide accurate extrapolation forecasts of time series. The Competition findings revealed that data models failed to beat naïve models, and established simple methods, with sufficient reliability to be of any practical interest to forecasters. In particular, the authors concluded from their analysis, "The six pure ML methods that were submitted in the M4 all performed poorly, with none of them being more accurate than Comb and only one being more accurate than Naïve2" (p. 803.) Over the past half-century, much has been learned about how to improve forecasting by conducting experiments to compare the performance of reasonable alternative methods. On the other hand, despite billions of dollars of expenditure, the various data modeling methods have not contributed to improving forecast accuracy. Nor can they do so, as we explain below.
This study focuses on the Federal Reserve and private forecasts of growth in real residential investment. The aim is to improve predictive accuracy by first evaluating these forecasts. The results for 1984–2015 reveal that the Federal Reserve and private forecasts are generally free of systematic bias, superior to the naïve benchmark, and predict directional change with high accuracy rates. However, these forecasts do not contain detailed information in consumer home-buying attitudes and expectations. Using a subset of such information and real-time data on residential investment, a knowledge model (KM) is constructed to produce comparable forecasts. The test results indicate that the KM forecasts of growth in residential investment contain distinct and useful predictive information, and the combined Federal Reserve, private, and KM forecasts show reductions in forecast errors that are more significant at longer horizons. As such, we conclude that consumer survey responses help improve forecast accuracy. Given that accurate forecasts contribute to the success of policy, more transparency in Federal Reserve Open Market Committee (FOMC) decisions is encouraged. With more transparency and clear communication, consumers are able to provide more informative responses, which can then be employed to produce more accurate forecasts of growth in residential investment.
