Research Program on Forecasting (RPF) Working Papers represent preliminary work circulated for
comment and discussion. Please contact the author(s) before citing this paper in any publications. The
views expressed in RPF Working Papers are solely those of the author(s) and do not necessarily represent
the views of RPF or George Washington University.
What Do We Know About G-7 Macro Forecasts?
Herman O. Stekler
Department of Economics
The George Washington University
Washington, DC 20052
RPF Working Paper No. 2008-009
August 18, 2008
RESEARCH PROGRAM ON FORECASTING
Center of Economic Research
Department of Economics
The George Washington University
Washington, DC 20052
Was wissen wir über die makroökonomischen
Vorhersagen für die Länder der G 7?
What Do We Know About G-7 Macro Forecasts?
Von Herman O. Stekler1
Fildes & Stekler’s (2002) survey of the state of knowledge about the quality of economic
forecasts focused primarily on US and UK data. This paper will draw on some of their
ﬁndings but it will not examine any additional US forecasts. The purpose is to determine
whether their results are robust by examining the predictions of other countries. The focus
will be on (1) directional errors, (2) the magnitude of the errors made in estimating growth
and inﬂation, (3) whether there were biases and systematic errors, (4) the sources of the
errors and (5) whether there has been an improvement in forecasting ability. It appears
that few of the papers that have analyzed any of the G7 forecasts have examined all of
these issues. A notable exception is a paper coauthored by Ullrich Heilemann, whom
we are honoring.2 However, a number of the papers that are cited here provided data
that give us a greater understanding of the forecasting process. Some the forecasts were
issued by international organizations such as the OECD and IMF. The others come from
institutions or from private forecasters as reported in Consensus Forecasts.
The next section examines directional errors followed by a discussion of the magni-
tude of the quantitative discrepancies. The third section questions whether the accuracy
of forecasts has improved. The subsequent sections discuss the existence of bias in the
various G7 countries and the sources of the errors. The ﬁnal section summarizes what
we have learned about forecasting in the G7 countries from the evaluation studies that
have been summarized here.
I Directional Errors
There is only limited information about forecasters’ ability to predict turning points because
most evaluations focused on the magnitude of the errors. Fildes & Stekler indicated that
forecasters were not able to forecast the cyclical turning points in the UK economy and
were only partially successful in identifying a turn when it occurred. The inﬂation surges
of 1979-80 and 1989-90 were also not predicted until a number of months after prices
had started rising rapidly; a similar result was observed when inﬂation tapered off.
As for other countries, the turning points in German GDP were not predicted. However,
the accelerations and decelerations of GDP but not of inﬂation were forecast accurately.
1 Prof. Herman O. Stekler, Research Professor of Economics, George Washington University, 2201 G
Street, NW, Washington, DC 20052 USA, firstname.lastname@example.org. I would like to thank Ulrich Fritsche
for his comments on an earlier draft.
2 Heilemann & Stekler (2003).
2 Herman O. Stekler
(Heilemann & Stekler, 2003; Dopke & Fritsche, 2006a). Loungani (2001) indicates that
there were 60 cyclical turns in his sample but private forecasters forecast only three of
them a year in advance.3 The evidence is similar for the forecasts of research institutes and
international organizations.4 (Ash et al.,1998; Öller & Barot, 2000; Blix et al., 2001; Vogel,
2007). In fact, Ashiya (2003) shows that the IMF’ forecasts of the direction of change
GDP made 15 months in advance are not signiﬁcantly related to the actual changes. Going
even further, Vuchelen & Guitierrez (2005b) found that the 24 month OECD forecasts
for the G7 countries were worthless.5 The preponderance of the evidence is that most
forecasters are not able to predict turning points in advance and even a suggestion that
they are not detected quickly.
II Magnitude of the Errors
Because most evaluations focus on the magnitude of the errors, there is considerable in-
formation that permits us to synthesize the results. Fildes & Stekler (2002) reported that
the Mean Absolute Error (MAE) of the UK annual GDP forecasts was around 1%, but
Öller & Barot (2000) found that the errors were larger for some of the other European
countries. Newer studies found similar results: the MAE of the GDP forecasts for the G7
countries was around 1% and the MAE of the one year-ahead inﬂation predictions was
as low as 0.33% (Mestre, 2007), but others found it to be between 0.5% and 0.75% (
Loungani, 2001; Heilemann & Stekler, 2003; Ashiya, 2006; Dopke & Fritsche, 2006a;
Isiklar & Lahiri, 2007; Bowles et al., 2007; Timmermann, 2007).6 The universality of
these results suggests, that given present knowledge and procedures, we cannot expect
to achieve higher levels of accuracy.7
III Have the Forecasts Improved?
Given the number of papers that have evaluated forecasts, it is surprising how few have
asked whether the quality of the forecasts has improved over time. There are sufﬁcient
data because we have 40 years worth of forecasts for some countries. The evidence from
a limited number of analyses is not deﬁnitive. Heilemann & Stekler (2003) examined Ger-
man forecasts and adjusted the errors for the difﬁculty in forecasting the relevant periods.
3 Loungani did not distinguish between the cyclical turns of developed and underdeveloped coun-
4 Batchelor (2001) found that the private sector forecasts were superior to those of the international
organizations, but Boweles et al (2007) indicated that the ECB SPF forecasts were comparable to
those of the private Consensus Forecasts.
5 But same authors (Vuchelen and Guitierrez, 2005a) and other studies had found that the one year
forecasts were useful.
6 Unfortunately, Campbell & Murphy (2006) showed that the range of the GDP forecasts made by
Canadian economists who provide information to the government encompassed the actual GDP
numbers only 10-35% of the time depending upon the sample period. The inﬂation rate forecasts
were better: the actual value was within the range of the forecasts 60% of the time.
7 An appropriate procedure would be to compare these errors to the mean absolute change or the
volatility of the series.
Was wissen wir über die makroökonomischen Vorhersagen für die G7-Länder? 3
The results were mixed. Dopke & Fritsche (2006a) surveyed previous studies of German
forecasts which had differing results but they suggest that accuracy may have improved.
Vogel (2007) shows that the RMSE of the OECD forecasts had declined over time, but
these statistics had not been adjusted for the changes in the volatility of the economies.
On the other hand, Timmerman (2007) indicates that the quality of the IMF forecasts had
declined since they had been last evaluated in the 1990s. Earlier Fildes & Stekler (2002)
had summarized the conclusions about US forecasts and found mixed results. Despite all
the efforts that have been devoted to forecasting, there is no clear evidence that accuracy
has improved. We need to ask why? Stock and Watson (2007) provided one explanation.
The decline in volatility has made it harder to forecast inﬂation.8
IV Bias, Rationality, and Systematic Errors
There are a number of possible explanations for the existence of these errors. While they
might be the lowest achievable errors, there are at least two alternative explanations. If
there were symmetric loss functions, the errors might be biased and thus irrational. Al-
ternatively, individuals might have asymmetric loss functions that explain the existence
of these errors. (Elliot & Timmermann, 2008). Most evaluations have focused on the
possibility that the forecasts were biased.
Many studies have indicated that forecasters make systematic errors and are biased.
(Fildes & Stekler, 2002). These biases consist of overestimates of growth rates during
slowdowns and recessions and underestimates during recoveries and booms. Similar
systematic errors are observed with regard to inﬂation forecasts: overpredictions occur
when inﬂation is low and underpredictions are observed when inﬂation is high.
Among non-US and UK studies cited by Fildes & Stekler, the IMF end of year forecasts
were rational for all countries Pons (2001) and the six month-ahead predictions of Ger-
man research institutes were rational while the year-ahead were not Kirchgassner (1993).
The OECD predictions did not include all available information (Ash et al. 1990) and
tended to underestimate changes in the trends of both real output growth and inﬂation.
(Anderson, 1997).9 As shown below, the newer studies do not reach a single deﬁnitive
Consensus Forecasts: Loungani (2001) found that the year-ahead Consensus Forecasts
of GDP had an optimistic bias that had diminished by the end of the year for which the
predictions were made. More recently and examining a longer time period, Batchelor
(2007) and Ager et al. (2007) reach different conclusions regarding the predictions of
GDP for the G7 countries. Batchelor ﬁnds that there is a difference between the US and
other G7 forecasters. The latter have large biases that diminish only gradually. Ager et
al. used the pooled method of Clements et al. (2007) and found that over all only the
Italian growth forecasts were biased. For other countries there were biases at some hori-
8 On the other hand, if individuals have expectations that are anchored on an inﬂation target, that
forecast may be sufﬁciently accurate and no further improvement in accuracy should be expected.
Secondly, the Great Moderation may mean that small forecast errors, while statistically signiﬁcant
are not economically meaningful. However, the economic events of 2007-2008 may signify the end
of the Great Moderation.
9 However, in periods when inﬂation was decelerating, the rate of inﬂation was overpredicted.
4 Herman O. Stekler
zons and some of the revisions were predictable. The two studies agree that the inﬂation
predictions were not biased.
Canadian Forecasts: Some Canadian forecasters are systematically optimistic or pes-
simistic. (Campbell & Murphy, 2006).
Eurozone Forecasts: The European Central Bank has selected a group of individuals
to prepare forecasts for the ECB. The evidence indicates that inﬂation forecasts made one
and two years in advance are both biased: the rate of inﬂation is underpredicted. Only the
two-year-ahead GDP estimates are biased. (Bowles et al., 2007; Garcia & Manzanares,
German Forecasts: Heilemann & Stekler (2003) indicated that there were systematic
errors in German predictions of inﬂation but that these errors were not observed in the
GDP forecasts. Dopke & Fritsche (2006a) agreed that the German GDP forecasts were
unbiased, but argued that the inﬂation forecasts were unbiased but inefﬁcient.
Japanese Forecasts: Ashiya (2006) found that some Japanese institutions were always
either optimistic or pessimistic. Moreover, they frequently overreact to new information,
make excessive revisions and thus are not rational. (Ashiya, 2003).
UK Forecasts: In order to provide more information about the probabilities of speciﬁc
rates of inﬂation, the Bank of England issues both a point forecast and also provides a fan
chart. The NIESR now also publishes a fan chart. The evidence indicates that the Bank’s
point forecasts are unbiased. (Wallis, 2003; Clements, 2004; Mitchell & Hall, 2005).
However, the Bank’s probability forecasts are biased because of the exaggerated concern
with upside risks but better than the appropriate benchmark. The Bank of England also
conducts a survey of professional forecasters. The evidence from that survey indicates
that the professional forecasters’ uncertainty about the rate of inﬂation has declined but
they also may not have noticed that the persistence of UK inﬂation had declined. (Boero
et al, 2008).
International Agency Forecasts: Timmermann (2007) showed that the IMF forecasts
of both growth and inﬂation for the G7 countries were signiﬁcantly biased. Economic
growth was overpredicted for all countries except the US while inﬂation was also over
predicted for four of the G7 countries. These forecasts are similar in quality to those made
by Consensus Forecasts, but the current year IMF forecasts are slightly less biased than
the others. Vogel’s (2007) analysis of the OECD found that the current year forecasts were
not biased, but that the year-ahead predictions were biased. They were overly optimistic
when growth was slowing down and were more biased than the Consensus estimates.
These ﬁndings are diverse making it hard to generalize, but there are two common
themes: (1) forecaster beliefs are persistent and make consistent over or under estimates
(Isiklar and Lahiri( 2008), and (2) they use information inefﬁciently Kirchgassner (1993)
and Dopke & Fritsche (2006a). Furthermore, there is a large variation among the forecast-
ers, with some perennially optimistic or pessimistic. In fact, their biases exist even at short
horizons. Ashiya (2006) reported this result for Japanese forecasters and Batchelor (2007)
noted this among the Consensus forecasters. This ﬁnding can be explained in a number
of ways: (1) the forecasters are irrational and do not learn from their perennial mistakes,
(2) they have asymmetric loss functions or (3) they exhibit this characteristic for strategic
reasons. Additional research is required to test these alternative explanations.10
10 However, it is important to note that weather forecasters have had biases but these systematic errors
were reduced when they had quick feedback on the quality of their predictions.
Was wissen wir über die makroökonomischen Vorhersagen für die G7-Länder? 5
V Sources of Errors
Our discussion has indicated that there are analytic difﬁculties associated with deter-
mining why the errors occurred. Nevertheless these newer studies have provided some
evidence about the sources of the errors. Heilemann & Stekler (2003) indicated that a
large portion of the German forecast errors were attributable to the failure to predict
recessions. Similarly, Dopke & Fritsche (2006b) found that the variability (dispersion) of
the German GDP forecasts was higher before and during recessions. They surmised that
forecasters must disagree about the importance of shocks and the economy’s response to
these shocks. Bowles et al. (2007) argue that the European SPF underestimates of inﬂation
were due to asymmetric shocks. All the shocks tended to raise the rate of inﬂation, and
when the forecasts were adjusted for the impact of these shocks, there was less evidence
of a bias.
VI What Have We Learned?
The evidence about the G7 macro forecasts that has been presented here is very similar to
the ﬁndings of Fildes & Stekler that primarily related only to US and UK forecasters. Both
studies found that recessions are not predicted in advance and account for a signiﬁcant
portion of the quantitative errors. Neither study was able to show that forecast accuracy
has improved and both found that there were systematic errors.
However, the studies that are summarized here provide us with some new insights.
There may be a limit beyond which forecast accuracy cannot be improved. (Heilemann
& Stekler, 2003; Isiklar & Lahiri, 2007). A second important insight is that forecasts
longer than 12-18 months might not be valuable. (Vuchelen & Guitierrez, 2005b; Isiklar
& Lahiri, 2007). While studies found systematic errors, they are now not necessarily
considered biases; they may be attributable to asymmetric loss functions. Finally, we now
have a somewhat better understanding of the forecasting process. Batchelor (2007) showed
how the systematic errors or “bias” was related to the forecasters’ optimism (pessimism)
and conservatism in revising their predictions. He notes that standard rationality tests are
not appropriate if there has been a structural break. The pattern of the errors can then
provide a way of understanding the forecasters’ learning process about the impact of this
structural break. Isiklar & Lahiri (2007) use forecast revisions to explain the behavioral
characteristics of forecasters, i.e. how they react to news and when is news important.11
We have also learned that there is much more work to be done in determining the impor-
tance of asymmetric losses, the sources of biases, the limits of accuracy, etc. Much can
be learned if we undertake more studies, about the sources of error, similar to the paper
published by our honoree. (Heilemann, 2002).
11 In the forecasts made for year t, the most important revisions occur at the end of t-1.
6 Herman O. Stekler
Ager. P., Kappler M., and Osterloh, S., 2007. The accuracy and efﬁciency of the Consensus Forecasts: A
further application and extension of the pooled approach, Centre for European Economic Research,
Discussion Paper No. 07–058.
Anderson, P. S., 1997, Forecast errors and ﬁnancial developments, Working Paper no 51, Bank for Inter-
national Settlements, Basle, Switzerland.
Ash, J.C.K., Smyth, D.J. and Heravi, S.M., 1990, The accuracy of OECD forecasts of the international
economy, International Journal of Forecasting, 6, 379–392.
Ash, J.C.K., Smyth, D.J. and Heravi, S.M., 1998, Are OECD forecasts useful? A directional analysis,
International Journal of Forecasting, 14, 381–391.
Ashiya, M., 2003, Testing the rationality of Japanese GDP forecasts: the sign of the forecast revision
matters, Journal of Economic Behavior and Organization, 50, 263–269.
Ashiya, M., 2003b, The directional accuracy of 15-months-ahead forecasts made by the IMF, Applied
Economics Letters, 10, 331–333.
Ashiya, M., 2006, Forecast accuracy and product differentiation of Japanese institutional forecasters,
International Journal of Forecasting, 22, 395–401.
Batchelor, R. 2001, How useful are the forecasts of international agencies? The IMF versus the consensus,
Applied Economics, 33, 225–235.
Batchelor, R. 2007, Bias in macroeconomic forecasts, International Journal of Forecasting, 23, 189–
Boero, G., Smith J. and Wallis K. F., 2008, Uncertainty and disagreement: the Bank of England Survey of
External Forecasters, Economic Journal, forthcoming.
Bowles, C., Friz, R., Genre, V., Keany, G., Meylen, A. and Rautanen, T., 2007, The ECB survey of professi-
onal forecasters (SPF), A review after eight years’ experience, ECB Occasional Paper Series No 59.
Campbell, B. and Murphy S., (2006), The recent performance of the Canadian forecasting industry,
Canadian Public Policy, 32, 23–40.
Clements, M. P., 2004, Evaluating the Bank of England density forecasts of inﬂation, The Economic
Journal, 114, 844–866.
Dopke, J. & Fritsche U., 2006a, Growth and inﬂation forecasts for Germany, Empirical Economics, 31,
Dopke, J. and Fritsche U., 2006b, When do forecasters disagree? An assessment of German growth and
inﬂation forecast dispersion, International Journal of Forecasting, 22, 125–135.
Elliott, G. and Timmermann, A., 2008, Economic forecasting, Journal of Economic Literature, 46,
Fildes, R. and Stekler, H., 2002, The state of macroeconomic forecasting, Journal of Macroeconomics,
Garcia, J. A. and Manzanares, A., 2007, Reporting biases and survey results, evidence from European
professional forecasters, European Central Bank Working Paper 836.
Heilemann, U., 2002, Increasing the transparency of macroeconometric forecasts: A report from the
trenches, International Journal of Forecasting, 18, 85–105.
Heilemann, U. and Stekler, H.O. (2003), Has the accuracy of German macroeconomic forecasts improved?,
Discussion paper of the German Research Council’s research centre, 475, 31/03.
Isiklar, G. and Lahiri, K., 2007, How far ahead can we forecast? Evidence from cross-country surveys,
International Journal of Forecasting, 23, 167–187.
Kirchgassner, G., 1993, Testing weak rationality of forecasts with different time horizons, Journal of
Forecasting, 12, 541–558.
Loungani, P., 2001, How accurate are private sector forecasts? Cross-country evidence from Consensus
Forecasts of output growth, International Journal of Forecasting, 17, 419–432.
Mestre, R., 2007, Are survey-based inﬂation expectations in the Euro area informative? European Central
Bank Working Paper 721.
Mitchell, J. and Hall S. G., 2005, Evaluating, comparing, and combining density forecasts using the KLIC
with an application to the Bank of England and NIESR ‘fan’ charts of inﬂation, Oxford Bulletin of
Economics and Statistics, 67, Supplement, 995–1033.
Öller, L-E. and Barot, B., 2000, Comparing the accuracy of European GDP forecasts, International Journal
of Forecasting, 16, 293–315.
Was wissen wir über die makroökonomischen Vorhersagen für die G7-Länder? 7
Pons, J., 2001, The rationality of price forecasts: a directional analysis, Applied Financial Economics,
Stock, J. H. and Watson, Mark W., 2007, Why has U.S. inﬂation become harder to forecast? Journal of
Money, Credit and Banking, Supplement 39, 3–33.
Timmerman, A., 2007, An evaluation of the World Economic Outlook, IMF Staff Papers, 54, 1–33.
Vogel, L., 2007. How do the OECD growth projections for the G7 economies perform? OECD Economics
Department Working Papers No. 573.
Vuchelen, J. and Hutierrez, M-I, 2005a, A direct test of the information content of the OECD growth
forecasts, International Journal of Forecasting, 21, 103–117.
Vuchelen, J. and Hutierrez, M-I, 2005b, Do the OECD 24 month horizon growth forecasts for the G7-
countries contain information?, Applied Economics, 37, 855–862.
Wallis, K. F., 2003, Chi-squared tests of interval and density forecasts and the Bank of England’s fan charts,
International Journal of Forecasting, 19, 165–175.