Content uploaded by Oscar Claveria
Author content
All content in this area was uploaded by Oscar Claveria on Nov 06, 2017
Content may be subject to copyright.
Content uploaded by Oscar Claveria
Author content
All content in this area was uploaded by Oscar Claveria on Apr 05, 2016
Content may be subject to copyright.
1
3rd CESIfo Conference on Survey Data in Economics –
Methodology and Applications
Munich, 6 November 2009
http://www.cesifo-group.de/ifoHome/events/Archive/conferences/2009/11/2009-11-06-event-
ConfIfo.html
How can the information content of neutral business
survey responses be exploited for forecasting
purposes?
Oscar Claveria† and Anna Stangl‡
oclaveria@ub.edu ; stangl@ifo.de
† Research Institute of Applied Economics (IREA), University of Barcelona, Barcelona (Spain)
‡ Ifo Institute for Economic Research, Munich (Germany)
Abstract
Business surveys are an essential tool for gathering information about the
development of the economy. Survey results are presented as weighted percentages of
respondents expecting a particular variable to rise, fall or remain unchanged. The aim of
the paper is to analyze whether taking into account the percentage of respondents
expecting a variable to remain constant helps to improve the forecasting performance of
survey results. With this objective we present a variation of the balance statistic
(weighted balance).
By means of a simulation experiment we test whether a variation of the balance
statistic outperforms the balance statistic in order to track the evolution of agents’
expectations and produce more accurate forecasts of the generated quantitative variable
used as a benchmark. We then use the information from a Business Survey in the
German manufacturing sector in autoregressions and Markov switching regime models
to analyze the predictive performance of the proposed methods in tracking the
Production Index.
The results of the forecasting evaluation permit us to conclude that taking into
account the fraction of respondents expecting a variable to remain constant improves the
forecasting performance of business survey results.
Keywords: Business surveys; Quantification; Expectations; Forecasting
JEL classification: C42, C51, C53, C82, D12, D84
2
1. INTRODUCTION
Business surveys are an essential tool for gathering information about the
development of the economy. As a rule, survey results are presented as percentages of
respondents expecting a particular variable to rise, remain constant or fall.
1
The aim of
this paper is to analyse whether taking into account the percentage of respondents
expecting a variable to remain constant helps to improve the usefulness of survey results
in forecasting macroeconomic variables. With this objective we evaluate the
performance of an alternative way of exploiting the information content of survey
responses for forecasting purposes: a variation of the balance statistic, known as
weighted balance (Claveria, 2010).
Balance statistics (fraction of positive responses minus the fraction of negative
responses) is the most popular quantification method of business expectations applied in
practice and used by Eurostat to track the official data on economic growth, such as the
production index. The balance statistic ignores the fraction of neutral responses and
considers the proportion of negative and the proportion of positive responses only. This
procedure is problematic as neutral responses exhibit the highest frequency of
occurrence in surveys. In business surveys their proportion not seldom lies at 60-70
percent. Ignoring neutral responses consequently results in vast information loss.
The information content of neutral responses is not explicit. Neutral responses
may reflect: (1) a true view of a respondent about a variable remaining constant, (2)
positive or negative responses, which have not reached a particular threshold, and/or (3)
“epistemic uncertainty”. Bruine de Bruin et al. (2000) argue that heaping at the middle
of a scale reflects uncertainty, and only seldom represents the true view of the
respondent. The neutral category subsequently functions as a surrogate for a “don’t
know” or “unsure about the future development” option (even if a “don’t know” option
is available within the question).
The role of uncertainty in influencing economic behavior of agents, such as
investment decisions or monetary-policy, has been recognized early (Knight, 1921):
Economic agents may delay investment and consumption decisions if they are uncertain
1
The measurement procedure of business surveys on economic expectations in the European Union has
been harmonized by the European Commission to three-category rating questions.
3
about the future economic development. Empirical findings confirm that uncertainty has
cyclical properties and is higher around cyclical turning points (Ramey and Ramey,
1995, Mitchell et al., 2005, Doepke and Fritsche, 2006, Guiso and Parigi, 1999). As the
information content of neutral responses encompasses an uncertainty component, we
use the fraction of neutral responses to weight the balance statistics (weighted balance)
and examine their properties by a simulation study and in an empirical analysis.
The simulation experiment is designed to test whether the proposed variation of
the balance statistic (weighted balance) outperforms the balance statistic in order to
track the evolution of agents’ expectations and produces more accurate forecasts of the
quantitative variable generated that is used as a benchmark. Survey results are as a rule
quantified making use of official data. The differences between the actual values of a
variable and quantified expectations may arise from three different sources (Lee, 1994):
measurement or conversion error due to the use of quantification methods, expectational
error due to the agents’ limited ability to predict the movements of the actual variable,
and sampling errors. Since survey data are approximations of unobservable
expectations, they inevitably entail a measurement error. Monte Carlo simulations allow
us to distinguish between these sources of error.
We also use survey results of a business survey in the German manufacturing
sector in autoregressions and Markov switching regime models to analyze the predictive
performance of the various proposed methods of assessing business surveys results to
track the evolution of the German Production Index.
The paper is organized as follows. The second section describes the variation of
the balance statistic. Section three presents the simulation experiment and analyzes the
relative forecasting performance of the variation of the balance statistic. Section four
presents the empirical data and the models and evaluates the relative forecasting
performance of the various methods of assessing business surveys results described in
section two. Section five concludes.
4
2. THE WEIGHTED BALANCE STATISTIC
Unlike other statistical series, survey results are weighted percentages of
respondents expecting an economic variable to increase, decrease or remain constant.
As a result, tendency surveys contain two pieces of independent information at time
t
,
t
R
and
t
F
, denoting the percentage of respondents at time
1t
expecting an economic
variable to rise or fall at time
t
. The information therefore refers to the direction of
change but not to its magnitude.
A variety of quantification methods have been proposed in the literature in order
to convert qualitative data on the direction of change into a quantitative measure of
agents’ expectations (see Claveria et al, 2006). The output of these quantification
procedures (estimated expectations) can be regarded as one period ahead forecasts of
the quantitative variable under consideration. In this paper we use agents’ expectations
about the future (prospective information) and compare the balance statistic to the
weighted balance.
The first attempt to quantify survey results is due to Anderson (1951). Assuming
that the expected percentage change in a variable remains constant over time for agents
reporting an increase and for those reporting a decrease, Anderson (1952) defined the
balance statistic (
tt FR
) as a measure of the average changes expected in the variable.
The balance statistic (
t
B
) does not take into account the percentage of
respondents expecting a variable to remain constant (
t
C
). As
t
C
usually shows the
highest proportions and high levels of dispersion, we propose a non-linear variation of
the balance statistic (
t
WB
, weighted balance) that accounts for this percentage of
respondents:
t
t
tt
tt
tC
B
FR FR
WB
1
(1)
Weighting the balance statistic by the proportion of respondents expecting a
variable to rise or fall allows discriminating between two equal values of the balance
statistic depending on the percentage of respondents expecting a variable to remain
constant.
5
3. THE SIMULATION EXPERIMENT
By Monte Carlo simulations we compare the forecasting performance of the two
quantification methods: the balance and the weighted balance. The experiment is
designed in five consecutive steps:
(i) The simulation begins by generating a series of actual changes of a variable. We
consider 500 agents and 250 time periods. Let
it
y
indicate the percentage change of
variable
it
Y
for agent
i
from time
1t
to time
t
. Additionally we suppose that the true
process behind the movement of
it
y
is given by:
ititit εdy
(2)
500,,1i
,
250,,1t
and
1,
tiit yφμd
, where
it
d
is the deterministic
component. The initial value,
85.0
0
i
y
, is assumed to be equal for all agents
2
.
it
ε
is an
identical and independent normally distributed random variable with mean zero and
variance
2
ε
σ
=0.5, 5, 50 . The average rate of change,
t
y
, is given by
iitiyy 500
1
.
The same weight is given to all agents. We assume different values of
μ
and
φ
.
(ii) Secondly, we generate a series of agents’ expectations about
t
y
under the
assumption that individuals are rational in Muth’s sense
3
:
itit
e
it ζdy
2
,0~ ζit σNζ
(3)
where
e
it
y
has the same deterministic part as
it
y
but a different stochastic term
it
ζ
. We
derive
i
e
it
e
iyy 5001
. Additionally, we assume that
1
22 ζζiσσ
. All the values
given to
2
ε
σ
and
2
ζ
σ
, and to the indifference interval are set to simulate actual business
survey series.
2
To check the robustness of the results, we chose different values for the autoregressive
parameter, ranging from 0 to 1 with an increase of 0.1 each time. As the final results did not vary
significantly from one specification to the other we presented the results for 0, 0.3, 0.6 and 0.9.
3
Muth (1961) assumed that rationality implied that expectations had to be generated by the same
stochastic process that generates the variable to be predicted.
6
(iii) The third step consists of constructing the answers to the business surveys. The
answers are given in terms of the direction of change, i.e., if the variable is expected to
increase, decrease or remain equal. We assume that agents’ answers deal with the next
period and that all agents have the same constant indifference interval
ba,
with
5 ba
. If
5.4
e
it
y
, agent
i
answers that
it
Y
will increase; if
5.4
e
it
y
,
i
expects
it
Y
to decrease; while the agent will report no change if
5.45.4 e
it
y
. With these
answers, qualitative variables
it
R
and
it
F
can be constructed.
it
R
(
it
F
) takes the value 1
(0) whenever the agent expects an increase (decrease) in
it
Y
.
t
R
and
t
F
are then
constructed by aggregation.
(iv) The fourth step of the simulation experiment consists of using the two different
quantification methods to trace back the series of actual changes of the generated
quantitative variable,
t
y
, from the qualitative variables. We will refer to these
expectations as estimated expectations in order to distinguish them from the
unobservable ones. With the aim of analysing the performance of the different proxy
series, we use the last 100 generated observations. Keeping the series of actual changes
fixed, the experiment of generating the rational expectations series as well as the proxy
series is replicated 1500 times
4
.
(v) To test the robustness of the results, we repeat the simulation experiment for
different values of
μ
, therefore assuming
it
y
is generated by a random walk and by an
autoregressive process with different drifts.
In order to evaluate the relative performance and the forecasting accuracy of the
different quantification procedures, we keep the series of actual changes fixed and we
replicate the experiment of generating the rational expectations series as well as the
qualitative variables
t
R
and
t
F
1500 times. The specification of the quantification
procedures is based on information up to the first 150 periods; models are then re-
estimated each period and forecasts are computed in a recursive way. In each
simulation, forecast errors for all methods are obtained for the last 100 periods.
4
All simulations are performed with Gauss for Windows 6.0.
7
In order to summarize this information, we calculate the Root Mean Squared
Error (RMSE), the Mean Error (ME), the Theil Coefficient (TC) and the three
components of the Mean Square Error (MSE): the bias proportion of the MSE (U1), the
variance proportion (U2) and the covariance proportion (U3). With the aim of testing
whether the reduction in RMSE when comparing both methods is statistically
significant, we calculate the measure of predictive accuracy proposed by Diebold-
Mariano (1995). Given these two competing forecasts and the series of actual changes
of the generated quantitative variable, we have calculated the DM measure which
compares the mean difference between a loss criteria (in this case, the root of the mean
squared error) for the two predictions using a long-run estimate of the variance of the
difference series.
Table 3.1 shows the results of an off-sample evaluation for the last 100 periods
when
0μ
. Table 3.2 and Table 3.3 show the results of an off-sample evaluation for
the last 100 periods when
1μ
and
1μ
respectively.
Table 3.1 shows that the weighted balance (WB) shows lower RMSE, ME and
TC in all cases. Although the proportion of systematic error (U1) is not very different,
the balance shows higher proportions of regression error (U2). As
2
ε
σ
increases,
forecasting results tend to worsen for both methods. Nevertheless if we look at the
results of the DM test, we can see that there is no significant difference between both
methods.
However, in Table 3.2 and Table 3.3, when
0μ
, the difference is significant
and is always in favor of the weighted balance, with the exception of the scenario in
which
9.0φ
and
50
2
ε
σ
. Comparison of the results in Table 1 with those in Table 2
and Table 3 highlights several differences, in particular regarding the forecasting results
as the value of
φ
increases from 0 to 0.9: while they worsen when
0μ
, this effect is
not clear when
0μ
. Another difference is that when
0μ
, as
2
ε
σ
increases the
forecasting results improve for both methods.
8
Table 3.1. Forecasts evaluation (
0μ
)
0μ
5.0
2
ε
σ
5
2
ε
σ
50
2
ε
σ
0φ
B
WB
B
WB
B
WB
RMSE a
3.60
2.35
3.56
2.43
3.11
2.80
ME b
0.02
0.01
0.00
0.00
0.18
0.15
% U1 c
1.0
1.0
0.9
0.9
0.6
0.6
% U2 c
98.9
98.7
97.8
96.3
92.6
90.8
% U3 c
0.1
0.3
1.3
2.8
6.8
8.6
TC d
13.15
5.64
12.97
6.13
8.94
7.06
DM e
1.04
0.02
0.94
3.0φ
B
WB
B
WB
B
WB
RMSE a
3.61
2.36
3.65
2.50
2.68
2.46
ME b
-0.04
-0.02
0.06
0.04
0.11
0.09
% U1 c
1.1
1.1
0.8
0.8
0.5
0.5
% U2 c
98.8
98.5
97.6
96.0
86.5
83.7
% U3 c
0.2
0.4
1.5
3.2
13.0
15.7
TC d
13.22
5.67
13.70
6.48
6.57
5.42
DM e
-1.57
0.29
0.99
6.0φ
B
WB
B
WB
B
WB
RMSE a
3.61
2.36
3.66
2.54
2.68
2.49
ME b
0.06
0.04
0.09
0.06
-0.16
-0.17
% U1 c
1.0
1.0
0.9
0.9
0.8
0.9
% U2 c
98.8
98.5
97.4
95.7
84.5
81.3
% U3 c
0.2
0.5
1.6
3.4
14.7
17.8
TC d
13.23
5.68
13.77
6.72
5.98
4.96
DM e
2.31
0.34
0.42
9.0φ
B
WB
B
WB
B
WB
RMSE a
3.61
2.36
3.80
2.85
3.96
3.97
ME b
-0.14
-0.09
0.70
0.50
0.15
0.12
% U1 c
1.2
1.1
3.9
3.5
1.2
0.9
% U2 c
98.6
98.2
94.4
93.3
3.7
7.5
% U3 c
0.3
0.7
1.8
3.2
95.1
91.7
TC d
13.24
5.71
14.81
8.36
2.61
2.70
DM e
4.11
1.74
-1.19
Notes: a RMSE = root mean square error
b ME = mean error
c Decomposition of the mean square error:
(i) %U1 = percentage of mean error (bias proportion of the MSE).
(ii) %U2 = percentage of regression error (variance proportion of the MSE).
(iii) %U3 = percentage of disturbance error (covariance proportion of the MSE).
d TC = Theil coefficient.
e DM = results of the Diebold-Mariano test. Statistic uses a NW estimator. Null hypothesis: the
difference between the two competing series is non-significant. A positive sign of the statistic
implies that the Balance has bigger errors, and is worse. When that t-stat is significant, the second
model is statistically better.
* Significant at the 5% level.
9
Table 3.2. Forecasts evaluation (
1μ
)
1μ
5.0
2
ε
σ
5
2
ε
σ
50
2
ε
σ
0φ
B
WB
B
WB
B
WB
RMSE a
7.17
4.41
6.67
4.33
2.69
2.44
ME b
-6.21
-3.72
-5.62
-3.56
-0.78
-0.66
% U1 c
74.74
70.93
70.68
66.93
10.38
9.47
% U2 c
25.22
28.96
28.91
32.09
74.84
71.63
% U3 c
0.04
0.11
0.41
0.98
14.78
18.90
TC d
51.67
19.58
44.84
18.99
6.13
4.79
DM e
361.33
42.50
4.14
3.0φ
B
WB
B
WB
B
WB
RMSE a
9.51
5.81
8.79
5.69
2.70
2.44
ME b
-8.81
-5.30
-8.02
-5.12
-0.84
-0.68
% U1 c
85.79
83.12
83.04
80.37
12.07
10.05
% U2 c
14.19
16.82
16.71
19.04
73.07
70.87
% U3 c
0.02
0.06
0.25
0.59
14.86
19.07
TC d
90.68
33.86
77.65
32.62
6.07
4.74
DM e
440.73
50.71
6.07
6.0φ
B
WB
B
WB
B
WB
RMSE a
15.70
9.63
13.32
8.77
2.89
2.62
ME b
-15.30
-9.31
-12.84
-8.40
-1.09
-0.86
% U1 c
94.85
93.48
92.75
91.35
17.55
13.92
% U2 c
5.14
6.49
7.12
8.37
69.96
70.07
% U3 c
0.01
0.03
0.12
0.28
12.49
16.00
TC d
246.87
92.83
177.82
77.25
6.91
5.40
DM e
614.10
68.99
7.50
9.0φ
B
WB
B
WB
B
WB
RMSE a
53.30
39.60
36.39
27.69
4.49
4.64
ME b
-53.22
-39.49
-36.26
-27.55
3.34
3.56
% U1 c
99.73
99.44
99.29
99.00
85.35
87.43
% U2 c
0.27
0.56
0.69
0.96
1.56
0.70
% U3 c
0.00
0.00
0.02
0.04
13.09
11.88
TC d
2840.68
1568.23
1324.37
766.85
13.11
14.47
DM e
4070.45
153.88
-9.38
Notes: a RMSE = root mean square error
b ME = mean error
c Decomposition of the mean square error:
(i) %U1 = percentage of mean error (bias proportion of the MSE).
(ii) %U2 = percentage of regression error (variance proportion of the MSE).
(iii) %U3 = percentage of disturbance error (covariance proportion of the MSE).
d TC = Theil coefficient.
e DM = results of the Diebold-Mariano test. Statistic uses a NW estimator. Null hypothesis: the
difference between the two competing series is non-significant. A positive sign of the statistic
implies that the Balance has bigger errors, and is worse. When that t-stat is significant, the second
model is statistically better.
* Significant at the 5% level.
10
Table 3.3. Forecasts evaluation (
1μ
)
1μ
5.0
2
ε
σ
5
2
ε
σ
50
2
ε
σ
0φ
B
WB
B
WB
B
WB
RMSE a
7.15
4.40
6.70
4.35
2.99
2.71
ME b
6.19
3.71
5.64
3.57
0.56
0.45
% U1 c
74.82
71.02
70.24
66.57
4.08
3.38
% U2 c
25.14
28.86
29.33
32.41
86.38
84.70
% U3 c
0.05
0.12
0.43
1.02
9.54
11.93
TC d
51.35
19.46
45.34
19.20
8.22
6.60
DM e
400.45
40.47
4.16
3.0φ
B
WB
B
WB
B
WB
RMSE a
9.55
5.83
8.45
5.47
2.87
2.58
ME b
8.84
5.32
7.67
4.89
1.09
0.87
% U1 c
85.68
82.98
82.00
79.20
17.54
14.62
% U2 c
14.29
16.95
17.74
20.17
69.51
68.55
% U3 c
0.03
0.07
0.26
0.62
12.95
16.83
TC d
91.39
34.13
71.82
30.23
6.93
5.34
DM e
427.04
59.15
8.00
6.0φ
B
WB
B
WB
B
WB
RMSE a
15.70
9.63
13.70
9.02
2.63
2.42
ME b
15.30
9.31
13.24
8.65
0.55
0.41
% U1 c
94.87
93.51
93.23
91.87
5.96
4.35
% U2 c
5.12
6.47
6.66
7.87
77.37
74.73
% U3 c
0.01
0.03
0.11
0.26
16.68
20.91
TC d
246.91
92.79
187.99
81.58
5.38
4.30
DM e
773.48
69.98
4.25
9.0φ
B
WB
B
WB
B
WB
RMSE a
53.12
39.42
38.25
29.27
5.04
5.24
ME b
53.05
39.31
38.13
29.14
-3.64
-3.92
% U1 c
99.73
99.43
99.37
99.07
83.28
85.47
% U2 c
0.27
0.56
0.61
0.89
0.66
0.29
% U3 c
0.00
0.00
0.02
0.04
16.06
14.24
TC d
2822.47
1554.43
1463.55
857.01
15.96
18.00
DM e
3429.23
262.86
-17.21
Notes: a RMSE = root mean square error
b ME = mean error
c Decomposition of the mean square error:
(i) %U1 = percentage of mean error (bias proportion of the MSE).
(ii) %U2 = percentage of regression error (variance proportion of the MSE).
(iii) %U3 = percentage of disturbance error (covariance proportion of the MSE).
d TC = Theil coefficient.
e DM = results of the Diebold-Mariano test. Statistic uses a NW estimator. Null hypothesis: the
difference between the two competing series is non-significant. A positive sign of the statistic
implies that the Balance has bigger errors, and is worse. When that t-stat is significant, the second
model is statistically better.
* Significant at the 5% level.
11
Although it is impossible to completely eliminate the measurement error
introduced when converting qualitative data on the direction of change into quantitative
estimations of agents’ expectations, the weighted balance shows lower measurement
errors and better forecasts. These results suggest that taking into account the percentage
of respondents expecting a variable to remain constant may improve the use of the
balance statistic with forecasting purposes.
4. EMPIRICAL ANALYSIS
In order to evaluate the relative forecasting accuracy of the business tendency
survey measures we use monthly data from the German Manufacturing Business Survey
to track the evolution of the Production Index. We incorporate business survey results in
autoregressive (AR) and Markov switching regime (MKTAR) models, and we compare
the results to those obtained without using business survey information (benchmark
models).
All models are estimated from 1993.01 until 2004.12 and forecasts for 1, 2, 3, 6
and 12 months ahead are computed
5
. The specification of the models is based on
information up to that date, and then, forecasts are computed up to 2008.12. Forecasts
errors are computed in a recursive way. In order to summarize this information, the root
mean square error (RMSE) is computed.
4.1. Data
Every month the Ifo Institute for Economic Research asks within its Business
Tendency Survey (BTS) a sample of companies in the German manufacturing sector for
their views on some key business variables, such as business expectations and present
situation. The panel involves business officials with a range of specializations within
their companies in management, finance, and other strategic business functions. The
survey participation is absolutely voluntary and derives entirely from the interest in the
5
All calculations are performed with Gauss for Windows 6.0.
12
survey results, as no other compensation is offered. Becker and Wohlrabe (2008)
provide a detailed description of the Ifo Business Survey micro data.
The companies covered by the sample include small, medium and big-size
companies. As BTS are designed to forecast the business cycle in a sector with respect
to output growth, particularly big economic players are involved in the survey. In the
December 2006 survey the overall sample contained 2,622 respondents, which is
approximately 1% of German enterprises in the manufacturing sector.
The dataset of individual responses covers 192 survey waves, beginning with the
January 1993 survey and continuing to the December 2008 survey. The data of the
study comprise a conventional incomplete panel dataset, as the sample changes from
time to time and not every respondent provides an expertise every months. However,
the estimated overlap of respondents who participate in two consecutive survey rounds
lies at around 80% or higher, indicating that the panel is sufficiently stable.
The performance of the manufacturing industry is proxied by the monthly year-
on-year growth rate of the production index (year 2000=100) for the German
manufacturing sector. As business survey respondents are asked to respond to the
questions on business expectations without taking into account differences in the length
of months or seasonal fluctuations, value adjusted for working day variations and
seasonally adjusted production index growth (PI) is used in the paper. Growth rates of
the production index are calculated as percentage change of the index compared to the
same month of the previous year. The German Statistical Office continuously revises
the production indices backward. The data used in the study were retrieved in August
2009.
4.2. Methodology
4.2.1. Benchmark models
In this work two different types of models (AR and MKTAR models) have been
proposed to obtain forecasts for the quantitative variable (Production Index of the
German manufacturing sector) expressed as year-on-year growth rates. As there are few
attempts in the literature to incorporate qualitative information in quantitative
forecasting models, both types of models have also been applied including qualitative
survey data.
13
4.2.1.1. Autoregressions (AR)
Autoregressions explain the behaviour of the endogenous variable as a linear
combination of its own past values:
tptpttt xxxx
...
2211
(4)
In order to determine the number of lags that should be included in the model,
we have selected the model with the lowest value of the Akaike Information Criteria
(AIC) considering models with a minimum number of 1 lag up to a maximum of 24
(including all the intermediate lags).
4.2.1.4. Markov switching regime models (MKTAR)
Time series regime-switching models assume that the distribution of the variable
is known conditional on a particular regime or state occurring. Hamilton (1989)
presented the Markov regime-switching model in which the unobserved regime evolves
over time as a first order Markov process.
In this analysis, we use a Markov-switching threshold autoregressive model
(MKTAR) allowing for different regime-dependent intercepts, autoregressive
parameters, and variances. Once we have estimated the probabilities of expansion and
recession using the Hamilton filter together with the smoothing filter of Kim (1994), we
construct the following model for the time series
t
x
using the estimated probabilities of
changing regime:
tt uxLB )·(
if
PxExpansionP kt
/
(5)
tt vxL )·(
if
PxExpansionP kt
/
(6)
where,
t
u
and
t
v
are white noises,
)(LB
and
)(L
are autoregressive polynomials,
k
is the value minimizing the sum of squared errors among 1 and 12 and the value
P
,
known as threshold, is given by the variation of the probability.
4.2.2. Models where business survey information is incorporated
One way to use the qualitative information of survey data on the direction of
change in order to improve the forecasts of the quantitative variables consists in
introducing qualitative information from business surveys as explanatory variables in
autoregressions. We have followed the same approach by incorporating the judgement
14
about the present business situation and expectations by the end of the next 6 months to
autoregressive (AR) and Markov switching regime (MKTAR) models in the form of
balances (B) and weighted balances (WB).
4.3. Results
The results of our forecasting competition are shown in Tables 4.1 and 4.2. The
tables present the values of the Root of the Mean Squared Error (RMSE) obtained from
recursive forecasts for 1, 2, 3, 6 and 12 months during the period 2004.01-2008.12 for
both, the benchmark models and the models including information from business survey
results
6
. Table 4.1 shows the results for the present business situation and Table 4.2 for
business expectations.
As expected, forecasts errors increase for longer horizons in all cases. Regarding
the forecast accuracy of the benchmark models compared to the models with business
survey information, results differ depending on whether one incorporates judgements
about the present business situation or expectations for 6 months ahead. In the former
case, while AR models with survey data about the present business situation show lower
RMSE than AR models just for longer horizons (Table 4.1), AR models where business
expectations are incorporated (Table 4.2) outperform benchmark models for all
forecasting horizons.
Table 4.1. Average RMSE - Recursive forecasts from January 2004 to December 2008.
Production Index. Year-on-year growth rate of seasonally adjusted series.
Survey results - Present business situation
Models without survey information
1 month
2 months
3 months
6 months
12 months
AR
2.06
2.54
2.87
3.97
5.63
MKTAR
-
-
-
-
-
Models with business survey information
1 month
2 months
3 months
6 months
12 months
AR_B
2.29
2.71
3.08
3.92
4.50
AR_WB
2.22
2.69
3.02
3.88
4.19
MKTAR_B
-
-
-
-
-
MKTAR_WB
-
-
-
-
-
Notes: a AR_B refers to the AR model with the balance; AR_WB refers to the AR model with the
weighted balance; MKTAR_B refers to the MKTAR model with the balance; MKTAR_WB refers to
the MKTAR model with the weighted balance.
b MKTAR models did not converge.
c Bold indicates best mode.
6
To check the consistency of the obtained results, we have chosen different time periods for the
forecasting evaluation, obtaining very similar results.
15
Table 4.2. Average RMSE - Recursive forecasts from January 2004 to December 2008.
Production Index. Year-on-year growth rate of seasonally adjusted series.
Survey results – Business expectations
Models without survey information
1 month
2 months
3 months
6 months
12 months
AR
2.06
2.54
2.87
3.97
5.63
MKTAR
-
-
-
-
-
Models with business survey information
1 month
2 months
3 months
6 months
12 months
AR_B
2.05
2.33
2.53
3.02
3.58
AR_WB
2.15
2.43
2.64
3.00
3.44
MKTAR_B
-
-
-
-
-
MKTAR_WB
2.35
2.88
3.29
3.86
5.06
Notes: a AR_B refers to the AR model with the balance; AR_WB refers to the AR model with the
weighted balance; MKTAR_B refers to the MKTAR model with the balance; MKTAR_WB refers to
the MKTAR model with the weighted balance.
b MKTAR models without business survey information did not converge. Neither did MKTAR models
with the Balance statistic.
c Bold indicates best model.
This difference in the results between the two different questions is also
observed when comparing models where business survey information is incorporated.
Thus, in the case of judgements about the present business situation AR models with the
balance statistic (AR_B) are outperformed by those incorporating the weighted balance
(AR_WB) for all forecasting horizons. For business expectations this only happens for
longer horizons (6 and 12 months).
The obtained results permit to conclude that incorporating business survey
information and taking into account the percentage of respondents expecting a variable
to remain constant when assessing survey results improve the forecasting accuracy,
specially for longer horizons (6 and 12 months).
5. CONCLUDING REMARKS
In this work we have presented a different way to assess survey results: a
variation of the balance statistic (weighted balance).This measure allow us to take into
account the percentage of respondents expecting no change in the evolution of an
economic variable.
By means of a simulation experiment we have tested whether this variation of
the balance statistic outperforms the balance statistic in order to track the evolution of
16
agents’ expectations and produces more accurate forecasts of the quantitative variable
generated used as a benchmark. In all cases, the weighted balance outperformed the
balance statistic and provided more accurate forecasts of the quantitative variable
generated as a benchmark.
In order to evaluate the relative forecasting accuracy of the different methods
proposed to assess survey results we have also used the information provided by the Ifo
Institute for Economic Research’s Manufacturing Business Survey to track the
evolution of the Production Index. We have incorporated business survey results in
autoregressive and Markov switching regime models, and we have compared the results
to those obtained without using business survey information (benchmark models). We
have found that models with business survey information outperform benchmark
models.
When comparing the relative forecasting accuracy of the different methods
proposed to assess survey results we have found that taking into account the percentage
of respondents expecting a variable to remain constant improves the forecasting
accuracy of business expectations.
REFERENCES
Anderson, O. (1951). Konjunkturtest und Statistik. Allgemeines Statistical Archives, 35, 209–
220.
Anderson, O. (1952). The business test of the IFO-Institute for economic research, Munich, and
its theoretical model. Revue de l’Institut International de Statistique, 20, 1–17.
Becker, S. O. and Wohlrabe, K. (2008). Micro data at the Ifo Institute for Economic Research -
the “Ifo Business Survey”, usage and access. Journal of Applied Social Science
Studies/Schmollers Jahrbuch, 128, 487–496.
Bruine de Bruin, W., Fischhoff, B., Halpern-Felsher, B. and Millstein, S. (2000). Expressing
epistemic uncertainty: It's a fifty-fifty chance. Organizational Behaviour and Human
Decision Processes, 81, 115–131.
Claveria, O. (2010). Qualitative survey data on expectations. Is there an alternative to the
balance statistic? In A. T. Molnar (Ed.), Economic Forecasting (pp. 181–190).
Hauppauge, NY: Nova Science Publishers.
Claveria, O., Pons, E. and Ramos, R. (2007). Business and consumer expectations and
macroeconomic forecasts. International Journal of Forecasting, 23, 47–69.
Claveria, O., Pons, E. and Suriñach, J. (2006). Quantification of expectations. Are they useful
for forecasting inflation? Economic Issues, 11, 19–38.
Diebold, F. X. and Mariano, R. (1995). Comparing Predictive Accuracy. Journal of Business
and Economic Statistics, 13, 253–263.
Doepke, J. and Fritsche, U. (2006). When do forecasters disagree? An assessment of German
growth and inflation forecast dispersion. International Journal of Forecasting, 22, 125–
135.
17
Guiso, L. and Parigi, G. (1999). Investment and Demand Uncertainty. The Quarterly Journal of
Economics, 114, 185–227.
Hamilton, J. (1989). A new approach to the economic analysis of nonstationary time series and
the business cycle. Econometrica, 57, 357-384.
Kim, C. J. (1994). Dynamic linear models with Markov-Switching. Journal of Econometrics,
60, 1–22.
Knight, F.H. (1921): Risk, Uncertainty, and Profit. Boston, MA: Hart, Schaffner & Marx;
Houghton Mifflin Company.
Lee, K.C. (1994). Formation of price and cost inflation expectations in British manufacturing
industries: a multi-sectoral analysis. Economic Journal, 104, 372–385.
Mitchell, J., Mouratidis, K. and Weale, M. (2005). Uncertainty in UK manufacturing: evidence
from qualitative survey data. Economics Letters, 94, 245–252.
Muth, J. (1961). Rational expectations and the theory of price movements. Econometrica, 29,
315–335.
Ramey, G. and Ramey, V. A. (1995). Cross-country evidence on the link between volatility and
growth. American Economic Review, 85, 1138–1151.
