Content uploaded by Adriaan Brebels

Author content

All content in this area was uploaded by Adriaan Brebels on Mar 24, 2016

Content may be subject to copyright.

(

)

()m

t tt

e yf= −

()

m

ft

1,

1

1

,

n

ii

in

i

MAE e mean e

n=

=

= =

∑

1,

,

i

in

MdAE median e

=

=

() ()

22

11,

1

,

n

ii

iin

MSE e mean e

n==

= =

∑

World Applied Sciences Journal 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

ISSN 1818-4952

© IDOSI Publications, 2013

DOI: 10.5829/idosi.wasj.2013.24.itmies.80032

Corresponding Author: Shcherbakov, Volgograd State Technical University, Lenin avenue, 28, 400005, Volgograd, Russia.

171

A Survey of Forecast Error Measures

Maxim Vladimirovich Shcherbakov, Adriaan Brebels,

Nataliya Lvovna Shcherbakova, Anton Pavlovich Tyukov,

Timur Alexandrovich Janovsky and Valeriy Anatol’evich Kamaev

Volgograd State Technical University, Volgograd, Russia

Submitted: Aug 7, 2013; Accepted: Sep 18, 2013; Published: Sep 25, 2013

Abstract: This article reviews the common used forecast error measurements. All error measurements have been

joined in the seven groups: absolute forecasting errors, measures based on percentage errors, symmetric errors,

measures based on relative errors, scaled errors, relative measures and other error measures. The formulas are

presented and drawbacks are discussed for every accuracy measurements. To reduce the impact of outliers, an

Integral Normalized Mean Square Error have been proposed. Due to the fact that each error measure has the

disadvantages that can lead to inaccurate evaluation of the forecasting results, it is impossible to choose only

one measure, the recommendations for selecting the appropriate error measurements are given.

Key words: Forecasting Forecast accuracy Forecast error measurements

INTRODUCTION

Different criteria such as forecast error measurements,

the speed of calculation, interpretability and others have where - y is the measured value at time t, - predicted

been used to assess the quality of forecasting [1-6].

Forecast error measures or forecast accuracy are the most

important in solving practical problems [6]. Typically, the

common used forecast error measurements are applied for

estimating the quality of forecasting methods and for

choosing the best forecasting mechanism in case of

multiple objects. A set of "traditional" error measurements

in every domain is applied despite on their drawbacks.

These error measurements are used as presets in domains

despite on drawbacks.

This paper provides an analysis of existing and quite

common forecast error measures that are used in

forecasting [4, 7-10]. Measures are divided into groups

according to the calculating method an value of error for

certain time t. The calculating formula, the description of

the drawbacks, the names of assessments are considered

for each error measure.

A Review

Absolute Forecasting Error: The first group is based on

the absolute error calculation. It includes estimates based

on the calculation of the value ei

(1)

t

value at time t, obtained from the use of the forecast

model m. Hereinafter referred to as the index of the model

(m) will be omitted.

Mean Absolute Error, MAE is given by:

(2)

where n –forecast horizon, mean(•) – a mean operation.

Median Absolute Error, MdAE is obtained using the

following formula

(3)

where mean(•) – operation for calculation of a median.

Mean Square Error, MSE is calculated by the formula

(4)

() ()

22

11,

1n

ii

iin

RMSE e mean e

n==

= =

∑

t

t

t

e

py

=

()

1,

1

1100 100

n

ii

in

i

MAPE p mean p

n=

=

= ⋅= ⋅

∑

(

)

1,

100 i

in

MdAPE median p

=

= ⋅

()

2

1,

10

0,

i

in

RMSPE mean p

=

= ⋅

()

2

1,

10

0,

i

in

RMdSPE median p

=

= ⋅

()

.

t

t

tt

e

syf

=+

()

1,

1

1200 20

0,

n

ii

in

i

sMAPE s mean s

n=

=

= ⋅= ⋅

∑

(

)

1,

20

0.

i

in

sMdAPE median s

=

= ⋅

()

1

1

,

/2

n

ii

ii i

i

yf

msMAPE nyf S

=

−

=++

∑

11

11

,.

11

11

11

ii

S yy y y

ik k

ii

ii

kk

−−

= −=

−−

−−

= =

∑∑

World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

172

hence, Root Mean Square Error, RMSE is calculated as:

(5) We note the following shortcomings.

These error measures are the most popular in various is equal to zero.

domains [8, 9]. However, absolute error measures have the Non-symmetrical issue - the error values differ

following shortcomings. whether the predicted value is bigger or smaller than

The main drawback is the scale dependency [9]. Outliers have significant impact on the result,

Therefore if the forecast task includes objects with particularly if outlier has a value much bigger then

different scales or magnitudes then absolute error the maximal value of the "normal" cases [4].

measures could not be applied. The error measures are biased. This can lead to an

The next drawback is the high influence of outliers in incorrect evaluation of the forecasting models

data on the forecast performance evaluation [11]. performance [15].

So, if data contain an outliers with maximal value

(this is common case in real world tasks), then Symmetric Errors: The criteria which have been included

absolute error measures provide conservative values. in this group are calculated based on the value:

RMSE,MSE have a low reliability: the results could

be different depending on different fraction of data

[4]. (11)

Measures Based on Percentage Errors: Percentage errors The group includes next measures. Symmetric

are calculated based on the value PMean Absolute Percentage Error, sMAPE is calculated

t

(6)

Also these errors are the most common in forecasting

domain. The group of percentage based errors includes and the median mean absolute percentage error

the following errors.

Mean Absolute Percentage Error, MAPE

(7) To avoid the problems associated with the division

Median Absolute Percentage Error, MdAPE is more has been proposed. Their denominators have an

resistant to outliers and calculated according to the additional member:

formula

(14)

(8)

Root Mean Square Percentage Error, RMSPE is

calculated according to:

(9)

and the median percentage error of the quadratic

(10)

Appearance division by zero when the actual value

the actual [12-14].

according to

(12)

(13)

by zero, a modified sMAPE - Modified sMAPE, msMAPE

where .

Developing the idea for the inclusion of an additional

terms, more sophisticated measures was presented [16]:

KL-N, KL-N1, KL-N2, KL-DE1, KL-DE2, IQR

()

*

,

t

t

tt

e

r

yf

=−

*

ft

*

t

tl

fy

−

=

1,

,

i

in

MRAE mean r

=

=

1,

,

i

in

MdRAE median r

=

=

1

2

.

1

1

t

tn

ii

i

e

q

yy

n−

=

=

−

−∑

1,

,

i

in

MASE mean q

=

=

(

)

2

1,

.

i

in

RMSSE mean q

=

=

*

,

MAE

RMAE

MAE

=

*

.

RMSE

RRMSE

RMSE

=

World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

173

The following disadvantages should be noted. If naive model has been chosen then division by zero

Despite its name, this error is also non-symmetric

[13].

Furthermore, if the actual value is equal to forecasted

value, but with opposite sign, or both of these values

are zero, then a divide by zero error occurs.

These criteria are affected by outliers in analogous

with the percentage errors.

If more complex estimations have been used, the

problem of interpretability of results occurs and this

fact slows their spread in practice [4].

Measures Based on Relative Errors: The basis for

calculation of errors in this group is the value determined

as follows:

(15)

where - the predictive value obtained using a reference

model prediction (benchmark model). The main practice is

to use a naive model as a reference model

,

(16)

where l - the value of the lag and l = 1.

The group includes the next measures. Mean Relative

Absolute Error, MRAE is given by the formula

(17)

Median Relative Absolute Error, MRAE is calculated

according to

(18)

and Geometric Mean Relative Absolute Error, GMRAE),

which is calculated similarly to (17), but instead of mean(•)

the geometric mean is obtained gmean(•).

It should be noted the following shortcomings.

Based the formulas (15-18), division by zero error

occurs, if the predictive value obtained by reference

model is equal to the actual value.

error occurs in case of continuous sequence of

identical values of the time series.

Scaled Error: As a basis for calculating the value of the

scaled errors q is given by

i

(19)

This group contains Mean Absolute Scaled Error,

MASE proposed in [9]. It is calculated according to:

(20)

Another evaluation of this group is Root Mean

Square Scaled Error, RMSSE is calculated by the formula

[10]:

(22)

These measures is symmetrical and resistant to

outliers. However, we can point to two drawbacks.

If the forecast horizon real values are equal to each

other, then division by zero occurs.

Besides it is possible to observe a weak bias

estimates if you do the experiments by analogy with

[15].

Relative Measures: This group contains of measures

calculated as a ratio of mentioned above error measures

obtained by estimated forecasting models and reference

models. Relative Mean Absolute Error, RelMAE is

calculated by the formula.

(23)

where MAE and MAE the mean absolute error for the

*

analyzed forecasting model and the reference model

respectively, calculated using the formula (2).

Relative Root Mean Square Error, RelRMSE is

calculated similarly to (23), except that the right side is

calculated by (5)

(24)

*

lo

g.

RMSE

LMR

RMSE

=

{

}

(

)

*

( ) 100

%.

PB MAE mean I MAE MAE=⋅<

*

0

,;

() 1.

if MAE MAE

I MAE <

=

(

)

2

1,

1

,

i

in

nRMSE mean e

y=

=

y

(

)

2

1,

1

1

.

i

n

in

i

i

inRSE mean e

y=

=

=

∑

()

()

2

1

2

1

,

n

i

i

n

i

i

e

inRSE

yy

=

=

=

−

∑

∑

1

1

n

yy

k

n

k

=

=

∑

()

2

1

_1

n

i

i

e MAE

Std AE n

=

−

=−

∑

()

2

1

_1

n

i

i

p MAPE

Std APE n

=

−

=−

∑

World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

174

In some situations it is reasonable to calculate the over the entire interval or time horizon or defined

logarithm of the ratio (23). In this case, the measure is

called the Log Mean Squared Error Ratio, (LMR)

(25)

Syntetos et al. proposed a more complex assessment

of the relative geometric standard deviation Relative

Geometric Root Mean Square Error, RGRMSE [17].

The next group of measures counts the number of

cases where the error of the model prediction error is

greater than the reference model. For instance, PB (MAE)

- Percentage Better (MAE), calculated by the formula:

(26)

where I{•} - the operator that yields the value of zero or

one, in accordance with the expression:

(27)

By analogy with PB (MAE), Percentage Better (MSE)

can be defined.

The disadvantages of these measures are the

following.

Division by zero error occurs if the reference forecast

error is equal to zero.

These criteria determine the number of cases when

the analyzed forecasting model superior to the base

but do not evaluate the value of difference.

Other Error Measures: This group includes measures

proposed in various studies to avoid the shortcomings of

existing and common measures.

To avoid the scale dependency, Normalized Root

Mean Square Error (nRMSE) has been proposed,

calculated by the formula:

(28)

where - the normalization factor, which is usually equal

to either the maximum measured value on the forecast

horizon, or the difference between the maximum and

minimum values. Normalization factor can be calculated

short interval of observation [18]. However, this

estimate is affected by influence of outliers, if outlier has

a value much bigger the maximal "normal" value. To

reduce the impact of outliers, Integral Normalized Mean

Square Error [19] have been proposed, calculated by the

formula:

(29)

Some research contains the the ways of NRMSE

calculation as [16]:

(30)

where .

Other measures are called normalized std_APE and

std_MAPE [20, 21] and calculated by the formula

(31)

and

(32)

respectively.

As a drawback, you can specify a division by zero

error if normalization factor is equal to zero.

Recommendations How to Choose Error Measures:

One of the most difficult issues is the question of

choosing the most appropriate measures out of the

groups. Due to the fact that each error measure has the

disadvantages that can lead to inaccurate evaluation of

the forecasting results, it is impossible to choose only one

measure [5].

175

We provide the following guidelines for choosing the ACKNOWLEDGMENTS

error measures.

If forecast performance is evaluated for time series research (Grants #12-07-31017, 12-01-00684).

with the same scale and the data preprocessing

procedures were performed (data cleaning, anomaly REFERENCES

detection), it is reasonable to choose MAE, MdAE,

RMSE. In case of different scales, these error 1. Tyukov, A., A. Brebels, M. Shcherbakov and

measures are not applicable. The following V. Kamaev, 2012. A concept of web-based energy

recommendations are provided for mutli-scales cases. data quality assurance and control system. In the

In spite of the fact that percentage errors are Proceedings of the 14th International Conference on

commonly used in real world forecast tasks, but due Information Integration and Web-based Applications

to the non-symmetry, they are not recommended. & Services, pp: 267-271.

If the range of the values lies in the positive 2. Kamaev, V.A., M.V. Shcherbakov, D.P. Panchenko,

half-plane and there are no outliers in the data, it is N.L. Shcherbakova and A. Brebels, 2012. Using

advisable to use symmetric error measures. Connectionist Systems for Electric Energy

If the data are "dirty", i.e. contain outliers, it is Consumption Forecasting in Shopping Centers.

advisable to apply the scaled measures such as Automation and Remote Control, 73(6): 1075-1084.

MASE, inRSE. In this case (i) the horizon should be 3. Owoeye, D., M. Shcherbakov and V. Kamaev, 2013.

large enough, (ii) no identical values should be, (iii) A photovoltaic output backcast and forecast method

the normalized factor should be not equal to zero. based on cloud cover and historical data. In the

If predicted data have seasonal or cyclical patterns, Proceedings of the The Sixth IASTED Asian

it is advisable to use the normalized error measures, Conference on Power and Energy Systems (AsiaPES

wherein the normalization factors could be calculated 2013), pp: 28-31.

within the interval equal to the cycle or season. 4. Armstrong, J.S. and F. Collopy, 1992. Error measures

If there is no results of prior analysis and a-prior for generalizing about forecasting methods: Empirical

information about the quality of the data, it comparisons. International Journal of Forecasting,

reasonable to use the defined set of error measures. 8(1): 69-80.

After calculating, the results are analyzed with 5. Mahmoud, E., 1984. Accuracy in forecasting: A

respect to division by zero errors and contradiction survey. Journal of Forecasting, 3(2): 139-159.

cases: 6. Yokuma, J.T. and J.S. Armstrong, 1995. Beyond

For the same time series the results for model maccuracy: Comparison of criteria used to select

1

is better than m, based on the one error forecasting methods. International Journal of

2

measure, but opposite for another one; Forecasting, 11(4): 591-597.

For different time series the results for model m7. Armstrong, J.S., 2001. Evaluating forecasting

1

is better in most cases, but worst for a few of methods. In Principles of forecasting: a handbook for

cases. researchers and practitioners. Norwell, MA: Kluwer

CONCLUSION 8. Gooijer, J.G.D. and R.J. Hyndman, 2006. 25 years of

The review contains the error measures for time series Forecasting, 22(3): 443-473.

forecasting models. All these measures are grouped into 9. Hyndman, R.J. and A.B. Koehler, 2006. Another look

seven groups: absolute forecasting error, percentage at measures of forecast accuracy. International

forecasting error, symmetrical forecasting error, measures Journal of Forecasting, 22(4): 679-688.

based on relative errors, scaled errors, relative errors and 10. Theodosiou, M., 2011. Forecasting monthly

other (modified). For each error measure the way of and quarterly time series using STL

calculation is presented. Also shortcomings are defined decomposition. International Journal of Forecasting,

for each of group. 27(4): 1178-1195.

Authors would like to thank RFBR for support of the

Academic Publishers, pp: 443-512.

time series forecasting. International Journal of

176

11. Shcherbakov, M.V. and A. Brebels, 2011. Outliers and 18. Tyukov, A., M. Shcherbakov and A. Brebels, 2011.

anomalies detection based on neural networks Automatic two way synchronization between server

forecast procedure. In the Proceedings of the 31 and multiple clients for HVAC system. In the

st

Annual International Symposium on Forecasting, Proceedings of The 13th International Conference on

ISF-2011, pp: 21-22. Information Integration and Web-based Applications

12. Goodwin, P. and R. Lawton, 1999. On the asymmetry & Services, pp: 467-470.

of the symmetric MAPE. International Journal of 19. Brebels, A., M.V. Shcherbakov and V.A. Kamaev,

Forecasting, 15(4): 405-408. 2010. Mathematical and statistical framework for

13. Koehler, A.B., 2001. The asymmetry of the sAPE comparison of neural network models with other

measure and other comments on the M3-competition. algorithms for prediction of Energy consumption in

International Journal of Forecasting, 17: 570-574. shopping centres. In the Proceedings of the 37 Int.

14. Makridakis, S., 1993. Accuracy measures: Theoretical Conf. Information Technology in Science Education

and practical concerns. International Journal of Telecommunication and Business, suppl. to Journal

Forecasting, 9: 527-529. Open Education, pp: 96-97.

15. Kolassa, S. and R. Martin, 2011. Percentage errors 20. Casella, G. and R. Berger, 1990. Statistical inference.

can ruin your day (and rolling the dice shows how). 2nd ed. Duxbury Press, pp: 686.

Foresight, (Fall): 21-27. 21. Kusiak, A., M. Li and Z. Zhang, 2010. A data-driven

16. Assessing Forecast Accuracy Measures. Date View approach for steam load prediction in buildings.

01.08.2013 http:// www.stat.iastate.edu/ preprint/ Applied Energy, 87(3): 925-933.

articles/2004-10.pdf

17. Syntetos, A.A. and J.E. Boylan, 2005. The accuracy

of intermittent demand estimates. International

Journal of Forecasting, 21(2): 303-314.