PosterPDF Available

Forecasting models for time-series: a comparative study between classical methodologies and Deep Learning

Authors:
  • FCUL - Faculdade de Ciências Universidade de Lisboa

Abstract

In a year where the word “forecast" has been extensively used, it's more important than ever to have accurate forecasting models. In particular, in economics, finance and business areas; forecasting techniques are used to support enterprises to decide future directions, which determine the success of the same enterprises. However, in order for the forecasting techniques to be efficient, these must be truly understood and tested in real data-driven context, by taking in account existing models and new approaches. Based on the scientific literature, the classical methodologies are the most utilised by professionals, the autoregressive moving average (e.g. ARMA) and the exponential smoothing models (e.g. ETS), are the classical methodologies which are the most utilised by professionals. Nonetheless, due to promising results, the literature has been keen on Deep Learning methodologies, in particular Deep Neural Networks (DNN). In fact, investigating what type of models should be used for each time-series based on their characteristics is the goal of this work. Three distinct models – ARMA, ETS and DNN – are assessed in the forecast of time-series with distinct patterns (see https://github.com/DidierRLopes/UnivariateTimeSeriesForecast). The discussion of the results will take into account not only the forecasting ability, but also its interpretability and computational cost. This study shows that the additional computational power required in more complex models may not justify the improved accuracy. Although in time-series with strong perturbations, advantages are recognised in DNN models (lower prediction error), in series with a clear trend and/or seasonality, classical methodologies (e.g. ETS) outweighs the former.
Didier Lopes
UNL and CEDOC-UNL
dro.lopes@campus.fct.unl.pt
Contacts
1. Aue, A., & Horváth, L. (2013). Structural breaks in time series. Journal of Time Series Analysis,34(1), 1-16.
2. Costa, A., Ramos, F., Mendes, D., & Mendes, V. (2019). Forecasting financial time series using deep learning techniques. In IO 2019 -XX Congresso da APDIO 2019. Instituto Politécnico de Tomar - Tomar.
3. Greff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., & Schmidhuber, J. (2015). LSTM: A Search Space Odyssey. IEEE Transactions on Neural Networks and Learning Systems,28(10), 22222232.
4. Hang, N. T. (2019). Research on a number of applicable forecasting techniques in economic analysis, supporting enterprises to decide management. World Scientific News,119,5267.
5. Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. In ICML - International Conference on Machine Learning.
6. Kirchgässner, G., & Wolters, J. (2007). Introduction to Modern Time Series Analysis (Springer). Berlin.
7. Lopes, D., Ramos, F. (2020) Univariate Time Series Forecast. Retrieved from https://github.com/DidierRLopes/UnivariateTimeSeriesForecast.
8. Ramos, F., Costa, A., Mendes, D., & Mendes, V. (2018). Forecasting financial time series: a comparative study. In JOCLAD 2018, XXIV Jornadas de Classificação e Análise de Dados. Escola Naval Alfeite.
9. Ravichandiran, S. (2019). Hands-On Deep Learning Algorithms with Python: Master deep learning algorithms with extensive math by implementing them using TensorFlow. Packt Publishing Ltd.
10. Tkác, M., Verner, R. (2016) Articial neural networks in business: Two decades of research. Applied Soft Computing,38,788-804. doi:10.1016/J.ASOC.2015.09.040
11. Wilson, J. H., Spralls III, S. A. (2018) What do business professionals say about forecasting in the marketing curriculum?. International Journal of Business, Marketing, Decision Science.11(1), 1-20.
12. Zhang, Y., Guo, Q., & Wang, J. (2017). Big data analysis using neural networks, 49, 918.
References
In a year where the word “forecast” has been
extensively used, it’s more important than ever to
have accurate forecasting models. In particular, in
economic-finance and business areas; these types of
forecasting techniques are used to support
enterprises to decide a company’s direction, which
helps determine the success of these same
enterprises. However, in order for the forecasting
techniques to be efficient, these must be truly
understood and tested in real context, not only to
improve already existing models but also test new
approaches. From the methodologies most cited in
the scientific literature, the classical models, such as
autoregressive moving average (ARMA) and
exponential smoothing (ETS), are the most utilised by
professionals. Nonetheless, due to promising results,
the literature has been keen on Deep Learning
methodologies, in particular Deep Neural Networks
(DNN). In fact, investigating what type of models
should be used for each time-series based on their
characteristics is the goal of this work. Three distinct
models ARMA, ETS and DNN are assessed in the
forecast of time-series with distinct patterns. The
discussion of the results will take into account not
only the forecasting ability, but also its interpretability
and computational cost.
Keywords: ARMA, DNN, ETS, forecasting, time-series
Introduction
Models
Forecast and Prediction error
Considering the time series  and the past
observations from period  , and being  an
unknown value in the future and  its
forecast, the prediction error corresponds to,
  
- Mean Absolute Error (MAE):  

- Mean Absolute Percentage Error (MAPE):


 


Where corresponds to the forecasting window, i.e.
number of observation in the forecasting samples
Computational implementation
For classical models, we relied on Akaike Information
Criteria (AIC), Bayesian Information Criterion (BIC)
and HannanQuinn Information Criterion (HQIC) to
assess the relative quality of the statistical model for
several different parameters for a given set of data.
For neural networks, we relied on the following cross-
validation and tuning methodology (Figure 2).
Methodology
Conclusion
Autoregressive Moving Average (ARMA)
-󰇛 󰇜 models: combination of an
Autoregressive model of order , , with a
Moving Average model of order ,󰇛󰇜.
-  models: ARIMA with an integration
component ,until the time-series achieves
stationarity.
-   models: seasonal
ARIMA used when time-series shows seasonality.
Exponential Smoothing (ETS)
Model that explicitly uses an exponentially decreasing
weight for past observations. These may contain a
level, trend and season component. In addition, the
model may be additive or multiplicative.
Deep Neural Networks (DNN)
Within DNN there are three main architectures,
where their difference is in the learning process, and
occurs at the neuron level (Figure 1).
.
- Multilayer Perceptron (MLP): Static model, where
each neuron receives an input vector
and a bias.
- Recurrent Neural Network (RNN): Dynamic model,
where in each training epoch, t,the neuron is fed by
an input vector
and a learning input  that is a
result from the previous training epoch, t-1.
- Long-Short Term Memory (LSTM): Based on RNN,
LSTMs utilize gates at the neuron level to allow for
long-term memory, for past data deemed important.
Figure 1. Comparison of a hidden cells of: MLP, RNN and LSTM
CPIAUSCL:Consumer Price Index for All Urban
Consumers All Items in U.S. City Average (873
monthly observations Figure 3 (A)).
VMT:Vehicle-Miles Travelled (597 monthly
observations Figure 3 (B))
PSI20:Portuguese Stock Index 20 (4534 daily
observations Figure 3 (C))
SPY:Standard & Poor's 500 Exchange-Traded Fund
(6715 daily observations Figure 3 (D))
The time-series represent a time period up of
September of 2019 (CPIAUSCL/VMT up to the 1st and
PSI20/SPY up to 27th). These are distinguished by
presenting different behaviors in terms of trend
and/or seasonality. However, all time-series are non-
stationary (ADF and KPSS Test), not i.i.d (BDS Test)
and a non-normal distribution (Jarque-Bera Test).
Results
In time series with strong perturbations, the
advantages of DNN models are evident.
In time series with a clear trend and/or
seasonality, the ARMA and ETS models are
adequate (and have low computational cost).
The additional computational power required in
more complex models is not always justified (e.g.
time-series CPIAUCSL and VMT).
Forecasting models for time-series:
a comparative study between classical
methodologies and Deep Learning
D.R. Lopes1; F.R. Ramos2; D.A. Mendes2; A.R. Costa3
1UNL and CEDOC-UNL, 2ISCTE-IUL and BRU-IUL, 3ISCTE-IUL and CMAF-CIO
Data
Figure 3. Time-series
Filipe Ramos
ISCTE-IUL and BRU-IUL
frjrs@iscte-iul.pt

 

RNNMLP




LSTM
250
200
150
50
300
Closing Price ($)
Business Days/Year
100
1992 1996 2000 2004 2008 20202012 2016
(A) CPIAUCSL (B) VMT
(C) PSI 20 (D) SPY
Past and future observations
290000
280000
270000
260000
230000
250000
240000
Vehicle Miles Travelled (millions)
-13 -5-22 5 8 12
-12 -11 -10 -9-8-7 -6 -4 -3 -1 0 1 3 4 6 7 9 10 11
SARIMA(1,1,1)n(1,1,1,12)
SARIMA(1,1,1)n(1,1,2,12)
SARIMA(1,1,4)n(1,1,2,12)
SARIMA(1,1,4)n(2,1,2,12)
SARIMA(4,1,1)n(1,1,2,12)
SARIMA(4,1,1)n(1,1,3,12)
-14
Real data
Past and future business days
-14 -12 0 6 14 20
-10 -8-6-4 -2 2 4 8 10 12 16 18
Closing Price ($)
300.0
297.5
295.0
292.5
285.0
290.0
287.5
ARIMA(1,1,1)
ARIMA(1,1,2)
ARIMA(4,1,2)
ARIMA(4,1,3)
ARIMA(4,1,4)
ARIMA(2,1,1)
Real data
4700
-14
5050
5000
4950
4900
4750
4850
4800
Past and future business days
-12 0 6 14 20
-10 -8-6-4 -2 2 4 8 10 12 16 18
Closing Price (€)
ARIMA(1,1,1)
ARIMA(2,1,1)
ARIMA(1,1,2)
ARIMA(3,1,3)
ARIMA(4,1,2)
ARIMA(4,1,4)
Real data
Past and future observations
258
256
254
252
246
244
250
248
Consumer Price Index
-14 -13 -5-22 5 8 12
-12 -11 -10 -9-8-7 -6 -4 -3 -1 0 1 3 4 6 7 9 10 11
ARIMA(1,1,2)
ARIMA(1,1,3)
ARIMA(3,1,1)
ARIMA(3,1,4)
ARIMA(4,1,3)
ARIMA(4,1,4)
Real data
(B)
(A)
(D)
(C)
-14 Past and future business days
-12 0 6 14 20
-10 -8-6-4 -2 2 4 8 10 12 16 18
4700
5050
5000
4950
4900
4750
4850
4800
Consumer Price Index
Past and future observations
256
254
252
246
244
250
248
-14 -13 -5-22 5 8 12
-12 -11 -10 -9-8-7 -6 -4 -3 -1 0 1 3 4 6 7 9 10 11
Real data
ETS (A, N)
ETS (Ad, N)
ETS (A, M)
290000
280000
270000
260000
230000
250000
240000
-5-22 5 8 12
-11 -10 -9-8-7 -6 -4 -3 -1 0 1 3 4 6 7 9 10 11
Vehicle Miles Travelled (millions)
Past and future observations
Real data
ETS (A, A)
ETS (Ad, A)
ETS (A, M)
ETS (Ad, M)
Closing Price (€)
ETS (N, N)
ETS (A, N)
ETS (Ad, N)
Real data
Past and future business days
-14 -12 0 6 14 20
-10 -8-6-4 -2 2 4 8 10 12 16 18
Closing Price ($)
300.0
297.5
295.0
292.5
285.0
290.0
287.5
Real data
ETS (A, N)
ETS (Ad, N)
(B)
(A)
(D)
(C)
Consumer Price Index
255
250
240
235
245
Past and future observations 0
Real data
In-Sample Forecast
Out-of-Sample Forecast
(A)
300000
280000
260000
220000
240000
Vehicle Miles Travelled (millions)
Past and future observations 0
(B)
Real data
In-Sample Forecast
Out-of-Sample Forecast
4700
5400
5300
5200
5100
4800
5000
4900
Closing Price (€)
Past and future observations 0
(C)
Real data
In-Sample Forecast
Out-of-Sample Forecast
305
300
295
290
275
285
280
Closing Price ($)
Past and future observations 0
(D)
Real data
In-Sample Forecast
Out-of-Sample Forecast
300000
280000
260000
220000
240000
Vehicle Miles Travelled (millions)
Past and future observations 0
Real data
In-Sample Forecast
Out-of-Sample Forecast
4700
5400
5300
5200
5100
4800
5000
4900
Closing Price (€)
Past and future observations 0
Real data
In-Sample Forecast
Out-of-Sample Forecast
305
300
295
290
275
285
280
Closing Price ($)
Past and future observations 0
Real data
In-Sample Forecast
Out-of-Sample Forecast
Consumer Price Index
255
250
240
235
245
Past and future observations 0
Real data
In-Sample Forecast
Out-of-Sample Forecast
(A)
(B)
(C)
(D)
ARMA Models ETS Models
MLP Models LSTM Models
Figure 2. Methodology for computational implementation (DNN)
Figure 4. Models fitting and forecasting of time-series:
(A) CPIAUCSL; (B) VMT; (C) PSI 20; (D) SPY
CPIAUCSL
VMT
SARIMA(4,1,1) × (1,1,3)
(:  / =12)
PSI 20
SPY
ARIMA(3,1,4)
ARIMA(4,1,4)
ARIMA(4,1,2)
Month
Quarter
Year
Month
Quarter
Year
Day
Week
Month
Day
Week
Month
0.12%
0.15%
0.37%
1.3%
0.9%
0.98%
1.45%
2.16%
3.29%
1.18%
0.91%
2.63%
(Time-
serie)
CPIAUCSL
VMT
PSI 20
SPY
(Model)
󰇛,󰇜
󰇛,󰇜
󰇛,󰇜
󰇛,󰇜
= 1 , = 0.12
= 0.37, = 0, = 1
= 1, = 0.29,= 0.24
= 0.95 , = 0
= 0.33, =12
Month
Quarter
Year
Month
Quarter
Year
Day
Week
Month
Day
Week
Month
MAPE
0.14%
0.14%
0.19%
0.74%
0.75%
0.92%
1.49%
2.12%
3.15%
1.28%
1.57%
2.83%
Table 1. Predictions errors of the ARMA models
Table 2. Predictions errors of the ETS models
* Minimum values - Maximum values (trimmed by 5%) obtained in a total of 60 runs
(Time-
Serie)
CPIAUCSL
VMT
PSI 20
SPY
Model




Month
Quarter
Year
Month
Quarter
Year
Day
Week
Month
Day
Week
Month
MAPE(*)
0.13%
0.26%
0.13%
0.27%
0.18%
0.29%
0.21%
0.38%
0.27%
0.51%
0.48%
0.82%
0.83%
1.39%
1.31%
1.84%
1.57%
2.01%
0.55%
0.94%
0.60%
1.03%
1.02%
1.82%
Table 3. Predictions errors of the MLP models
Table 4. Predictions errors of LSTM models
* Minimum values - Maximum values (trimmed by 5%) obtained in a total of 20 runs
(Time-
Serie)
CPIAUCSL
VMT
PSI 20
SPY
Model




Month
Quarter
Year
Month
Quarter
Year
Day
Week
Month
Day
Week
Month
MAPE(*)
0.11%
0.21%
0.12%
0.19%
0.16%
0.24%
0.04%
0.10%
0.16%
0.25%
0.35%
0.71%
0.87%
1.41%
1.19%
1.63%
0.97%
1.52%
0.48%
0.83%
0.50%
0.96%
0.79%
1.12%

Supplementary resource (1)

... For example, regarding stock price forecasting, Costa et al. (2019), Lopes et al. (2021) and Ramos et al. (2018) report that some Recurrent Neural Networks (RNN) models, e.g., Long Short-Term Memory networks-LSTM, can be promising for modeling and forecasting time series with structure breaks, or with very irregular behavior (such as time series related to financial markets). However, despite the good forecasting quality, Lopes et al. (2021) and Ramos et al. (2021) note that these neural network architectures have a significant computational cost. ...
... For example, regarding stock price forecasting, Costa et al. (2019), Lopes et al. (2021) and Ramos et al. (2018) report that some Recurrent Neural Networks (RNN) models, e.g., Long Short-Term Memory networks-LSTM, can be promising for modeling and forecasting time series with structure breaks, or with very irregular behavior (such as time series related to financial markets). However, despite the good forecasting quality, Lopes et al. (2021) and Ramos et al. (2021) note that these neural network architectures have a significant computational cost. Due to the facts mentioned by these authors, further reflection is important, combining the prediction power and computational cost of DNN models. ...
... These results are consistent with literature, which suggests that LSTM architecture may underperform when Deep learning methodologies seem to show advantages over classical methodologies in terms of forecast quality, since nonlinear dependencies of the data are better captured. However, it is noteworthy that these models are associated with considerably higher computational costs and greater implementation complexity compared to classical techniques (corroborating with the scientific literature- Lopes et al., 2021 andRamos et al., 2021). Despite these limitations, implementation of deep learning models in the present study yielded a substantial reduction in prediction errors. ...
Article
Full-text available
This study aims to evaluate forecasting properties of classic methodologies (ARCH and GARCH models) in comparison with deep learning methodologies (MLP, RNN, and LSTM architectures) for predicting Bitcoin's volatility. As a new asset class with unique characteristics, Bitcoin's high volatility and structural breaks make forecasting challenging. Based on 2753 observations from 08-09-2014 to 01-05-2022, this study focuses on Bitcoin logarithmic returns. Results show that deep learning methodologies have advantages in terms of forecast quality, although significant computational costs are required. Although both MLP and RNN models produce smoother forecasts with less fluctuation, they fail to capture large spikes. The LSTM architecture, on the other hand, reacts strongly to such movements and tries to adjust its forecast accordingly. To compare forecasting accuracy at different horizons MAPE, MAE metrics are used. Diebold–Mariano tests were conducted to compare the forecast, confirming the superiority of deep learning methodologies. Overall, this study suggests that deep learning methodologies could provide a promising tool for forecasting Bitcoin returns (and therefore volatility), especially for short-term horizons.
... However, as the complexity of the data increases, more sophisticated modeling approaches are required to capture the dynamic and complex nature of the time series data. On the other hand, given the limitations pointed out in the scientific literature for classical methodologies [ 4,6], several new approaches in the field of artificial intelligence have contributed a lot to advances in predictive analysis [ 7]. Under this paradigm, Artificial Neural Networks (ANN), in particular Deep Neural Networks (DNN), have been pointed out in the scientific literature as a very promising option [ 2,8,9]. ...
... In addition to the implementation of the routines referring to the BJ-DNN model, only forecast values were obtained from the LSTM model (resulting from the LSTM architecture). According to the literature, LSTM models show better predictive quality in data with disturbances in historical data, compared to models resulting from the MLP and RNN architectures [ 6,26]. In each case, a multi-grid was used to explore several possible combinations to define an accurate model. 1 To evaluate a model, it is necessary to compare its forecasted values against the actual price data that the model has not encountered before (test set). ...
Chapter
Forecasting plays a crucial role in enhancing the efficiency and effectiveness of logistics and supply chain management in the healthcare sector, particularly in financial management within healthcare facilities. Modeling and forecasting techniques serve as valuable tools in this domain, with Artificial Neural Networks (ANN), especially Deep Neural Networks (DNN), emerging as promising options, as indicated by the scientific literature. Furthermore, combining ANN with other methodologies has been a subject of frequent discussion. This study builds on previous research that used historical data to predict expenditure on medicines in Portuguese NHS Hospitals. The focus of this study is to evaluate advantages of Deep Learning methodologies. In addition to traditional approaches as Exponential Smoothing (ES), hybrid models are explored, specifically the combination of DNN with the Box and Jenkins methodology (BJ-DNN). A comparative analysis is conducted to assess the predictive quality and computational cost associated with the forecast models. The findings reveal that ES models provide overall suitability and low computational cost. However, considering the Mean Absolute Percentage Error (MAPE), the BJ-DNN model demonstrates robust forecasting accuracy. In conclusion, this study highlights the potential of Deep Learning methodologies for improving expenditure forecasting in healthcare, while maintaining the favorable attributes of traditional Exponential Smoothing models. These findings contribute to the broader understanding of forecasting techniques in the healthcare sector.
Chapter
Full-text available
The articulation of statistics, mathematical and computational techniques for modelling and forecasting of time series can help in the decision-making process. When dealing with the intrinsic challenges for financial time series, Machine Learning methodologies, in particular Deep Learning, was pointed out as being a promising option. Previous works, highlight the potential of Deep Neural Network architectures, but also their limitations with regards to computational complexity. Some of these limitations are analysed in this work, where a hybrid approach is proposed in order to benefit from the knowledge and solidity of Box&Jenkins methodologies and the viability of applying robust cross-validation of the neural network – Group k-Fold. Through the construction of complete and automated computational routines, the proposed model is tested with the modelling of two financial time series with disturbances on their historical data: Portuguese Stock Index 20 (PSI 20) and Standard & Poor’s 500 Exchange-Traded Fund (SPY). The approach is compared to neural network models with Multilayer Perceptron and Long Short-Term Memory architectures. Besides reducing the implicit computational time by 20%, by considering Mean Absolute Percentage Errors, the proposed model shows forecasting quality.
ResearchGate has not been able to resolve any references for this publication.