ArticlePDF Available

Abstract and Figures

We perform an extensive comparison between four stochastic and two machine learning (ML) forecasting algorithms by conducting a multiple-case study. The latter is composed by 50 single-case studies, which use time series of total monthly precipitation and mean monthly temperature observed in Greece. We apply a fixed methodology to each individual case and, subsequently, we perform a cross-case synthesis to facilitate the detection of systematic patterns. The stochastic algorithms include the Autoregressive order one model, an algorithm from the family of Autoregressive Fractionally Integrated Moving Average models, an Exponential Smoothing State Space algorithm and the Theta algorithm, while the ML algorithms are Neural Networks and Support Vector Machines. We also use the last observation as a Naïve benchmark in the comparisons. We apply the forecasting methods to the deseasonalized time series. We compare the one-step ahead as also the multi-step ahead forecasting properties of the algorithms. Regarding the one-step ahead forecasting properties, the assessment is based on the absolute error of the forecast of the last observation. For the comparison of the multi-step ahead forecasting properties we use five metrics applied to the test set (last twelve observations), i.e. the root mean square error, the Nash-Sutcliffe efficiency, the ratio of standard deviations, the index of agreement and the coefficient of correlation. Concerning the ML algorithms, we also perform a sensitivity analysis for time lag selection. Additionally, we compare more sophisticated ML methods as regards to the hyperparameter optimization to simple ones.
Content may be subject to copyright.
European Water 59: 161-168, 2017.
© 2017 E.W. Publications
Forecasting of geophysical processes using stochastic and machine learning
algorithms
G.A. Papacharalampous*, H. Tyralis and D. Koutsoyiannis
Department of Water Resources and Environmental Engineering, School of Civil Engineering, National Technical University of
Athens, Iroon Polytechniou 5, 157 80 Zografou, Greece
* e-mail: papacharalampous.georgia@gmail.com
Abstract: We perform an extensive comparison between four stochastic and two machine learning (ML) forecasting algorithms
by conducting a multiple-case study. The latter is composed by 50 single-case studies, which use time series of total
monthly precipitation and mean monthly temperature observed in Greece. We apply a fixed methodology to each
individual case and, subsequently, we perform a cross-case synthesis to facilitate the detection of systematic patterns.
The stochastic algorithms include the Autoregressive order one model, an algorithm from the family of
Autoregressive Fractionally Integrated Moving Average models, an Exponential Smoothing State Space algorithm
and the Theta algorithm, while the ML algorithms are Neural Networks and Support Vector Machines. We also use
the last observation as a Naïve benchmark in the comparisons. We apply the forecasting methods to the
deseasonalized time series. We compare the one-step ahead as also the multi-step ahead forecasting properties of the
algorithms. Regarding the one-step ahead forecasting properties, the assessment is based on the absolute error of the
forecast of the last observation. For the comparison of the multi-step ahead forecasting properties we use five metrics
applied to the test set (last twelve observations), i.e. the root mean square error, the Nash-Sutcliffe efficiency, the ratio
of standard deviations, the index of agreement and the coefficient of correlation. Concerning the ML algorithms, we
also perform a sensitivity analysis for time lag selection. Additionally, we compare more sophisticated ML methods
as regards to the hyperparameter optimization to simple ones.
Key words: Stochastic algorithms, machine learning algorithms, multiple-case study, cross-case synthesis, multi-step ahead
forecasting, one-step ahead forecasting, time lag selection, hyperparameter optimization
1. INTRODUCTION
Machine learning (ML) algorithms are widely used for the forecasting of geophysical processes
as an alternative to stochastic algorithms. Popular ML algorithms are the rather well established
Neural Networks (NN) and the new-entrant in most scientific fields Support Vector Machines
(SVM). The latter was presented in its current form by Cortes and Vapnik (1995; see also Vapnik,
1995, 1999). The large number of the relevant applications is imprinted in Maier and Dandy (2000)
and Raghavendra and Deka (2014).
As a result, the research in geophysical sciences often focuses on comparing stochastic to ML
forecasting algorithms. The comparisons performed are usually based on single-case studies (e.g.
Koutsoyiannis et al., 2008; Valipour et al., 2013), which offer the benefit of studying the
phenomena in detail as also in their context and thus can provide interesting insights. On the other
hand, single-case studies do not allow generalizations to any extent (Achen and Snidal, 1989).
Generalizations could be derived by examining a sufficient number of different cases, as
implemented in Papacharalampous et al. (2017). Within the latter study large-scale computational
experiments based on simulations are conducted to compare several stochastic and ML methods
regarding their multi-step ahead forecasting properties. A statistical analysis is performed and the
results are presented accordingly.
Here we conduct a multiple-case study composed by 50 individual cases, each of them based on
geophysical time series data from Greece. We apply a fixed methodology to each individual case
for the comparison between several stochastic and ML methods regarding their one-step ahead and
G.A. Papacharalampous et al.
162
multi-step ahead forecasting properties. Concerning the ML methods, we also perform a sensitivity
analysis for time lag selection. Additionally, we compare more sophisticated ML methods as
regards to the hyperparameter optimization to simple ones. Finally, we perform a cross-case
synthesis to facilitate the detection of systematic patterns. We believe that the multiple-case study
method can be useful for the comparative assessment of forecasting methods, as it can provide a
form of generalization named “contingent empirical generalization”, while retaining the immediacy
of the single-case study method (Achen and Snidal, 1989).
2. DATA AND METHODS
2.1 Time series
We use 50 time series of total monthly precipitation and mean monthly temperature observed in
Greece. We select only those with few missing values (blocks with length equal or less than one).
Subsequently, we use the Kalman filter algorithm from the zoo R package (Zeileis and
Grothendieck, 2005) for filling in the missing values. We use the deseasonalized time series for the
application of the forecasting methods for the improvement of the forecasting quality, as suggested
in Taieb et al. (2012). The deseasonalization is performed using a multiplicative model of time
series decomposition.
The basic information about the time series is provided in Table 1. To describe the long-term
persistence of the deseasonalized time series, we estimate the Hurst parameter H for each of them
using the maximum likelihood method (Tyralis and Koutsoyiannis, 2011) implemented with the
HKprocess R package (Tyralis, 2016).
2.2 Forecasting methods
We use four stochastic and two ML forecasting algorithms. The stochastic algorithms include the
Autoregressive order one model (AR(1)), an algorithm from the family of Autoregressive
Fractionally Integrated Moving Average models (auto_ARFIMA), an Exponential Smoothing State
Space algorithm (BATS) and the Theta algorithm. The ML algorithms are Neural Networks (NN)
and Support Vector Machines (SVM). We also use the last observation as a Naïve benchmark in the
comparisons. We apply the stochastic algorithms using the forecast R package (Hyndman and
Khandakar, 2008; Hyndman et al., 2016) and the ML using the rminer R package (Cortez, 2010,
2015). The Naïve, AR(1), auto_ARFIMA and BATS algorithms apply Box-Cox transformation to
the input data before fitting a model to them.
While the stochastic forecasting methods are simply defined by the stochastic algorithm, the ML
methods are defined by the set {ML algorithm, hyperparameter selection procedure, time lags}. We
compare two procedures for hyperparameter selection, i.e. predefined hyperparameters or defined
after optimization, and 21 regression matrices, each using the first n time lags, n = 1, 2, …, 21. The
hyperparameter optimization is performed with the hold-out method.
Hereafter, we consider that the ML models are used with predefined hyperparameters and that
the regression matrix is built only by the first time lag, unless mentioned differently. We use two
ML forecasting methods (one for each algorithm) in the comparisons conducted between stochastic
and machine learning. We also use 42 forecasting methods (21 for each algorithm) to perform a
sensitivity analysis for time lag selection and four ML forecasting methods (two for each algorithm)
for the investigation of the effect of the hyperparameter optimization.
2.3 Metrics
Regarding the one-step ahead forecasting properties, the assessment is based on the absolute
European Water 59 (2017)
163
error (AE) of the forecast of the last observation. For the comparison of the multi-step ahead
forecasting properties we use the root mean square error (RMSE), the Nash-Sutcliffe efficiency
(NSE), the ratio of standard deviations (rSD), the index of agreement (d) and the coefficient of
correlation (Pr) applied to the test set. These metrics quantify the forecasting methods’ performance
according to several criteria related to the accuracy, the capture of the variance and the correlation
between the forecasted and their respective observed values. For the definitions of the metrics NSE,
d and Pr the reader is referred to Krause et al. (2005), while for the definition of the rSD to
Zambrano-Bigiarini (2014).
Table 1. Time series examined. The Hurst parameter H is estimated for the deseasonalized time series.
s/n
Process
Code
Station id
Reference
Start
End
Length
(months)
H
1
Precipitation
prec_1
16672000
Peterson
and Vose
(1997)
Jan 1956
Dec 1987
384
0.48
2
prec_2
16627000
Jan 1951
Dec 1990
480
0.59
3
prec_3
16674000
Jan 1907
Dec 1990
1008
0.53
4
prec_4
16754001
Jan 1919
Dec 1939
252
0.52
5
prec_5
16754001
Jan 1950
Dec 1979
360
0.53
6
prec_6
16687000
Jan 1949
Dec 2000
624
0.51
7
prec_7
16714000
Jan 1860
Dec 1881
264
0.48
8
prec_8
16714000
Jan 1887
Dec 2005
1428
0.53
9
prec_9
16716000
Jan 1929
Dec 1945
204
0.52
10
prec_10
16715001
Jan 1926
Dec 1990
780
0.54
11
prec_11
16754000
Jan 1946
Dec 1990
540
0.50
12
prec_12
16641001
Jan 1951
Dec 1990
480
0.49
13
prec_13
16642000
Jan 1951
Dec 1990
480
0.58
14
prec_14
16726000
Jan 1956
Dec 1970
180
0.51
15
prec_15
16756001
Jan 1950
Dec 1984
420
0.50
16
prec_16
16760001
Jan 1949
Dec 1976
336
0.55
17
prec_17
16641000
Jan 1952
Dec 1996
540
0.51
18
prec_18
16743000
Jan 1951
Dec 1973
276
0.48
19
prec_19
16742000
Jan 1958
Dec 1990
396
0.49
20
prec_20
16632000
Jan 1955
Dec 1987
396
0.57
21
prec_21
16648000
Jan 1951
Dec 1997
564
0.55
22
prec_22
16650001
Jan 1951
Dec 2000
600
0.52
23
prec_23
16734000
Jan 1951
Dec 1991
492
0.49
24
prec_24
16738000
Jan 1951
Dec 1990
480
0.57
25
prec_25
16667000
Jan 1952
Dec 1990
468
0.55
26
prec_26
16732000
Jan 1955
Dec 1971
204
0.46
27
prec_27
16689000
Jan 1901
Dec 1984
1008
0.52
28
prec_28
16757000
Jan 1960
Dec 1983
288
0.56
29
prec_29
16684000
Jan 1955
Dec 1987
396
0.50
30
prec_30
16622000
Jan 1931
Dec 1997
804
0.58
31
prec_31
16622002
Jan 1961
Dec 1970
120
0.56
32
prec_32
16645001
Jan 1951
Dec 1990
480
0.56
33
prec_33
16710000
Jan 1951
Dec 1985
420
0.53
34
Temperature
temp_1
16687001
Lawrimore
et al. (2011)
Jan 1951
Dec 1980
360
0.66
35
temp_2
16714000
Jan 1858
Dec 1975
1416
0.67
36
temp_3
16714000
Jan 1989
Dec 2001
156
0.68
37
temp_4
16716000
Jan 1951
Dec 2012
744
0.65
38
temp_5
16754000
Jan 1950
Dec 2015
792
0.69
39
temp_6
16726000
Jan 1956
Dec 2015
720
0.74
40
temp_7
16641000
Jan 1951
Dec 2016
792
0.67
41
temp_8
16648000
Jan 1899
Dec 2016
1416
0.64
42
temp_9
16650000
Jan 1951
Dec 1998
576
0.75
43
temp_10
16734000
Jan 1951
Dec 1972
264
0.59
44
temp_11
16734000
Jan 1975
Dec 2000
312
0.61
45
temp_12
16689000
Jan 1951
Dec 1989
468
0.69
46
temp_13
16723000
Jan 1955
Dec 1969
180
0.64
47
temp_14
16723000
Jan 1974
Dec 2003
360
0.64
48
temp_15
16746000
Jan 1961
Dec 2015
660
0.71
49
temp_16
16622000
Jan 1892
Dec 2016
1500
0.67
50
temp_17
16622001
Jan 1961
Dec 1970
120
0.48
G.A. Papacharalampous et al.
164
2.4 Methodology outline
We conduct 50 single-case studies by applying a fixed methodology to each time series (see
Section 2.1), as explained subsequently. First, we split the time series into a fitting and a test set.
The latter is the last observation for the one-step ahead forecasting experiments and the last 12
observations for the multi-step ahead forecasting experiments. Second, we fit the models to the
deseasonalized fitting set, within the context determined by each forecasting method (see Section
2.2), and make predictions corresponding to the test set. Third, we add the seasonality to the
predicted values and compare them to their corresponding observed using the metrics (see Section
2.3). Finally, we conduct the cross-case synthesis presented in Section 3 to demonstrate similarities
and differences between the single-case studies conducted.
3. RESULTS AND DISCUSSION
We visualize the results within and across the individual cases using heatmaps. For the
quantitative form of the latter graphs, as well as for Figures S1 and S2, the reader is referred to the
Supplementary material, which is available at: https://doi.org/10.17632/p8sw8pzkcd.3.
As regards the heatmaps of the present study, they are formed under the following conditions: a)
the darker the colour the better the forecasts and b) the scaling is performed in the row direction.
White color rows indicate that no scaling is taking place. The latter happens when the forecasting
methods under comparison perform equally well regarding the criterion tested.
In Figures S1 and 1 we present the heatmaps formed for the comparison between the stochastic
and two of the ML forecasting methods on precipitation and temperature time series data
respectively. As we observe, the results of the single-case studies vary significantly. We also
observe that in every individual case examined the following applies. There is no best or worst
forecasting method regarding all the criteria set simultaneously. In other words, none of the
forecasting methods is uniformly better or worse than the rest. The former observations apply
equally to the stochastic and the ML forecasting methods, while it is noteworthy that the Naïve
benchmark is as competent as the forecasting methods regarding all the criteria set.
The observations outlined above are particularly important, because they reveal that the
forecasting quality is subject to limitations. Each forecasting method has some specific theoretical
properties and, due to the latter, it performs better or worse than other forecasting methods
regarding specific criteria and in specific cases. Thus, the conduct of a single-case study using
fewer criteria would have led to a very different overall picture. We note that the metrics RMSE and
NSE give almost the same information about the forecast quality regarding the multi-step ahead
forecasting experiments, a fact that does not apply to any other pair of metrics.
It is also interesting that the forecasting methods AR(1) and auto_ARFIMA are the least proper
to use on precipitation data, while they are competent on the temperature time series data. This is
actually a systematic pattern, which can be explained, when tracing back to the single-case studies
using precipitation time series data. In more detail, those two forecasting methods predict zero
precipitation in contrast to the rest, as a result to the zero precipitation observations in the summer
months.
Finally, by studying the numerical results we note that the forecasts for temperature are
remarkably better than the forecasts for precipitation. This may be explained by the fact that the
variability in temperature is more regular than that in precipitation.
In Figures 2 and S2 we present the heatmaps formed for the sensitivity analysis on the time lags
in time series forecasting using the NN and the SVM algorithms respectively. In both figures we
observe significant variations in the results across the individual cases, in an extent that it is
impossible to decide on a best or worst ML forecasting method among the single-case studies.
Regarding the SVM algorithm (Figure S2), we observe no systematic patterns and the variations
seem to be rather random, while for the case of the NN algorithm (Figure 2) we observe that the left
parts of the heatmaps are smoother with no white cells.
European Water 59 (2017)
165
Figure 1. Heatmaps for the comparison between stochastic and ML methods on temperature time series.
(1) one-step ahead forecasting - AE (2) multi-step ahead forecasting - RMSE (3) multi-step ahead forecasting - NSE
(4) multi-step ahead forecasting - rSD (5) multi-step ahead forecasting - d (6) multi-step ahead forecasting - Pr
Time series
Time series!
Forecasting method
Forecasting method!
Forecasting method
(1)
(2)
(3)
(4)
(5)
(6)
G.A. Papacharalampous et al.
166
Figure 2. Heatmaps for the sensitivity analysis on the time lags in time series forecasting using the NN algorithm.
In Figure 3 we present the heatmaps formed for the investigation of the effect of the
hyperparameter optimization. The results vary across the single-case studies in a rather random
manner, which indicates that the hyperparameter optimization does not necessary lead to better
forecasts for the NN and SVM algorithms.
4. CONCLUSIONS
The multiple-case study conducted must be encountered as a contingent empirical evidence on
several issues that have drawn the attention in the field of time series forecasting. The findings
suggest that the stochastic and ML methods can perform equally well, but always under limitations.
The best forecasting method depends on the case examined and the criterion of interest, while it can
be either stochastic or ML. However, the ML methods are computationally intensive. Regarding the
time lag selection, the best choice seems to depend mainly on the case, while the ML algorithm
one-step ahead forecasting - AE
multi-step ahead forecasting - RMSE
!
!
Time series
!
!
Time series
!
Time lags
Time lags
European Water 59 (2017)
167
might has also some effect. Finally, for the algorithms used in the present study hyperparameter
optimization does not necessarily lead to better forecasts.
Figure 3. Heatmaps for the investigation of the effect of hyperparameter optimization on the forecast quality. The
symbol * in the name of a forecasting method denotes that the model’s hyperparameters have been optimized.
one-step ahead forecasting - AE
multi-step ahead forecasting - RMSE
Time series
Time series
Forecasting method
Forecasting method
G.A. Papacharalampous et al.
168
REFERENCES
Achen, C. H., Snidal D., 1989. Rational deterrence theory and comparative case studies. World Polit.; 41(2): 143-169.
doi:10.2307/2010405
Cortes, C., Vapnik, V., 1995. Support-vector networks. Mach. Learn.; 20(3): 273-297. doi:10.1007/BF00994018
Cortez, P., 2010. Data mining with neural networks and support vector machines using the R/rminer tool. In: Perner P. (eds)
Advances in Data Mining. Applications and Theoretical Aspects. Springer Berlin Heidelberg, pp 572-583. doi:10.1007/978-3-
642-14400-4_44
Cortez, P., 2015. rminer: Data mining classification and regression methods. R package version 1.4.1. https://cran.r-
project.org/web/packages/rminer/index.html
Hyndman, R. J., O'Hara-Wild, M., Bergmeir, C., Razbash, S., Wang, E., 2016. forecast: Forecasting functions for time series and
linear models. R package version 7.1. https://CRAN.R-project.org/package=forecast
Hyndman, R. J., Khandakar, Y., 2008. Automatic time series forecasting: the forecast package for R. J. Stat. Softw.; 27(3): 1-22.
doi:10.18637/jss.v027.i03
Koutsoyiannis, D., Yao, H., Georgakakos, A., 2008, Medium-range flow prediction for the Nile: a comparison of stochastic and
deterministic methods. Hydrolog. Sci. J.; 53(1). doi:10.1623/hysj.53.1.142
Krause, P., Boyle, D. P., Bäse, F., 2005. Comparison of different efficiency criteria for hydrological model assessment. Adv. Geosci.;
5: 89-97
Lawrimore, J. H., Menne, M. J., Gleason, B. E., Williams C. N., Wuertz, D. B., Vose, R. S., Rennie, J., 2011. An overview of the
Global Historical Climatology Network monthly mean temperature data set, version 3. J. Geophys. Res.; 116(D19121).
doi:10.1029/2011JD016187
Maier, H. R., Dandy, G. C., 2000. Neural networks for the prediction and forecasting of water resources variables: a review of
modelling issues and applications. Environ. Model. Softw.; 15(1): 101-124. doi:10.1016/S1364-8152(99)00007-9
Papacharalampous, G. A., Tyralis, H., Koutsoyiannis, D., 2017. Comparison of stochastic and machine learning methods for multi-
step ahead forecasting of hydrological processes. Preprints, doi:10.20944/preprints201710.0133.v1
Peterson, T. C., Vose, R. S., 1997. An overview of the Global Historical Climatology Network temperature database. Bull. Am.
Meteorol. Soc.; 78(12): 2837-2849. doi:10.1175/1520-0477(1997)078<2837:AOOTGH>2.0.CO;2
Raghavendra, N. S., Deka, P. C., 2014. Support vector machine applications in the field of hydrology: a review. Appl. Soft Comput.;
19: 372-386. doi:10.1016/j.asoc.2014.02.002
Taieb, S. B., Bontempi, G., Atiya, A. F., Sorjamaa, A., 2012. A review and comparison of strategies for multi-step ahead time series
forecasting based on the NN5 forecasting competition. Expert Syst. Appl; 39(8): 7067-7083. doi:10.1016/j.eswa.2012.01.039
Tyralis, H., 2016. HKprocess: Hurst-Kolmogorov Process. R package version 0.0-2. https://cran.r-project.org/web/packages/
HKprocess/index.html
Tyralis, H., Koutsoyiannis, D., 2011. Simultaneous estimation of the parameters of the HurstKolmogorov stochastic process. Stoch.
Env. Res. Risk.; 25(1): 21-33. doi:10.1007/s00477-010-0408-x
Valipour, M., Banihabib, M. E., Behbahani, S. M. R., 2013. Comparison of the ARMA, ARIMA, and the autoregressive artificial
neural network models in forecasting the monthly inflow of Dez dam reservoir. J. Hydrol.; 476: 433-441.
doi:10.1016/j.jhydrol.2012.11.017
Vapnik, V. N., 1995. The nature of statistical learning theory, first edition. Springer-Verlag New York. doi:10.1007/978-1-4757-
3264-1
Vapnik, V. N., 1999. An overview of statistical learning theory. IEEE Trans Neural Netw; 10(5): 988-999. doi:10.1109/72.788640
Zambrano-Bigiarini, M., 2014. hydroGOF: Goodness-of-fit functions for comparison of simulated and observed hydrological time
series. R package version 0.3-8. https://cran.r-project.org/web/packages/hydroGOF/index.html
Zeileis, A., Grothendieck, G., 2005. zoo: S3 infrastructure for regular and irregular time series. J. Stat. Softw.; 14(6): 1-27.
doi:10.18637/jss.v014.i06
... Despite its far more challenging nature in comparison to one-step ahead forecasting, multi-step ahead forecasting is a common practice in hydrology (e.g. [3,4]) and beyond. Moreover, multi-step ahead streamflow forecasting is of particular interest, due to the large number of relevant applications (e.g. ...
... Finally, by examining the results with respect to the standard deviation of the time series, we sense that forecast uncertainty is rather high than negligible and that its proper modelling (e.g. in [14,15]) could perhaps be useful. The forecasting limitations have been also emphasized by [4,11]. ...
Conference Paper
Full-text available
Multi-step ahead streamflow forecasting is of practical interest. We examine the error evolution in multi-step ahead forecasting by conducting six simulation experiments. Within each of these experiments we compare the error evolution patterns created by 16 forecasting methods, when the latter are applied to 2 000 time series. Our findings suggest that the error evolution can differ significantly from the one forecasting method to the other and that some forecasting methods are more useful than others. However, the errors computed at each time step of a forecast horizon for a specific single-case study strongly depend on the case examined and can be either small or large, regardless of the used forecasting method and the time step of interest. This fact is illustrated with a comparative case study using 92 monthly time series of streamflow.
... Traditional analysis approaches are not suitable to investigate or utilize such massive data products. Hence, alternatives revolving around machine learning methods that can be implemented for forecasting or classification are becoming increasingly popular (Lary et al., 2016;Papacharalampous et al., 2017;Tyralis et al., 2019). ...
Article
Self-organizing maps provide a powerful, non-linear technique of dimensionality reduction that can be used to identify clusters with similar attributes. Here, they were constructed from a 1000-year-long gridded palaeoclimatic dataset, namely the Old World Drought Atlas, to detect regions of homogeneous hydroclimatic variability across the European continent. A classification scheme of 10 regions was found to describe most efficiently the spatial properties of Europe’s hydroclimate. These regions were mainly divided into a northern and a southern subset, linked together with a northwest-to-southeast orientation. Further analysis of the classification scheme with complex networks confirmed the divergence between the northern and southern components of European hydroclimate, also revealing that is not strongly correlated to the Iberian Peninsula. On the contrary, the region covering the British Isles, France and Germany appeared to be linked to both branches, implying links of hydroclimate with atmospheric/oceanic circulation.
... Traditional analysis approaches are not suitable to investigate or utilize such massive data products. Hence, alternatives revolving around machine learning methods that can be implemented for forecasting or classification are becoming increasingly popular (Lary et al., 2016;Papacharalampous et al., 2017;Tyralis et al., 2019). ...
Preprint
Full-text available
Self-Organizing Maps provide a powerful, non-linear technique of dimensionality reduction that can be used to identify clusters with similar attributes. Here, they were constructed from a 1000-year-long gridded palaeoclimatic dataset, namely the Old World Drought Atlas, to detect regions of homogeneous hydroclimatic variability across the European continent. A classification scheme of 10 regions was found to describe most efficiently the spatial properties of Europe's hydroclimate. These regions were mainly divided into a northern and a southern subset, linked together with a northwest-to-southeast orientation. Further analysis of the classification scheme with complex networks confirmed the divergence between the northern and southern components of European hydroclimate, also revealing that is not strongly correlated to the Iberian peninsula. On the contrary, the region covering British Isles, France and Germany, appeared to be linked to both branches, implying links of hydroclimate with atmospheric/oceanic circulation.
... One column of the input data matrix holds information about the predictand variable and the remaining columns information about lagged (predictor) variables that are assumed to be informative about the predictand. Variable selection (or feature selection) is known as a factor that might affect the performance of ML algorithms in both regression and forecasting applications (see, e.g., Anctil et al. 2009;Papacharalampous et al. 2017c). Thus, many studies specifically focus on the examination of this problem (e.g., Kohavi and John 1997;Tyralis and Papacharalampous 2017). ...
Article
Full-text available
Research within the field of hydrology often focuses on the statistical problem of comparing stochastic to machine learning (ML) forecasting methods. The performed comparisons are based on case studies, while a study providing large-scale results on the subject is missing. Herein, we compare 11 stochastic and 9 ML methods regarding their multi-step ahead forecasting properties by conducting 12 extensive computational experiments based on simulations. Each of these experiments uses 2000 time series generated by linear stationary stochastic processes. We conduct each simulation experiment twice; the first time using time series of 100 values and the second time using time series of 300 values. Additionally, we conduct a real-world experiment using 405 mean annual river discharge time series of 100 values. We quantify the forecasting performance of the methods using 18 metrics. The results indicate that stochastic and ML methods may produce equally useful forecasts.
... Acknowledgements A previous shorter version of the paper has been presented in the 10th World Congress of EWRA "Panta Rei" Athens, Greece, 5-9 July, 2017 under the title "Forecasting of geophysical processes using stochastic and machine learning algorithms" (Papacharalampous et al. 2017b). We thank the Scientific and Organizing Committees for selecting this research. ...
Article
Full-text available
We provide contingent empirical evidence on the solutions to three problems associated with univariate time series forecasting using machine learning (ML) algorithms by conducting an extensive multiple-case study. These problems are: (a) lagged variable selection, (b) hyperparameter handling, and (c) comparison between ML and classical algorithms. The multiple-case study is composed by 50 single-case studies, which use time series of mean monthly temperature and total monthly precipitation observed in Greece. We focus on two ML algorithms, i.e. neural networks and support vector machines, while we also include four classical algorithms and a naïve benchmark in the comparisons. We apply a fixed methodology to each individual case and, subsequently, we perform a cross-case synthesis to facilitate the detection of systematic patterns. We fit the models to the deseasonalized time series. We compare the one- and multi-step ahead forecasting performance of the algorithms. Regarding the one-step ahead forecasting performance, the assessment is based on the absolute error of the forecast of the last monthly observation. For the quantification of the multi-step ahead forecasting performance we compute five metrics on the test set (last year’s monthly observations), i.e. the root mean square error, the Nash-Sutcliffe efficiency, the ratio of standard deviations, the coefficient of correlation and the index of agreement. The evidence derived by the experiments can be summarized as follows: (a) the results mostly favour using less recent lagged variables, (b) hyperparameter optimization does not necessarily lead to better forecasts, (c) the ML and classical algorithms seem to be equally competitive.
Article
Investigations of changing the flow regime and flow prediction are of vital societal and hydro-ecological importance in a transboundary river like Punarbhaba river between India and Bangladesh. The present paper investigated flow regime though advance periodicity models (Morlet’s wavelet transformation) at the seasonal scale. Flow prediction using advanced machine learning techniques like Support vector machine (SVM), Artificial Neural Network (ANN), Hybrid wavelet ANN (W-ANN), Random forest(RF) capturing the periodicity, duration, cyclic or semi-cyclic nature of flow wave or wavelet from the historical time series data (1978–2017) is very crucial for environmental flow management and estimating present and future states of environmental flow. Flow alteration modeling (using heat map) and estimation of environmental flow (using duration curve shifting and RVA methods) are another vital objectives of this work to know the present and future hydro-ecological state. The result of periodicity clearly identified two distinct flow regimes before and after 1992-93 triggered by damming over there in 1992. All the prediction models identified the declining trend of flow in all the seasons, however hybrid wavelet ANN model could be treated as the best suited because of its very high-performance level. The degree of hydrological alteration is identified very high in the post-hydrological alteration (PHA) (post-1992) period and it is likely to be intensified predicted period. The estimated environmental flow state in PHA falls under moderate to critically modified states but if alteration goes on in this way ecological deficit will be the obvious result. For the survival of the river estimated environmental flow could be released primarily.
Thesis
Full-text available
This thesis falls into the scientific areas of stochastic hydrology, hydrological modelling and hydroinformatics. It contributes with new practical solutions, new methodologies and large-scale results to predictive modelling of hydrological processes, specifically to solving two interrelated technical problems with emphasis on the latter. These problems are: (A) hydrological time series forecasting by exclusively using endogenous predictor variables (hereafter, referred to simply as “hydrological time series forecasting”); and (B) stochastic process-based modelling of hydrological systems via probabilistic post-processing (hereafter, referred to simply as “probabilistic hydrological post-processing”). For the investigation of these technical problems, the thesis forms and exploits a novel predictive modelling and benchmarking toolbox. This toolbox is consisted of: (i) approximately 6 000 hydrological time series (sourced from larger freely available datasets), (ii) over 45 ready-made automatic models and algorithms mostly originating from the four major families of stochastic, (machine learning) regression, (machine learning) quantile regression, and conceptual process-based models, (iii) seven flexible methodologies (which together with the ready-made automatic models and algorithms consist the basis of our modelling solutions), and (iv) approximately 30 predictive performance evaluation metrics. Novel model combinations coupled with different algorithmic argument choices result in numerous model variants, many of which could be perceived as new methods. All the utilized models (i.e., the ones already available in open software, as well as those automated and proposed in the context of the thesis) are flexible, computationally convenient and fast; thus, they are appropriate for large-sample (even global-scale) hydrological investigations. Such investigations are implied by the (mainly) algorithmic nature of the methodologies of the thesis. In spite of this nature, the thesis also provides innovative theoretical supplements to its practical and methodological contribution. Technical problem (A) is examined in four stages. During the first stage, a detailed framework for assessing forecasting techniques in hydrology is introduced. Complying with the principles of forecasting and contrary to the existing hydrological (and, more generally, geophysical) time series forecasting literature (in which forecasting performance is usually assessed within case studies), the introduced framework incorporates large-scale benchmarking. The latter relies on big hydrological datasets, large-scale time series simulation by using classical stationary stochastic models, many automatic forecasting models and algorithms (including benchmarks), and many forecast quality metrics. The new framework is exploited (by utilizing part of the predictive modelling and benchmarking toolbox of the thesis) to provide large-scale results and useful insights on the comparison of stochastic and machine learning forecasting methods for the case of hydrological time series forecasting at large temporal scales (e.g., the annual and monthly ones), with emphasis on annual river discharge processes. The related investigations focus on multi-step ahead forecasting. During the second stage of the investigation of technical problem (A), the work conducted during the previous stage is expanded by exploring the one-step ahead forecasting properties of its methods, when the latter are applied to non-seasonal geophysical time series. Emphasis is put on the examination of two real-world datasets, an annual temperature dataset and an annual precipitation dataset. These datasets are examined in both their original and standardized forms to reveal the most and least accurate methods for long-run one-step ahead forecasting applications, and to provide rough benchmarks for the one-year ahead predictability of temperature and precipitation. The third stage of the investigation of technical problem (A) includes both the examination-quantification of predictability of monthly temperature and monthly precipitation at global scale, and the comparison of a large number of (mostly stochastic) automatic time series forecasting methods for monthly geophysical time series. The related investigations focus on multi-step ahead forecasting by using the largest real-world data sample ever used so far in hydrology for assessing the performance of time series forecasting methods. With the fourth (and last) stage of the investigation of technical problem (A), the multiple-case study research strategy is introduced −in its large-scale version− as an innovative alternative to conducting single- or few-case studies in the field of geophysical time series forecasting. To explore three sub-problems associated with hydrological time series forecasting using machine learning algorithms, an extensive multiple-case study is conducted. This multiple-case study is composed by a sufficient number of single-case studies, which exploit monthly temperature and monthly precipitation time series observed in Greece. The explored sub-problems are lagged variable selection, hyperparameter handling, and comparison of machine learning and stochastic algorithms. Technical problem (B) is examined in three stages. During the first stage, a novel two-stage probabilistic hydrological post-processing methodology is developed by using a theoretically consistent probabilistic hydrological modelling blueprint as a starting point. The usefulness of this methodology is demonstrated by conducting toy model investigations. The same investigations also demonstrate how our understanding of the system to be modelled can guide us to achieve better predictive modelling when using the proposed methodology. During the second stage of the investigation of technical problem (B), the probabilistic hydrological modelling methodology proposed during the previous stage is validated. The validation is made by conducting a large-scale real-world experiment at monthly timescale. In this experiment, the increased robustness of the investigated methodology with respect to the combined (by this methodology) individual predictors and, by extension, to basic two-stage post-processing methodologies is demonstrated. The ability to “harness the wisdom of the crowd” is also empirically proven. Finally, during the third stage of the investigation of technical problem (B), the thesis introduces the largest range of probabilistic hydrological post-processing methods ever introduced in a single work, and additionally conducts at daily timescale the largest benchmark experiment ever conducted in the field. Additionally, it assesses several theoretical and qualitative aspects of the examined problem and the application of the proposed algorithms to answer the following research question: Why and how to combine process-based models and machine learning quantile regression algorithms for probabilistic hydrological modelling?
Preprint
Full-text available
Research within the field of hydrology often focuses on comparing stochastic to machine learning (ML) forecasting methods. The comparisons performed are all based on case studies, while an extensive study aiming to provide generalized results on the subject is missing. Herein, we compare 11 stochastic and 9 ML methods regarding their multi-step ahead forecasting properties by conducting 12 large-scale computational experiments based on simulations. Each of these experiments uses 2 000 time series generated by linear stationary stochastic processes. We conduct each simulation experiment twice; the first time using time series of 100 values and the second time using time series of 300 values. Additionally, we conduct a real-world experiment using 405 mean annual river discharge time series of 100 values. We quantify the performance of the methods using 18 metrics. The results indicate that stochastic and ML methods perform equally well.
Article
Full-text available
We assess the performance of random forests and Prophet in forecasting daily streamflow up to seven days ahead in a river in the US. Both the assessed forecasting methods use past streamflow observations, while random forests additionally use past precipitation information. For benchmarking purposes we also implement a naïve method based on the previous streamflow observation, as well as a multiple linear regression model utilizing the same information as random forests. Our aim is to illustrate important points about the forecasting methods when implemented for the examined problem. Therefore, the assessment is made in detail at a sufficient number of starting points and for several forecast horizons. The results suggest that random forests perform better in general terms, while Prophet outperforms the naïve method for forecast horizons longer than three days. Finally, random forests forecast the abrupt streamflow fluctuations more satisfactorily than the three other methods.
Article
Full-text available
We assess the performance of the recently introduced Prophet model in multi-step ahead forecasting of monthly streamflow by using a large dataset. Our aim is to compare the results derived through two different approaches. The first approach uses past information about the time series to be forecasted only (standard approach), while the second approach uses exogenous predictor variables alongside with the use of the endogenous ones. The additional information used in the fitting and forecasting processes includes monthly precipitation and/or temperature time series, and their forecasts respectively. Specifically, the exploited exogenous (observed or forecasted) information considered at each time step exclusively concerns the time of interest. The algorithms based on the Prophet model are in total four. Their forecasts are also compared with those obtained using two classical algorithms and two benchmarks. The comparison is performed in terms of four metrics. The findings suggest that the compared approaches are equally useful.
Preprint
Full-text available
We perform an extensive comparison between 11 stochastic to 9 machine learning methods regarding their multi-step ahead forecasting properties by conducting 12 large-scale computational experiments. Each of these experiments uses 2 000 time series generated by linear stationary stochastic processes. We conduct each simulation experiment twice; the first time using time series of 110 values and the second time using time series of 310 values. Additionally, we conduct 92 real-world case studies using mean monthly time series of streamflow and particularly focus on one of them to reinforce the findings and highlight important facts. We quantify the performance of the methods using 18 metrics. The results indicate that the machine learning methods do not differ dramatically from the stochastic, while none of the methods under comparison is uniformly better or worse than the rest. However, there are methods that are regularly better or worse than others according to specific metrics.
Code
Full-text available
Methods to make inference about the Hurst-Kolmogorov and the AR(1) process.
Presentation
Full-text available
Machine learning (ML) is considered to be a promising approach to hydrological processes forecasting. We conduct a comparison between several stochastic and ML point estimation methods by performing large-scale computational experiments based on simulations. The purpose is to provide generalized results, while the respective comparisons in the literature are usually based on case studies. The stochastic methods used include simple methods, models from the frequently used families of Autoregressive Moving Average (ARMA), Autoregressive Fractionally Integrated Moving Average (ARFIMA) and Exponential Smoothing models. The ML methods used are Random Forests (RF), Support Vector Machines (SVM) and Neural Networks (NN). The comparison refers to the multi-step ahead forecasting properties of the methods. A total of 20 methods are used, among which 9 are the ML methods. 12 simulation experiments are performed, while each of them uses 2 000 simulated time series of 310 observations. The time series are simulated using stochastic processes from the families of ARMA and ARFIMA models. Each time series is split into a fitting (first 300 observations) and a testing set (last 10 observations). The comparative assessment of the methods is based on 18 metrics, that quantify the methods’ performance according to several criteria related to the accurate forecasting of the testing set, the capturing of its variation and the correlation between the testing and forecasted values. The most important outcome of this study is that there is not a uniformly better or worse method. However, there are methods that are regularly better or worse than others with respect to specific metrics. It appears that, although a general ranking of the methods is not possible, their classification based on their similar or contrasting performance in the various metrics is possible to some extent. Another important conclusion is that more sophisticated methods do not necessarily provide better forecasts compared to simpler methods. It is pointed out that the ML methods do not differ dramatically from the stochastic methods, while it is interesting that the NN, RF and SVM algorithms used in this study offer potentially very good performance in terms of accuracy. It should be noted that, although this study focuses on hydrological processes, the results are of general scientific interest. Another important point in this study is the use of several methods and metrics. Using fewer methods and fewer metrics would have led to a very different overall picture, particularly if those fewer metrics corresponded to fewer criteria. For this reason, we consider that the proposed methodology is appropriate for the evaluation of forecasting methods.
Article
Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data.High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
Article
In the recent few decades there has been very significant developments in the theoretical understanding of Support vector machines (SVMs) as well as algorithmic strategies for implementing them, and applications of the approach to practical problems. SVMs introduced by Vapnik & others in the early 90s are machine learning systems that utilize a hypothesis space of linear functions in a high dimensional feature space, trained with optimization algorithms that implements a learning bias derived from statistical learning theory. This paper reviews the state-of-the-art and focuses over a wide range of applications of SVMs in the field of hydrology. To use SVM aided hydrological models, which have increasingly extended during the last years; comprehensive knowledge about their theory and modeling approaches seems to be necessary. Furthermore, this review provides a brief synopsis of the techniques of SVMs and other emerging ones (hybrid models), which have proven useful in the analysis of the various hydrological parameters. Moreover, various examples of successful applications of SVMs for modeling different hydrological processes are also provided
Article
Several recent books have argued that comparative case studies of crises demonstrate the failure of rational-deterrence theory; they have offered certain empirical generalizations as substitutes. This paper shows that such contentions are unwarranted. First, the empirical generalizations are impressive as historical insights, but they do not meet the standards for theory set out by the most sophisticated case-study analysts themselves. Second, the “tests” of rational deterrence used in the case studies violate standard principles of inference, and the ensuing procedures are so biased as to be useless. Rational deterrence, then, is a more successful theory than portrayed in this literature, and it remains the only intellectually powerful alternative available. Case studies are essential to theory building: more efficiently than any other methods, they find suitable variables, suggest middle-range generalizations for theory to explain, and provide the prior knowledge that statistical tests require. Their loose constraints on admissible propositions and suitable evidence are appropriate and even necessary for these tasks. These same characteristics, however, inevitably undermine all attempts to construe case-study generalizations as bodies of theory or tests of hypotheses.