No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Time-Series Methods and Recursive Estimation in Hydrological Systems Analysis
Abstract
Previous chapters have shown how models of hydrological systems can be formulated in many different ways and with various levels of complexity. In this chapter, we will see how these kinds of models can be considered within a unified stochastic setting and how it is then possible to treat model calibration as a problem of time-series analysis. In this manner, powerful time-series techniques, such as recursive estimation (Young 1984) can be used in the identification, estimation and validation of the models. And, because ot their inherently stochastic nature, such models can subsequently provide a natural vehicle for real-time flow forecasting. Moreover, the recursive approach to estimation allows for continuous updating of the model parameter estimates and the possibility of more advanced “self-adaptive” forecasting and control procedures.
... • Metric Models, which are based primarily on observational data and seek to characterize the flow response largely on the basis of these data, using some form of statistical estimation or optimization (e.g. Wood & O'Connell, 1985;Young and Wallis, 1985;Young, 1986). These include purely black-box, timeseries models, such as discrete and continuous-time transfer functions, neural network and neuro-fuzzy representations (e.g Tokar & Johnson, 1999;Jang et al., 1997). ...
... The model will receive inputs from the rainfall-flow models discussed above and, in examples such as the Dumfries flood warning model, from flow gauges far upstream which provide advance warning of impending flow changes. A typical early example of such a model is that used for studies of the Bedford-Ouse river system (Whitehead et al., 1976); more recent examples are the simple River Wyre model (Young, 1986); the much more spatially complex Dumfries model (Lees et al., 1994) and other models discussed in Cluckie (1993). ...
The Chapter discusses the modelling of rainfall-flow (rainfall-runoff) and flow routing processes in river systems within the context of real-time flood forecasting. It is argued that deterministic, reductionist (or 'bottom-up') models are inappro-priate for real-time forecasting because of the inherent uncertainty that character-izes river catchment dynamics and the problems of model over-parametrization. The advantages of alternative, efficiently parameterized Data-Based Mechanistic (DBM) models, identified and estimated using statistical methods, are discussed. It is shown that such models are in an ideal form for incorporation in a real-time, adaptive data assimilation and forecasting system based on recursive state space estimation (an adaptive version of the stochastic Kalman Filter algorithm). An il-lustrative example, based on the analysis of daily data from the ephemeral Canning River in SW Australia, demonstrates the utility of this methodology and illustrates the advantages of incorporating real-time state and parameter adaption.
... Genetic algorithms are in the same family (Wang, 1991). Other techniques are based on maximum likelihood estimation (in combination with a Kalman filter in the recursive parameter calibration techniques of Jakeman et al. (1990) and Young (1986)). Parameter uncertainty estimates were calculated and discussed for hydrological models by Johnston & Pilgrim (1976), Kuczera (1983), Kuczera & Mroczkowski (1998). ...
... Goodness-of-fit statistics used in the optimisation are the mean squared error, the coefficient of efficiency (Nash & Sutcliffe, 1970), Young's Information Criterion (Young, 1986), etc.; for an overview see Legates and McCabe (1999). ...
Errors and uncertainties in the prediction of rainfall–runoff are often substantial. Consideration of these errors is therefore crucially important. When water decision-making is based on the rainfall–runoff prediction, for instance, by means of rainfall–runoff, uncertainties affect model outcomes and, consequently, the decisions. Hence, good modeling practice provides not only the results of model predictions but also the accuracy of these results. It supplies decision makers with important additional information on the uncertainty in the data and information that they use as a basis for their decisions. After quantification of the runoff prediction errors, water policies can be set up for which the efficiency can be guaranteed up to specified acceptable risk levels. Combined with the uncertainty analysis, a sensitivity analysis provides the modeler and decision maker with information about the importance of various types of model limitations and sources of uncertainty.
... The effects of nonlinearity (partly caused by variations in antecedent conditions) are removed by the joint use of baseflow separation and the calculation of an 'effective rainfall' input ensuring that the inputs equal the predicted outputs. In recent years, however, there has been a move towards modelling the whole of the outflow using linear transfer functions, with the effects of nonlinearity reduced by pre-processing the input signal through for example DBM (e.g., Young 1986;Young & Beven 1994;Beven et al. 2011;Chappell et al. 2017) or IHACRES (e.g., Jakeman et al. 1990;Littlewood 2021). For a comparative discussion of the two approaches, see Littlewood et al. (2010). ...
While it is known that antecedent conditions and rainfall profiles contribute to the nonlinearity of streamflow response and that hydrograph shape can be dependent on the nature of rainfall inputs, how antecedent conditions (with similar rainfall inputs) impact hydrograph shape is less known. Here, a data-based mechanistic (DBM) approach is applied to quantify hydrograph shape, in terms of timing and volume, for the purposes of comparing hydrographs across 17 micro-basins at selected localities in upland UK over a 4-year period. The analysis demonstrates the nonlinearity of storm response for small catchments and revealed that with low antecedent conditions and/or small rainfall inputs there was a high variance in hydrograph shape quantifiers and that these variances decrease (at rates micro-basin dependent) as the micro-basins became wetter or as the storms increased in size, potentially converging to a more stable response.
... This calls for the use of models that can handle time-series records with many gaps or missing data, and quantify the effects of sometimes poor data quality on model structures and parameters to be interpreted Young, 2001;Young et al., 1999). One of such approaches is the relatively new Data-Based Mechanistic (DBM) modelling routines (Beck & Young, 1975;Whitehead & Young, 1975;Young, 1978Young, , 1983Young, , 1984Young, , 1986Young, , 1992Young, , 1993Young & Minchin, 1991;Young & Lees, 1993;Young & Beven, 1994;Chappell et al., 1999;Lees, 2000;Amisigo, 2005;Ampadu et al., 2013aAmpadu et al., , 2015. ...
Climatic impacts on the environment, land cover and land use change such as urbanisation and deforestation affect rainfall and change riverflow responsiveness and seasonal flows as a result of their influence on nonlinear behaviour of the catchments of rivers. Water supply is seriously affected which invariably impacts on domestic activities and agriculture. In some countries river transportation, hydropower, irrigation and fish farming are at risk due to these impacts. Within this paper, a review of existing hydrological and climatic studies in Africa and Ghana in particular is presented. The paper highlights the knowledge gaps identified in the review, such as rainfall-riverflow processes and their controls in the country, estimation of cycles in evapo-transpiration and solar radiation time series, together with the complete characterisation of the temporal and spatial fluctuations within the climate cycles. The establishment of a monograph of catchment response characteristic across the Ghanaian latitudinal gradient and if possible across the tropics is recommended. This could be used for the prediction of hydrologic response of ungauged catchments in the country.
... There are other approaches to obtain the data-driven model, in the paper hydrometeorogical example is taken. In this area such approaches are existing for several decades, for example, the classical data-driven models [13] and the modern ones [12,6]. We note that the areas, where the data-driven models for time-series are applied, are countless, therefore, we do not stop on the different applications. ...
In the literature, vast amounts of methods of time-series modeling are described. Most for the methods, either classical or machine learning, left interpretation to the expert. Even though the interpretation is sometimes possible, usually, it is done only in a very narrow range of the applications. In the article approach to the extended time-series model interpretation is proposed. The algorithm of time-series model discovery in the form of the algebraic expression in a closed-form is described. The resulting algorithm utilizes the flexibility of the evolutionary optimization and possibility of the sparse regression to make concise models.
... The derivation of equation (17) in this manner is perfectly credible in physical terms: indeed the derivation, including the assumption (19), is the basis of the physically-based "lag-and-route" models that have been used for some considerable time by hydrologists (see e.g. Young, 1986). ...
Although rainfall-flow processes have received much attention in the hydrological literature, the nature of the nonlinear processes involved in the relationship between rainfall and river flow still remains rather unclear. This paper outlines the first author's Data-Based Mechanistic (DBM) approach to model structure identification and parameter estimation for linear and nonlinear dynamic systems and uses it to explore afresh the nonlinear relationship between measured rainfall and flow in two typical catchments. Exploiting the power of recursive estimation, state dependent nonlinearities are identified objectively from the time-series data and used as the basis for the estimation of nonlinear transfer function models of the rainfall-flow dynamics. These objectively identified models not only explain the data in a parametrically efficient manner but also reveal the possible parallel nature of the underlying physical processes within the catchments. The DBM modelling approach provides a useful tool for the further investigation of rainfall-flow processes, as well as other linear and nonlinear environmental systems. Moreover, because DBM modelling uses recursive estimation, it provides a powerful vehicle for the design of real-time, self adaptive environmental management systems. Finally, the paper points out how DBM models can often be interpreted directly in terms dynamic conservation equations (mass, energy or momentum) associated with environmental flow processes and stresses the importance of parallel processes in this connection.
... In order to face these "reductionist philosophy" problems, "top-down" approaches were proposed : the model is determined using the measured data and despite some slight vocabulary divergences, these approaches link directly to the field of system identification. Among the models used in the literature, the linear transfer function models are widely used [You86], [Doo59], [TDV91]. Even though the simplicity of these linear models, they are still used nowadays and offer some good solution in urban context for example [PL09], [KLG + 09]. ...
La procédure d'identification consiste à rechercher un modèle mathématique adéquat pour un système dynamique donné à partir de données expérimentales. Alors que l'identification de système est orientée majoritairement pour répondre aux problèmes de commande depuis les années 90, l'identification de systèmes naturels reste cruciale pour une meilleure compréhension de notre environnement. Cette thèse vise à apporter une solution au problème de modélisation de la relation pluie/débit dans un bassin versant rural. Un bassin versant est défini comme la portion de territoire délimitée par des lignes de crête, dont les eaux alimentent un exutoire commun : cours d'eau, lac, mer, etc. L'identification de la relation pluie/débit est un problème stimulant, de par la complexité à trouver une structure de modèle définissant le comportement du bassin dans son ensemble. De plus, dans les bassins ruraux, il y a une grande variabilité spatio-temporelle des propriétés du sol tant au niveau de la végétation, du type de sol ou de l'évapotranspiration et seulement une partie de la pluie totale ruisselle et contribue au débit à l'exutoire. Dans ce cas, les modèles linéaires ne sont pas adaptés et ne peuvent délivrer de modèle acceptable pour la relation pluie/débit. A cet effet, deux structures de modèles non-linéaires sont étudiées : les modèles Hammerstein et les modèles Linéaires à Paramètres variants (LPV). La contribution principale de cette thèse réside dans le développement de méthodes dédiées à l'estimation de ces modèles, à temps discret ou continu, opérant en boucle ouverte ou fermée, en se concentrant sur le cas réaliste où le bruit de sortie est coloré et indépendant du processus étudié : le cas Box--Jenkins (BJ). De plus, les méthodes proposées ont été conçues spécialement pour fournir des résultats utiles dans le cas réel où le modèle de bruit est inconnu ou mal évalué. Finalement, ces méthodes sont utilisées sur des données réelles, acquises sur un bassin versant rural situé à Rouffach, Alsace, France et un processus d'identification innovant est proposé pour la modélisation de la relation pluie/débit.
... In order to face these "reductionist philosophy" problems, "top-down" approaches were proposed : the model is determined using the measured data and despite some slight vocabulary divergences, these approaches link directly to the field of system identification. Among the models used in the literature, the linear transfer function models are widely used [You86], [Doo59], [TDV91]. Even though the simplicity of these linear models, they are still used nowadays and offer some good solution in urban context for example [PL09], [KLG + 09]. ...
System identification is an established field in the aera of system analysis and control. It aims at determining mathematical models for dynamical systems based measured data. The identification of natural systems is crucial for a better understanding of our environment and this work aims at solving the modelling problem of the rainfall/flow relationship in rural catchments. In order to achieve this goal, two nonlinear model structures are studied: the Hammerstein and the Linear Parameter Varying (LPV) models. The contribution of this work lies in the development of identification methods dealing robustly with estimation problem of such models, both in discrete-time and continuous-time, in open-loop and closed-loop configuration, focusing on the realistic Box--Jenkins (BJ) case. Moreover, the methods are especially designed to result in relevant estimates in case the noise model is unknown, which is the case in most practical applications. The first chapter is an introduction defining the problems encountered with natural systems and motivating the theoretical work induced. The second chapter presents a suboptimal Refined Instrumental Variable based method for Hammerstein BJ models. The third chapter focuses on the identification of LPV-BJ models, highlights the problems encountered by the existing methods and proposes a solution via a reformulation of the model. Finally, the last chapter is dedicated to the application of the presented methods on some real rainfall/flow data set acquired from a rural catchment situated in Rouffach, Alsace, France for the identification of the rainfall/runoff relationship
... Οι μέθοδοι της δεύτερης κατηγορίας στηρίζονται στη θεωρία ανάλυσης χρονοσειρών (βλ. Young, 1986). ...
Το βιβλίο καλύπτει θέματα σχεδιασμού και λειτουργικού ελέγχου των δικτύων αποχέτευσης. Περιλαμβάνει στοιχεία από την ελληνική τεχνολογική εμπειρία στα αποχετευτικά έργα (μεθοδολογίες και προδιαγραφές) με παράλληλη αναφορά σε στοιχεία της διεθνούς βιβλιογραφίας. Δίνει έμφαση στις επιστημονικές αρχές που αιτιολογούν τις προδιαγραφές και τις μεθοδολογίες. Επίσης, ενσωματώνει και ορισμένα πρωτότυπα στοιχεία. Αποτελείται από οκτώ κεφάλαια. Στο πρώτο δίνονται οι βασικές έννοιες και οι ορισμοί μαζί με ένα σύντομο ιστορικό και μια γενική περιγραφή των μελετών αποχέτευσης. Στο δεύτερο κεφάλαιο αναφέρονται οι μέθοδοι εκτίμησης των παροχών ακαθάρτων, με παράθεση χρήσιμων δεδομένων, ελληνικών και διεθνών. Στο τρίτο κεφάλαιο δίνεται η τυπική μεθοδολογία εκτίμησης των παροχών ομβρίων, με ιδιαίτερη αναφορά στις ελληνικές συνθήκες (παράθεση τυπικών όμβριων καμπυλών, κτλ.). Το τέταρτο κεφάλαιο καλύπτει τους υδραυλικούς υπολογισμούς που είναι απαραίτητοι για το σχεδιασμό, τον έλεγχο επάρκειας και την εκτίμηση των χαρακτηριστικών ροής των αγωγών αποχέτευσης. Έμφαση δίνεται στην αντιμετώπιση ορισμένων μη τυπικών καταστάσεων λειτουργίας των αγωγών (πχ. πολύ μεγάλες ή πολύ μικρές κλίσεις και ταχύτητες), οι οποίες μπορεί να προκαλέσουν σοβαρά προβλήματα στη λειτουργία των δικτύων. Στο πέμπτο κεφάλαιο παρουσιάζονται οι πιο σύγχρονες αντιλήψεις και υπολογιστικές μέθοδοι των αποχετευτικών δικτύων που βασίζονται στην συνολική προσομοίωση της λειτουργίας τους (υδρολογική και υδραυλική). Το έκτο κεφάλαιο καλύπτει ορισμένα ποιοτικά θέματα τα οποία συναρτώνται με το σχεδιασμό και τον έλεγχο της λειτουργίας των δικτύων, όπως είναι η παραγωγή και απελευθέρωση υδροθείου στους αγωγούς και οι συνέπειές της. Στο έβδομο κεφάλαιο περιλαμβάνονται τεχνολογικά θέματα, όπως η επιλογή υλικών για τους προκατασκευασμένους σωλήνες, ο τρόπος κατασκευής των χυτών επί τόπου αγωγών και η αντιδιαβρωτική προστασία των αγωγών λυμάτων. Τέλος, στο όγδοο κεφάλαιο δίνονται πληροφορίες για κάποια εναλλακτικά συστήματα αποχέτευσης. Μερικές από τις πρωτότυπες αναλύσεις που περιλαμβάνονται στο βιβλίο είναι: (α) η στατιστική θεμελίωση και μοντελοποίηση της διακύμανσης των παροχών ακαθάρτων, (β) η κατάρτιση αλγορίθμων αριθμητικής επίλυσης διάφορων υδραυλικών προβλημάτων, (γ) η εξαγωγή αναλυτικών σχέσεων που μπορούν να αντικαταστήσουν διάφορα εμπειρικά νομογραφήματα ή πίνακες της βιβλιογραφίας (π.χ. εξίσωση μεταβολής της τραχύτητας συναρτήσει του αδιαστατοποιημένου βάθους ροής σε κυκλικούς αγωγούς, αντίστοιχη εξίσωση για τον άμεσο προσεγγιστικό υπολογισμό του κρίσιμου βάθους σε κυκλικούς αγωγούς, εξίσωση εκτίμησης του συντελεστή τοπικών απωλειών σε συμβολές αποχετευτικών αγωγών), και (δ) η συστηματοποίηση και μεταφορά στο σύστημα μονάδων SI διάφορων εμπειρικών εξισώσεων της βιβλιογραφίας (π.χ. εξισώσεις εκτίμησης χρόνων συρροής ομβρίων, εξισώσεις παραγωγής υδροθείου, κ.ά.).
... In order to face these "reductionist philosophy" problems, "top-down" approaches were proposed : the model is determined using the measured data and despite some slight vocabulary divergences, these approaches link directly to the field of system identification. Among the models used in the literature, the linear transfer function models are widely used [You86], [Doo59], [TDV91]. Even though the simplicity of these linear models, they are still used nowadays and offer some good solution in urban context for example [PL09], [KLG + 09]. ...
System identification is an established field in the aera of system analysis and control. It aims at determining mathematical models for dynamical systems based measured data. The identification of natural systems is crucial for a better understanding of our environment and this work aims at solving the modelling problem of the rainfall/flow relationship in rural catchments. In order to achieve this goal, two nonlinear model structures are studied: the Hammerstein and the Linear Parameter Varying (LPV) models. The contribution of this work lies in the development of identification methods dealing robustly with estimation problem of such models, both in discrete-time and continuous-time, in open-loop and closed-loop configuration, focusing on the realistic Box--Jenkins (BJ) case. Moreover, the methods are especially designed to result in relevant estimates in case the noise model is unknown, which is the case in most practical applications. The first chapter is an introduction defining the problems encountered with natural systems and motivating the theoretical work induced. The second chapter presents a suboptimal Refined Instrumental Variable based method for Hammerstein BJ models. The third chapter focuses on the identification of LPV-BJ models, highlights the problems encountered by the existing methods and proposes a solution via a reformulation of the model. Finally, the last chapter is dedicated to the application of the presented methods on some real rainfall/flow data set acquired from a rural catchment situated in Rouffach, Alsace, France for the identification of the rainfall/runoff relationship.
... Fortunately, as mentioned in section 1, previous research on the DBM modeling of rainfall-flow (and flow-routing) systems has shown that the SDP nonlinearity, in this case, is a simple function of the measured flow, acting as a surrogate measure of soil moisture. Moreover, as we shall see in the later practical example of section 4 and the associated Figure 5, the linear transfer function (TF) part of the model can be reduced to a hydrologically interpretable combination of lumped, conceptual ''storage tanks'' (surface water stores or reservoirs) that are likely to be much more familiar to many readers (see also the discussion on this in Young [1986Young [ , 1992aYoung [ , 1992b). ...
[1] The paper introduces a logical extension to data-based mechanistic (DBM) modeling, which provides hypothetico-inductive (HI-DBM) bridge between conceptual models, derived in a hypothetico-deductive manner, and the DBM model identified inductively from the same time-series data. The approach is illustrated by a quite detailed example of HI-DBM analysis applied to the well-known Leaf River data set and the associated HyMOD conceptual model. The HI-DBM model significantly improves the explanation of the Leaf River data and enhances the performance of the original DBM model. However, on the basis of various diagnostic tests, including recursive time-variable and state-dependent parameter estimation, it is suggested that the model should be capable of further improvement, particularly as regards the conceptual effective rainfall mechanism, which is based on the probability distributed model hypothesis. In order to verify the efficacy of the HI-DBM analysis in a situation where the actual model generating the data is completely known, the analysis is also applied to a stochastic simulation model based on a modified HyMOD model.
... Thus, a comprehensive review of such work is beyond the scope of this paper. The Kalman Filter (KF) algorithm used here is formulated in the predictivecorrective form (e.g., [5], and the earlier references therein) which underlies the general unobserved component approach to state space estimation and forecasting [ by Bras and Rodriguez-Iturbe [lo] contains an overview of the KF and other statistical techniques in hydrology (see also [11,12]), while an inventory of recent hydrological applications of the KF, especially for streamflow forecasting, can be found in [13], where the KF method is also applied for the prediction of monthly flow for two catchments in Turkey and the USA, and for monthly rainfall prediction in Saudi Arabia. ...
In [1], the IHACRES rainfall-runoff model is calibrated for the purpose of predicting streamflow discharge in ten catchments of the Upper Murray Basin using a four-hourly time step. A map and description of the basin can be found in [1,2]. The major aim of the present paper is to describe the subsequent development and testing of a four-hourly time step flow forecasting model which exploits the Kalman Filter (KF) algorithm to upgrade the IHACRES models from a simple predictive to a real-time forecasting capability. In [2], the IHACRES model and a self-adaptive filtering approach, based on the autoregressive integrated moving average (ARIMA) representation of the model residuals, were combined and utilized for forecasting daily streamflow in nine catchments of the Upper Murray Basin. Such linear filtering of the model residuals provided a considerable improvement in forecasting both low and high values of streamflow. A KF forecasting algorithm, incorporating the subdaily Upper Murray Basin IHACRES model, has been used in this second stage of the project as a tool for operational streamflow forecasting because it provides a more flexible approach and yields even better results (in terms of Nash-Sutcliffe efficiency statistics [3] and relative errors) than the ARIMA linear filtering approach.
... the paper helps to confirm, yet again, the way in which such objective time-series analysis procedures can lead to models which not only provide an excellent explanation of hydrological data (e.g. Young, 1986) but which also can be interpreted in physically meaningful terms, in this case to help define the nature of a parallel pathway between rainfall excess and streamflow. ...
... [3] While considerable progress has been made in the development and application of automated calibration methods, many of these approaches treat the underlying uncertainty in the input-output representation of the model as being primarily (and explicitly) due to uncertainty in the parameter estimates. A few papers do discuss the treatment of input, state, and model structural uncertainties for environmental models [e.g., Young and Beck, 1974;Beck and Young, 1975;Bras, 1980a, 1980b;Beck, 1985Beck, , 1987Hebson and Wood, 1985;Young, 1986], but such approaches have not become common practice for nonlinear watershed models. Clearly, uncertainties in the modeling procedure stem not only from uncertainties in the parameter estimates, but also from measurement errors associated with the system input (forcing) and output, and from model structural errors arising from the aggregation of spatially distributed real-world processes into a mathematical model. ...
... [3] While considerable progress has been made in the development and application of automated calibration methods, many of these approaches treat the underlying uncertainty in the input-output representation of the model as being primarily (and explicitly) due to uncertainty in the parameter estimates. A few papers do discuss the treatment of input, state, and model structural uncertainties for environmental models [e.g., Young and Beck, 1974;Beck and Young, 1975;Bras, 1980a, 1980b;Beck, 1985Beck, , 1987Hebson and Wood, 1985;Young, 1986], but such approaches have not become common practice for nonlinear watershed models. Clearly, uncertainties in the modeling procedure stem not only from uncertainties in the parameter estimates, but also from measurement errors associated with the system input (forcing) and output, and from model structural errors arising from the aggregation of spatially distributed real-world processes into a mathematical model. ...
... Instead of assuming parametric invariance, suppose that the parameters vary in a manner that can be described by a Gauss-Markov stochastic difference equation: where $ is a (2n+l)x(2n+l) transition matrix and T is a (2n+l)x(2n+l) input matrix, both of which may be time variable; q k is a (2n+l)xl white noise vector with zero mean and covariance matrix Q. The simplest example of this model is the random walk model: In the case of the random walk model, the IV algorithm can be modified to become (Young, 1986): Here Q is a (2n+l)x(2n+l) matrix which allows for possible parametric variations. ...
Operational flood forecasting on the River Meuse in Belgium is based on both hydrologie and hydraulic modelling. The hydrologie models take into account the rainfall-runoff relationships for the main subcatchments. This allows to handle the spatially-vari able hydrologie behaviour of the total basin. Classical modelling procedures such as a time-invariant stochastic approach, cannot handle problems related to the changing dynamics of the hydrologie processes. This paper discusses a methodology based on on-line estimation of parameters, to model short variations in the response of the main subcatchments of the Meuse. The variation of the para meters is modelled by means of a random walk method. Some objective criteria for evaluation forecasting performance are intro duced. It can be concluded that, in general, adaptive modelling improves the real-time performance within the linear framework.
... Earlier studies have demonstrated the importance of reducing uncertainties from the input, initial model states and model structure (Young and Beck 1974;Kitanidis and Bras 1980;Hebson and Wood 1985;Young 1986), but the importance of reducing uncertainty has focused on application of hydrologic models for a given watershed (Vrugt et al. 2006;Fan et al. 2015). Recently, efforts have focused on reducing model uncertainty using multimodel combination algorithm, which is aimed to optimally combine multiple model predictions so that developed multimodel prediction could result in improved predictability. ...
Model errors are the inevitable part in any prediction exercise. One approach that is currently gaining attention to reduce model errors is by optimally combining multiple models to develop improved predictions. The rationale behind this approach primarily lies on the premise that optimal weights could be derived for each model so that the developed multimodel predictions will result in improved predictability. On the other hand, Data assimilation aims to reduce the prediction error by updating the state variables by minimizing the error covariance matrix between the model predictions and observations. In this study, we compare the performance of multimodel combination and data assimilation in improving the streamflow predictions at daily and monthly time scales. For multimodel combination, we consider two hydrological models, 'abcd' model and Variable Infiltration Capacity (VIC) model, with each model's parameter being estimated by two different objective functions to develop multimodel streamflow predictions. In addition, an ensemble representation of the latter approach, the Ensemble Kalman Filter (EnKF) is also adopted for `abcd' model to demonstrate its usefulness on streamflow predictions. The performance of both these techniques - multimodel combination and data assimilation - are compared based on their ability to predict the streamflow under known model and initial conditions with the errors in observed flows being homoscedastic/heteroscedastic. Results from the study show that, under monthly time scale, streamflow simulated from individual models performed better than multimodels under almost no model error. Under increased model error, the multimodel consistently performed better than the single model predictions and the EnKF predictions in terms of all performance measures. However, under daily time scale, the multimodel predictions are no better than the EnKF predictions under different model errors.
... Linear transfer function models, of the type considered in the present paper, have been considered by various authors over many years (e.g. Dooge, 1959, 1986; Young, 1986; Troch et al., 1991; Cluckie, 1993 and prior references on the topic cited therein). The Bedford–Ouse model (e.g. ...
The paper discusses the problems associated with environmental modelling and the need to develop simple, 'top-down', stochastic models that match the information content of the data. It introduces the con- cept of Data-Based Mechanistic (DBM) modelling and contrasts its inductive approach with the hypothetico- deductive approaches that dominate most environmental modelling research at the present time. The major methodological procedures utilized in DBM modelling are outlined and two practical examples illustrate how it has been applied in a hydrological and water quality context. The use of this same methodology as a basis for the evaluation and simplification of large deterministic simulation models is also discussed briefly.
... This appendix gives only the basic recursive formulae of the self-tuning predictor after Ambrus (1980). The reader can find further details about the algorithm in studies published by Kalman (1960), Wittenmark (1974), Ganendra (1976), Ambrus & Szollôsi-Nagy (1981), Young (1986) etc. ...
This paper deals with adaptive flow forecasting from discharge observations, with catchment rainfall as an additional input component. The precipitation input is pre-processed by approximate expressions of excess rainfall and routed by a linear cascade model. The basic equations of the algorithm and the general pattern of the computer program are presented. Examples are given for a case study area (Västerdalälven River in Sweden) where rainfall input in addition to the flow observations was used to improve the forecast accuracy. The results are discussed.
... • Metric Models, which are based primarily on observational data and seek to characterise the flow response largely on the basis of these data, using some form of statistical estimation or optimisation (e.g. Young, 1986). These include purely black-box, time-series models, such as the discrete-time transfer function or the neural network representations. ...
... Nevertheless, because of various short-term and long-term changes in the watershed, complex hydrological and hydraulic phenomena , and the incomplete information about the system, the problem of real-time estimation and the forecasting of Pakistan's rivers is still very uncertain. A variety of hydrological time series models, mostly stochastic in nature, are available (O'Donnell, 1986; Young, 1986). The drawback of these models is that they ignore possible nonlinearities inherent in the system (Roberson et al., 1998). ...
One of the world’s largest irrigation networks, based on the Indus River system in Pakistan, faces serious scarcity of water in one season and disastrous floods in another. The system is dominated both by
monsoon and by snow and glacier dynamics, which confer strong seasonal and inter-annual variability. In
this paper two different forecasting methods are utilized to analyse the long-term seasonal behaviour of the
Indus River. The study also assesses whether the strong seasonal behaviour is dominated by the presence of low-dimensional nonlinear dynamics, or whether the periodic behaviour is simply immersed in random
fluctuations. Forecasts obtained by nonlinear prediction (NLP) and the seasonal autoregressive integrated
moving average (SARIMA) methods show that the performance of NLP is relatively better than the SARIMA method. This, along with the low values of the correlation dimension, is indicative of lowdimensional
nonlinear behaviour of the hydrological dynamics. A relatively better performance of NLP, using an inverse technique, may also be indicative of the low-dimensional behaviour. Moreover, the embedding dimension of the best NLP forecasts is in good agreement with the estimated correlation dimension. This provides evidence that the nonlinearity inherent in the monthly river flow due to the snowmelt and the monsoon variations dominate over the high-dimensional components and might be exploited for prediction and modelling of the complex hydrological system.
... Such TF models, which can be formulated in discrete or continuous time, are particularly useful for modelling¯uvial systems, since they can characterize the dynamics of serial, parallel and feedback processes, whilst allowing for pure delays between system inputs and outputs (e.g. Young, 1986Young and Wallis, 1985). ...
This paper outlines the application of a new data-based mechanistic (DBM) modelling methodology to the characterization of the sediment transmission dynamics in a small upland reservoir, Wyresdale Park, Lancashire. The DBM modelling strategy exploits advanced statistical procedures to infer the dynamic model structure and its associated parameters directly from the instrumented data, producing a parametrically efficient, continuous time, transfer function model which relates suspended sediment load at the reservoir inflow to the outflow at the event scale. The associated differential equation model parameters have physical attributes which can be interpreted in terms of sediment transmission processes and associated reservoir trap efficiency. Sedigraph analysis suggests that wind-induced resuspension episodically supplies an additional load to the reservoir outlet. The stochastic nature of the DBM model makes it ideal for evaluating the effects of uncertainty through Monte Carlo simulations (MCS) for discharge and sediment transmission. Copyright © 2000 John Wiley & Sons, Ltd.
... Dooge, 1959(e.g. Dooge, , 1986Young, 1986;Troch et al., 1991;Cluckie, 1993 and prior references on the topic cited therein). The Bedford-Ouse model (e.g. ...
The data-based mechanistic (DBM) approach to modelling has developed as a stochastic, ‘top-down’ response to the problems associated with the deterministic, ‘bottom-up’ approach. As such, it can be compared with the deterministic, top-down modelling methods that have been attracting attention recently in the hydrological literature. Using catchment-scale rainfall–flow modelling as an example, this paper compares the inductive DBM approach with its hypothetico-deductive, deterministic alternative and shows how they can be used to identify and estimate low-order, nonlinear models of the rainfall–flow dynamics in the River Hodder catchment of northwest England based on a limited set of rainfall–flow data. Copyright © 2003 John Wiley & Sons, Ltd.
... Sometimes, piece-wise linearity has been assumed to handle some aspects of the nonlinear behaviour (e.g. Natale and Todini, 1976a,b); while, in other cases, the model parameters have been allowed to vary with time (Young, 1974;Whitehead and Young, 1975;Whitehead et al., 1976;Young and Wallis, 1985;Young, 1986). ...
This article discusses continuous and discrete-time transfer function (TF) models within the context of the Data-Based Mechanistic (DBM) modeling of hydrological systems. Although, at a superficial level, TF models simply provide a convenient and concise way of presenting differential or difference equations in an input-output form, they are much more than this. In particular, they allow for the consideration of linear dynamic systems in simple algebraic terms and for their analysis within a dynamic systems and control context. This is useful in various ways, particularly where a higher order TF model is identified and estimated directly from hydrological data. Often, this data-based model can then be decomposed into serial, parallel, and feedback connections of first-order systems that can be interpreted in hydrologically meaningful terms, thereby facilitating the model's use in hydrological systems analysis. The article also shows that TF models can be considered in time-variable and state-dependent parameter (SDP) form, so allowing for them to describe nonstationary and nonlinear systems. And, since it is straightforward to consider all TF models in stochastic terms, they provide a powerful vehicle for uncertainty, forecasting, and risk analysis. The practical utility and power of DBM TF models is illustrated by two real hydrological examples.
Keywords:
transfer functions;
differential equations;
continuous time;
discrete time;
transfer function decomposition;
data-based mechanistic models
... The derivation of equation (17) in this manner is perfectly credible in physical terms: indeed the derivation, including the assumption (19), is the basis of the physically-based "lag-and-route" models that have been used for some considerable time by hydrologists (see e.g. Young, 1986). ...
Although rainfall-flow processes have received much attention in the hydrological literature, the nature of the non-linear processes involved in the relationship between rainfall and river flow still remains rather unclear. This paper outlines the first author's data-based mechanistic (DBM) approach to model structure identification and parameter estimation for linear and non-linear dynamic systems and uses it to explore afresh the non-linear relationship between measured rainfall and flow in two typical catchments. Exploiting the power of recursive estimation, state dependent non-linearities are identified objectively from the time series data and used as the basis for the estimation of non-linear transfer function models of the rainfall-flow dynamics. These objectively identified models not only explain the data in a parametrically efficient manner but also reveal the possible parallel nature of the underlying physical processes within the catchments. The DBM modelling approach provides a useful tool for the further investigation of rainfall-flow processes, as well as other linear and non-linear environmental systems. Moreover, because DBM modelling uses recursive estimation, it provides a powerful vehicle for the design of real-time, self-adaptive environmental management systems. Finally, the paper points out how DBM models can often be interpreted directly in terms of dynamic conservation equations (mass, energy or momentum) associated with environmental flow processes and stresses the importance of parallel processes in this connection.
Time-series modeling is a well-studied topic of classical analysis and machine learning. However, large datasets are required to obtain the model with a better prediction quality with the increasing model complexity. Therefore, some applications demand synthetic datasets that are preserving modeling-sensitive properties. Another application of synthetic data is data anonymization. The synthetic data generation algorithm may be split into two parts: the time-series modeling and the synthetic data generation parts. The model must be interpretable to obtain the synthetic data with good quality. The model parameter interpretation allows controlling generation by adding noise to different groups of parameters. In the paper, the evolutionary multi-objective closed-form algebraic expressions discovery approach that allows obtaining the model in the form that may be analyzed using the mathematics is proposed. The analysis allows the interpretation of the model parameters for the controllable generation of the synthetic data. The notion of synthetic data quality is discussed. The examples of the synthetic time-series generation based on two datasets with different properties are shown.
Most of the studies related to the rainfall-runoff modeling of rivers consist of data-driven models, given that the corresponding physical modeling approaches are based on a thorough geological knowledge of the river in addition to a time consuming simulation. Indeed, flood forecasting services have the difficult task of avoiding natural and human disasters and choose for that to use input-output or grey box models for their simplicity and easy calibration updates. However, these models are not evolving according to the variations of environmental conditions or need at least the evapotranspiration and the soil humidity measurements in addition to the rainfall quantity. This paper gives an alternative approach to the existing rainfall/runoff linear and nonlinear models by the utilization of a hybrid system consisting in a Piecewise Auto-Regressive eXogeneous (PWARX) structure identified using an approach that alternates between data assignment and parameter estimation. The usage of this special kind of nonlinear systems bears a potential to handle the nonlinearities and varying-time delays mainly induced by the soil water storage and evapotranspiration.
This work regards the improvement of water asset management and modelling strategies proposed recently for hydrographical networks in a context of global warming which exacerbates extreme events. Indeed, hydrographical systems, and more specifically those including inland waterways, are large scale systems that involve mass energy transport phenomena consisting in the amount of water in excess induced by extreme rainy events. The water excess has to be dispatched over the complete network in order to anticipate the effects of potential floods and rejected to the sea either by gravity flow heeding the tides or by the utilization of pumps. The latter solution leads to big operation costs and has to be minimized. These goals are fulfilled by means of an integrated model within a flow-based network and a quadratic optimization based on constraints. The recent improvements in the field of predictive rainfall/runoff modelling approaches helps to enhance the operational goals of this water management strategy. A particular example of these approaches are those related to the identification of black-box ARX models in its linear and nonlinear structures. A performance comparison between the recursive least square and the recursive instrumental variable estimation of an ARX model and the estimation of a Linear Parameter Varying ARX-based model is carried out on a river located in the north of France with a prediction horizon of 24 h. Further, the performance of the resource management methodology is tested by means of a realistic case study based on a portion of the real inland waterways located in the north of France.
An implementation of a precise flood forecasting is becoming increasingly urgent considering the growing number of climatic disasters. Several solutions of water asset management based or not on optimization and control exist but they require primarily on a good prediction of the volumes triggered by rainfall after a storm. Numerous studies are dedicated to this challenging issue that have this main objective of how to predict the runoff by using the right model with taking into account the nonlinearities induced by the geographical features of the catchment. This paper present an alternative to the large scale hydrological mathematical model based on offline and recursive/online parameter estimation of data-driven linear and nonlinear models. The updating of the model parameters allows to handle the variations due to the past rainfall quantities and to the weather forecasting.
By ‘environmetrics’, we mean the application of statistical and systems methods to the analysis and modelling of environmental data. In this chapter, we consider a particular class of environmetric methods; namely the analysis of environmental time-series. Although such time-series can be obtained from planned experiments, they are more often obtained by passively monitoring the environmental variables over long periods of time. Not surprisingly, therefore, the statistical characteristics of such series can change considerably over the observation interval, so that the series can be considered nonstationary in a statistical sense. Figure 1(a), for example, shows a topical and important environmental time-series—the variations of atmospheric CO2 measured at Mauna Loa in Hawaii over the period 1974 to 1987. This series exhibits a clear upward trend, together with pronounced annual periodicity. The trend behaviour is a classic example of statistical nonstationarity of the mean, with the local mean value of the series changing markedly over the observation interval.
Durch die Korrelation werden Zusammenhänge zwischen mehreren Kollektiven von Zufallsgrößen untersucht. Dieser Zusammenhang ist nicht streng funktional, wird aber in der Form y = f(x) ausgedrückt und verbindet hydrologische Beobachtungen, zwischen denen eine sachlogische Beziehung hergestellt werden soll. So hängen die Wasserstände längs eines Flusses voneinander ab. Im allgemeinen hat das Steigen des Wasserstandes an einem Punkt auch einen steigenden Wasserstand an einem flußabwärts gelegenen Punkt zur Folge. Niederschläge im Zwischeneinzugsgebiet, seitliche Zuflüsse und andere Störgrößen sind Ursachen dafür, daß einem bestimmten Wasserstand am ersten Pegel nicht eindeutig ein bestimmter Wasserstand am zweiten unterhalb gelegenen Pegel entspricht. Vielmehr kann dem Wasserstand am ersten Pegel nur ein Kollektiv von Wasserständen am zweiten Pegel zugeordnet werden. Die Vorhersage des Wasserstandes am unterhalb gelegenen Pegel kann mit der Korrelationsrechnung gelöst werden.
Time series modelling has already taken place in hydrological technology. ARMA, AR models, etc. are devoted to preserving statistical properties from the stochastic process underlying a given sample, to generate long undistinguishable synthetic samples to provide for better analysis or derived processes. These models are characterized by the use of information from the analyzed series.
Many researchers in the field of real-time flood forecasting recognize the time-dependent nature of the rainfall-runoff relationship. However, for isolated storm events, the effects of this time-variant behaviour are very often neglected, leading to the definition of linear time-invariant systems or non-linear time-invariant systems. In the latter case, the non-linearity can be introduced using some form of rainfall separation (e.g. threshold parameters) or as an additional deterministic signal (e.g. Catchment Wetness Index). More advanced modelling techniques to deal with non-linearity are Volterra and Wiener series expansion (Napiorkowski, 1986). This paper deals with linear time-variant (adaptive) modelling and forecasting. This modelling procedure is based on an on-line identification of the model parameters using the recursive instrumental variable estimator. The variation of the parameters is modelled using a random walk model. Some objective hydrological criteria for the evaluation of forecasting performance are introduced. This modelling technique is then compared, using real-world data from the catchment of the River Vesdre, with linear time-invariant modelling and forecasting. It can be concluded that, in general, adaptive modelling can improve the real-time forecasting performance within the linear framework.
This thesis reports work undertaken to improve modelling of faecal coliform
dynamics in river systems. A broad literature review introduces the many
environmental processes that influence faecal coliform concentrations in rivers and
highlights approaches to modelling. Channel flushing and resuspension of organisms
from the riverbed can significantly lower the quality of the overlying water. A series
of experiments investigated the impact of reservoir releases on the entrainment of
organisms from storage within the channel. The results suggest three phases of
transport: wavefront transport by near bed steep-fronted waves, wavefront
entrainment with transport at the slower mean velocity, and burst-sweep cycle
entrainment maintaining elevated concentrations during continued high flow. A
model is presented and demonstrated to reproduce the timing and concentration of
the entrainment episodes to a high degree of accuracy. The model has the potential
for application to the transient movement of other particle-associated contaminants.
Sunlight is the major influence on faecal coliform die-off. Direct measurements of
solar irradiance are commonly not available a method is presented for estimating
irradiance from daily sunshine hours. The estimated irradiance is incorporated into
water quality models that effectively reproduce the variations in faecal coliform
concentration over a seven-year period. Measurements of irradiance or sunlight
hours should be one of the prime considerations for any monitoring programme
concerned with bacterial concentrations in environmental waters.
Operational flood forecasting requires accurate forecasts with a suitable lead time, in order to be able to issue appropriate warnings and take appropriate emergency actions. Recent improvements in both flood plain characterization and computational capabilities have made the use of distributed flood inundation models more common. However, problems remain with the application of such models. There are still uncertainties associated with the identifiability of parameters; with the computational burden of calculating distributed estimates of predictive uncertainty; and with the adaptive use of such models for operational , real-time flood inundation forecasting. Moreover, the application of distributed models is complex, costly and requires high degrees of skill. This paper presents an alternative to distributed inun-dation models for real-time flood forecasting that provides fast and accurate, medium to short-term forecasts. The Data Based Mechanistic (DBM) methodology exploits a State Dependent Parameter (SDP) modelling approach to derive a nonlinear dependence between the water levels measured at gauging stations along the river. The transformation of water levels depends on the relative geometry of the channel cross-sections, without the need to apply rating curve transformations to the discharge. The relationship obtained is used to transform water levels as an input to a linear, on-line, real-time and adaptive stochas-tic DBM model. The approach provides an estimate of the prediction uncertainties, including allowing for heterescadasticity of the multi-step-ahead forecasting errors. The approach is illustrated using an 80 km reach of the River Severn, in the UK.
In this Paper, initial results obtained from a detailed study of the aggregated dead zone (ADZ) model for longitudinal solute transport and dispersion in river and channel flow are presented. This model is based on a novel approach to the description of advection and dispersion which is able to reproduce observed solute concentrations to a high degree of accuracy. During the past three years, the study team has conducted many tracer experiments in a number of rivers in the north-west of England, using a specially developed microcomputer-based fluorometric data logging and analysis system. These data have been used to calibrate the ADZ model, using advanced methods of statistical time series analysis both to identifiy the appropriate model structure and to estimate the associated model parameters. The complete analysis of the results from four short reaches shows that the model parameters exhibit a sensible relationship with river discharge and, moreover, that they can be combined in a simple, non-dimensional ratio which remains essentially constant for each reach over the complete range of discharges encountered during the study.
Computational experiments are carried out with the rainfall field model described in Part 1 of this paper and a physically based distributed modelling system, the Système Hydrologique Européen (SHE), to explore the interaction between spatial variability in rainfall and other factors controlling catchment response; both models have been calibrated for the small upland Wye catchment (area 10.55 km2). Simulated rainfall fields are used to provide fully distributed rainfall inputs to the SHE; the corresponding ‘true’ catchment responses are then compared with those derived from incomplete sampling of the rainfall fields. The differences in simulated peak discharges and runoff volumes are assessed as a function of antecedent catchment conditions, network density and the level of spatial correlation in the rainfall input. A piecewise linear transfer function model with an averaged rainfall input is used to approximate SHE responses to fully distributed rainfall inputs, to provide insight into simple lumped model performance.
The full project provides a full report with technical details of the choice of catchments, methods of analysis and full results described in 5 technical reports.
The purpose of the study was to further the understanding of unsteady transport of coarse bedload. A particular focus was the interaction of the load with the river bed material and the effects of transport on stream hydraulics. It was anticipated that new methodology would be developed and that data sets suitable for further modelling would result. The nature of coarse gravel transport through a short reach of a mountain river was examined using novel methodology; primarily, magnetic particle detection; acoustic bedload detection, electromagnetic current meters and fine-resolution bed elevation surveys.
This Chapter outlines a unified approach to the identification, estimation and control of linear, continuous-time, stochastic, dynamic systems which can be described by delta (δ) operator models with constant or time-variable parameters. It shows how recursive refined instrumental variable estimation algorithms can prove effective both in off-line model identification and estimation, and in the implementation of self-tuning or self-adaptive True Digital Control (TDC) systems which exploit a special Non-Minimum State Space (NMSS) formulation of the δ operator models.
The SMAR-AR Model is used for real-time riverflow forecasting on the Blue Nile catchment at Eldeim Station, which is located near the Sudanese-Ethiopian border. The SMAR-AR model is calibrated using data for the period 1992–1994 and verified by the data for the year 1995. The results of the SMAR-AR model are compared with those of the SAMFIL model that is currently used by the Sudanese Ministry of Irrigation and Water Resources (MOIWR) for riverflow forecasting at Eldeim Station. The SAMFIL was developed and brought into operation in Sudan by Delft Hydraulics of the Netherlands. The comparison of results is carried out during four flood seasons for the years 1992–1995. The discharge forecasts of the SAMFIL model are obtained from the archives of the MOIWR. The results of the comparison show that the discharge forecasts of the SMAR-AR model are far more reliable than those of the SAMFIL.
This contribution considers the methods available to hydrologists to simulate flow routing and inundation in natural channels. Whilst the fluid dynamics of free surface flow in rivers and over floodplains may be complex, the methods available to treat such problems range widely in complexity from simple empirical models to full solutions of three-dimensional Navier?Stokes equations with sophisticated representations of flow turbulence. These methods are reviewed in detail, followed by a consideration of their requirements for topography, friction, initial condition, boundary condition, and validation data. In particular, the contribution discusses how these needs are being met increasingly using remote sensing techniques. Lastly, the contribution considers techniques to evaluate uncertainties in flow modeling.
Most well-known time-series methods treat the system as a univariate, bivariate or multivariate ‘black box’whose parameters provide a convenient and concise description of the data. This is in contrast to physically based, mechanistic models, whose parameters normally have an identifiable physical interpretation. The present paper describes a unified ‘data-based mechanistic’approach to the modelling of dynamic systems from time-series data using continuous or discrete-time transfer function models in the time derivative, backward shift or delta operator. This approach, which exploits recursive methods of parameter estimation, represents a useful compromise between the physically based methods of mechanistic modelling and the ‘black box’methods of time-series analysis. It provides a powerful tool for the objective investigation of environmental dynamic systems when time-series data are available for analysis. Its practical potential is illustrated by several real examples concerned with the objective investigation of parallel processes in hydrology and water quality.
The instrumental variable-approximate maximum likelihood (IV0AML) method provides a technique to develop better models for short-time increment hydrologic data. In this method, a recursive input-output model, which consists of a deterministic model and a stochastic noise model are used. These models handle the system and measurement noise separately. The instrumental variable method has been developed to eliminate the bias in parameter estimates.
The IV-AML method is investigated in the present study. Parameters of daily rainfall-runoff models are estimated by the IV-AML and by least squares methods and compared. The effects of a rainfall filter on parameter estimates are also investigated. Forecast accuracies of models whose parameters are estimated by IV-AML and least squares methods are compared.
The results indicate that the forecast accuracy of models whose parameters are estimated by least squares method is comparable to that of models whose parameters are estimated by IV-AML method. The rainfall filter, on the other hand, reduces the parameter variation and improves forecasts.
This paper describes an adaptive hydrologic modelling technique for real-time flood forecasting. The modelling approach is based on a linear stochastic time-varying representation of the rainfall-runoff process and on the Muskingum routing method formulated as an optimal linear filtering problem. The most general stochastic rainfall-runoff model used for linear forecasting is known as the transfer function noise model. An on-line identification procedure based on an extension of the recursive Instrumental Variable estimator is discussed. The routing procedure, based on the Muskingum method, is written in a state-space representation. This allows real-time updating of the state and the system parameters by means of Kalman filtering. The described method is used to forecast extreme flood events for the River Ourthe (drainage basin: approx 3626 km2), one of the main tributaries of the River Meuse, Belgium. The method is compared with stationary modelling procedures and its superiority based on objective forecasting criteria is demonstrated.
The purpose of this paper is two-fold: firstly, it is meant to be partly tutorial, to survey recursive least squares analysis in all its forms; secondly, it discusses two recent developments in the area of recursive least squares analysis, both of which have particular relevance to the estimation of parameters characterising discrete and continuous-time series models of stochastic dynamic systems.
Samples of recent sediments from 18 sites in Lake Illawarra showed marked enrichment of copper, zinc, cadmium and lead. The highest concentrations were found in Griffins Bay in the north-east corner of the lake, which is closest to the Port Kembla industrial complex. Maximum enrichments for copper, zinc and lead are respectively 4, 34 and 11 times the background concentrations found in deep cores. By comparison of vertical profiles of heavy metal concentration with historical data on the growth of metal smelting industries in the Illawarra region, the recent sedimentation rate has been estimated as 2.5-3 mm/year. A depth of 40-50 cm below the surface of the sediment marks the beginning of human influence on the lake.
Surface and bottom water salinities were determined at 10 monitoring stations over a 2-year period. Mean salinity is controlled primarily by rainfall and varied from 12.8 to 31.3%, during the monitoring period.
The rate of flushing of Lake Illawarra by tidal action and by freshwater entry was estimated from salinity variations and by mathematical modelling; for dry periods a flushing time (for reduction by a factor of e) of about 60 days was calculated for a conservative substance introduced into the lake. Mean tidal volume was 1 % of the lake volume.
Vertical mixing by wave action is complete within 2-4 weeks. Although the major creeks are on the western side of the lake, no significant east-west salinity gradient was found. Lake Illawarra is characterized by the small ratio of catchment to lake area and the short distance from the very elevated watershed to the lake shores, resulting in rapid response to rainfall. The volume of direct precipitation was estimated to be comparable with creek input.
This paper describes the refined IVAML algorithm for estimating the parameters in the multivariable analogue of the single input,. single output model considered in Part I. It also shows that similar refined algorithms can be constructed for other multivariable model forms, including the ARMAX, dynamic adjustment (DA) with autoregressive errors and the multivariable transfer function (MTF). The performance of the algorithm is evaluated by Monte Carlo analysis applied to four simulation models with between 25 and 39 parameters and it is carried out for various sample sizes and signal/noise ratios. As in the SISO model version, the analysis indicates that the algorithm yields asymptotically efficient estimation results, whilst providing low variance estimates of the basic system parameters for medium sample sizes.
This paper is the first in a series concerned with a comprehensive evaluation of the refined instrumental variable-approximate maximum likelihood (IVAML) method of time-series analysis. The implementation of a recursive/iterative version of the refined IVAML algorithm for single input, single output systems is discussed in detail and the performance of the algorithm is evaluated by Monte-Carlo simulation analysis applied to five simulated stochastic systems. As conjectured, the algorithm appears to yield asymptotically efficient estimates of the time-series model parameters and, indeed, it seems to approach minimum variance estimation of the basic system model parameters for even low sample size and low signal/noise ratios. The noise model parameters are not estimated so well at the smaller sample sizes but the estimation performance appears similar to that of other competing methods of analysis, such as recursive maximum likelihood (RML). Subsequent papers on this same general topic will deal with extensions of the refined IVAML procedure to handle multivariate systems, time-variable parameters and the estimation of continuous-time systems described by ordinary differential equations.
It. is becoming increasingly clear that in the area of environmental systems analysis, where systems are often badly defined and models are subject to considerable uncertainty, recursive methods of time-series analysis can be particularly useful both in the initial and critical stages of system identification, as well as in the final stages of parameter estimation and model validation. Of the existing methods of time-series analysis, those based upon the method of instrumental variables are amongst the simplest and. at the same time, demonstrably most useful in practical applications. This paper discusses the instrumental variable (IV) and approximate maximum likelihood (AML) methods of recursive time-series analysis and shows how they can be unified to some extent within the context of maximum likelihood estimation. In this manner the virtues and limitations of the existing IV and AML techniques become more apparent and possible procedures for improving their statistical efficiency are exposed.
The paper describes a method of automatic process parameter estimation which has been mechanised with the aid of hybrid (analogue-digital) equipment. The technique is characterised by it’s simplicity, and differs from earlier schemes of this type in not requiring direct measurement of the input and output time derivatives of the process. The performance of the system is discussed, with particular reference to the effect of uncertainty on the sampled data caused by spurious noise contamination. Finally, a simple identification-adaptive control system for a nonstationary process is described, which utilises the hybrid parameter estimation techniques.
This paper surveys the convergence analysis and the convergence properties of recursive identification methods for dynamical, stochastic systems. For the survey, two classes of recursive estimators are recognized, to which most of the commonly used methods belong. These are recursive prediction error methods and pseudo linear regressions, respectively. The former possess the general property that they converge to a local minimum of a certain criterion, while convergence of the latter class is tied to positive realness of certain transfer functions associated with the true system.
When time-series methods of parameter estimation are used to model the concentration of a pollutant in a natural waterway, the results may be grouped into a restricted class of models which are termed (j, n, m) models. It is shown that the dispersion as represented in conservative forms of these models has a temporal variance that increases linearly with time (thus mimicking the variance of a Fickian process) and is consistent with dead zone mixing within rivers and streams. Data from a variety of rivers are used to demonstrate that the (j, n, m) model provides an explanation of actual data that is superior to those obtained by manipulation of a Fickian equation. In addition, the model greatly simplifies computation of dead zone dispersive effects, is able to simply deal with non-conservative pollutants and the results suggest that residence times are more meaningful physical parameters than dispersion coefficients in explaining dispersive phenomena.
The Extended Kalman Filter (EKF) provides a logical statistically based extension to those existing approaches to model fitting based on deterministic model response error (surface) minimization. Its crucial feature as a basis for identification, however, is the recursive nature of the algorithm which permits the estimation of possible variations in the model parameters. Depending upon whether such estimated variations are realistic or not, bearing in mind the physical nature of the dynamic system, it is possible to formulate criteria for model adequacy. This approach to model identification was applied to the problem of modeling DO BOD interaction in a freshwater river system based on daily field data. It was found that the basic model structure, as defined by a dynamic version of the Streeter Phelps equations, was inadequate and, in the case of the river considered, it was necessary to introduce additional 'sustained sunlight' terms to account for the effects of floating algal populations.
If the dictionary definition were the sole criterion, a model would be considered valid if it was found to be well grounded, sound, cogent, logical, and incontestable. Similarly, a model would be deemed credible if it was deserving of or entitled to belief, or if it was plausible, tenable, or reasonable. All of these characteristics are, of course, desirable in a mathematical model of a physical system; but when used as the basis for the definition of model adequacy, they are clearly too subjective to provide useful and rigorous criteria for model evaluation.
The paper describes a simple instrumental variable method for identifying the structure of a wide class of time-series models. The method is aimed at providing a parametrically efficient (parsimonious) model structure which will lead to a combination of low residual error variance, i.e. a good explanation of the data, and low parametric estimation error variance (as measured by some norm associated with the covariance matrix of the estimation errors). It can be applied to single input-single output and multivariable systems using either discrete or continuous-time series models. It can also function as a recursive (on-line) test for reduction in model order.
The problem of identifying a dynamic process from its normal operating data has received considerable attention in recent years. The various techniques developed range from largely deterministic procedures to sophisticated statistical methods based on the results of optimal estimation theory. The instrumental variable (IV) technique outlined in this paper is intended as a compromise between these two extremes; it has a basis in classical statistical estimation theory, but does not require a priori information on the signal and noise statistics.
Edited versions of a portion of the proceedings of the ''International Symposium on Real-Time Operation of Hydrosystems'' are presented. Many aspects leading to the physical operation of these systems are discussed, for example, operation in real time, forecasting and long-range system performance.
This paper. the first of a two part description of the modelling activities associated with the Bedford-Ouse River Study, concentrates on streamflow characterisation. The streamflow models are of a stochastic-dynamic type in which a simple lumped parameter differential equation model for mainstream flow is enhanced by stochastic time-series descriptions of rainfall-runoff behaviour. The models have been developed using a new systematic approach to the modelling of badly defined dynamic systems which is centred around the exploitation of recursive methods of parameter estimation and time-series analysis. At the same time. the models have quite strong links with more conventional models used previously in hydrological systems analysis and the implications of the modelling results can easily be interpreted in conventional hydrologic terms. An important aspect of the modelling exercises described in the paper is that they are objective orientated and, in the Bedford-Ouse Study. the models are developed specifically with operational control and management applications in mind.
This is the final paper in a series of three which have been concerned with the comprehensive evaluation of the refined instrumental variable (IV) method of recursive time-series analysis. The paper shows how the refined IV procedure can be extended in various important directions and how it can provide the basis for the synthesis of optimal generalized equation error (GEE) algorithms for a wide class of stochastic dynamic systems. The topics discussed include the estimation of parameters in continuous-time differential equation models from continuous or discrete data; the estimation of time-variable parameters in continuous or discrete-time models of dynamic systems ; the design of stochastic state reconstruction (Wiener-Kalman) filters direct from data ; the estimation of parameters in multi-input, single output (MISO) transfer function models ; the design of simple stochastic approximation (SA) implementations of the refined IV algorithms ; and the use of the recursive algorithms in self-adaptive (self tuning) control.
As the width of a vertically well-mixed estuary increases, the longitudinal dispersion coefficient eventually reduces from a value associated with the transverse shear of the tidal current to a much smaller value associated with the oscillatory vertical shear. In this paper the final stages of this transition are investigated, with particular emphasis on buoyancy effects due to the salinity distribution. The central mathematical result relates the long-term longitudinal dispersion coefficient to the local unsteady horizontal dispersion coefficients and to the residual horizontal circulation. A useful consequence of this result is a demarcation of the parameter regimes in which the various mass-transport mechanisms are dominant. The Thames downstream of London Bridge is revealed to be buoyancy dominated.
The paper describes a method of automatic process parameter estimation which has been mechanised with the aid of hybrid (analogue-digital) equipment. The technique is characterised by it’s simplicity, and differs from earlier schemes of this type in not requiring direct measurement of the input and output time derivatives of the process. The performance of the system is discussed, with particular reference to the effect of uncertainty on the sampled data caused by spurious noise contamination. Finally, a simple identification-adaptive control system for a nonstationary process is described, which utilises the hybrid parameter estimation techniques.
The classical filtering and prediction problem is re-examined using the Bode-Sliannon representation of random processes and the “state-transition” method of analysis of dynamic systems. New results are: (1) The formulation and methods of solution of the problem apply without modification to stationary and nonstationary statistics and to growing-memory and infinitememory filters. (2) A nonlinear difference (or differential) equation is derived for the covariance matrix of the optimal estimation error. From the solution of this equation the coefficients of the difference (or differential) equation of the optimal linear filter are obtained without further calculations. (3) The filtering problem is shown to be the dual of the noise-free regulator problem. The new method developed here is applied to two well-known problems, confirming and extending earlier results. The discussion is largely self-contained and proceeds from first principles; basic concepts of the theory of random processes are reviewed in the Appendix.
Mathematical models of hydrological and water-resource systems have been formulated in many different ways and with various levels of complexity. There are advantages to be gained, therefore, by trying to unify some of the more common models within a statistical framework which will allow for more objective methods of model calibration. In this paper, we consider the general class of linear, dynamic models, as applied to the characterisation of flow and dispersion behavior in rivers, and show how these can be unified within the context of recursive time-series analysis and estimation. This allows not only for more objective, data-based approaches to stochastic model structure identification, but also for improved statistical estimation and the development of both constant parameter and self-adaptive, Kalman-filter-based forecasting procedures. The unified approach presented in the paper is being applied successfully in other environmental areas, such as soil science, climatic data analysis, meterological forecasting, and plant physiology.
The paper reviews the progress of research on parameter estimation for continuous-time models of dynamic systems over the period 1958–1980. Major developments are considered in historical order and within a classification system which conforms as closely as possible to that which has arisen naturally over the past two decades. While every attempt is made to consider progress in other related scientific disciplines, such as econometrics, the major accent is on research in the control and systems field.
This book is intended to introduce the reader to aspects of the theory and practice of recursive estimation, including its use in time-series analysis. A number of other excellent books, dealing at least in part with recursive estimation, have been available for some time (e.g. Sage, 1968; Bryson and Ho, 1969; Jazwinski, 1970; Gelb, 1974 etc.), but these tend to be orientated towards state estimation and Kalman filtering, rather than parameter estimation. Others (e.g. Tsypkin, 1971; Mendel, 1973; Eykhoff, 1974; Sorenson, 1980 etc.) deal in part with recursive parameter estimation but do not consider, in depth, the problems of recursive analysis applied to time-series models.
Three case studies in Aircraft Parameter Identification using simulated data for X-22 VTOL aircraft and flight test data for HL-10 and M2/F3 lifting bodies. After a brief discussion of the previous techniques and their limitations, a technique based on the Maximum Likelihood criterion is described. The problems of identifiability and uniqueness in determining the Stability and Control derivatives from flight test data are discussed and several methods for alleviating these problems are presented. The flight test data is analyzed in several different ways for obtaining physically meaningful estimates for the aircraft parameters.
It is shown how the refined instrumental variable (r.i.v.) method of recursive parameter estimation can be modified simply so that it functions as an optimal adaptive filter and state-estimation algorithm.
This paper explores the possibility of using the instrumental variable method to estimate the parameters of linear time-invariant discrete-time systems. The existence of optimal estimates is established, methods for their approximate computation are given, and an on-line identification scheme based on recursive computation is proposed. Experimental results are included.
A Comparative study of dynamic models for D0-B0D- Algae interaction in a freshwater river
- M B Beck
- MB Beck
Systems analysis of an estuary: the Peel-Harvey estuarine study (1976-1980)
- R B Humphries
- P C Young
- T Beer
- RB Humphries
Convergence of recursive estimators Identification and System Parameter Estimation
- L Ljung
Analysis of hydrological data using microCAPTAIN
- J R Moll
- JR Moll
An on-line algorithm for approximate maximum likelihood identification of linear dynamic systems
- T Soderstrom
Mutually interactive state/parametric (MISP) estimation in hydrological applications
- E Todini
Control System Synthesis
- T G Truxal
- TG Truxal