No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A robust sparse identification method for nonlinear dynamic systems affected by non-stationary noise
Chaos
Abstract
In the field of science and engineering, identifying the nonlinear dynamics of systems from data is a significant yet challenging task. In practice, the collected data are often contaminated by noise, which often severely reduce the accuracy of the identification results. To address the issue of inaccurate identification induced by non-stationary noise in data, this paper proposes a method called weighted ℓ1-regularized and insensitive loss function-based sparse identification of dynamics. Specifically, the robust identification problem is formulated using a sparse identification mathematical model that takes into account the presence of non-stationary noise in a quantitative manner. Then, a novel weighted ℓ1-regularized and insensitive loss function is proposed to account for the nature of non-stationary noise. Compared to traditional loss functions like least squares and least absolute deviation, the proposed method can mitigate the adverse effects of non-stationary noise and better promote the sparsity of results, thereby enhancing the accuracy of identification. Third, to overcome the non-smooth nature of the objective function induced by the inclusion of loss and regularization terms, a smooth approximation of the non-smooth objective function is presented, and the alternating direction multiplier method is utilized to develop an efficient optimization algorithm. Finally, the robustness of the proposed method is verified by extensive experiments under different types of nonlinear dynamical systems. Compared to some state-of-the-art methods, the proposed method achieves better identification accuracy.
ResearchGate has not been able to resolve any citations for this publication.
We study collective failures in biologically realistic networks that consist of coupled excitable units. The networks have broad-scale degree distribution, high modularity, and small-world properties, while the excitable dynamics is determined by the paradigmatic FitzHugh–Nagumo model. We consider different coupling strengths, bifurcation distances, and various aging scenarios as potential culprits of collective failure. We find that for intermediate coupling strengths, the network remains globally active the longest if the high-degree nodes are first targets for inactivation. This agrees well with previously published results, which showed that oscillatory networks can be highly fragile to the targeted inactivation of low-degree nodes, especially under weak coupling. However, we also show that the most efficient strategy to enact collective failure does not only non-monotonically depend on the coupling strength, but it also depends on the distance from the bifurcation point to the oscillatory behavior of individual excitable units. Altogether, we provide a comprehensive account of determinants of collective failure in excitable networks, and we hope this will prove useful for better understanding breakdowns in systems that are subject to such dynamics.
Percolation establishes the connectivity of complex networks and is one of the most fundamental critical phenomena for the study of complex systems. On simple networks, percolation displays a second-order phase transition; on multiplex networks, the percolation transition can become discontinuous. However, little is known about percolation in networks with higher-order interactions. Here, we show that percolation can be turned into a fully fledged dynamical process when higher-order interactions are taken into account. By introducing signed triadic interactions, in which a node can regulate the interactions between two other nodes, we define triadic percolation. We uncover that in this paradigmatic model the connectivity of the network changes in time and that the order parameter undergoes a period doubling and a route to chaos. We provide a general theory for triadic percolation which accurately predicts the full phase diagram on random graphs as confirmed by extensive numerical simulations. We find that triadic percolation on real network topologies reveals a similar phenomenology. These results radically change our understanding of percolation and may be used to study complex systems in which the functional connectivity is changing in time dynamically and in a non-trivial way, such as in neural and climate networks. Triadic interactions are higher-order interactions relevant to many real complex systems. The authors develop a percolation theory for networks with triadic interactions and identify basic mechanisms for observing dynamical changes of the giant component such as the ones occurring in neuronal and climate networks.
This work develops a regularized least absolute deviation-based sparse identification of dynamics (RLAD-SID) method to address outlier problems in the classical metric-based loss function and the sparsity constraint framework. Our method uses absolute derivation loss as a substitute of Euclidean loss. Moreover, a corresponding computationally efficient optimization algorithm is derived on the basis of the alternating direction method of multipliers due to the non-smoothness of both the new proposed loss function and the regularization term. Numerical experiments are performed to evaluate the effectiveness of RLAD-SID using several exemplary nonlinear dynamical systems, such as the van der Pol equation, the Lorenz system, and the 1D discrete logistic map. Furthermore, detailed numerical comparisons are provided with other existing methods in metric-based sparse regression. Numerical results demonstrate that (1) RLAD-SID shows significant robustness toward a large outlier and (2) RLAD-SID can be seen as a particular metric-based sparse regression strategy that exhibits the effectiveness of the metric-based sparse regression framework for solving outlier problems in a dynamical system identification.
The sparse identification of nonlinear dynamics (SINDy) is a regression framework for the discovery of parsimonious dynamic models and governing equations from time-series data. As with all system identification methods, noisy measurements compromise the accuracy and robustness of the model discovery procedure. In this work we develop a variant of the SINDy algorithm that integrates automatic differentiation and recent time-stepping constrained motivated by Rudy et al. [1] for simultaneously (i) denoising the data, (ii) learning and parametrizing the noise probability distribution, and (iii) identifying the underlying parsimonious dynamical system responsible for generating the time-series data. Thus within an integrated optimization framework, noise can be separated from signal, resulting in an architecture that is approximately twice as robust to noise as state-of-the-art methods, handling as much as 40% noise on a given time-series signal and explicitly parametrizing the noise probability distribution. We demonstrate this approach on several numerical examples, from Lotka-Volterra models to the spatio-temporal Lorenz 96 model. Further, we show the method can learn a diversity of probability distributions for the measurement noise, including Gaussian, uniform, Gamma, and Rayleigh distributions.
We present two approaches to system identification, i.e. the identification of partial differential equations (PDEs) from measurement data. The first is a regression-based variational system identification procedure that is advantageous in not requiring repeated forward model solves and has good scalability to large number of differential operators. However it has strict data type requirements needing the ability to directly represent the operators through the available data. The second is a Bayesian inference framework highly valuable for providing uncertainty quantification, and flexible for accommodating sparse and noisy data that may also be indirect quantities of interest. However, it also requires repeated forward solutions of the PDE models which is expensive and hinders scalability. We provide illustrations of results on a model problem for pattern formation dynamics, and discuss merits of the presented methods.
Discovering governing physical laws from noisy data is a grand challenge in many science and engineering research areas. We present a new approach to data-driven discovery of ordinary differential equations (ODEs) and partial differential equations (PDEs), in explicit or implicit form. We demonstrate our approach on a wide range of problems, including shallow water equations and Navier–Stokes equations. The key idea is to select candidate terms for the underlying equations using dimensional analysis, and to approximate the weights of the terms with error bars using our threshold sparse Bayesian regression. This new algorithm employs Bayesian inference to tune the hyperparameters automatically. Our approach is effective, robust and able to quantify uncertainties by providing an error bar for each discovered candidate equation. The effectiveness of our algorithm is demonstrated through a collection of classical ODEs and PDEs. Numerical experiments demonstrate the robustness of our algorithm with respect to noisy data and its ability to discover various candidate equations with error bars that represent the quantified uncertainties. Detailed comparisons with the sequential threshold least-squares algorithm and the lasso algorithm are studied from noisy time-series measurements and indicate that the proposed method provides more robust and accurate results. In addition, the data-driven prediction of dynamics with error bars using discovered governing physical laws is more accurate and robust than classical polynomial regressions. © 2018 The Author(s) Published by the Royal Society. All rights reserved.
It is generally known that the states of network nodes are stable and have strong correlations in a linear network system. We find that without the control input, the method of compressed sensing can not succeed in reconstructing complex networks in which the states of nodes are generated through the linear network system. However, noise can drive the dynamics between nodes to break the stability of the system state. Therefore, a new method integrating QR decomposition and compressed sensing is proposed to solve the reconstruction problem of complex networks under the assistance of the input noise. The state matrix of the system is decomposed by QR decomposition. We construct the measurement matrix with the aid of Gaussian noise so that the sparse input matrix can be reconstructed by compressed sensing. We also discover that noise can build a bridge between the dynamics and the topological structure. Experiments are presented to show that the proposed method is more accurate and more efficient to reconstruct four model networks and six real networks by the comparisons between the proposed method and only compressed sensing. In addition, the proposed method can reconstruct not only the sparse complex networks, but also the dense complex networks.
The ability to discover physical laws and governing equations from data is
one of humankind's greatest intellectual achievements. A quantitative
understanding of dynamic constraints and balances in nature has facilitated
rapid development of knowledge and enabled advanced technological achievements,
including aircraft, combustion engines, satellites, and electrical power. In
this work, we combine sparsity-promoting techniques and machine learning with
nonlinear dynamical systems to discover governing physical equations from
measurement data. The only assumption about the structure of the model is that
there are only a few important terms that govern the dynamics, so that the
equations are sparse in the space of possible functions; this assumption holds
for many physical systems. In particular, we use sparse regression to determine
the fewest terms in the dynamic governing equations required to accurately
represent the data. The resulting models are parsimonious, balancing model
complexity with descriptive ability while avoiding overfitting. We demonstrate
the algorithm on a wide range of problems, from simple canonical systems,
including linear and nonlinear oscillators and the chaotic Lorenz system, to
the fluid vortex shedding behind an obstacle. The fluid example illustrates the
ability of this method to discover the underlying dynamics of a system that
took experts in the community nearly 30 years to resolve. We also show that
this method generalizes to parameterized, time-varying, or externally forced
systems.
Evolutionary games model a common type of interactions in a variety of complex, networked, natural systems and social systems. Given such a system, uncovering the interacting structure of the underlying network is key to understanding its collective dynamics. Based on compressive sensing, we develop an efficient approach to reconstructing complex networks under game-based interactions from small amounts of data. The method is validated by using a variety of model networks and by conducting an actual experiment to reconstruct a social network. While most existing methods in this area assume oscillator networks that generate continuous-time data, our work successfully demonstrates that the extremely challenging problem of reverse engineering of complex networks can also be addressed even when the underlying dynamical processes are governed by realistic, evolutionary-game type of interactions in discrete time.
We consider the problem of differentiating a function specified
by noisy data. Regularizing the differentiation process
avoids the noise amplification of finite-difference methods.
We use total-variation regularization, which allows for discontinuous
solutions. The resulting simple algorithm accurately
differentiates noisy functions, including those which
have a discontinuous derivative.
First-order difference equations arise in many contexts in the biological, economic and social sciences. Such equations, even though simple and deterministic, can exhibit a surprising array of dynamical behaviour, from stable points, to a bifurcating hiearchy of stable cycles, to apparently random fluctuations. There are consequently many fascinating problems, some concerned with delicate mathematical aspects of the fine structure of the trajectories, and some concerned with the practical implications and applications. This is an interpretive review of them.
Many problems of recent interest in statistics and machine learning can be posed in the framework of convex optimization. Due to the explosion in size and complexity of modern datasets, it is increasingly important to be able to solve problems with a very large number of features or training examples. As a result, both the decentralized collection or storage of these datasets as well as accompanying distributed solution methods are either necessary or at least highly desirable. In this review, we argue that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas. The method was developed in the 1970s, with roots in the 1950s, and is equivalent or closely related to many other algorithms, such as dual decomposition, the method of multipliers, Douglas–Rachford splitting, Spingarn's method of partial inverses, Dykstra's alternating projections, Bregman iterative algorithms for ℓ1 problems, proximal methods, and others. After briefly surveying the theory and history of the algorithm, we discuss applications to a wide variety of statistical and machine learning problems of recent interest, including the lasso, sparse logistic regression, basis pursuit, covariance selection, support vector machines, and many others. We also discuss general distributed optimization, extensions to the nonconvex setting, and efficient implementation, including some details on distributed MPI and Hadoop MapReduce implementations.
An extremely challenging problem of significant interest is to predict
catastrophes in advance of their occurrences. We present a general approach to
predicting catastrophes in nonlinear dynamical systems under the assumption
that the system equations are completely unknown and only time series
reflecting the evolution of the dynamical variables of the system are
available. Our idea is to expand the vector field or map of the underlying
system into a suitable function series and then to use the compressive-sensing
technique to accurately estimate the various terms in the expansion. Examples
using paradigmatic chaotic systems are provided to demonstrate our idea.
Given n noisy observations g i of the same quantity f , it is common use to give an estimate of f by minimizing the function P n i=1 (g i Gamma f) 2 . From a statistical point of view this corresponds to computing the Maximum Likelihood estimate, under the assumption of Gaussian noise. However, it is well known that this choice leads to results that are very sensitive to the presence of outliers in the data. For this reason it has been proposed to minimize functions of the form P n i=1 V (g i Gamma f ), where V is a function that increases less rapidly than the square. Several choices for V have been proposed and successfully used to obtain "robust" estimates. In this paper we show that, for a class of functions V , using these robust estimators corresponds to assuming that data are corrupted by Gaussian noise whose variance fluctuates according to some given probability distribution, that uniquely determines the shape of V . c fl Massachusetts Institute of Technology, 1996 Th...
Support Vector Machines Regression (SVMR) is a learning technique where the goodness of fit is measured not by the usual quadratic loss function (the mean square error), but by a different loss function called the ∈-Insensitive Loss Function (ILF), which is similar to loss functions used in the field of robust statistics. The quadratic loss function is well justified under the assumption of Gaussian additive noise. However, the noise model underlying the choice of the ILF is not clear. In this paper the use of the ILF is justified under the assumption that the noise is additive and Gaussian, where the variance and mean of the Gaussian are random variables. The probability distributions for the variance and mean will be stated explicitly. While this work is presented in the framework of SVMR, it can be extended to justify non-quadratic loss functions in any Maximum Likelihood or Maximum AP osteriori approach. It applies not only to the ILF, but to a much broader class of loss functions.
The complex phase interactions of the two-phase flow are a key factor in understanding the flow pattern evolutional mechanisms, yet these complex flow behaviors have not been well understood. In this paper, we employ a series of gas-liquid two-phase flow multivariate fluctuation signals as observations and propose a novel interconnected ordinal pattern network to investigate the spatial coupling behaviors of the gas-liquid two-phase flow patterns. In addition, we use two network indices, which are the global subnetwork mutual information (I) and the global subnetwork clustering coefficient (C), to quantitatively measure the spatial coupling strength of different gas-liquid flow patterns. The gas-liquid two-phase flow pattern evolutionary behaviors are further characterized by calculating the two proposed coupling indices under different flow conditions. The proposed interconnected ordinal pattern network provides a novel tool for a deeper understanding of the evolutional mechanisms of the multi-phase flow system, and it can also be used to investigate the coupling behaviors of other complex systems with multiple observations.
The temperature of rotary kiln, as one of the essential equipment in zinc smelting process, determines the product quality and resource utilization. However, since rotary kiln is a large-scale and highly coupled system, there are plenty of variables affecting the rotary kiln temperature, bringing too much redundant information to describe rotary kiln dynamics. In addition, due to the variations in raw materials, production load, and market demand, there exist various operation conditions, making it difficult to achieve stability control of the rotary kiln temperature with traditional control methods. To solve these problems, an error-triggered adaptive model predictive control (ET-AMPC) is proposed in this article. Specifically, since rotary kiln temperature is hard to regulate due to redundancy among variables and strong nonlinearity, an orthogonal maximum mutual information coefficient feature selection (OMICFS) method is first proposed to determine vital variables affecting temperature most. Then, aiming at the problem of changing operating conditions of rotary kilns, an ET-AMPC method is proposed, which can precisely adapt to different operating conditions and achieve stability control. Finally, experiments on a numerical simulation case and an industrial rotary kiln show the strength and reliability of the proposed method, which reduces 10%–20% trajectory tracking error in the period of operating condition changing and improves the control accuracy effectively.
Reputation and reciprocity are key mechanisms for cooperation in human societies, often going hand in hand to favor prosocial behavior over selfish actions. Here we review recent researches at the interface of physics and evolutionary game theory that explored these two mechanisms. We focus on image scoring as the bearer of reputation, as well as on various types of reciprocity, including direct, indirect, and network reciprocity. We review different definitions of reputation and reciprocity dynamics, and we show how these affect the evolution of cooperation in social dilemmas. We consider first-order, second-order, as well as higher-order models in well-mixed and structured populations, and we review experimental works that support and inform the results of mathematical modeling and simulations. We also provide a synthesis of the reviewed researches along with an outlook in terms of six directions that seem particularly promising to explore in the future.
With the digital transformation of process manufacturing, identifying the system model from process data and then applying to predictive control has become the most dominant approach in process control. However, the controlled plant often operates under changing operating conditions. What is more, there are often unknown operating conditions such as first appearance operating conditions, which make traditional predictive control methods based on identified model difficult to adapt to changing operating conditions. Moreover, the control accuracy is low during operating condition switching. To solve these problems, this article proposes an error-triggered adaptive sparse identification for predictive control (ETASI4PC) method. Specifically, an initial model is established based on sparse identification. Then, a prediction error-triggered mechanism is proposed to monitor operating condition changes in real time. Next, the previously identified model is updated with the fewest modifications by identifying parameter change, structural change, and combination of changes in the dynamical equations, thus achieving precise control to multiple operating conditions. Considering the problem of low control accuracy during the operating condition switching, a novel elastic feedback correction strategy is proposed to significantly improve the control accuracy in the transition period and ensure accurate control under full operating conditions. To verify the superiority of the proposed method, a numerical simulation case and a continuous stirred tank reactor (CSTR) case are designed. Compared with some state-of-the-art methods, the proposed method can rapidly adapt to frequent changes in operating conditions, and it can achieve real-time control effects even for unknown operating conditions such as first appearance operating conditions.
Cooperation is a widespread phenomenon in human society and plays a significant role in achieving synchronization of various systems. However, there has been limited progress in studying the co-evolution of synchronization and cooperation. In this manuscript, we investigate how reinforcement learning affects the evolution of synchronization and cooperation. Namely, the payoff of an agent depends not only on the cooperation dynamic but also on the synchronization dynamic. Agents have the option to either cooperate or defect. While cooperation promotes synchronization among agents, defection does not. We report that the dynamic feature, which indicates the action switching frequency of the agent during interactions, promotes synchronization. We also find that cooperation and synchronization are mutually reinforcing. Furthermore, we thoroughly analyze the potential reasons for synchronization promotion due to the dynamic feature from both macro- and microperspectives. Additionally, we conduct experiments to illustrate the differences in the synchronization-promoting effects of cooperation and dynamic features.
We introduce a new model consisting of globally coupled high-dimensional generalized limit-cycle oscillators, which explicitly incorporates the role of amplitude dynamics of individual units in the collective dynamics. In the limit of weak coupling, our model reduces to the D-dimensional Kuramoto phase model, akin to a similar classic construction of the well-known Kuramoto phase model from weakly coupled two-dimensional limit-cycle oscillators. For the practically important case of D=3, the incoherence of the model is rigorously proved to be stable for negative coupling (K<0) but unstable for positive coupling (K>0); the locked states are shown to exist if K>0; in particular, the onset of amplitude death is theoretically predicted. For D≥2, the discrete and continuous spectra for both locked states and amplitude death are governed by two general formulas. Our proposed D-dimensional model is physically more reasonable, because it is no longer constrained by fixed amplitude dynamics, which puts the recent studies of the D-dimensional Kuramoto phase model on a stronger footing by providing a more general framework for D-dimensional limit-cycle oscillators.
A data-driven sparse identification method is developed to discover the underlying governing equations from noisy measurement data through the minimization of Multi-Step-Accumulation (MSA) in error. The method focuses on the multi-step model, while conventional sparse regression methods, such as the Sparse Identification of Nonlinear Dynamics method (SINDy), are one-step models. We adopt sparse representation and assume that the underlying equations involve only a small number of functions among possible candidates in a library. The new development in MSA is to use a multi-step model, i.e., predictions from an approximate evolution scheme based on initial points. Accordingly, the loss function comprises the total error at all time steps between the measured series and predicted series with the same initial point. This enables MSA to capture the dynamics directly from the noisy measurements, resisting the corruption of noise. By use of several numerical examples, we demonstrate the robustness and accuracy of the proposed MSA method, including a two-dimensional chaotic map, the logistic map, a two-dimensional damped oscillator, the Lorenz system, and a reduced order model of a self-sustaining process in turbulent shear flows. We also perform further studies under challenging conditions, such as noisy measurements, missing data, and large time step sizes. Furthermore, in order to resolve the difficulty of the nonlinear optimization, we suggest an adaptive training strategy, namely, by gradually increasing the length of time series for training. Higher prediction accuracy is achieved in an illustrative example of the chaotic map by the adaptive strategy.
How to predict and control the collective decision-making dynamics in networked populations is of great significance for various applications in engineering, social and natural sciences, and attracts burgeoning interdisciplinary researches across networked systems and control theory. In this paper, we investigate the asynchronous best-response dynamics in networks of anticoordinating agents. To identify the influence of threshold of the anticoordinating model, we consider the homogeneous-threshold networks and explore how the threshold affects the convergence time and network equilibrium. Results on the convergence time show that upper bound of the total number of strategy switches is determined by three factors: the number of network edges, the number of network nodes, and the value of network threshold. Based on the Lyapunov method, asymptotic stability analyses of the network equilibrium are performed. Meanwhile, by introducing relevant payoff incentives (i.e., reward or punishment) during the game playing, the maximal anticoordinating equilibrium with neighboring agents in different strategies will be achieved in finite time to acquire more stable system-wide outcomes.
The way of information diffusion among individuals can be quite complicated, and it is not only limited to one type of communication, but also impacted by multiple channels. Meanwhile, it is easier for an agent to accept an idea once the proportion of their friends who take it goes beyond a specific threshold. Furthermore, in social networks, some higher-order structures, such as simplicial complexes and hypergraph, can describe more abundant and realistic phenomena. Therefore, based on the classical multiplex network model coupling the infectious disease with its relevant information, we propose a novel epidemic model, in which the lower layer represents the physical contact network depicting the epidemic dissemination, while the upper layer stands for the online social network picturing the diffusion of information. In particular, the upper layer is generated by random simplicial complexes, among which the herd-like threshold model is adopted to characterize the information diffusion, and the unaware–aware–unaware model is also considered simultaneously. Using the microscopic Markov chain approach, we analyze the epidemic threshold of the proposed epidemic model and further check the results with numerous Monte Carlo simulations. It is discovered that the threshold model based on the random simplicial complexes network may still cause abrupt transitions on the epidemic threshold. It is also found that simplicial complexes may greatly influence the epidemic size at a steady state.
In real industrial processes, factors, such as the change in manufacturing strategy and production technology lead to the creation of multimode industrial processes and the continuous emergence of new modes. Although the industrial SCADA system has accumulated a large amount of historical data, which can be used for modeling and monitoring multimode processes to a certain extent, it is difficult for the model learned from historical data to adapt to emerging modes, resulting in the model mismatch. On the other hand, updating the model with data from new modes allows the model to continuously match the new modes, but it may cause the model to lose the ability to represent the historical modes, resulting in "catastrophic forgetting." To address these problems, this article proposed a jointly mode-matching and similarity-preserving dictionary learning (JMSDL) method, which updated the model by learning the data of new modes, so that the model can adaptively match the newly emerged modes. At the same time, a similarity metric was put forward to guarantee the representation ability of the proposed method for historical data. A numerical simulation experiment, the CSTH process experiment, and an industrial roasting process experiment indicated that the proposed JMSDL method can match new modes while maintaining its performance on the historical modes accurately. In addition, the proposed method significantly outperforms the state-of-the-art methods in terms of fault detection and false alarm rate.
This paper proposes a non-convex penalty regression method to identify governing equations of nonlinear dynamical systems from noisy state measurements. The idea to connect the non-convex penalty function instead of the l1 - norm with least squares is due to the fact that the l1 - norm excessively penalizes large coefficients and may incur estimation bias. The purpose of this work is to improve the accuracy and robustness in regression tasks. A threshold non-convex penalty sparse least squares optimization algorithm is developed, wherein the threshold parameter is selected using the L-curve criterion. With two examples of nonlinear dynamical systems, we illustrate the accuracy and robustness of the non-convex penalty least squares on noisy state measurements, indicating the validity of our method in a wide range of potential applications.
Recent decades have seen a rise in the use of physics methods to study different societal phenomena. This development has been due to physicists venturing outside of their traditional domains of interest, but also due to scientists from other disciplines taking from physics the methods that have proven so successful throughout the 19th and the 20th century. Here we characterise the field with the term ‘social physics’ and pay our respect to intellectual mavericks who nurtured it to maturity. We do so by reviewing the current state of the art. Starting with a set of topics that are at the heart of modern human societies, we review research dedicated to urban development and traffic, the functioning of financial markets, cooperation as the basis for our evolutionary success, the structure of social networks, and the integration of intelligent machines into these networks. We then shift our attention to a set of topics that explore potential threats to society. These include criminal behaviour, large-scale migration, epidemics, environmental challenges, and climate change. We end the coverage of each topic with promising directions for future research. Based on this, we conclude that the future for social physics is bright. Physicists studying societal phenomena are no longer a curiosity, but rather a force to be reckoned with. Notwithstanding, it remains of the utmost importance that we continue to foster constructive dialogue and mutual respect at the interfaces of different scientific disciplines.
Long-time prediction of future states has been challenging in data-driven modeling of nonlinear dynamical systems as the prediction error accumulates over the prediction horizon. One of the potential reasons is the lack of robustness for the data-driven model. In this study we present a recurrent neural network (RNN) framework with an adaptive training strategy to model nonlinear dynamical systems from data for long-time prediction of future states. Specifically, we exploit the recurrence of network to improve the model robustness by explicitly incorporating the multi-step prediction with error accumulation into model training. Furthermore, we introduce an adaptive training strategy, where the prediction horizon gradually increases from a small value to facilitate the RNN training. We demonstrate the proposed approach on a family of Duffing oscillators, including autonomous and non-autonomous systems with various attractors, and discuss its advantages and limitations.
This work proposes an iterative sparse-regularized regression method to recover governing equations of nonlinear dynamical systems from noisy state measurements. The method is inspired by the Sparse Identification of Nonlinear Dynamics (SINDy) approach of Brunton et al. (2016), which relies on two main assumptions: the state variables are known a priori and the governing equations lend themselves to sparse, linear expansions in a (nonlinear) basis of the state variables. The aim of this work is to improve the accuracy and robustness of SINDy in the presence of state measurement noise. To this end, a reweighted ℓ1-regularized least squares solver is developed, wherein the regularization parameter is selected from the corner point of a Pareto curve. The idea behind using weighted ℓ1-norm for regularization – instead of the standard ℓ1-norm – is to better promote sparsity in the recovery of the governing equations and, in turn, mitigate the effect of noise in the state variables. We also present a method to recover single physical constraints from state measurements. Through several examples of well-known nonlinear dynamical systems, we demonstrate empirically the accuracy and robustness of the reweighted ℓ1-regularized least squares strategy with respect to state measurement noise, thus illustrating its viability for a wide range of potential applications.
Intrinsic predictability is imperative to quantify inherent information contained in a time series and assists in evaluating the performance of different forecasting methods to get the best possible prediction. Model forecasting performance is the measure of the probability of success. Nevertheless, model performance or the model does not provide understanding for improvement in prediction. Intuitively, intrinsic predictability delivers the highest level of predictability for a time series and informative in unfolding whether the system is unpredictable or the chosen model is a poor choice. We introduce a novel measure, the Wavelet Entropy Energy Measure (WEEM), based on wavelet transformation and information entropy for quantification of intrinsic predictability of time series. To investigate the efficiency and reliability of the proposed measure, model forecast performance was evaluated via a wavelet networks approach. The proposed measure uses the wavelet energy distribution of a time series at different scales and compares it with the wavelet energy distribution of white noise to quantify a time series as deterministic or random. We test the WEEM using a wide variety of time series ranging from deterministic, non-stationary, and ones contaminated with white noise with different noise-signal ratios. Furthermore, a relationship is developed between the WEEM and Nash–Sutcliffe Efficiency, one of the widely known measures of forecast performance. The reliability of WEEM is demonstrated by exploring the relationship to logistic map and real-world data.
We investigate the impact of a stochastic forcing, comprised of a sum of time-lagged copies of a single source of noise, on the system dynamics. This type of stochastic forcing could be made artificially, or it could be the result of shared upstream inputs to a system through different channel lengths. By means of a rigorous mathematical framework, we show that such a system is, in fact, equivalent to the classical case of a stochastically-driven dynamical system with time-delayed intrinsic dynamics but without a time lag in the input noise. We also observe a resonancelike effect between the intrinsic period of the oscillation and the time lag of the stochastic forcing, which may be used to determine the intrinsic period of oscillations or the inherent time delay in dynamical systems with oscillatory behavior or delays. As another useful application of imposing time-lagged stochastic forcing, we show that the dynamics of a system can be controlled by changing the time lag of this stochastic forcing, in a fashion similar to the classical case of Pyragas control via delayed feedback. To confirm these results experimentally, we set up a laser diode system with such stochastic inputs, which effectively behaves as a Langevin system. As in the theory, a peak emerged in the autocorrelation function of the output signal that could be tuned by the lag of the stochastic input. Our findings, thus, indicate a new approach for controlling useful instabilities in dynamical systems.
Complex network has proven to be a general model to characterize interactions of practical complex systems. Recently, reconstructing the structure of complex networks with limited and noisy data attracts much research attention and has gradually become a hotspot. However, the collected data are often contaminated by unknown outliers inevitably, which seriously affects the accuracy of network reconstruction. Unfortunately, the existence of outliers is hard to be identified and always ignored in the network structure reconstruction task. To address this issue, here we propose a novel method which involves the outliers from the Bayesian perspective. The accuracy and the robustness of the proposed method have been verified in network reconstruction with payoff data contaminated with some outliers on both artificial networks and empirical networks. Extensive simulation results demonstrate the superiority of the proposed method. Thus, it can be concluded that since the proposed method can identify and get rid of outliers in observation data, it is conducive to improve the performance of network reconstruction.
First-principles-based multiscale models provide mechanistic insight and allow screening of large materials spaces to find promising new catalysts. In this Review, Reuter and co-workers discuss methodological cornerstones of existing approaches and highlight successes and ongoing developments in the field.
The inner workings of a biological cell or a chemical reactor can be rationalized by the network of reactions, whose structure reveals the most important functional mechanisms. For complex systems, these reaction networks are not known a priori and cannot be efficiently computed with ab initio methods; therefore, an important goal is to estimate effective reaction networks from observations, such as time series of the main species. Reaction networks estimated with standard machine learning techniques such as least-squares regression may fit the observations but will typically contain spurious reactions. Here we extend the sparse identification of nonlinear dynamics (SINDy) method to vector-valued ansatz functions, each describing a particular reaction process. The resulting sparse tensor regression method "reactive SINDy" is able to estimate a parsimonious reaction network. We illustrate that a gene regulation network can be correctly estimated from observed time series.
We propose a new method for estimation in linear models. The ‘lasso’ minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models. Our simulation studies suggest that the lasso enjoys some of the favourable properties of both subset selection and ridge regression. It produces interpretable models like subset selection and exhibits the stability of ridge regression. There is also an interesting relationship with recent work in adaptive function estimation by Donoho and Johnstone. The lasso idea is quite general and can be applied in a variety of statistical models: extensions to generalized regression models and tree‐based models are briefly described.
Data-driven dynamical systems is a burgeoning field—it connects how measurements of nonlinear dynamical systems and/or complex systems can be used with well-established methods in dynamical systems theory. This is a critically important new direction because the governing equations of many problems under consideration by practitioners in various scientific fields are not typically known. Thus, using data alone to help derive, in an optimal sense, the best dynamical system representation of a given application allows for important new insights. The recently developed dynamic mode decomposition (DMD) is an innovative tool for integrating data with dynamical systems theory. The DMD has deep connections with traditional dynamical systems theory and many recent innovations in compressed sensing and machine learning.
Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems, the first book to address the DMD algorithm,
* presents a pedagogical and comprehensive approach to all aspects of DMD currently developed or under development;
* blends theoretical development, example codes, and applications to showcase the theory and its many innovations and uses;
* highlights the numerous innovations around the DMD algorithm and demonstrates its efficacy using example problems from engineering and the physical and biological sciences; and
* provides extensive MATLAB code, data for intuitive examples of key methods, and graphical presentations.
This paper introduces an L1-norm based probabilistic principal component analysis model on 2D data (L1-2DPPCA) based on the assumption of the Laplacian noise model. The Laplacian or L1 density function can be expressed as a superposition of an infinite number of Gaussian distributions. Under this expression, a Bayesian inference can be established based on the variational EM approach. All the key parameters in the probabilistic model can be learned by the proposed variational algorithm. It has been experimentally demonstrated that the newly introduced hidden variables in the superposition can serve as an effective indicator for data outliers. Experiments on some publicly available databases show that the performance of L1-2DPPCA has been largely improved after identifying and removing sample outliers, resulting in more accurate image reconstruction than the existing PCAbased methods. The performance of feature extraction of the proposed method generally outperforms other existing algorithms in terms of reconstruction errors and classification accuracy.
Traditional Symbolic Regression applications are a form of supervised learning, where a label y is provided for every
[(x)\vec]\vec{x}
and an explicit symbolic relationship of the form
y = f([(x)\vec])y = f(\vec{x})
is sought. This chapter explores the use of symbolic regression to perform unsupervised learning by searching for implicit
relationships of the form
f([(x)\vec], y) = 0f(\vec{x}, y) = 0
. Implicit relationships are more general and more expressive than explicit equations in that they can also represent closed
surfaces, as well as continuous and discontinuous multi-dimensional manifolds. However, searching these types of equations
is particularly challenging because an error metric is difficult to define. We studied several direct and indirect techniques,
and present a successful method based on implicit derivatives. Our experiments identified implicit relationships found in
a variety of datasets, such as equations of circles, elliptic curves, spheres, equations of motion, and energy manifolds.
The lasso is a popular technique for simultaneous estimation and variable selection. Lasso variable selection has been shown to be consistent under certain conditions. In this work we derive a necessary condition for the lasso variable selection to be consistent. Consequently, there exist certain scenarios where the lasso is inconsistent for variable selection. We then propose a new version of the lasso, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the l1 penalty. We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance. Similar to the lasso, the adaptive lasso is shown to be near-minimax optimal. Furthermore, the adaptive lasso can be solved by the same efficient algorithm for solving the lasso. We also discuss the extension of the adaptive lasso in generalized linear models and show that the oracle properties still hold under mild regularity conditions. As a byproduct of our theory, the nonnegative garotte is shown to be consistent for variable selection.
This paper addresses selection of the loss function for regression problems with finite data. It is well-known (under standard regression formulation) that for a known noise density there exist an optimal loss function under an asymptotic setting (large number of samples), i.e. squared loss is optimal for Gaussian noise density. However, in real-life applications the noise density is unknown and the number of training samples is finite. For such practical situations, we suggest using Vapnik's ε-insensitive loss function. We use practical method for setting the value of ε as a function of known number of samples and (known or estimated) noise variance (V. Cherkassky and Y. Ma, (2004), (2002)). We consider commonly used noise densities (such as Gaussian, Uniform and Laplacian noise). Empirical comparisons for several representative linear regression problems indicate that Vapnik's ε-insensitive loss yields more robust performance and improved prediction accuracy, in comparison with squared loss and least-modulus loss, especially for noisy high-dimensional data sets.
The Time-Frequency and Time-Scale communities have recently developed a large number of overcomplete waveform dictionaries --- stationary wavelets, wavelet packets, cosine packets, chirplets, and warplets, to name a few. Decomposition into overcomplete systems is not unique, and several methods for decomposition have been proposed, including the Method of Frames (MOF), Matching Pursuit (MP), and, for special dictionaries, the Best Orthogonal Basis (BOB). Basis Pursuit (BP) is a principle for decomposing a signal into an "optimal" superposition of dictionary elements, where optimal means having the smallest l 1 norm of coefficients among all such decompositions. We give examples exhibiting several advantages over MOF, MP! and BOB, including better sparsity, and super-resolution. BP has interesting relations to ideas in areas as diverse as ill-posed problems, in abstract harmonic analysis, total variation de-noising, and multi-scale edge de-noising. Basis Pursuit in highly ...