No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Variational approach to rare event simulation using least-squares regression
Chaos
Abstract
We propose an adaptive importance sampling scheme for the simulation of rare events when the underlying dynamics is given by diffusion. The scheme is based on a Gibbs variational principle that is used to determine the optimal (i.e., zero-variance) change of measure and exploits the fact that the latter can be rephrased as a stochastic optimal control problem. The control problem can be solved by a stochastic approximation algorithm, using the Feynman–Kac representation of the associated dynamic programming equations, and we discuss numerical aspects for high-dimensional problems along with simple toy examples.
... Similarly, [80] studies potential problems that appear when an optimal importance sampling proposal is not available, in particular when the time horizon of the problem is large. A non-asymptotic variant of the aforementioned approaches for finite noise diffusions is based on the stochastic control formulation of the optimal change of measure [127,130]. Furthermore we should note that there have been many attempts to find good (low-dimensional) proposals by taking advantage of specific structures of the problem at hand, using simplified models that approximate a complicated multiscale system [79,128,133,273]. Recently, the scaling properties of certain approximations to control-based importance sampling estimators with the system dimension have been analyzed in [217], suggesting that the empirical loss function that is used to numerically approximate the optimal proposal distribution is essential. ...
... For a recent numerical attempt to approach variance minimization based on neural networks we refer the reader to [213,Section 5.2], for a theoretical analysis of convergence rates we refer to [4], and for a general overview regarding adaptive importance sampling techniques we refer to [44]. The relationship between optimal control and importance sampling (see Theorem 1.2) has been exploited by various authors to construct efficient samplers [161,279], in particular also with a view towards the sampling based estimation of hitting times, in which case optimal controls are governed by elliptic rather than parabolic PDEs [126,127,131,132]. Similar sampling problems have been addressed in the context of sequential Monte Carlo [66,136] and generative models [283,284]. ...
... Conditioned diffusions (Problem 1.2) have been considered in a large deviation context [81] as well as in a variational setting [127,130] motivated by free energy computations, building on earlier work in [40,61], see also [9,55,60,95]. The simulation of diffusion bridges has been studied in [207] and conditioning via Doob's h-transform has been employed in a sequential Monte Carlo context [136]. ...
Motivated by computing functionals of high-dimensional, potentially metastable diffusion processes, this thesis studies robustness issues appearing in the numerical approximation of expectation values and their gradients. A major challenge being high variances of corresponding estimators, we investigate importance sampling of stochastic processes for improving statistical properties and provide novel nonasymptotic bounds on the relative error of corresponding estimators depending on deviations from optimality. Numerical strategies that aim to come close to those optimal sampling strategies can be encompassed in the framework of path space measures, and minimizing suitable divergences between those measures suggests a variational formulation that can be addressed in the spirit of machine learning. A key observation is that while several natural choices of divergences have the same unique minimizer, their finite sample properties differ vastly. We provide the novel log-variance divergence, which turns out to have favorable robustness properties that we investigate theoretically and apply in the context of path space measures as well as in the context of densities, for instance offering promising applications in Bayesian variational inference. Aiming for optimal importance sampling of diffusions is (more or less) equivalent to solving Hamilton-Jacobi- Bellman PDEs and it turns out that our numerical methods can be equally applied for the approximation of rather general high-dimensional semi-linear PDEs. Motivated by stochastic representations of elliptic and parabolic boundary value problems we refine variational methods based on backward SDEs and provide the novel diffusion loss, which can be related to other state-of-the-art attempts, while offering certain numerical advantages.
... The idea of approximating solutions to PDEs by solving BSDEs has been studied extensively [14,23,61], where first approaches were regression based, relying on iterations backwards in time. These ideas do not seem to be straightforwardly applicable to the case when Ω is bounded, as the trajectories of the forward process (10a) are not all of the same length (but see [13] and [31]). A global variational strategy using neural networks has first been introduced in [19], where however in contrast to Definition 2.5, the initial condition (X 0 , t 0 ) is deterministic (and fixed) and only parabolic problems on Ω = R d are considered. ...
... Proof. It is clear that 1. implies 2. by the construction of (31). For the converse direction, notice that (33) implies ϕ(x c ) = 1 as well as (29), that is, ϕ is an eigenfunction with eigenvalue λ. ...
... In this section we provide two examples for the approximation of principal eigenvalues and corresponding eigenfunctions. The first one is a linear problem and therefore Proposition 4.2 assures that the minimization of an appropriate loss as in (31) leads to the desired solution. The second example is a nonlinear eigenvalue problem, for which we can numerically show that our algorithm still provides the correct solution. ...
Solving high-dimensional partial differential equations is a recurrent challenge in economics, science and engineering. In recent years, a great number of computational approaches have been developed, most of them relying on a combination of Monte Carlo sampling and deep learning based approximation. For elliptic and parabolic problems, existing methods can broadly be classified into those resting on reformulations in terms of (BSDEs) and those aiming to minimize a regression-type -error (, PINNs). In this paper, we review the literature and suggest a methodology based on the novel that interpolates between BSDEs and PINNs. Our contribution opens the door towards a unified understanding of numerical approaches for high-dimensional PDEs, as well as for implementations that combine the strengths of BSDEs and PINNs. We also provide generalizations to eigenvalue problems and perform extensive numerical studies, including calculations of the ground state for nonlinear Schr\"odinger operators and committor functions relevant in molecular dynamics.
... Conditioned diffusions (Problem 2.4) have been considered in a large deviation context [35] as well as in a variational setting [56,58] motivated by free energy computations, building on earlier work in [16,30], see also [3,26,29,43]. The simulation of diffusion bridges has been studied in [86] and conditioning via Doob's h-transform has been employed in a sequential Monte Carlo context [61]. ...
... We refer the reader to [88, Section 5.2] for a recent attempt based on neural networks, to [2] for a theoretical analysis of convergence rates, to [57] for potential non-robustness issues, and to [18] for a general overview regarding adaptive importance sampling techniques. The relationship between optimal control and importance sampling (see Theorem 2.2) has been exploited by various authors to construct efficient samplers [74,114], in particular also with a view towards the sampling based estimation of hitting times, in which case optimal controls are governed by elliptic rather than parabolic PDEs [55,56,59,60]. Similar sampling problems have been addressed in the context of sequential Monte Carlo [31,61] and generative models [116,117]. ...
... In this respect, we may also mention the development of more elaborate schemes to update the control for the forward dynamics. Second, one may attempt to generalise the current framework to other types of control problems and PDEs (for instance to elliptic PDEs and hitting time problems as considered in [55,56,59,60], or to the Schrödinger problem as discussed in [104]). Deeper understanding of the design of IDO algorithms could be achieved by extending our stability analysis beyond the product case and for controls that differ greatly from the optimal one. ...
Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton–Jacobi–Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of iterative diffusion optimisation techniques, in particular considering applications in importance sampling and rare event simulation, and focusing on problems without diffusion control, with linearly controlled drift and running costs that depend quadratically on the control. More generally, our methods apply to nonlinear parabolic PDEs with a certain shift invariance. The choice of an appropriate loss function being a central element in the algorithmic design, we develop a principled framework based on divergences between path measures, encompassing various existing methods. Motivated by connections to forward-backward SDEs, we propose and study the novel log-variance divergence, showing favourable properties of corresponding Monte Carlo estimators. The promise of the developed approach is exemplified by a range of high-dimensional and metastable numerical examples.
... This approach was first noted in queuing theory [22], but has also been used for problems in the physical sciences. These enhancements are also related to variational approaches to importance sampling, where the alternative SDE is posed as the solution to a stochastic optimal control problem-which, in principle, can yield zero-variance estimators [23,24,25,26]. The drawbacks of these approaches are also wellnoted. ...
... A variation of this problem considers path-dependent quantities, which involve functionals of sample trajectories. These problems are well-studied in the computational chemistry community, where one seeks rare paths between long-lived molecular configurations [5,24]. This quantity of interest is associated with the solution of a boundary value problem, and its approximation can also be used for sampling. ...
... We can also contrast our approach with rare event simulation methods based on stochastic optimal control [23,24,26]. The goal of these efforts is the same as ours: to find a controller for the dynamical system that approximates the zero-variance importance sampling estimator. ...
We exploit the relationship between the stochastic Koopman operator and the Kolmogorov backward equation to construct importance sampling schemes for stochastic differential equations. Specifically, we propose using eigenfunctions of the stochastic Koopman operator to approximate the Doob transform for an observable of interest (e.g., associated with a rare event) which in turn yields an approximation of the corresponding zero-variance importance sampling estimator. Our approach is broadly applicable and systematic, treating non-normal systems, non-gradient systems, and systems with oscillatory dynamics or rank-deficient noise in a common framework. In nonlinear settings where the stochastic Koopman eigenfunctions cannot be derived analytically, we use dynamic mode decomposition (DMD) methods to compute them numerically, but the framework is agnostic to the particular numerical method employed. Numerical experiments demonstrate that even coarse approximations of a few eigenfunctions, where the latter are built from non-rare trajectories, can produce effective importance sampling schemes for rare events.
... [1][2][3]. Subsequent to the latter in [4,5] this optimal control setting has been applied to the characterization of free energy of an uncontrolled dynamical system. Different numerical methods have been developed and are now widely used and further investigated, see arXiv:2010.04465v1 [math.OC] 9 Oct 2020 e.g. ...
... with Dirichlet boundary condition v * (x) = 0 on ∂Ξ. We later employ the Policy Iteration algorithm to solve this coupled equation by alternating between the value updates given by (4) and the policy updates given by (5). In preparation to that we first notice that by fixing a policy α this coupled equation becomes uncoupled and a linear function equation is remaining ...
... Here, the curse of dimensions comes into play. Choosing a polynomial degree of 6 and a tensor train rank of [5,5,5,5,5] allows us to reduce the ansatz space from 46656 degrees of freedom to 770. Again, setting N = 10 · dof , M = 100 and τ = 0.1 we visualize the resulting controller in figure 9. We start the trajectory at [−1, · · · − 1] ∈ R n . ...
We consider a stochastic optimal exit time feedback control problem. The Bellman equation is solved approximatively via the Policy Iteration algorithm on a polynomial ansatz space by a sequence of linear equations. As high degree multi-polynomials are needed, the corresponding equations suffer from the curse of dimensionality even in moderate dimensions. We employ tensor-train methods to account for this problem. The approximation process within the Policy Iteration is done via a Least-Squares ansatz and the integration is done via Monte-Carlo methods. Numerical evidences are given for the (multi dimensional) double well potential and a three-hole potential.
... The identities (24) connect key quantities pertaining to the problem formulations 2.1, 2.2, 2.3 and 2.4. The fact that J(u * ; x init , 0) = − log Z can moreover be understood in terms of the Donsker-Varadhan formula [16], as discussed in [29,30,52]. ...
... Conditioned diffusions (Problem 2.4) have been considered in a large deviation context [35] as well as in a variational setting [52,53] motivated by free energy computations, building on earlier work in [16,30], see also [3,26,29,40]. The simulation of diffusion bridges has been studied in [78] and conditioning via Doob's h-transform has been employed in a sequential Monte Carlo context [56]. ...
... We refer the reader to [80, Section 5.2] for a recent attempt based on neural networks, to [2] for a theoretical analysis of convergence rates, and to [18] for a general overview regarding adaptive importance sampling techniques. The relationship between optimal control and importance sampling (see Theorem 2.2) has been exploited by various authors to construct efficient samplers [66,103], in particular also with a view towards the sampling based estimation of hitting times, in which case optimal controls are governed by elliptic rather than parabolic PDEs [51,52,54,55]. Similar sampling problems have been addressed in the context of sequential Monte Carlo [31,56] and generative models [105,106]. ...
Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton-Jacobi-Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of iterative diffusion optimisation techniques, in particular considering applications in importance sampling and rare event simulation. The choice of an appropriate loss function being a central element in the algorithmic design, we develop a principled framework based on divergences between path measures, encompassing various existing methods. Motivated by connections to forward-backward SDEs, we propose and study the novel log-variance divergence, showing favourable properties of corresponding Monte Carlo estimators. The promise of the developed approach is exemplified by a range of high-dimensional and metastable numerical examples.
... Various approaches have been proposed to numerically solve (23) and obtain an approximate control. The d-dimensional HJB PDE (23) has been solved using least-squares regression [22] and model reduction techniques [25] for higher dimensions. Neural networks have also been employed to solve the HJB PDE in higher dimensions with stochastic gradient [23] and cross-entropy [41] learning methods for the stochastic optimal control formulation (25). ...
... Assuming u(t, x) > 0, ∀(t, x) ∈ [0, T ] × R d , we obtain the minimizer for (86) as in (22). Substituting the optimal control ζ * (t, x) in (86), produces (21), which solves for value function u. ...
This paper investigates Monte Carlo methods to estimate probabilities of rare events associated with solutions to the d-dimensional McKean-Vlasov stochastic differential equation. The equation is usually approximated using a stochastic interacting P-particle system, a set of P coupled d-dimensional stochastic differential equations (SDEs). Importance sampling (IS) is a common technique to reduce high relative variance of Monte Carlo estimators of rare event probabilities. In the SDE context, optimal measure change is derived using stochastic optimal control theory to minimize estimator variance, which when applied to stochastic particle systems yields a -dimensional partial differential control equation, which is cumbersome to solve. The work in [15] circumvented this problem by a decoupling approach, producing a d-dimensional control PDE. Based on the decoupling approach, we develop a computationally efficient double loop Monte Carlo (DLMC) estimator. We offer a systematic approach to our DLMC estimator by providing a comprehensive error and work analysis and formulating optimal computational complexity. Subsequently, we propose an adaptive DLMC method combined with IS to estimate rare event probabilities, significantly reducing relative variance and computational runtimes required to achieve a given relative tolerance compared with standard Monte Carlo estimators without IS. The proposed estimator has computational complexity with significantly reduced constant. Numerical experiments, which are performed on the Kuramoto model from statistical physics, show substantial computational gains achieved by our estimator.
... Furthermore, when the dimensionality of the state-space X becomes larger, standard numerical methods for solving Partial Differential Equations (PDEs) such as Finite Differences or the Finite Element Method become impractical. For these reasons, we propose instead to approximate the control function c with neural networks, and employ methods based on automatic differentiation and the nonlinear Feynman-Kac approach to solve semilinear PDEs [19,20,24,13,6,22,23,1,18]. ...
... CDT algorithm. The following outlines our training procedure to learn neural networks N 0 and N that satisfy (20). To minimize the loss function (22), any stochastic gradient algorithm can be used with a user-specified mini-batch size of J ≥ 1. ...
This paper is concerned with online filtering of discretely observed nonlinear diffusion processes. Our approach is based on the fully adapted auxiliary particle filter, which involves Doob's h-transforms that are typically intractable. We propose a computational framework to approximate these h-transforms by solving the underlying backward Kolmogorov equations using nonlinear Feynman-Kac formulas and neural networks. The methodology allows one to train a locally optimal particle filter prior to the data-assimilation procedure. Numerical experiments illustrate that the proposed approach can be orders of magnitude more efficient than the bootstrap particle filter in the regime of highly informative observations, when the observations are extreme under the model, and if the state dimension is large.
... Various approaches have been proposed to numerically solve (23) and obtain an approximate control. (Hartmann et al. 2019) solved the d-dimensional HJB PDE (23) using least-squares regression, whereas (Hartmann et al. 2016) solved it using model-reduction techniques for higher dimensions. Neural networks have also been employed to solve the HJB PDE in higher dimensions with stochastic gradient (Hartmann et al. 2017) and cross-entropy (Zhang et al. 2014) learning methods for the stochastic optimal control formulation (25). ...
This paper investigates Monte Carlo (MC) methods to estimate probabilities of rare events associated with the solution to the d-dimensional McKean–Vlasov stochastic differential equation (MV-SDE). MV-SDEs are usually approximated using a stochastic interacting P-particle system, which is a set of P coupled d-dimensional stochastic differential equations (SDEs). Importance sampling (IS) is a common technique for reducing high relative variance of MC estimators of rare-event probabilities. We first derive a zero-variance IS change of measure for the quantity of interest by using stochastic optimal control theory. However, when this change of measure is applied to stochastic particle systems, it yields a -dimensional partial differential control equation (PDE), which is computationally expensive to solve. To address this issue, we use the decoupling approach introduced in (dos Reis et al. 2023), generating a d-dimensional control PDE for a zero-variance estimator of the decoupled SDE. Based on this approach, we develop a computationally efficient double loop MC (DLMC) estimator. We conduct a comprehensive numerical error and work analysis of the DLMC estimator. As a result, we show optimal complexity of with a significantly reduced constant to achieve a prescribed relative error tolerance . Subsequently, we propose an adaptive DLMC method combined with IS to numerically estimate rare-event probabilities, substantially reducing relative variance and computational runtimes required to achieve a given compared with standard MC estimators in the absence of IS. Numerical experiments are performed on the Kuramoto model from statistical physics.
... Stochastic Optimal Control (SOC) is designed to steer a noisy system toward a desired state by minimizing a specific cost function. This methodology finds extensive applications across various fields in science and engineering, including rate event simulation [29,31], stochastic filtering and data assimilation [43,65,49], non-convex optimization [9], modeling population dynamics [8,39]. SOC is also related to diffusion-based sampling methods that are predominant in machine learning literature. ...
Recent advancements in diffusion models and diffusion bridges primarily focus on finite-dimensional spaces, yet many real-world problems necessitate operations in infinite-dimensional function spaces for more natural and interpretable formulations. In this paper, we present a theory of stochastic optimal control (SOC) tailored to infinite-dimensional spaces, aiming to extend diffusion-based algorithms to function spaces. Specifically, we demonstrate how Doob's h-transform, the fundamental tool for constructing diffusion bridges, can be derived from the SOC perspective and expanded to infinite dimensions. This expansion presents a challenge, as infinite-dimensional spaces typically lack closed-form densities. Leveraging our theory, we establish that solving the optimal control problem with a specific objective function choice is equivalent to learning diffusion-based generative models. We propose two applications: (1) learning bridges between two infinite-dimensional distributions and (2) generative models for sampling from an infinite-dimensional distribution. Our approach proves effective for diverse problems involving continuous function space representations, such as resolution-free images, time-series data, and probability density functions.
... These methods, however, are not well suited for non-linear problems Morio et al. (2014). The use of surrogate models ranges from the introduction of a control term in the dynamic equations to guide trajectories Bouchet et al. (2019); Hartmann et al. (2019) to the rare event set 1 to problem-specific surrogate models or classical deterministic surrogates as polynomials, splines or the Kriging method. ...
Appropriate methods for the conjunction analysis between orbital objects are at the core of the provision of Space Surveillance and Tracking services. The increasing number of RSOs and the associated requirements on safety and automation requires a search for alternative approaches in conjunction analysis to improve the current state-of-the-art. Classical methods are based on several assumptions that must be held for the computation to be accurate enough. Exploration of new approaches are required to find a suitable balance between accuracy and computational cost. In other fields, the accurate computation of rare events has been a matter of extensive research. The novel method proposed in this communication is based on the extreme event theory, extensively used in other contexts, and an approximate dynamical model rooted in the insight of the Keplerian approach. The probability of collision can then be computed from the use of a Monte Carlo approach leveraging from the efficiency of the approximate method. Finally, an importance sampling approach can be included to reduce further the computational cost of the method.
... The control v in the forward process may be beneficial from the computational point of view, pushing the process into regions of interest or reducing the variance of Monte Carlo estimators, cf. Hartmann et al. (2019). ...
The numerical approximation of partial differential equations (PDEs) poses formidable challenges in high dimensions since classical grid-based methods suffer from the so-called curse of dimensionality. Recent attempts rely on a combination of Monte Carlo methods and variational formulations, using neural networks for function approximation. Extending previous work (Richter et al., 2021), we argue that tensor trains provide an appealing framework for parabolic PDEs: The combination of reformulations in terms of backward stochastic differential equations and regression-type methods holds the promise of leveraging latent low-rank structures, enabling both compression and efficient computation. Emphasizing a continuous-time viewpoint, we develop iterative schemes, which differ in terms of computational efficiency and robustness. We demonstrate both theoretically and numerically that our methods can achieve a favorable trade-off between accuracy and computational efficiency. While previous methods have been either accurate or fast, we have identified a novel numerical strategy that can often combine both of these aspects.
... has zero variance under the probability measure Q = Q generated by the controlled process , where is any adapted and square-integrable process. The main conclusion is that we can change the drift of the forward SDE by modifying the control without affecting the variance of the free energy estimator Richter 2019). Having a zero variance estimator is of course only useful under the assumption that it is possible to (approximately) solve the BSDE associated with (6.52) or (6.56). ...
One of the main challenges in molecular dynamics is overcoming the ‘timescale barrier’: in many realistic molecular systems, biologically important rare transitions occur on timescales that are not accessible to direct numerical simulation, even on the largest or specifically dedicated supercomputers. This article discusses how to circumvent the timescale barrier by a collection of transfer operator-based techniques that have emerged from dynamical systems theory, numerical mathematics and machine learning over the last two decades. We will focus on how transfer operators can be used to approximate the dynamical behaviour on long timescales, review the introduction of this approach into molecular dynamics, and outline the respective theory, as well as the algorithmic development, from the early numerics-based methods, via variational reformulations, to modern data-based techniques utilizing and improving concepts from machine learning. Furthermore, its relation to rare event simulation techniques will be explained, revealing a broad equivalence of variational principles for long-time quantities in molecular dynamics. The article will mainly take a mathematical perspective and will leave the application to real-world molecular systems to the more than 1000 research articles already written on this subject.
... The novelty of this work is establishing a connection between IS and SOC in the context of pure jump processes, particularly for SRNs, with an emphasis on related practical and numerical aspects. Note that some previous studies [7,17,20,[28][29][30][31]33,41,49] have established a similar connection, mainly in the diffusion dynamics context, with less focus on pure jump dynamics. In this work, the proposed methodology is based on an approximate explicit TL scheme, which could and be subsequently extended in future work to continuous-time formulation (exact schemes), and implicit TL schemes which are relevant for systems with fast and slow time scales. ...
We explore efficient estimation of statistical quantities, particularly rare event probabilities, for stochastic reaction networks. Consequently, we propose an importance sampling (IS) approach to improve the Monte Carlo (MC) estimator efficiency based on an approximate tau-leap scheme. The crucial step in the IS framework is choosing an appropriate change of probability measure to achieve substantial variance reduction. This task is typically challenging and often requires insights into the underlying problem. Therefore, we propose an automated approach to obtain a highly efficient path-dependent measure change based on an original connection in the stochastic reaction network context between finding optimal IS parameters within a class of probability measures and a stochastic optimal control formulation. Optimal IS parameters are obtained by solving a variance minimization problem. First, we derive an associated dynamic programming equation. Analytically solving this backward equation is challenging, hence we propose an approximate dynamic programming formulation to find near-optimal control parameters. To mitigate the curse of dimensionality, we propose a learning-based method to approximate the value function using a neural network, where the parameters are determined via a stochastic optimization algorithm. Our analysis and numerical experiments verify that the proposed learning-based IS approach substantially reduces MC estimator variance, resulting in a lower computational complexity in the rare event regime, compared with standard tau-leap MC estimators.
... The distribution is then unbiased to obtain the true probability of the event. Different kinds of importance sampling methods have been developed to sample rare events, such as instanton based importance sampling [33] and adaptive importance sampling [34]. In splitting algorithms, events close to the rare event of interest are realized many times while other events are allowed with a certain probability, in the course of the simulation. ...
We develop a biased Monte Carlo algorithm to measure probabilities of rare events in cluster-cluster aggregation for arbitrary collision kernels. Given a trajectory with a fixed number of collisions, the algorithm modifies both the waiting times between collisions, as well as the sequence of collisions, using local moves. We show that the algorithm is ergodic by giving a protocol that transforms an arbitrary trajectory to a standard trajectory using valid Monte Carlo moves. The algorithm can sample rare events with probabilities of the order of and lower. The algorithm's effectiveness in sampling low-probability events is established by showing that the numerical results for the large deviation function of constant-kernel aggregation reproduce the exact results. It is shown that the algorithm can obtain the large deviation functions for other kernels, including gelling ones, as well as the instanton trajectories for atypical times. The dependence of the autocorrelation times, both temporal and configurational, on the different parameters of the algorithm is also characterized.
... R) being the solution to the parabolic PDE in(77), hold true independent of the choice of v(Hartmann et al., 2019). ...
The combination of Monte Carlo methods and deep learning has recently led to efficient algorithms for solving partial differential equations (PDEs) in high dimensions. Related learning problems are often stated as variational formulations based on associated stochastic differential equations (SDEs), which allow the minimization of corresponding losses using gradient-based optimization methods. In respective numerical implementations it is therefore crucial to rely on adequate gradient estimators that exhibit low variance in order to reach convergence accurately and swiftly. In this article, we rigorously investigate corresponding numerical aspects that appear in the context of linear Kolmogorov PDEs. In particular, we systematically compare existing deep learning approaches and provide theoretical explanations for their performances. Subsequently, we suggest novel methods that can be shown to be more robust both theoretically and numerically, leading to substantial performance improvements.
... R) being the solution to the parabolic PDE in(77), hold true independent of the choice of v(Hartmann et al., 2019). ...
The combination of Monte Carlo methods and deep learning has recently led to efficient algorithms for solving partial differential equations (PDEs) in high dimensions. Related learning problems are often stated as variational formulations based on associated stochastic differential equations (SDEs), which allow the minimization of corresponding losses using gradient-based optimization methods. In respective numerical implementations it is therefore crucial to rely on adequate gradient estimators that exhibit low variance in order to reach convergence accurately and swiftly. In this article, we rigorously investigate corresponding numerical aspects that appear in the context of linear Kolmogorov PDEs. In particular, we systematically compare existing deep learning approaches and provide theoretical explanations for their performances. Subsequently, we suggest novel methods that can be shown to be more robust both theoretically and numerically, leading to substantial performance improvements.
... Further variational perspectives have been suggested in [33], putting additional emphasis on certain numerical robustness properties and allowing for high-dimensional applications by modeling the control with neural networks. For strategies that are based on backward stochastic differential equations we refer for instance to [15]. The optimal control attempt has also been combined with model reduction techniques in [19,20] and [45], noting that one of the main drawbacks of this approach is the placing of ansatz functions over the domain of interest. ...
Sampling rare events in metastable dynamical systems is often a computationally expensive task and one needs to resort to enhanced sampling methods such as importance sampling. Since we can formulate the problem of finding optimal importance sampling controls as a stochastic optimization problem, this then brings additional numerical challenges and the convergence of corresponding algorithms might as well suffer from metastabilty. In this article, we address this issue by combining systematic control approaches with the heuristic adaptive metadynamics method. Crucially, we approximate the importance sampling control by a neural network, which makes the algorithm in principle feasible for high dimensional applications. We can numerically demonstrate in relevant metastable problems that our algorithm is more effective than previous attempts and that only the combination of the two approaches leads to a satisfying convergence and therefore to an efficient sampling in certain metastable settings.
... Note that the representation of (25) by a 2BSDE is not unique, even though its solution is (cf. [2]). ...
... where U is the set of admissible controls, a control process that verify (1.3) is called optimal. In the recent years the framework of G-expectation has found increasing application in the domain of finance and economics, e.g., Epstein and Ji [16,17] study the asset pricing with ambiguity preferences, Beissner [5] who studies the equilibrium theory with ambiguous volatility, and many others see e.g [6,48,49], also see [25][26][27]. for numerical methods. The motivation is that many systems are subject to model uncertainty or ambiguity due to incomplete information, or vague concepts and principles. ...
This paper is concerned with optimal control of systems driven by G-stochastic differential equations (G-SDEs), with controlled jump term. We study the relaxed problem, in which admissible controls are measurevalued processes and the state variable is governed by an G-SDE driven by a counting measure valued process called relaxed Poisson measure such that the compensator is a product measure. Under some conditions on the coefficients, using the G-chattering lemma, we show that the strict and the relaxed control problems have the same value function. Additionally, we derive a maximum principle for this relaxed problem.
... where U is the set of admissible controls, a control process that verify (1.3) is called optimal. In the recent years the framework of G-expectation has found increasing application in the domain of finance and economics, e.g., Epstein and Ji [16,17] study the asset pricing with ambiguity preferences, Beissner [5] who studies the equilibrium theory with ambiguous volatility, and many others see e.g [6,48,49], also see [25][26][27]. for numerical methods. The motivation is that many systems are subject to model uncertainty or ambiguity due to incomplete information, or vague concepts and principles. ...
This paper is concerned with optimal control of systems driven by G-stochastic differential equations (G-SDEs), with controlled jump term. We study the relaxed problem, in which admissible controls are measurevalued processes and the state variable is governed by an G-SDE driven by a counting measure valued process called relaxed Poisson measure such that the compensator is a product measure. Under some conditions on the coefficients, using the G-chattering lemma, we show that the strict and the relaxed control problems have the same value function. Additionally, we derive a maximum principle for this relaxed problem.
... The novelty of this work is establishing a connection between IS and SOC in the context of pure jump processes, particularly for SRNs, with an emphasis on related practical and numerical aspects. Note that some previous studies [20,17,7,28,50,30,33,31,29,42] have established a similar connection, mainly in the diffusion dynamics context, with less focus on pure jump dynamics. In this work, the proposed methodology is based on an approximate explicit TL scheme, which could and be subsequently extended in future work to continuous-time formulation (exact schemes), and implicit TL schemes which are relevant for systems with fast and slow time scales. ...
We explore efficient estimation of statistical quantities, particularly rare event probabilities, for stochastic reaction networks. Consequently, we propose an importance sampling (IS) approach to improve the Monte Carlo (MC) estimator efficiency based on an approximate tau-leap scheme. The crucial step in the IS framework is choosing an appropriate change of probability measure to achieve substantial variance reduction. This task is typically challenging and often requires insights into the underlying problem. Therefore, we propose an automated approach to obtain a highly efficient path-dependent measure change based on an original connection in the stochastic reaction network context between finding optimal IS parameters within a class of probability measures and a stochastic optimal control formulation. Optimal IS parameters are obtained by solving a variance minimization problem. First, we derive an associated dynamic programming equation. Analytically solving this backward equation is challenging, hence we propose an approximate dynamic programming formulation to find near-optimal control parameters. To mitigate the curse of dimensionality, we propose a learning-based method to approximate the value function using a neural network, where the parameters are determined via a stochastic optimization algorithm. Our analysis and numerical experiments verify that the proposed learning-based IS approach substantially reduces MC estimator variance, resulting in a lower computational complexity in the rare event regime, compared with standard tau-leap MC estimators.
... Note that the representation of (25) by a 2BSDE is not unique, even though its solution is (cf. [2]). ...
In this article we propose a -hypergeometric model with uncertain volatility (UV) where we derive a worst-case scenario for option pricing. The approach is based on the connexion between a certain class of nonlinear partial differential equations of HJB-type (G-HJB equations), that govern the nonlinear expectation of the UV model and that provide an alternative to the difficult model calibration problem of UV models, and second-order backward stochastic differential equations (2BSDEs). Using asymptotic analysis for the G-HJB equation and the equivalent 2BSDE representation, we derive a limit model that provides an accurate description of the worst-case price scenario in cases when the bounds of the UV model are slowly varying. The analytical results are tested by numerical simulations using a deep learning based approximation of the underlying 2BSDE.
... with V : R d ×[0, T ] → R being the solution to the parabolic PDE (1), hold true independent of the choice of v (Hartmann et al., 2019). Our algorithms readily transfer to this change in sampling the forward process by adapting the backward process and the corresponding loss functionals (10) and (11) accordingly. ...
High-dimensional partial differential equations (PDEs) are ubiquitous in economics, science and engineering. However, their numerical treatment poses formidable challenges since traditional grid-based methods tend to be frustrated by the curse of dimensionality. In this paper, we argue that tensor trains provide an appealing approximation framework for parabolic PDEs: the combination of reformulations in terms of backward stochastic differential equations and regression-type methods in the tensor format holds the promise of leveraging latent low-rank structures enabling both compression and efficient computation. Following this paradigm, we develop novel iterative schemes, involving either explicit and fast or implicit and accurate updates. We demonstrate in a number of examples that our methods achieve a favorable trade-off between accuracy and computational efficiency in comparison with state-of-the-art neural network based approaches.
... Pre-asymptotic approximations to the optimal proposal are necessary when studying escape problems, for which the time horizon of the problem is either indefinite or infinitely large, a case that has been analysed in [18]. A non-asymptotic variant of the aforementioned approaches for finite noise diffusions is based on the stochastic control formulation of the optimal change of measure [28,30]. Furthermore we should note that there have been many attempts to find good (low-dimensional) proposal by taking advantage of specific structures of the problem at hand, using simplified models that approximate a complicated multiscale system [17,29,32,50]. ...
Importance sampling is a popular variance reduction method for Monte Carlo estimation, where a notorious question is how to design good proposal distributions. While in most cases optimal (zero-variance) estimators are theoretically possible, in practice only suboptimal proposal distributions are available and it can often be observed numerically that those can reduce statistical performance significantly, leading to large relative errors and therefore counteracting the original intention. In this article, we provide nonasymptotic lower and upper bounds on the relative error in importance sampling that depend on the deviation of the actual proposal from optimality, and we thus identify potential robustness issues that importance sampling may have, especially in high dimensions. We focus on path sampling problems for diffusion processes, for which generating good proposals comes with additional technical challenges, and we provide numerous numerical examples that support our findings.
... In many applications of stochastic optimal control, solving the corresponding HJB equation with probabilistic methods is required, especially in high dimensional problems see e.g. [14]. In these cases where we do not know the exact value of the volatility, but only a range of it, like the case of finance, the corresponding HJB equation in a fully non-linear G-PDE equation, and in case of high dimension we can't solve this end by the usual methods like the finite difference, so a probabilistic representation is required, and when the control enter the diffusion see e.g. ...
In this paper, we study the existence and the uniqueness of solution of coupled G-forward-backward stochastic differential equations (G-FBDSEs in short). Our systems are described by coupled multi-dimensional G-FBDSEs. We construct a mapping for which the fixed point is the solution of our G-FBSDE, where we prove that this mapping is a contraction. In this paper we do not require the monotonicity condition to prove the existence.
... Hartmann et al. 12 present an adaptive dynamical importance sampling method. Here, a stochastic di erential equation of the form dx = b(x)dt + σ (x)dW is modi ed to include a control drift term u, becoming dx = (b(x) + σ (x)u(t))dt + σ (x)dW. ...
Stochastic optimal control (SOC) aims to direct the behavior of noisy systems and has widespread applications in science, engineering, and artificial intelligence. In particular, reward fine-tuning of diffusion and flow matching models and sampling from unnormalized methods can be recast as SOC problems. A recent work has introduced Adjoint Matching (Domingo-Enrich et al., 2024), a loss function for SOC problems that vastly outperforms existing loss functions in the reward fine-tuning setup. The goal of this work is to clarify the connections between all the existing (and some new) SOC loss functions. Namely, we show that SOC loss functions can be grouped into classes that share the same gradient in expectation, which means that their optimization landscape is the same; they only differ in their gradient variance. We perform simple SOC experiments to understand the strengths and weaknesses of different loss functions.
We consider a stochastic optimal exit time feedback control problem. The Bellman equation is solved approximatively via the Policy Iteration algorithm on a polynomial ansatz space by a sequence of linear equations. As high degree multipolynomials are needed, the corresponding equations suffer from the curse of dimensionality even in moderate dimensions. We employ Tensor-Train methods to account for this problem. The approximation process within the Policy Iteration is done via a Least-Squares ansatz and the integration is done via Monte-Carlo methods. Numerical evidences are given for the (multidimensional) double-well potential, a three-hole potential, and a 40-dimensional stochastic Van der Pol oscillator.
We exploit the relationship between the stochastic Koopman operator and the Kolmogorov backward equation to construct importance sampling schemes for stochastic differential equations. Specifically, we propose using eigenfunctions of the stochastic Koopman operator to approximate the Doob transform for an observable of interest (e.g., associated with a rare event) which in turn yields an approximation of the corresponding zero-variance importance sampling estimator. Our approach is broadly applicable and systematic, treating non-normal systems, non-gradient systems, and systems with oscillatory dynamics or rank-deficient noise in a common framework. In nonlinear settings where the stochastic Koopman eigenfunctions cannot be derived analytically, we use dynamic mode decomposition (DMD) methods to approximate them numerically, but the framework is agnostic to the particular numerical method employed. Numerical experiments demonstrate that even coarse approximations of a few eigenfunctions, where the latter are built from non-rare trajectories, can produce effective importance sampling schemes for rare events.
This text presents one of the first successful applications of a rare events sampling method for the study of multistability in a turbulent flow without stochastic energy injection. The trajectories of collapse of turbulence in plane Couette flow, and their probability and rate of occurrence are systematically computed using adaptive multilevel splitting (AMS). The AMS computations are performed in a system of size at Reynolds number R=370 with an acceleration by a factor with respect to direct numerical simulations (DNS) and in a system of size at Reynolds number R=377 with an acceleration by a factor . The AMS results are validated by a comparison with DNS in the smaller system. Visualisations indicate that turbulence collapses because the self-sustaining process of turbulence fails locally. The streamwise vortices decay first in streamwise elongated holes, leaving streamwise invariant streamwise velocity tubes that experience viscous decay. These holes then extend in the spanwise direction. The examination of more than a thousand trajectories in the plane in the smaller system confirms the faster decay of streamwise vortices and shows concentration of trajectories. This hints at an instanton phenomenology in the large size limit. The computation of turning point states, beyond which laminarisation is certain, confirms the hole formation scenario and shows that it is more pronounced in larger systems. Finally, the examination of non-reactive trajectories indicates that both the vortices and the streaks reform concomitantly when the laminar holes close.
In this article, we study the relaxed control problem where the admissible controls are measure-valued processes and the state variable is governed by a G-stochastic differential equation (SDEs) driven by a relaxed Poisson measure where the compensator is a product measure. The control variable appears in the drift and in the jump term. We prove that every solution of our SDE associated to a relaxed control can be written as a limit of a sequence of solutions of SDEs associated to strict controls (stability results). In the end, we show the existence of our relaxed control.
This paper develops algorithms for high-dimensional stochastic control problems based on deep learning and dynamic programming. Unlike classical approximate dynamic programming approaches, we first approximate the optimal policy by means of neural networks in the spirit of deep reinforcement learning, and then the value function by Monte Carlo regression. This is achieved in the dynamic programming recursion by performance or hybrid iteration, and regress now methods from numerical probabilities. We provide a theoretical justification of these algorithms. Consistency and rate of convergence for the control and value function estimates are analyzed and expressed in terms of the universal approximation error of the neural networks, and of the statistical error when estimating network function, leaving aside the optimization error. Numerical results on various applications are presented in a companion paper (Bachouch et al. 2021) and illustrate the performance of the proposed algorithms.
This paper presents several numerical applications of deep learning-based algorithms for discrete-time stochastic control problems in finite time horizon that have been introduced in [Huré et al. 2021]. Numerical and comparative tests using TensorFlow illustrate the performance of our different algorithms, namely control learning by performance iteration (algorithms NNcontPI and ClassifPI), control learning by hybrid iteration
(algorithms Hybrid-Now and Hybrid-LaterQ),
on the 100-dimensional nonlinear PDEs examples from [E et al. 2017] and on quadratic backward stochastic differential equations as in [Chassagneux and Richou 2016].
We also performed tests on low-dimension control problems such as an option hedging problem in finance, as well as energy storage problems arising in the valuation of gas storage and in microgrid management. Numerical results and comparisons to quantization-type algorithms Qknn, as an efficient algorithm to numerically solve low-dimensional control problems, are also provided.
Data assimilation addresses the general problem of how to combine model-based predictions with partial and noisy observations of the process in an optimal manner. This survey focuses on sequential data assimilation techniques using probabilistic particle-based algorithms. In addition to surveying recent developments for discrete- and continuous-time data assimilation, both in terms of mathematical foundations and algorithmic implementations, we also provide a unifying framework from the perspective of coupling of measures, and Schrödinger’s boundary value problem for stochastic processes in particular.
The aim of this work is to present a novel sampling-based numerical scheme designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). By means of a nonlinear version of the Feynman–Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton–Jacobi–Bellman equation, expressed in the form of a system of decoupled FBSDEs. This system of FBSDEs can be solved by employing linear regression techniques. The proposed framework relaxes some of the restrictive conditions present in recent sampling based methods within the Linearly Solvable Optimal Control framework, and furthermore addresses problems in which the time horizon is not prespecified. To enhance the efficiency of the proposed scheme when treating more complex nonlinear systems, we then derive an iterative algorithm based on Girsanov’s theorem on the change of measure, which features importance sampling. This scheme is shown to be capable of learning the optimal control without requiring an initial guess.
The article surveys and extends variational formulations of the thermodynamic free energy and discusses their information-theoretic content from the perspective of mathematical statistics. We revisit the well-known Jarzynski equality for nonequilibrium free energy sampling within the framework of importance sampling and Girsanov change-of-measure transformations. The implications of the different variational formulations for designing efficient stochastic optimization and nonequilibrium simulation algorithms for computing free energies are discussed and illustrated.
We propose a new algorithm for solving parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs) in high dimension, by making an analogy between the BSDE and reinforcement learning with the gradient of the solution playing the role of the policy function, and the loss function given by the error between the prescribed terminal condition and the solution of the BSDE. The policy function is then approximated by a neural network, as is done in deep reinforcement learning. Numerical results using TensorFlow illustrate the efficiency and accuracy of the proposed algorithms for several 100-dimensional nonlinear PDEs from physics and finance such as the Allen-Cahn equation, the Hamilton-Jacobi-Bellman equation, and a nonlinear pricing model for financial derivatives.
Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feynman–Kac PI and can be estimated using Monte Carlo sampling. In this contribution we review PI control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. We review the most commonly used methods in robotics and control. Within the PI theory, the question of how to compute becomes the question of importance sampling. Efficient importance samplers are state feedback controllers and the use of these requires an efficient representation. Learning and representing effective state-feedback controllers for non-linear stochastic control problems is a very challenging, and largely unsolved, problem. We show how to learn and represent such controllers using ideas from the cross entropy method. We derive a gradient descent method that allows to learn feed-back controllers using an arbitrary parametrisation. We refer to this method as the path integral cross entropy method or PICE. We illustrate this method for some simple examples. The PI control methods can be used to estimate the posterior distribution in latent state models. In neuroscience these problems arise when estimating connectivity from neural recording data using EM. We demonstrate the PI control method as an accurate alternative to particle filtering.
Der Fokus dieser Dissertation liegt darauf, effiziente numerische Methode für ungekoppelte lokal Lipschitz-stetige und quadratische stochastische Vorwärts-Rückwärtsdifferenzialgleichungen (BSDE) mit Endbedingungen von schwacher Regularität zu entwickeln. Obwohl BSDE viele Anwendungen in der Theorie der Finanzmathematik, der stochastischen Kontrolle und der partiellen Differenzialgleichungen haben, gibt es bisher nur wenige numerische Methoden. Drei neue auf Monte-Carlo- Simulationen basierende Algorithmen werden entwickelt. Die in der zeitdiskreten Approximation zu lösenden bedingten Erwartungen werden mittels der Methode der kleinsten Quadrate näherungsweise berechnet. Ein Vorteil dieser Algorithmen ist, dass sie als Eingabe nur Simulationen eines Vorwärtsprozesses X und der Brownschen Bewegung benötigen. Da sie auf modellfreien Abschätzungen aufbauen, benötigen die hier vorgestellten Verfahren nur sehr schwache Bedingungen an den Prozess X. Daher können sie auf sehr allgemeinen Wahrscheinlichkeitsräumen angewendet werden. Für die drei numerischen Algorithmen werden explizite maximale Fehlerabschätzungen berechnet. Die Algorithmen werden dann auf Basis dieser maximalen Fehler kalibriert und die Komplexität der Algorithmen wird berechnet. Mithilfe einer zeitlich lokalen Abschneidung des Treibers der BSDE werden quadratische BSDE auf lokal Lipschitz-stetige BSDE zurückgeführt. Es wird gezeigt, dass die Komplexität der Algorithmen im lokal Lipschitz-stetigen Fall vergleichbar zu ihrer Komplexität im global Lipschitz-stetigen Fall ist. Es wird auch gezeigt, dass der Vergleich mit bereits für Lipschitz-stetige BSDE existierenden Methoden für die hier vorgestellten Algorithmen positiv ausfällt.
An important sampling method for certain rare event problems involving small noise diffusions is proposed. Standard Monte Carlo schemes for these problems behave exponentially poorly in the small noise limit. Previous work in rare event simulation has focused on developing estimators with optimal exponential variance decay rates. This criterion still allows for exponential growth of the statistical relative error. We show that an estimator related to a deterministic control problem not only has an optimal variance decay rate but can have vanishingly small relative statistical error in the small noise limit. The sampling method based on this estimator can be seen as the limit of the zero variance importance sampling scheme, which uses the solution of the second-order partial differential equation (PDE) associated with the diffusion. In the scheme proposed here this PDE is replaced by a Hamilton-Jacobi equation whose solution is computed pointwise on the fly from its variational formulation, an operation that remains practical even in high-dimensional problems. We test the scheme on several simple illustrative examples as well as a stochastic PDE, the noisy Allen-Cahn equation. © 2012 Wiley Periodicals, Inc.
We present a reformulation of the stochastic optimal control problem in terms of KL divergence minimisation, not only providing a unifying perspective of previous approaches in this area, but also demonstrating that the formalism leads to novel practical approaches to the control problem. Specifically, a natural relaxation of the dual formulation gives rise to exact iterative solutions to the finite and infinite horizon stochastic optimal control problem, while direct application of Bayesian inference methods yields instances of risk sensitive control.
In this chapter we review some of the recent developments for efficient estimation of rare-events, most of which involve application of importance sampling techniques to achieve variance reduction. The zero-variance importance sampling measure is well known and in many cases has a simple representation. Though not implementable, it proves useful in selecting good and implementable importance sampling changes of measure that are in some sense close to it and thus provides a unifying framework for such selections. Specifically, we consider rare events associated with: 1) multi-dimensional light-tailed random walks, 2) with certain events involving heavy-tailed random variables and 3) queues and queueing networks. In addition, we review the recent literature on development of adaptive importance sampling techniques to quickly estimate common performance measures associated with finite-state Markov chains. We also discuss the application of rare-event simulation techniques to problems in financial engineering. The discussion in this chapter is non-measure theoretic and kept sufficiently simple so that the key ideas are accessible to beginners. References are provided for more advanced treatments.
This paper discusses a novel strategy for simulating rare events and an associated Monte Carlo estimation of tail probabilities.
Our method uses a system of interacting particles and exploits a Feynman-Kac representation of that system to analyze their
fluctuations. Our precise analysis of the variance of a standard multilevel splitting algorithm reveals an opportunity for
improvement. This leads to a novel method that relies on adaptive levels and produces, in the limit of an idealized version
of the algorithm, estimates with optimal variance. The motivation for this theoretical work comes from problems occurring
in watermarking and fingerprinting of digital contents, which represents a new field of applications of rare event simulation
techniques. Some numerical results show performance close to the idealized version of our technique for these practical applications.
KeywordsRare event–Sequential importance sampling–Feynman-Kac formula–Metropolis-Hastings–Fingerprinting–Watermarking
We consider duality relations between risk-sensitive stochastic control problems and dynamic games. They are derived from
two basic duality results, the first involving free energy and relative entropy and resulting from a Legendre-type transformation,
the second involving power functions. Our approach allows us to treat, in essentially the same way, continuous- and discrete-time
problems, with complete and partial state observation, and leads to a very natural formal justification of the structure of
the cost functional of the dual. It also allows us to obtain the solution of a stochastic game problem by solving a risk-sensitive
control problem.
In [Probab. Theory Related Fields 141 (2008) 543-567], the authors proved the uniqueness among the solutions of quadratic BSDEs with convex generators and unbounded terminal conditions which admit every exponential moments. In this paper, we prove that uniqueness holds among solutions which admit some given exponential moments. These exponential moments are natural as they are given by the existence theorem. Thanks to this uniqueness result we can strengthen the nonlinear Feynman-Kac formula proved in [Probab. Theory Related Fields 141 (2008) 543-567]. © Association des Publications de l'Institut Henri Poincaré, 2011.
We reformulate a class of non-linear stochastic optimal control problems
introduced by Todorov (2007) as a Kullback-Leibler (KL) minimization problem.
As a result, the optimal control computation reduces to an inference
computation and approximate inference methods can be applied to efficiently
compute approximate optimal controls. We show how this KL control theory
contains the path integral control method as a special case. We provide an
example of a block stacking task and a multi-agent cooperative game where we
demonstrate how approximate inference can be successfully applied to instances
that are too complex for exact computation. We discuss the relation of the KL
control approach to other inference approaches to control.
This paper presents a method for fast simulation of rare events, called RESTART (REpetitive Simulation Trials After Reaching Thresholds). The method obtains dramatic computer time savings for an equal confidence of the results. The paper describes the method in its multiple thresholds version, shows the efficiency of the method and gives criteria to choose its parameters. It presents a simulation library (ASTRO) in which RESTART is implemented and, finally, application examples to teletraffic and reliability problems are provided.
We are concerned with the numerical resolution of backward stochastic differential equations. We propose a new numerical scheme based on iterative regressions on function bases, which coefficients are evaluated using Monte Carlo simulations. A full convergence analysis is derived. Numerical experiments about finance are included, in particular, concerning option pricing with differential interest rates.
This article develops a class of Monte Carlo (MC) methods for simulating conditioned diffusion sample paths, with special emphasis on importance sampling schemes. We restrict attention to a particular type of conditioned diffusion, the so-called diffusion bridge processes. The diffusion bridge is the process obtained by conditioning a diffusion to start and finish at specific values at two consecutive times t0 < t1.
We describe an adaptive importance sampling algorithm for rare events that is based on a dual stochastic control formulation of a path sampling problem. Specifically, we focus on path functionals that have the form of cumulate generating functions, which appear relevant in the context of, e.g.~molecular dynamics, and we discuss the construction of an optimal (i.e. minimum variance) change of measure by solving a stochastic control problem. We show that the associated semi-linear dynamic programming equations admit an equivalent formulation as a system of uncoupled forward-backward stochastic differential equations that can be solved efficiently by a least squares Monte Carlo algorithm. We illustrate the approach with a suitable numerical example and discuss the extension of the algorithm to high-dimensional systems.
This chapter provides a relatively low-level introduction to the problem of rare event simulation with Monte Carlo methods and to a class of methods known as variance reduction techniques that have been devised to deal with this problem. Special emphasis is given to importance sampling, but several other techniques are also presented, including the cross-entropy method, rejection sampling, and Markov chain Monte Carlo methods such as the Metropolis method and Gibbs sampling. A brief discussion is also given about asymptotic efficiency and the connections with large deviations theory.
In the host of numerical schemes devised to calculate free energy differences by way of geometric transformations, the adaptive biasing force algorithm has emerged as a promising route to map complex free-energy landscapes. It relies upon the simple concept that as a simulation progresses, a continuously updated biasing force is added to the equations of motion, such that in the long-time limit it yields a Hamiltonian devoid of an average force acting along the transition coordinate of interest. This means that sampling proceeds uniformly on a flat free-energy surface, thus providing reliable free-energy estimates. Much of the appeal of the algorithm to the practitioner is in its physically intuitive underlying ideas and the absence of any requirements for prior knowledge about free-energy landscapes. Since its inception in 2001, the adaptive biasing force scheme has been the subject of considerable attention, from in-depth mathematical analysis of convergence properties to novel developments and extensions. The method has also been successfully applied to many challenging problems in chemistry and biology. In this contribution, the method is presented in a comprehensive, self-contained fashion, discussing with a critical eye its properties, applicability and inherent limitations, as well as introducing novel extensions. Through free-energy calculations prototypical molecular systems, many methodological aspects are examined, from stratification strategies to overcoming the so-called hidden barriers in orthogonal space, relevant not only to the adaptive biasing force algorithm, but also to other importance-sampling schemes. On the basis of the discussions in this paper, a number of good practices for improving the efficiency and reliability of the computed free-energy differences are proposed.
A numerical scheme for solving high-dimensional stochastic control problems on an infinite time horizon that appear relevant in the context of molecular dynamics is outlined. The scheme rests on the interpretation of the corresponding Hamilton–Jacobi–Bellman equation as a nonlinear eigenvalue problem that, using a logarithmic transformation, can be recast as a linear eigenvalue problem, for which the principal eigenvalue and its eigenfunction are sought. The latter can be computed efficiently by approximating the underlying stochastic process with a coarse-grained Markov state model for the dominant metastable sets. We illustrate our method with two numerical examples, one of which involves the task of maximizing the population of α-helices in an ensemble of small biomolecules (alanine dipeptide), and discuss the relation to the large deviation principle of Donsker and Varadhan.
The estimation of rare event probability is a crucial issue in areas such as reliability, telecommunications, aircraft management. In complex systems, analytical study is out of question and one has to use Monte Carlo methods. When rare is really rare, which means a probability less than 10^−9, naive Monte Carlo becomes unreasonable. A widespread technique consists in multilevel splitting, but this method requires enough knowledge about the system to decide where to put the levels at hand. This is unfortunately not always possible. In this paper, we propose an adaptive algorithm to cope with this problem: the estimation is asymptotically consistent, costs just a little bit more than classical multilevel splitting and has the same efficiency in terms of asymptotic variance. In the one dimensional case, we prove rigorously the a.s. convergence and the asymptotic normality of our estimator, with the same variance as with other algorithms that use fixed crossing levels. In our proofs we mainly use tools from the theory of empirical processes, which seems to be quite new in the field of rare events.
Stochastic control problems on an infinite time horizon with exponential cost criteria are considered. The Donsker--Varadhan large deviation rate is used as a criterion to be optimized. The optimum rate is characterized as the value of an associated stochastic differential game, with an ergodic (expected average cost per unit time) cost criterion. If we take a small-noise limit, a deterministic differential game with average cost per unit time cost criterion is obtained. This differential game is related to robust control of nonlinear systems.
A heuristic that has emerged in the area of importance sampling is that the changes of measure used to prove large deviation lower bounds give good performance when used for importance sampling. Recent work, however, has suggested that the heuristic is incorrect in many situations. The perspective put forth in the present paper is that large deviation theory suggests many changes of measure, and that not all are suitable for importance sampling. In the setting of Cramer's Theorem, the traditional interpretation of the heuristic suggests a fixed change of distribution on the underlying independent and identically distributed summands. In contrast, we consider importance sampling schemes where the exponential change of measure is adaptive, in the sense that it depends on the historical empirical mean. The existence of asymptotically optimal schemes within this class is demonstrated. The result indicates that an adaptive change of measure, rather than a static change of measure, is what the large deviations analysis truly suggests. The proofs utilize a control-theoretic approach to large deviations, which naturally leads to the construction of asymptotically optimal adaptive schemes in terms of a limit Bellman equation. Numerical examples contrasting the adaptive and standard schemes are presented, as well as an interpretation of their different performances in terms of differential games.
Simulating rare events in telecommunication networks such as estimation for cell loss probability in asynchronous transfer mode (ATM) networks requires a major simulation effort due to the slight chance of buffer overflow. Importance sampling (IS) is applied to accelerate the occurrence of rare events. Importance sampling depends on a biasing scheme to make the estimator from IS unbiased. Adaptive importance sampling (AIS) employs an estimated sampling distribution of IS to the system of interest during the course of simulation. In this study, we propose a nonparametric adaptive importance sampling (NAIS) technique, a non-parametrically modified version of AIS, and estimate the probability of rare event occurrence in an M/M/1 queueing model. Compared with classical Monte Carlo simulation and AIS, the computational efficiency and variance reductions gained via NAIS are reasonable. A possible extension of NAIS with regards to random number generation is also discussed.
Rare event simulation and estimation for systems in equilibrium are among the
most challenging topics in molecular dynamics. As was shown by Jarzynski and
others, nonequilibrium forcing can theoretically be used to obtain equilibrium
rare event statistics. The advantage seems to be that the external force can
speed up the sampling of the rare events by biasing the equilibrium
distribution towards a distribution under which the rare events is no longer
rare. Yet algorithmic methods based on Jarzynski's and related results often
fail to be efficient because they are based on sampling in path space. We
present a new method that replaces the path sampling problem by minimization of
a cross-entropy-like functional which boils down to finding the optimal
nonequilibrium forcing. We show how to solve the related optimization problem
in an efficient way by using an iterative strategy based on milestoning.
In this paper we first give a review of the least-squares Monte Carlo approach for approximating the solution of backward stochastic differ-ential equations (BSDEs) first suggested by Gobet, Lemor, and Warin (Ann. Appl. Probab., 15, 2005, 2172–2202). We then propose the use of basis functions, which form a system of martingales, and explain how the least-squares Monte Carlo scheme can be simplified by exploiting the martingale property of the basis functions. We partially compare the convergence behavior of the original scheme and the scheme based on martingale basis functions, and provide several numerical examples related to option pricing problems under different interest rates for borrowing and investing.
In this paper we consider a finite horizon, nonlinear, stochastic, risk-sensitive optimal control problem with complete state information, and show that it is equivalent to a stochastic differential game. Risk-sensitivity and small noise parameters are introduced, and the limits are analyzed as these parameters tend to zero. First-order expansions are obtained which show that the risk-sensitive controller consists of a standard deterministic controller, plus terms due to stochastic and game-theoretic methods of controller design. The results of this paper relate to the design of robust controllers for nonlinear systems.
This paper is concerned with Markov diffusion processes which obey stochastic differential equations depending on a small parameter. The parameter enters as a coefficient in the noise term of the stochastic differential equation. The Ventcel-Freidlin estimates give asymptotic formulas (as0) for such quantities as the probability of exit from a regionD through a given portionN of the boundary D, the mean exit time, and the probability of exit by a given timeT. A new method to obtain such estimates is given, using ideas from stochastic control theory.
An exponential function of cost is adopted as a risk-sensitive criterion, and reasons given that this choice should be a natural one. It is shown that the analysis leads to risk-sensitive versions of the certainty-equivalence principle (separation principle) and of the maximum principle, and that these have a validity even outside the usual linear/quadratic/Gaussian framework. The methods are applied to some simple examples of economic interest.
It was established in (6, 7) that importance sampling algorithms for estimating rare-event probabilities are intimately connected with two-person zero-sum dierential games and the associated Isaacs equa- tion. This game interpretation shows that dynamic or state-dependent schemes are needed in order to attain asymptotic optimality in a gen- eral setting. The purpose of the present paper is to show that classical subsolutions of the Isaacs equation can be used as a basic and flexible tool for the construction and analysis of ecient dynamic importance sampling schemes. There are two main contributions. The first is a basic theoretical result characterizing the asymptotic performance of importance sampling estimators based on subsolutions. The second is an explicit method for constructing classical subsolutions as a mollifi- cation of piecewise ane functions. Numerical examples are included for illustration and to demonstrate that simple, nearly asymptotically optimal importance sampling schemes can be obtained for a variety of problems via the subsolution approach.
The asymptotic behaviors of the principal eigenvalue and the corresponding normalized eigenfunction of the operator for small are studied. Under some conditions, the first order expansions for them are obtained. Two applications to risk-sensitive control problems are also mentioned.
In this paper we show that the variational representation holds, where W is a standard d-dimensional Brownian motion, f is any bounded measurable function that maps into and the infimum is over all processes v that are progressively measurable with respect to the augmentation of the filtration generated by W. An application is made to a problem concerned with large deviations, and an extension to unbounded functions is given.
We provide existence, comparison and stability results for one- dimensional backward stochastic differential equations (BSDEs) when the coefficient (or generator) is continuous and has a quadratic growth in Z and the terminal condition is bounded.e also give, in this framework, the links between the solutions of BSDEs set on a diffusion and viscosity or Sobolev solutions of the corresponding semilinear partial differential equations.
A guiding principle in the efficient estimation of rare-event probabilities by Monte Carlo is that importance sampling based on the change of measure suggested by a large deviations analysis can reduce variance by many orders of magnitude. In a variety of settings, this approach has led to estimators that are optimal in an asymptotic sense. We give examples, however, in which importance sampling estimators based on a large deviations change of measure have provably poor performance. The estimators can have variance that decreases at a slower rate than a naive estimator, variance that increases with the rarity of the event, and even infinite variance. For each example, we provide an alternative estimator with provably efficient performance. A common feature of our examples is that they allow more than one way for a rare event to occur; our alternative estimators give explicit weight to lower probability paths neglected by leading-term asymptotics.
We introduce a forward scheme to simulate backward SDEs. Compared to existing schemes, we avoid high order nestings of conditional expectations backwards in time. In this way the error, when approximating the conditional expectation, in dependence of the time partition is significantly reduced. Besides this generic result, we present an implementable algorithm and provide an error analysis for it. Finally, we demonstrate the strength of the new algorithm by solving some financial problems numerically.
Optimal choice of actions is a fundamental problem relevant to fields as diverse as neuroscience, psychology, economics, computer science, and control engineering. Despite this broad relevance the abstract setting is similar: we have an agent choosing actions over time, an uncertain dynamical system whose state is affected by those actions, and a performance criterion that the agent seeks to optimize. Solving problems of this kind remains hard, in part, because of overly generic formulations. Here, we propose a more structured formulation that greatly simplifies the construction of optimal control laws in both discrete and continuous domains. An exhaustive search over actions is avoided and the problem becomes linear. This yields algorithms that outperform Dynamic Programming and Reinforcement Learning, and thereby solve traditional problems more efficiently. Our framework also enables computations that were not possible before: composing optimal control laws by mixing primitives, applying deterministic methods to stochastic systems, quantifying the benefits of error tolerance, and inferring goals from behavioral data via convex optimization. Development of a general class of easily solvable problems tends to accelerate progress--as linear systems theory has done, for example. Our framework may have similar impact in fields where optimal choice of actions is relevant.
In this article, we explain how the importance sampling technique can be generalized from simulating expectations to computing the initial value of backward stochastic differential equations (SDEs) with Lipschitz continuous driver. By means of a measure transformation we introduce a variance reduced version of the forward approximation scheme by Bender and Denk [44.
Bender , C. , and
Denk , R. 2007 . A forward scheme for backward SDEs . Stochastic Processes and their Applications 117 ( 12 ): 1793 – 1812 . View all references] for simulating backward SDEs. A fully implementable algorithm using the least-squares Monte Carlo approach is developed and its convergence is proved. The success of the generalized importance sampling is illustrated by numerical examples in the context of Asian option pricing under different interest rates for borrowing and lending.
This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton-Jacobi-Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton-Jacobi equation to the Schr\"odinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process, that can be computed by stochastic integration or by the evaluation of a path integral. It is shown, how in the deterministic limit the PMP formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as MC sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise. Comment: 21 pages, 6 figures, submitted to JSTAT
Linear regression MDP scheme for discrete backward stochastic differential equations under general conditions
A method for stochastic optimization
- D P Kingma
- J B Adam
Risk-sensitivity a strangely pervasive concept