No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Optimal stopping of a piecewise-deterministic Markov process
Abstract
This paper deals with the optimal stopping problem for a class of strong, nonstandard Markov processes the paths of which follow continuous deterministic trajectories but for random jumps at random times. The gain function, defined on the state space, is assumed to be bounded, measurable and, roughly speaking, continuous along the trajectories of the deterministic drift Combining continuous-time deterministic maximization and discrete-time dynamic programming yields a functional operator solving the single-jump problem.
... O D'une part, on étudie un problème d'arrêt optimal. L'arrêt optimal est un cas particulier de problème de contrôle stochastique, développé par exemple dans [Dav93,Gug86] pour les PDMP. En considérant un PDMP (X t ) t 0 , le but est de trouver le meilleur temps pour arrêter notre processus, afin de maximiser l'espérance d'une fonctionnelle du processus à l'instant où il est arrêté. ...
... On dit alors que de tels temps d'arrêt sont ε-optimaux. L'étude des fonctions valeurs sur des PDMP finidimensionnels a été développée par exemple dans [Dav93,Gug86] ou encore [dSDZ15]. Ici, on se propose de l'étendre aux PDMP à valeurs mesures, et de montrer qu'on peut caractériser la fonction valeur par itération d'opérateurs. ...
... où S est un temps d'arrêt dans M. On a choisit de détailler ce point important de démonstration, non explicité dans [Gug86]. On en déduit alors : ...
Les processus markoviens déterministes par morceaux (PDMP) forment une vaste classe de processus stochastiques caractérisés par une évolution déterministe entre des sauts à mécanisme aléatoire. Ce sont des processus de type hybride, avec une composante discrète de mode et une composante d’état qui évolue dans un espace continu. Entre les sauts du processus, la composante continue évolue de façon déterministe, puis au moment du saut un noyau markovien sélectionne la nouvelle valeur des composantes discrète et continue. Dans cette thèse, nous construisons des PDMP évoluant dans des espaces de mesures (de dimension infinie), pour modéliser des population de cellules en tenant compte des caractéristiques individuelles de chaque cellule. Nous exposons notre construction des PDMP sur des espaces de mesure, et nous établissons leur caractère markovien. Sur ces processus à valeur mesure, nous étudions un problème d'arrêt optimal. Un problème d'arrêt optimal revient à choisir le meilleur temps d'arrêt pour optimiser l'espérance d'une certaine fonctionnelle de notre processus, ce qu'on appelle fonction valeur. On montre que cette fonction valeur est solution des équations de programmation dynamique et on construit une famille de temps d'arrêt $epsilon$-optimaux. Dans un second temps, nous nous intéressons à un PDMP en dimension finie, le TCP, pour lequel on construit un schéma d'Euler afin de l'approcher. Ce choix de modèle simple permet d'estimer différents types d'erreurs. Nous présentons des simulations numériques illustrant les résultats obtenus.
... The simplest form of impulse control is optimal stopping, where the decision maker selects only one intervention time when the process is stopped. Optimal stopping for PDMPs has been studied in particular in [14,3,9]. ...
... This construction is based on the iteration of a single-jump-or-intervention operator associated to the PDMP. It builds on the explicit construction of ǫ-optimal stopping times developed by U.S. Gugerli [14] for the optimal stopping problem. However, for the general impulse control problem, one must also optimally choose the new starting points of the process, which is a significant source of additional difficulties. ...
... corresponding exactly to the definition (14) ...
This paper deals with the general discounted impulse control problem of a piecewise deterministic Markov process. We investigate a new family of epsilon-optimal strategies. The construction of such strategies is explicit and only necessitates the previous knowledge of the cost of the no-impulse strategy. In particular, it does not require the resolution of auxiliary optimal stopping problem or the computation of the value function at each point of the state space. This approach is based on the iteration of a single-jump-or-intervention operator associated to the piecewise deterministic Markov process.
... Optimal stopping problems have been studied for PDMP's in Costa and Davis (1988); Costa et al. (2000); Davis (1993); Gatarek (1991); Gugerli (1986); Lenhart and Liao (1985). In Gugerli (1986) the author defines an operator related to the first jump time of the process, and shows that the value function of the optimal stopping problem is a fixed point for this operator. ...
... Optimal stopping problems have been studied for PDMP's in Costa and Davis (1988); Costa et al. (2000); Davis (1993); Gatarek (1991); Gugerli (1986); Lenhart and Liao (1985). In Gugerli (1986) the author defines an operator related to the first jump time of the process, and shows that the value function of the optimal stopping problem is a fixed point for this operator. The basic assumption in this case is that the final cost function is continuous along trajectories, and it is shown that the value function will also have this property. ...
... As regards PDMP's, it was shown in Gugerli (1986) that the value function of the optimal stopping problem can be calculated by iterating a functional operator, labeled L (see equation (1) for its definition), which is related to a continuous-time maximization and a discrete-time dynamic programming formula. Thus, in order to approximate the value function of the optimal stopping problem of a PDMP {X(t)}, a natural approach would have been to follow the same lines as in Bally and Pagès (2003); Bally et al. (2005); Pagès et al. (2004b). ...
We propose a numerical method to approximate the value function for the optimal stopping problem of a general class of hybrid processes, namely piecewise deterministic Markov processes (PDMP's). Our approach is based on quantization of the post jump location inter-arrival time Markov chain naturally embedded in the PDMP, and path-adapted time discretization grids. It allows us to derive bounds for the convergence rate of the algorithm and to provide a computable -optimal stopping time.
... On peut montrer que la fonction valeur du problème d'arrêt optimal pour les PDP peut se construire en itérant un opérateur de programmation dynamique, voir [Gug86] et l'équation (2.1). Cet opérateur fait intervenir des espérances conditionnelles qui sont compliquées à calculer et surtout à itérer numériquement. ...
... Gugerli a montré dans [Gug86] que les équations de la programmation dynamique sont les suivantes pour la fonction valeur du problème d'arrêt optimal pour les PDP v N = g, v n = L(v n+1 , g) pour 0 ≤ n ≤ N − 1, ...
... A partir du temps d'arrêt -optimal construit par Gugerli dans [Gug86], nous proposons également une règle d'arrêt calculable en pratique. Elle a en outre l'avantage de n'utiliser que des quantités déjà calculées pour l'approximation de la fonction valeur. ...
Depuis septembre 2006, je suis maître de conférences à l'université Montesquieu Bordeaux IV en mathématiques appliquées. Je suis membre du GREThA (groupement de recherche en économie théorique et appliquée), de l'IMB (Institut de Mathématiques de Bordeaux, équipe probabilités et statistique) et de l'équipe Inria CQFD (contrôle de qualité et fiabilité dynamique). Depuis mon arrivée à Bordeaux, mes travaux de recherche se poursuivent dans deux directions principales : l'une concerne les processus auto-régressifs de bifurcation sur des arbres binaires et l'autre le développement d'outils numériques pour le contrôle des processus markoviens déterministes par morceaux. Il s'agit de processus en temps continu qui changent de régime de façon markovienne au cours du temps. Ce mémoire présente uniquement mes travaux réalisés à Bordeaux depuis 2006.
... Les PAO ontétéétudiés pour les PDMP dans [16,19,21,38,42,53],à horizon infini (noté PAO ∞ ) lorsque l'ensemble des F t -temps d'arrêt τ , noté M, vérifie P x (τ < ∞) = 1 pour tout x ∈ E, ouà horizon aléatoire (noté PAO n ), lorsque l'ensemble des F t -temps d'arrêt τ est dominé par le n-ème temps de saut T n pour n ∈ N, avec F t la filtration naturelle du PDMP (voir Chapitre 1). A partir de la définition qu'en donne M.H.A. Davis, on peut définir le PAO ∞ comme un triplet (M ∞ , g, v) où 1. M ∞ = {τ ∈ M, P x (τ < ∞) = 1 pour tout x ∈ E}, 2. la fonction g, mesurable continue et bornée, détermine le gain g(x) obtenu lorsque le processus est arrêté au point x ∈ E. ...
... Dans [42], l'auteur définit un opérateur relatif au premier temps de saut du processus, et montre que la fonction valeur du PAOà horizon infini est un point fixe pour cet opérateur. L'hypothèse de base dans ce cas est que la fonction gain est continue le long des trajectoires, et il est montré que la fonction valeur a la même propriété. ...
... Notre but est d'utiliser les résultats de [42] et [16] pour les adapter au cas de l'horizon aléatoire, ainsi que des méthodes numériques basées sur la quantification (voir par exemple [3,4,41,60,61,63,62]) et développées dans le cadre des processus de diffusion, pourécrire un schéma de dicrétisation adapté aux PDMP. En effet, la structure particulière des PDMP ne permet pas d'appliquer directement les méthodes existantes, notamment celles développées pour les diffusions. ...
Piecewise Deterministic Markov Processes (PDMP's) have been introduced in the literature by M.H.A. Davis as a general class of stochastics models. PDMP's are a family of Markov processes involving deterministic motion punctuated by random jumps. In a rst part, PDMP's are used to compute probabilities of top events for a case-study of dynamic reliability (the heated tank system) with two di erent methods : the rst one is based on the resolution of the di erential system giving the physical evolution of the tank and the second uses the computation of the functional of a PDMP by a system of integro-di erential equations. In the second part, we propose a numerical method to approximate the value function for the optimal stopping problem of a PDMP. Our approach is based on quantization of the postjump location and inter-arrival time of the Markov chain naturally embedded in the PDMP, and path-adapted time discretization grids. It allows us to derive bounds for the convergence rate of the algorithm and to provide a computable epsilon-optimal stopping time.
... The general mathematical problem of optimal stopping corresponding to this maintenance problem can be found in [12,10,11]. It is now briefly recalled. ...
... The general mathematical problem of optimal stopping corresponding to this maintenance problem can be found in [11,5,6]. It is now briefly recalled. ...
... Define then -optimal stopping times as achieving optimal value minus , i.e. v(z) − . Under fairly weak regularity conditions, Gugerli has shown in [11] that the value function v can be calculated iteratively as follows. First, choose the computational horizon N such that after N jumps, the running time t has reached T f for almost all trajectories. ...
We present a numerical method to compute an optimal maintenance date for the
test case of the heated hold-up tank. The system consists of a tank containing
a fluid whose level is controlled by three components: two inlet pumps and one
outlet valve. A thermal power source heats up the fluid. The failure rates of
the components depends on the temperature, the position of the three components
monitors the liquid level in the tank and the liquid level determines the
temperature. Therefore, this system can be modeled by a hybrid process where
the discrete (components) and continuous (level, temperature) parts interact in
a closed loop. We model the system by a piecewise deterministic Markov process,
propose and implement a numerical method to compute the optimal maintenance
date to repair the components before the total failure of the system.
... Moreover, it is different from the optimal stopping problem of a PDMP under complete observation mainly because the new state variables given by the Markov chain (Π n , S n ) n≥0 are not the underlying Markov chain of some PDMP. Therefore the results of the literature [9,13,18] cannot be used. Finally, a natural way to proceed with the numerical approximation is then to follow the ideas developed in [9,18] namely to replace the filter Π n and the interjump time S n by some finite state space approximations in the dynamic programming equation. ...
... Moreover, it is different from the optimal stopping problem of a PDMP under complete observation mainly because the new state variables given by the Markov chain (Π n , S n ) n≥0 are not the underlying Markov chain of some PDMP. Therefore the results of the literature [9,13] cannot be used. ...
... Notice that V N is known and the expression of V n involves only V n+1 and the Markov chain (Π n , S n ). Thus, the sequence (V n ) 0≤n≤N is completely characterized by the system (13). In addition, V 0 = v 0 (Π 0 ) = v(Π 0 ). ...
This paper deals with the optimal stopping problem under partial observation
for piecewise-deterministic Markov processes. We first obtain a recursive
formulation of the optimal filter process and derive the dynamic programming
equation of the partially observed optimal stopping problem. Then, we propose a
numerical method, based on the quantization of the discrete-time filter process
and the inter-jump times, to approximate the value function and to compute an
actual $\epsilon$-optimal stopping time. We prove the convergence of the
algorithms and bound the rates of convergence.
... Optimal stopping provides a balance between these two extremes, by providing the optimal replacement time given a cost function. Optimal stopping problems have been studied for PDMP's in Costa and Davis (1988); Costa et al. (2000); Davis (1993); Gatarek (1991); Gugerli (1986); Lenhart and Liao (1985). In Gugerli (1986) the author defines an operator related to the first jump time of the process, and shows that the value function of the optimal stopping problem is a fixed point for this operator. ...
... Optimal stopping problems have been studied for PDMP's in Costa and Davis (1988); Costa et al. (2000); Davis (1993); Gatarek (1991); Gugerli (1986); Lenhart and Liao (1985). In Gugerli (1986) the author defines an operator related to the first jump time of the process, and shows that the value function of the optimal stopping problem is a fixed point for this operator. The basic assumption in this case is that the final cost function is continuous along trajectories, and it is shown that the value function will also have this property. ...
... Providing the cost function and the Markov kernel are Lipschitz, some bounds and rates of convergence are obtained (see for example section 2.2.2 in Bally and Pagès (2003)). As regards PDMP's, it was shown in Gugerli (1986) that the value function of the optimal stopping problem can be calculated by iterating a functional operator, labeled L (see equation (1) for its definition), which is related to a continuous-time maximization and a discrete-time dynamic programming formula. Thus, in order to approximate the value function of the optimal stopping problem of a PDMP {X(t)}, a natural approach would have been to follow the same lines as in Bally and Pagès (2003); Bally et al. (2005); Pagès et al. (2004b). ...
We propose a numerical method to approximate the value function for the optimal stopping problem of a general class of hybrid processes, namely piecewise deterministic Markov processes (PDMP's). Our approach is based on quantization of the post jump location – inter-arrival time Markov chain naturally embedded in the PDMP, and path-adapted time discretization grids. It allows us to derive bounds for the convergence rate of the algorithm and to provide a computable -optimal stopping time.
... For any pair of arrival rates λ 0 and λ 1 and mark distributions ν 0 (·) and ν 1 (·), we describe explicitly a quickest detection rule. These rules depend on the some F-adapted odds-ratio process Φ = {Φ t ; t ≥ 0}; see (11). At every t ≥ 0, the random variable Φ t is the conditional odds-ratio of the event {θ ≤ t} that disorder has happened at or before time t given past and present observations F t of the process X. ...
... On the other hand, the functions V n (·) can be found easily by an iterative algorithm. We shall calculate the V n 's by adapting to our problem a method of Gugerli [11] and Davis [7,Chapter 5]. Developed for optimal stopping of general piecewise-deterministic Markov processes with an undiscounted terminal reward, the results of U. Gugerli and M. Davis do not apply here immediately. ...
... 4. Sample paths and bounds on the optimal alarm time. A brief study of sample paths of the sufficient statistic Φ in (11)(12)(13) gives simple lower and upper bounds on the optimal alarm time U 0 in (36). In several special cases, the lower bound becomes optimal. ...
In the compound Poisson disorder problem, arrival rate and/or jump distribution of some compound Poisson process changes suddenly at some unknown and unobservable time. The problem is to detect the change (or disorder) time as quickly as possible. A sudden regime shift may require some countermeasures be taken promptly, and a quickest detection rule can help with those efforts. We describe complete solution of the compound Poisson disorder problem with several standard Bayesian risk measures. Solution methods are feasible for numerical implementation and are illustrated by examples.
... The theoretical problem of optimal stopping for PDMP's is well understood, see e.g. Gugerli [7]. However, there are surprisingly few works in the literature presenting practical algorithms to compute the optimal cost and optimal stopping time. ...
... Under fairly weak regularity conditions, Gugerli has shown in [7] that the value function v can be calculated iteratively as follows. Let v N = g be the reward function, and we iterate an operator L backwards. ...
... However, we can see that this operator depends only on the discrete time Markov chain (Z n , S n ). Gugerli also proposes an iterative construction of -optimal stopping times, which is a bit too tedious and technical to be described here, see [7] for details. ...
We present a numerical method to compute the optimal maintenance time for a
complex dynamic system applied to an example of maintenance of a metallic
structure subject to corrosion. An arbitrarily early intervention may be
uselessly costly, but a late one may lead to a partial/complete failure of the
system, which has to be avoided. One must therefore find a balance between
these too simple maintenance policies. To achieve this aim, we model the system
by a stochastic hybrid process. The maintenance problem thus corresponds to an
optimal stopping problem. We propose a numerical method to solve the optimal
stopping problem and optimize the maintenance time for this kind of processes.
... A suitable choice of the state space and the local characteristics φ, λ and Q provide stochastic models covering a great number of problems of operations research [6]. Optimal stopping problems have been studied for PDMPs in [3,5,6,9,11,13]. In [11] the author defines an operator related to the first jump time of the process and shows that the value function of the optimal stopping problem is a fixed point for this operator. ...
... Optimal stopping problems have been studied for PDMPs in [3,5,6,9,11,13]. In [11] the author defines an operator related to the first jump time of the process and shows that the value function of the optimal stopping problem is a fixed point for this operator. The basic assumption in this case is that the final cost function is continuous along trajectories, and it is shown that the value function will also have this property. ...
... As regards PDMPs, it was shown in [11] that the value function of the optimal stopping problem can be calculated by iterating a functional operator, labeled L [see (3.5) for its definition], which is related to a continuous-time maximization and a discrete-time dynamic programming formula. Thus, in order to approximate the value function of the optimal stopping problem of a PDMP {X(t)}, a natural approach would have been to follow the same lines as in [1,2,17]. ...
We propose a numerical method to approximate the value function for the optimal stopping problem of a piecewise deterministic Markov process (PDMP). Our approach is based on quantization of the post jump location -- inter-arrival time Markov chain naturally embedded in the PDMP, and path-adapted time discretization grids. It allows us to derive bounds for the convergence rate of the algorithm and to provide a computable epsilon-optimal stopping time. The paper is illustrated by a numerical example.
... A suitable choice of the state space and the local characteristics φ, λ, and Q provides stochastic models covering a great number of problems of operations research [6]. Optimal stopping problems have been studied for PDMP's in [3,5,6,9,11,13]. In [11] the author defines an operator related to the first jump time of the process, and shows that the value function of the optimal stopping problem is a fixed point for this operator. ...
... Optimal stopping problems have been studied for PDMP's in [3,5,6,9,11,13]. In [11] the author defines an operator related to the first jump time of the process, and shows that the value function of the optimal stopping problem is a fixed point for this operator. The basic assumption in this case is that the final cost function is continuous along trajectories, and it is shown that the value function will also have this property. ...
... Providing the cost function and the Markov kernel are Lipschitz, some bounds and rates of convergence are obtained (see for example section 2.2.2 in [1]). As regards PDMP's, it was shown in [11] that the value function of the optimal stopping problem can be calculated by iterating a functional operator, labeled L (see equation (3.5) for its definition), which is related to a continuous-time maximization and a discrete-time dynamic programming formula. Thus, in order to approximate the value function of the optimal stopping problem of a PDMP {X(t)}, a natural approach would have been to follow the same lines as in [1,2,17]. ...
... In the first one, the switching decision can be made at any point in time. This gives rise to a continuous stopping problem which is relatively more difficult to analyze; see Gugerli (1986) and (Davis 1993, Chapter 5) for the theory of optimal stopping for piecewise deterministic Markov processes. In the second formulation, on the other hand, the stopping times are assumed to take values on some pre-determined discrete set = {0, t 1 , t 2 , . . . ...
... In the first, the stopping time is continuous in the sense that it can take any values on [0, T ]. Solving the problem for this class requires the methods and techniques of Gugerli (1986) and (Davis 1993, Chapter 5) developed for piecewise deterministic Markov processes. In the second class of stopping times, we restrict ourselves to the set of stopping times taking values on a discrete set . ...
We consider the end-of-life inventory problem for the supplier of a product in its final phase of the service life cycle. This phase starts when the production of the items stops and continues until the warranty of the last sold item expires. At the beginning of this phase the supplier places a final order for spare parts to serve customers coming with defective items. At any time during the final phase the supplier may also decide to switch to an alternative and more cost effective service policy. This alternative policy may be in the form of replacing defective items with substitutable products or offering discounts/rebates on the new generation ones. In this setup, the objective is to find a final order quantity and a time to switch to an alternative policy which will minimize the total expected discounted costs of the supplier. The switching time is a stopping time and is based on the realization of the arrival process of defective items. In this paper, we study this problem under a general cost structure in a continuous-time framework where the arrival of customers is given by a non-homogeneous Poisson process. We show in detail how to compute the value function, and illustrate our approach on numerical examples.
... We also exhibit a sequence of ε-optimal stopping times. Our approach is based on [15] that solved the optimal stopping problem for finite-dimensional PDMPs. ...
... Then we prove that the value functions can be constructed by iteration of the dynamic programming operators and that the stopping times are ε-optimal. This section is inspired from the study of the optimal stopping problem for finite (fixed) dimension PDMPs derived in [15]. ...
This paper investigates the random horizon optimal stopping problem for measure-valued piecewise deterministic Markov processes (PDMPs). This is motivated by population dynamics applications, when one wants to monitor some characteristics of the individuals in a small population. The population and its individual characteristics can be represented by a point measure. We first define a PDMP on a space of locally finite measures. Then we define a sequence of random horizon optimal stopping problems for such processes. We prove that the value function of the problems can be obtained by iterating some dynamic programming operator. Finally we prove on a simple counter-example that controlling the whole population is not equivalent to controlling a random lineage.
... In this subsection, furthermore, we investigate the stochastic reachability, as an optimal stopping problem, for PDMP. The motivation for doing this is the fact, for PDMP, characterizations of the OSP abound in the literature [14, 15, 26, 34]. The approach of [34], extended then in [33], is inspired by the theory of viscosity solutions associated to first order integro-differential operators. ...
... However, the optimal control problems for Feller-Markov processes are well understood now [22], and many other results can be derived in this particular setting. We are more interested in the approach developed in [26], and generalized in [15], and then in [14]. Mainly, in these papers the value function of the optimal stopping problem is characterized as the unique fixed point of the first jump operator. ...
Reachability analysis is the core of model checking of time systems. For stochastic hybrid systems, this safety verification method is very little supported mainly because of complexity and difficulty of the associated mathematical problems. In this paper, we develop two main directions of studying stochastic reachability as an optimal stopping problem. The first approach studies the hypotheses for the dynamic program-ming corresponding with the optimal stopping problem for stochastic hybrid systems. In the second approach, we investigate the reachability problem considering approxima-tions of stochastic hybrid systems. The main difficulty arises when we have to prove the convergence of the value functions of the approximating processes to the value function of the initial process. An original proof is provided.
... In other words, the DM beliefs evolve deterministically between arrivals of new information, and experience random jumps at event times. From the control perspective, various aspects of optimal stopping of PDP's have been studied by [26], [20] and [7]. In this paper, we study a class of finite-horizon decision-making problems within the PDP framework by considering a general regime-switching model with Poisson information arrivals. ...
... Hence, solving the equation (−ρ + L)V (s, π) + C( π) = 0 and determining the boundary of the region { π ∈ D : V (T, π) = H( π)} is not easy even when n = 2; see, for example, [32] who solve free-boundary problems similar to (3.1) for infinite horizon problems, and with n = 2. Instead of studying the problem in (3.1), we will employ a sequential approximation technique to compute the value function following [20] and [10, Chapter 5]. Similar approach is also taken in [3] and [12] for disorder-detection and hypothesis-testing problems respectively in infinite horizon. ...
We study decision timing problems on finite horizon with Poissonian
information arrivals. In our model, a decision maker wishes to optimally time
her action in order to maximize her expected reward. The reward depends on an
unobservable Markovian environment, and information about the environment is
collected through a (compound) Poisson observation process. Examples of such
systems arise in investment timing, reliability theory, Bayesian regime
detection and technology adoption models. We solve the problem by studying an
optimal stopping problem for a piecewise-deterministic process which gives the
posterior likelihoods of the unobservable environment. Our method lends itself
to simple numerical implementation and we present several illustrative
numerical examples.
... In the setting with partial observations, an iterative approximation scheme was employed in [5] to study the Poisson disorder detection problem with unknown post-disorder intensity, then later, in [9], to analyse a combined Poisson-Wiener disorder detection problem, and, more recently, in [4], to investigate the Wiener disorder detection under discrete observations. In the fully observable setting, such iterative approximations go back to at least as early as [19], which deals with a Markovian optimal stopping problem with a piecewise deterministic underlying. In Financial Mathematics, iteratively constructed approximations were used in [2,3] to study the value functions of finite and perpetual American put options, respectively, for a jump diffusion. ...
Optimal liquidation of an asset with unknown constant drift and stochastic regime-switching volatility is studied. The uncertainty about the drift is represented by an arbitrary probability distribution, the stochastic volatility is modelled by $m$-state Markov chain. Using filtering theory, an equivalent reformulation of the original problem as a four-dimensional optimal stopping problem is found and then analysed by constructing approximating sequences of three-dimensional optimal stopping problems. An optimal liquidation strategy and various structural properties of the problem are determined. Analysis of the two-point prior case is presented in detail, building on which, an outline of the extension to the general prior case is given.
... This book is focused on the computation of expectation of functionals of PDMPs with applications to the evaluation of service times and on solving optimal control problems with applications to maintenance optimization. Optimal control of PDMPs [55,24,25], and optimal stopping have been studied, [31]. Results in the control theory for the long-run average continuous control problem of PDMPs are presented in [20] processes, with related reliability studies and applications. ...
This chapter concerns a stochastic differential system that describes the evolution of a degradation mechanism, the fatigue crack propagation. A Markov or a semi-Markov process will be considered as the perturbing process of the system that models the crack evolution. With the help of Markov renewal theory, we study the reliability of a structure and propose for it a new analytical solution. The method we propose reduces the complexity of the reliability calculus compared with the previous resolution method. As numerical applications, we tested our method on a numerical example and on an experimental data set, which gave results in good agreement with a Monte Carlo estimation.
... Thanks to these methods one can solve optimal stopping problems [CD88,dSDG10,Gug86], impulse control problems [Cos93,CD89,dSD12], approximate distributions of exit times [BDSD12b] and compute expectations of functionals of PDMP's [BdSD12a]. ...
We give a short overview of recent results on a specific class of Markov
process: the Piecewise Deterministic Markov Processes (PDMPs). We first recall
the definition of these processes and give some general results. On more
specific cases such as the TCP model or a model of switched vector fields,
better results can be proved, especially as regards long time behaviour. We
continue our review with an infinite dimensional example of neuronal activity.
From the statistical point of view, these models provide specific challenges:
we illustrate this point with the example of the estimation of the distribution
of the inter-jumping times. We conclude with a short overview on numerical
methods used for simulating PDMPs.
... By means of suitable dynamic programming operator, the continuous-time optimal stopping problem is reduced to an essentially discrete-time optimal stopping problem. This approach is based on the stochastic dynamic optimization theory for piecewise deterministic Markov processes; see, for example, Gugerli [13] and Davis [8]. The dynamic programming operator maps every bounded function to another bounded function, whose value at every point in the domain is obtained as the solution of a straightforward deterministic optimization problem. ...
The quickest detection of the unknown and unobservable disorder time, when the arrival rate and mark distribution of a compound Poisson process suddenly changes, is formulated in a Bayesian setting, where the detection delay penalty is a general smooth function of the detection delay time. Under suitable conditions, the problem is shown to be equivalent to the optimal stopping of a finite-dimensional piecewise-deterministic strongly Markov sufficient statistic. The solution of the optimal stopping problem is described in detail for the compound Poisson disorder problem with polynomial detection delay penalty function of arbitrary but fixed degree. The results are illustrated for the case of the quadratic detection delay penalty function.
... The analysis of this section parallels the general framework of impulse control of piecewise deterministic processes (pdp) developed by Costa and Davis [1989], Lenhart and Liao [1988]. It is also related to optimal stopping of pdp's studied in Gugerli [1986], Costa and Davis [1988]. ...
We study finite horizon optimal switching problems for hidden Markov chain models with point process observations. The controller possesses a finite range of strategies and attempts to track the state of the unobserved state variable using Bayesian updates over the discrete observations. Such a model has applications in economic policy making, staffing under variable demand levels and generalized Poisson disorder problems. We show regularity of the value function and explicitly characterize an optimal strategy. We also provide an efficient numerical scheme and illustrate our results with several computational examples.
This paper investigates the random horizon optimal stopping problem for measure-valued piecewise deterministic Markov processes (PDMPs). This is motivated by population dynamics applications, when one wants to monitor some characteristics of the individuals in a small population. The population and its individual characteristics can be represented by a point measure. We first define a PDMP on a space of locally finite measures. Then we define a sequence of random horizon optimal stopping problems for such processes. We prove that the value function of the problems can be obtained by iterating some dynamic programming operator. Finally we prove via a simple counter-example that controlling the whole population is not equivalent to controlling a random lineage.
This chapter presents the computational method developed for the control of the piecewise‐deterministic Markov process (PDMP) and explains how similar ideas can be used for statistical inference. It deals with the most simple form of control, namely optimal stopping. The chapter gives the exact form of the dynamic programming operators and introduces the quantized approximation of the value function as well as an optimal stopping time. It presents the exit time distribution estimation procedure and defines the exit time problem and then the recursive formulation of the problem. The dynamics of the thickness loss for the structure can be described by a PDMP. The chapter focuses on the optimal quantization approach developed for numerical probability. Optimal quantization involves replacing a continuous state space random variable with the best discrete one in the sense that the L2‐norm of the difference is minimum.
This paper presents a state space and time discretization for the general average impulse control of piecewise deterministic Markov processes (PDPs). By combining several previous results we show that under some continuity, boundedness and compactness conditions on the parameters of the process, boundedness of the discretizations, and compactness of the state space, the discretized problem will converge uniformly to the original one. An application to optimal capacity expansion under uncertainty is given.
Momentum is the notion that an asset that has performed well in the past will continue to do so for some period. We study the optimal liquidation strategy for a momentum trade in a setting where the drift of the asset drops from a high value to a smaller one at some random change-point. This change-point is not directly observable for the trader, but it is partially observable in the sense that it coincides with one of the jump times of some exogenous Poisson process representing external shocks, and these jump times are assumed to be observable. Comparisons with existing results for momentum trading under incomplete information show that the assumption that the disappearance of the momentum effect is triggered by observable external shocks significantly improves the optimal strategy.
This chapter presents analytic methods for stochastic reachability using the connections with well-studied stochastic control problems (like the optimal stopping problem, obstacle problem and exit problem). These methods have to be adapted such that we capture the interaction between continuous dynamics and discrete dynamics, which is characteristic to a stochastic hybrid system. This interaction is illustrated mainly by the structure of the infinitesimal generator of the stochastic hybrid process, which is an integro-differential operator that satisfies some boundary conditions. For the characterisation of reach set probabilities, variational inequalities are derived. Variational inequalities based on energy forms, Dirichlet boundary problems and Hamilton-Jacobi-Bellman equations provide solutions for the estimation of reach set probabilities. The novelty of all these approaches consists of the ability of using the infinitesimal generator associated to a stochastic hybrid process. Simpler solutions of such problems might be obtained by ways to “approximate” this generator by operators that have richer properties and are easier to be handled in dynamic programming associated equations.
This paper presents a state space and time discretization for the general average impulse control of piecewise deterministic Markov processes (PDPs). By combining several previous results we show that under some continuity, boundedness and compactness conditions on the parameters of the process, boundedness of the discretizations, and compactness of the state space, the discretized problem will converge uniformly to the original one. An application to optimal capacity expansion under uncertainty is given.
Nous présentons d'abord les problématiques liées à l'utilisation des processus pour la modélisation des modèles d'histoire de vie et de survie, écriture de vraisemblance, définition d'indépendance locale entre processus et interprétation causale. De manière indépendante, nous présentons ensuite des modèles de processus de bifurcation, les méthodes d'estimation associées avec application à la division cellulaire. Enfin nous regardons des problèmes liés aux PDMP : modélisation de propagation de fissures, de HUMS et estimation du taux de saut. Quelques exemples de collaborations avec des chercheurs d'autres disciplines sont donnés dans le dernier chapitre.
In this chapter, we discuss Markov chains on continuous state space. We first analyze a discrete-time Markov chain on continuous state space, and then discuss a discrete-time Markov chain on a bivariate state space. Applying the censoring technique, we provide expression for the RG-factorizations, which are used to derive the stationary probability of the Markov chain. Further, we consider a continuous-time Markov chain on continuous state space. Specifically, we deal with a continuous-time level-dependent QBD process with continuous phase variable, and provide orthonormal representations for the R-, U- and G-measures, which lead to the matrix-structured computation of the stationary probability. As an application, we introduce continuousphase type (CPH) distribution and continuous-phase Markovian arrival process (CMAP), and then analyze a CMAP/CPH/1 queue. Finally, we study a piecewise deterministic Markov process, which is applied to deal with more general queues such as the GI/G/c queue.
We study de Finetti’s optimal dividend problem, also known as the optimal harvesting problem, in the dual model. In this model, the firm value is affected both by continuous fluctuations and by upward directed jumps. We use a fixed point method to show that the solution of the optimal dividend problem with jumps can be obtained as the limit of a sequence of stochastic control problems for a diffusion. In each problem, the optimal dividend strategy is of barrier type, and the rate of convergence of the barrier and the corresponding value function is exponential.
"Constructive Computation in Stochastic Models with Applications: The RG-Factorizations" provides a unified, constructive and algorithmic framework for numerical computation of many practical stochastic systems. It summarizes recent important advances in computational study of stochastic models from several crucial directions, such as stationary computation, transient solution, asymptotic analysis, reward processes, decision processes, sensitivity analysis as well as game theory. Graduate students, researchers and practicing engineers in the field of operations research, management sciences, applied probability, computer networks, manufacturing systems, transportation systems, insurance and finance, risk management and biological sciences will find this book valuable. Dr. Quan-Lin Li is an Associate Professor at the Department of Industrial Engineering of Tsinghua University, China. © Tsinghua University Press, Beijing and Springer-Verlag Berlin Heidelberg 2010. All rights are reserved.
We consider the existence and uniqueness of viscosity solutions to a system of integro-differential equations with bilateral implicit obstacles. The system is the dynamic programming equation associated with a switching game for two players, in which the underlying state processes are piecewise-deterministic processes. We prove the probabilistic representation of the viscosity solution as the saddle point of the game for the R case
The paper deals with value functions for optimal stopping and impulsive control for piecewise-deterministic processes with discounted cost. The associated dynamic programming equations are variational and quasi-variational inequalities with integral and first-order differential terms The technique used is to approximate the value functions for an optimal stopping (impulsive control. switching control) problem for a piecewise-deterministic process by value functions for optimal stopping (impulsive control, switching control) problems for Feller piecewise-deterministic processes
The paper deals with value functions for optimal stopping and impulsive control for piecewise-deterministic processes with long run average cost. The associated dynamic programming equations are variational and quasi-variation inequalities with integral and first order differential terms. Two important examples are given.
In this paper we consider the problem of optimal stopping and continuous control on some local parameters of a piecewise-deterministic Markov processes (PDP's). Optimality equations are obtained in terms of a set of variational inequalities as well as on the first jump time operator of the PDP. It is shown that if the final cost function is absolutely continuous along trajectories then so is the value function of the optimal stopping problem with continuous control. These results unify and generalize previous ones in the current literature.
Suppose that there are finitely many simple hypotheses about the unknown arrival rate and mark distribution of a compound Poisson process, and that exactly one of them is correct. The objective is to determine the correct hypothesis with minimal error probability and as soon as possible after the observation of the process starts. This problem is formulated in a Bayesian framework, and its solution is presented. Provably convergent numerical methods and practical near-optimal strategies are described and illustrated on various examples.
A finite collection of piecewise-deterministic processes are controlled in order to minimize the expected value of a performance functional with continuous operating cost and discrete switching control costs. The solution of the associated dynamic programming equation is obtained by an iterative approximation using optimal stopping time problems.
Controlled piecewise-deterministic Markov processes have deterministic trajectories punctuated by random jumps, at which the sample path is right-continuous. By considering the sequence of states visited by the process at its jump times, it is shown that a discounted infinite horizon control problem can be reformulated as a discrete-time Markov decision problem (the positive case). Under certain continuity assumptions it is shown that an optimal stationary policy exists in relaxed controls.
In this paper we formulate new general optimality conditions for impulsive control of piecewise-deterministic processes. We prove continuity of the value functions for optimal stopping, discounted impulsive control, and impulsive control with long run average cost. We study conditions for optimal and nearly optimal policies for the corresponding impulsive control problems. We give variational formulations of the optimality conditions.
This paper deals with approximation techniques for the optimal stopping of a piecewise-deterministic Markov process (P.D.P.).
Such processes consist of a mixture of deterministic motion and random jumps. In the first part of the paper (Section 3) we
study the optimal stopping problem with lower semianalytic gain function; our main result is the construction of ε-optimal
stopping times. In the second part (Section 4) we consider a P.D.P. satisfying some smoothness conditions, and forN integer we construct a discretized P.D.P. which retains the main characteristics of the original process. By iterations of
the single jump operator from ℝ
N
to ℝ
N
, each iteration consisting ofN one-dimensional minimizations, we can calculate the payoff function of the discretized process. We demonstrate the convergence
of the payoff functions, and for the case when the state space is compact we construct ε-optimal stopping times for the original
problem using the payoff function of the discretized problem. A numerical example is presented.
The paper deals with value functions for impulsive control for piecewise-deterministic processes. The associated dynamic programming equations are quasivariational inequalities with integral and first order differential terms. Here we study different regularity properties of the cost function and existence of optimal policies.
In this paper we consider the long run average continuous control problem of piecewise-deterministic Markov processes (PDP's for short). The control variable acts on the jump rate λ and transition measure Q of the PDP. We consider relaxed open loop policies which choose, at each jump time, randomized (rather than deterministic) control actions. The advantage of allowing randomized actions is that the optimality equation for the continuous-time problem can be re-written as a discrete-time Markov decision process with compact action space. The main goal of this paper is to show the compactness proprieties of the action space for discrete-time problem as well as to prove the equivalence between the optimality equations of the continuous and discrete-time problems.
Unlike the pure discrete or continuous cases, for stochastic hybrid systems the reachability analysis is still a difficult problem. The early mathematical approaches to the stochastic reachability turned out into constructive verification methods with tool support from dynamic programming. This line of research suggests the need for a deeper mathematical investigation of connections between reachability based verification and optimal control for stochastic hybrid systems. In this paper, we prove a new characterization of the stochastic reachability problem in terms of variational inequalities on Hilbert spaces. These inequalities are well studied in stochastic optimal control, for which theoretical and numerical methods exist.
We present a numerical method to compute the survival function and the
moments of the exit time for a piecewise-deterministic Markov process (PDMP).
Our approach is based on the quantization of an underlying discrete-time Markov
chain related to the PDMP. The approximation we propose is easily computable
and is even flexible with respect to the exit time we consider. We prove the
convergence of the algorithm and obtain bounds for the rate of convergence in
the case of the moments. An academic example and a model from the reliability
field illustrate the paper.
In this paper, we consider the average impulse control of piecewise deterministic Markov processes (PDPs). In the first part
of the paper, we present a policy-iteration technique which consists of solving a sequence of optimal stopping problems that
allow only a finite number of jumps. In the second part, we present a discretization of the state space which, along with
the technique mentioned above, reduces the average impulse control of PDPs to a sequence of one-dimensional minimizations.
A numerical example is presented.
We consider the problem of optimal stopping and continuous control
on some local parameters of a piecewise-deterministic Markov processes
(PDPs). Optimality equations are obtained in terms of the infinitesimal
operator as well as on the first jump time operator of the PDP. It is
shown that if the final cost function is absolutely continuous along
trajectories then so is the value function of the optimal stopping
problem with continuous control. These results unify and generalize
previous ones in the current literature
It is shown that all local martingales of the sigma -fields generated by a jump process of very general type can be represented as stochastic integrals with respect to a fundamental family of martingales associated with the jump process.
The representation of martingales of jump processes
- R M Blumenthal
- R K Getoor
- Markov Processe
I] R. M. Blumenthal and R. K. Getoor, Markov Processe.~ and Potrntiol Theory, Academic Press, New York, 1968. L2] P. BrBmoud. Poiiit Processes and Queues. Springer-Verlag. New York; !OR!. i3l M. H. A. Davis. The representation of martingales of jump processes, SlAM J. Control and Optimization 14 (19763, 623-638.
Piecewise-delerministir Tvlnrkov procew.>: A grrrerai ciass of non-dln'usion stochastic nlt~cielr
- M H A Davis
M. H. A. Davis. Piecewise-delerministir Tvlnrkov procew.>: A grrrerai ciass of non-dln'usion stochastic nlt~cielr,.I K n w ! Statrstica: Soi. (B) 36 (1984). 353-388.
161 N. El Karoui, I.es aspects probabilisles du contrSie slochastique Markov processes: Ray processes and right processes Integro-differential equations associated with optimal stopping time of a piecewise-deterministic process, Stochastics, to appear
- [ Sj
- E B Dynkin
- K K Getoorxi
- S M Lenhart
- Yu-Chung
- Liao
[SJ E. B. Dynkin. Markov Proc~rses, Springer Verlag, New York. 1965. 161 N. El Karoui, I.es aspects probabilisles du contrSie slochastique. Lxcture Notes in Mathematics 876. Springer-Verlag, Berlin. 1981. [71 K. K. Getoor, Markov processes: Ray processes and right processes, Lrcture Notes in Mathematics 440, Springer-Verlag, Berlin, 1975. [XI S. M. Lenhart and Yu-Chung Liao, Integro-differential equations associated with optimal stopping time of a piecewise-deterministic process, Stochastics, to appear. [9J A. N. Shiryayev, Optimal Stopping Rules, Springer-Verlag. New York, !97R. jlG] J. L. Sneii, Applications of martingale system theorems, Trans. Amer. Math. Soc.