Article

# Bounding Procedures for Stochastic Dynamic Programs with Application to the Perimeter Patrol Problem

08/2011; abs/1108.3299. DOI: 10.1109/ACC.2012.6314780
Source: DBLP

ABSTRACT One often encounters the curse of dimensionality in the application of
dynamic programming to determine optimal policies for controlled Markov chains.
In this paper, we provide a method to construct sub-optimal policies along with
a bound for the deviation of such a policy from the optimum via a linear
programming approach. The state-space is partitioned and the optimal cost-to-go
or value function is approximated by a constant over each partition. By
minimizing a non-negative cost function defined on the partitions, one can
construct an approximate value function which also happens to be an upper bound
for the optimal value function of the original Markov Decision Process (MDP).
As a key result, we show that this approximate value function is {\it
independent} of the non-negative cost function (or state dependent weights as
it is referred to in the literature) and moreover, this is the least upper
bound that one can obtain once the partitions are specified. Furthermore, we
show that the restricted system of linear inequalities also embeds a family of
MDPs of lower dimension, one of which can be used to construct a lower bound on
the optimal value function. The construction of the lower bound requires the
solution to a combinatorial problem. We apply the linear programming approach
to a perimeter surveillance stochastic optimal control problem and obtain
numerical results that corroborate the efficacy of the proposed methodology.

Download full-text

### Full-text

Available from: Meir Pachter, Jun 28, 2015
0 Followers
·
99 Views
• ##### Article: State partitioning based linear program for stochastic dynamic programs: An invariance property
[Hide abstract]
ABSTRACT: A common approximate dynamic programming method entails state partitioning and the use of linear programming, i.e., the state-space is partitioned and the optimal value function is approximated by a constant over each partition. By minimizing a positive cost function defined on the partitions, one can construct an upper bound for the optimal value function. We show that this approximate value function is independent of the positive cost function and that it is the least upper bound, given the partitions.
Operations Research Letters 11/2012; 40(6):487–491. DOI:10.1016/j.orl.2012.08.006 · 0.62 Impact Factor