Abstract:We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

Asymptotic Perron's Method and Simple Markov Strategies in Stochastic Games and Control

Stochastic Perron's Method and Elementary Strategies for Zero-Sum Differential Games

Marginal Values of a Stochastic Game

A Fixed-Point Policy-Iteration-Type Algorithm for Symmetric Nonzero-Sum Stochastic Impulse Control Games

Stochastic Near-Optimal Controls for Path-Dependent Systems

Numerical approximations of the value of zero-sum stochastic differential impulse controls game in finite horizon

A Generic Strategy Improvement Method for Simple Stochastic Games

Optimal strategies for impulse control of piecewise deterministic Markov processes

A tutorial on Zero-sum Stochastic Games

Zero-sum Risk-sensitive Stochastic Games for Continuous Time Markov Chains

Zero-Sum Games for piecewise deterministic Markov decision processes with risk-sensitive finite-horizon cost criterion

Nonzero-sum Discrete-time Stochastic Games with Risk-sensitive Ergodic Cost Criterion

Zero-Sum Stopper Versus Singular-Controller Games with Constrained Control Directions

On asymptotic optimality of Merton's myopic portfolio strategies for discrete time market

Stochastic Perron for stochastic target games

A theory of Markovian time-inconsistent stochastic control in discrete time

Long Run Stochastic Control Problems with General Discounting

Zero-sum stopper vs. singular-controller games with constrained control directions

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Two sided long-time optimization singular control problems for Lévy processes and Dynkin's games

Singular mean-field control games with applications to optimal harvesting and investment problems