Abstract:We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

On Strategic Measures and Optimality Properties in Discrete-Time Stochastic Control with Universally Measurable Policies

Finite Horizon Partially Observable Semi-Markov Decision Processes under Risk Probability Criteria

Risk-sensitive Continuous-Time Markov Decision Processes with Unbounded Rates and Borel Spaces

Nonstationary discrete-time deterministic and stochastic control systems: Bounded and unbounded cases

Minimax Control for Discrete-Time Time-Varying Stochastic Systems

Maximizing the probability of visiting a set infinitely often for a Markov decision process with Borel state and action spaces

Brief Minimax control for discrete-time time-varying stochastic systems

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Nonstationary Discrete-Time Deterministic And Stochastic Control Systems With Infinite Horizon

Risk-sensitive Markov control processes

Blackwell optimality and policy stability for long-run risk sensitive stochastic control

Finite-horizon Optimality for Continuous-Time Markov Decision Processes with Unbounded Transition Rates

Existence of Markov equilibrium control in discrete time

On Borkar and Young Relaxed Control Topologies and Continuous Dependence of Invariant Measures on Control Policy

Risk Probability Minimization Problems for Continuous-Time Markov Decision Processes on Finite Horizon

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

Robust Equilibrium Strategies for Time-Inconsistent Stochastic Optimal Control Problems with Applications

Optimal Control of Uncertain Stochastic Systems with Markovian Switching and Its Applications to Portfolio Decisions

A Semimartingale Characterization of Average Optimal Stationary Policies for Markov Decision Processes

Convexity and monotonicity in nonlinear optimal control under uncertainty

Continuous time Stochastic optimal control under discrete time partial observations