Abstract:We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

Average Cost Optimality of Partially Observed MDPS: Contraction of Non-linear Filters, Optimal Solutions and Approximations

Average Cost Optimality of Partially Observed MDPs: Contraction of Nonlinear Filters and Existence of Optimal Solutions and Approximations

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Partially Observed Optimal Stochastic Control: Regularity, Optimality, Approximations, and Learning

Nonfragile Near-Optimal Control of Stochastic Time-Varying Multiagent Systems with Control- and State-Dependent Noises

Approximation Schemes for POMPDs with Continuous Spaces and Their Near Optimality

Average cost optimal control under weak ergodicity hypotheses: Relative value iterations

On Stochastic Optimal Control of Partially Observable Nonlinear Quasi Hamiltonian Systems

Necessary Optimality Conditions For Average Cost Minimization Problems

Q-Learning for Continuous State and Action MDPs under Average Cost Criteria

Discrete-Time Approximation of Stochastic Optimal Control with Partial Observation

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

Average-Cost MDPs with Infinite State and Action Sets: New Sufficient Conditions for Optimality Inequalities and Equations

Discretized Approximations for POMDP with Average Cost

Average Optimality for Markov Decision Processes in Borel Spaces: a New Condition and Approach

Partially Observed Optimal Control of Local and Remote Controllers

A Multilevel Approach for Stochastic Nonlinear Optimal Control

Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimality

Controlled Diffusions under Full, Partial and Decentralized Information: Existence of Optimal Policies and Discrete-Time Approximations

A Branching Particle System Approximation for Solving Partially Observed Stochastic Optimal Control Problems Via Stochastic Maximum Principle

Regularity of Optimal Solutions and the Optimal Cost for Hybrid Dynamical Systems via Reachability Analysis