Abstract:We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

Convex Stochastic Dominance in Bayesian Localization, Filtering and Controlled Sensing POMDPs

Stochastic Steepest-Descent Optimization Of Multiple-Objective Mobile Sensor Coverage

Partially Observed Optimal Stochastic Control: Regularity, Optimality, Approximations, and Learning

Multivariate Robust Second-Order Stochastic Dominance and Resulting Risk-Averse Optimization

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Stochastic maximum principle for hybrid optimal control problems under partial observation

Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking

Kullback-Leibler-Quadratic Optimal Control

Control Theory Meets POMDPs: A Hybrid Systems Approach

Observation-Based Optimization for POMDPs with Continuous State, Observation, and Action Spaces.

Partial Information Near-Optimal Control of Forward-Backward Stochastic Differential System with Observation Noise

Convexity and monotonicity in nonlinear optimal control under uncertainty

The Global Maximum Principle for Progressive Optimal Control of Partially Observed Forward-Backward Stochastic Systems with Random Jumps

Partial Stochastic Dominance via Optimal Transport

Log-concave posterior densities arising in continuous filtering and a maximum a posteriori algorithm

Convex duality for stochastic singular control problems

Benign Nonconvex Landscapes in Optimal and Robust Control, Part I: Global Optimality

Observer-based distributed convex optimization of bipartite containment control for higher order nonlinear uncertain multi-agent systems

Variational Bayesian Learning based Joint Localization and Path Loss Exponent with Distance-dependent Noise in Wireless Sensor Network

Distributionally Robust Second-Order Stochastic Dominance Constrained Optimization with Wasserstein Ball

Optimality of Myopic Sensing in Multi-Channel Opportunistic Access