Abstract:We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

Partial Extended Observability Certification and Optimal Design of Moving Horizon Estimators Under Uncertainties

On Stochastic Optimal Control of Partially Observable Nonlinear Quasi Hamiltonian Systems

Multi-sensor Optimal Linear Estimation with Unobservable Measurement Losses

MHE under parametric uncertainty -- Robust state estimation without informative data

Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs

Regret-Optimal Control under Partial Observability

Optimal Bounds on Nonlinear Partial Differential Equations in Model Certification, Validation, and Experimental Design

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Robust Observer Design and Fuzzy Optimization for Uncertain Dynamic Systems

Observability Inequality from Measurable Sets and the Shape Design Problem for Stochastic Parabolic Equations

Convexity and monotonicity in nonlinear optimal control under uncertainty

Interval Observer Design for Continuous-Time Linear Parameter-Varying Systems

Optimality and robustness in path-planning under initial uncertainty

Optimal Interval Observers for Bounded Jacobian Nonlinear Dynamical Systems

Myopically Verifiable Probabilistic Certificates for Safe Control and Learning

Continuous time Stochastic optimal control under discrete time partial observations

Optimal Sampling for Uncertainty-of-Information Minimization in a Remote Monitoring System

Interval Observer Design for Uncertain Discrete-Time Linear Systems

A Unified Framework for Adjustable Robust Optimization with Endogenous Uncertainty

Interval Observers for Continuous-Time Systems with Parametric Uncertainties

Observer Design For Systems With Unknown Inputs And Missing Measurements