Abstract:We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with control actions and the state process causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies. In particular, while in the belief-MDP reduction of POMDPs, weak Feller condition requirement imposes total variation continuity on either the system kernel or the measurement kernel, with the approach of this paper only weak continuity of both the transition kernel and the measurement kernel is needed (and total variation continuity is not) together with regularity conditions related to filter stability. For the average cost setup, we provide a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for non-linear filter processes, where we show that unique ergodicity of the measurement process and a measurability condition related to filter stability leads to unique ergodicity.

Partially observable optimal control using exponential cost criterion

Suboptimal Reliable Guaranteed Cost Control for Continuous-Time Systems with Multi-Criterion Constraints

Lmi Approach To Optimal Guaranteed Cost Control For A Class Of Linear Uncertain Discrete Systems

On Stochastic Optimal Control of Partially Observable Nonlinear Quasi Hamiltonian Systems

A stochastic optimal control strategy for partially observable nonlinear systems

A Minimax Optimal Control Strategy for Partially Observable Uncertain Quasi-Hamiltonian Systems

General Necessary Conditions for Partially Observed Optimal Stochastic Controls

Regret-Optimal Control under Partial Observability

Nonlinear stochastic optimal control of partially observable linear structures

Stochastic Minimax Vibration Control for Uncertain Nonlinear Quasi-Hamiltonian Systems with Noisy Observations

Stochastic Optimal Vibration Control of Partially Observable Nonlinear Quasi Hamiltonian Systems with Actuator Saturation

Robust Optimization for Quantum Reinforcement Learning Control Using Partial Observations

Partial Information Near-Optimal Control of Forward-Backward Stochastic Differential System with Observation Noise

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

Nonlinear Stochastic Optimal Control of MDOF Partially Observable Linear Systems Excited by Combined Harmonic and Wide-Band Noises

The Maximum Principle for Partially Observed Optimal Control of Stochastic Differential Equations

A Multilevel Approach for Stochastic Nonlinear Optimal Control

Optimal Control in Partially Observable Complex Social Systems.

Optimal Adaptive Control of Linear Stochastic Systems with Quadratic Cost Function

Discrete-Time Approximation of Stochastic Optimal Control with Partial Observation

The necessary and sufficient conditions for stochastic differential systems with multi-time states cost functional