Abstract:Partially observable Markov decision processes (POMDPs) provide a modeling framework for a variety of sequential decision making under uncertainty scenarios in artificial intelligence (AI). Since the states are not directly observable in a POMDP, decision making has to be performed based on the output of a Bayesian filter (continuous beliefs). Hence, POMDPs are often computationally intractable to solve exactly and researchers resort to approximate methods often using discretizations of the continuous belief space. These approximate solutions are, however, prone to discretization errors, which has made POMDPs ineffective in applications, wherein guarantees for safety, optimality, or performance are required. To overcome the complexity challenge of POMDPs, we apply notions from control theory. The goal is to determine the reachable belief space of a POMDP, that is, the set of all possible evolutions given an initial belief distribution over the states and a set of actions and observations. We begin by casting the problem of analyzing a POMDP into analyzing the behavior of a discrete-time switched system. For estimating the reachable belief space, we find over-approximations in terms of sub-level sets of Lyapunov functions. Furthermore, in order to verify safety and optimality requirements of a given POMDP, we formulate a barrier certificate theorem, wherein we show that if there exists a barrier certificate satisfying a set of inequalities along with the belief update equation of the POMDP, the safety and optimality properties are guaranteed to hold. In both cases, we show how the calculations can be decomposed into smaller problems that can be solved in parallel. The conditions we formulate can be computationally implemented as a set of sum-of-squares programs. We illustrate the applicability of our method by addressing two problems in active ad scheduling and machine teaching.

The Bayesian process control with multiple assignable causes

Chance Constrained Nonlinear Model Predictive Control

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Control Theory Meets POMDPs: A Hybrid Systems Approach

Episodic Bayesian Optimal Control with Unknown Randomness Distributions

Hierarchical Task-Space Optimal Covariance Control With Chance Constraints

Remarks on Bayesian Control Charts

Optimal Control of Logically Constrained Partially Observable and Multi-Agent Markov Decision Processes

Chance Constrained Stochastic Optimal Control for Arbitrarily Disturbed LTI Systems Via the One-Sided Vysochanskij-Petunin Inequality

Continuous time Stochastic optimal control under discrete time partial observations

Probabilistic Control and Majorization of Optimal Control

Chance Constrained Stochastic Optimal Control for Linear Systems with a Time Varying Random Control Matrix

Chance-Constrained Information-Theoretic Stochastic Model Predictive Control with Safety Shielding

Chance Constrained Stochastic Optimal Control Based on Sample Statistics With Almost Surely Probabilistic Guarantees

Controlled Markov Processes With Safety State Constraints

Extended mean-field control problems with multi-dimensional singular controls

Stochastic Model Predictive Control with Dynamic Chance Constraints

Stochastic Optimal Control in Continuous Space-Time Multi-Agent Systems

Safe Bayesian Optimization for Complex Control Systems via Additive Gaussian Processes

A Bayesian Theory of Change Detection in Statistically Periodic Random Processes

A Bayesian perspective on classical control