Abstract:A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviations from the expected behavior may lead to mission failure. In this paper, we consider the problem of designing policies for MDPs and POMDPs with objectives and constraints in terms of dynamic coherent risk measures, which we refer to as the constrained risk-averse problem. For MDPs, we reformulate the problem into a infsup problem via the Lagrangian framework and propose an optimization-based method to synthesize Markovian policies. For MDPs, we demonstrate that the formulated optimization problems are in the form of difference convex programs (DCPs) and can be solved by the disciplined convex-concave programming (DCCP) framework. We show that these results generalize linear programs for constrained MDPs with total discounted expected costs and constraints. For POMDPs, we show that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies. For stochastic finite-state controllers (FSCs), we show that the latter optimization simplifies to a (finite-dimensional) DCP and can be solved by the DCCP framework. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs.

Strong Simple Policies for POMDPs

Strategy Synthesis in POMDPs via Game-Based Abstractions

Bounded Policy Synthesis for POMDPs with Safe-Reachability Objectives

Explainable Finite-Memory Policies for Partially Observable Markov Decision Processes

Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints

Tableaux for Policy Synthesis for MDPs with PCTL* Constraints

Point-Based Methods for Model Checking in Partially Observable Markov Decision Processes

Stochastic Finite State Control of POMDPs with LTL Specifications

Policies Grow on Trees: Model Checking Families of MDPs

Learning Explainable and Better Performing Representations of POMDP Strategies

Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach

Agent-state based policies in POMDPs: Beyond belief-state MDPs

Bridging the Gap between Partially Observable Stochastic Games and Sparse POMDP Methods

Imprecise Probabilities Meet Partial Observability: Game Semantics for Robust POMDPs

Search and Explore: Symbiotic Policy Synthesis in POMDPs

Learning Robust Policies for Uncertain Parametric Markov Decision Processes

Robust Almost-Sure Reachability in Multi-Environment MDPs

Policy Synthesis for Factored MDPs with Graph Temporal Logic Specifications

Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments

Risk-Averse Decision Making Under Uncertainty

Optimal Control of Logically Constrained Partially Observable and Multi-Agent Markov Decision Processes