Abstract:A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviations from the expected behavior may lead to mission failure. In this paper, we consider the problem of designing policies for MDPs and POMDPs with objectives and constraints in terms of dynamic coherent risk measures, which we refer to as the constrained risk-averse problem. For MDPs, we reformulate the problem into a infsup problem via the Lagrangian framework and propose an optimization-based method to synthesize Markovian policies. For MDPs, we demonstrate that the formulated optimization problems are in the form of difference convex programs (DCPs) and can be solved by the disciplined convex-concave programming (DCCP) framework. We show that these results generalize linear programs for constrained MDPs with total discounted expected costs and constraints. For POMDPs, we show that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies. For stochastic finite-state controllers (FSCs), we show that the latter optimization simplifies to a (finite-dimensional) DCP and can be solved by the DCCP framework. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs.

Non-Deterministic Policies in Markovian Decision Processes

Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes

Decision Making in Non-Stationary Environments with Policy-Augmented Search

Decision Making in Non-Stationary Environments with Policy-Augmented Monte Carlo Tree Search

Acting in Delayed Environments with Non-Stationary Markov Policies

Markov Decision Processes under External Temporal Processes

Decision-Theoretic Planning with non-Markovian Rewards

Risk-Averse Decision Making Under Uncertainty

Learning Non-Markovian Decision-Making from State-only Sequences

Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes

Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning

Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time

Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs

Learning to Switch Among Agents in a Team via 2-Layer Markov Decision Processes

Intrinsically Motivated Hierarchical Policy Learning in Multi-objective Markov Decision Processes

Safe Reinforcement Learning in Constrained Markov Decision Processes

Online Reinforcement Learning for Real-Time Exploration in Continuous State and Action Markov Decision Processes

Monitored Markov Decision Processes

Markov Abstractions for PAC Reinforcement Learning in Non-Markov Decision Processes

Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments

SPO: Sequential Monte Carlo Policy Optimisation