Abstract:A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviations from the expected behavior may lead to mission failure. In this paper, we consider the problem of designing policies for MDPs and POMDPs with objectives and constraints in terms of dynamic coherent risk measures, which we refer to as the constrained risk-averse problem. For MDPs, we reformulate the problem into a infsup problem via the Lagrangian framework and propose an optimization-based method to synthesize Markovian policies. For MDPs, we demonstrate that the formulated optimization problems are in the form of difference convex programs (DCPs) and can be solved by the disciplined convex-concave programming (DCCP) framework. We show that these results generalize linear programs for constrained MDPs with total discounted expected costs and constraints. For POMDPs, we show that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies. For stochastic finite-state controllers (FSCs), we show that the latter optimization simplifies to a (finite-dimensional) DCP and can be solved by the DCCP framework. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs.

Risk Sensitivity in Markov Games and Multi-Agent Reinforcement Learning: A Systematic Review

Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Games

A Game-Theoretic Framework for Managing Risk in Multi-Agent Systems

Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learning

Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Robust Multi-Agent Reinforcement Learning with State Uncertainty

Model and Reinforcement Learning for Markov Games with Risk Preferences

Risk-Sensitive Cooperative Games for Human-Machine Systems

RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization

A Survey of Risk-Aware Multi-Armed Bandits

Risk-Averse Decision Making Under Uncertainty

Tractable Equilibrium Computation in Markov Games through Risk Aversion

Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertainty

Risk-Aware Distributed Multi-Agent Reinforcement Learning

Modeling Risk in Reinforcement Learning: A Literature Mapping

LQG Risk-Sensitive Single-Agent and Major-Minor Mean Field Game Systems: A Variational Framework

Risk-Averse Bayes-Adaptive Reinforcement Learning

Policy Evaluation and Seeking for Multi-Agent Reinforcement Learning Via Best Response