Abstract:The objective in a traditional reinforcement learning (RL) problem is to find a policy that optimizes the expected value of a performance metric such as the infinite-horizon cumulative discounted or long-run average cost/reward. In practice, optimizing the expected value alone may not be satisfactory, in that it may be desirable to incorporate the notion of risk into the optimization problem formulation, either in the objective or as a constraint. Various risk measures have been proposed in the literature, e.g., exponential utility, variance, percentile performance, chance constraints, value at risk (quantile), conditional value-at-risk, prospect theory and its later enhancement, cumulative prospect theory. In this book, we consider risk-sensitive RL in two settings: one where the goal is to find a policy that optimizes the usual expected value objective while ensuring that a risk constraint is satisfied, and the other where the risk measure is the objective. We survey some of the recent work in this area specifically where policy gradient search is the solution approach. In the first risk-sensitive RL setting, we cover popular risk measures based on variance, conditional value-at-risk, and chance constraints, and present a template for policy gradient-based risk-sensitive RL algorithms using a Lagrangian formulation. For the setting where risk is incorporated directly into the objective function, we consider an exponential utility formulation, cumulative prospect theory, and coherent risk measures. This non-exhaustive survey aims to give a flavor of the challenges involved in solving risk-sensitive RL problems using policy gradient methods, as well as outlining some potential future research directions.

A nearly Blackwell-optimal policy gradient method

Convergence Rate of Primal-Dual Approach to Constrained Reinforcement Learning with Softmax Policy

On the Convergence of Discounted Policy Gradient Methods

Policy Gradient for Reinforcement Learning with General Utilities

Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learning

Variational Policy Gradient Method for Reinforcement Learning with General Utilities

A Payoff-Based Policy Gradient Method in Stochastic Games with Long-Run Average Payoffs

Optimal Control-Based Baseline for Guided Exploration in Policy Gradient Methods

Risk-Sensitive Reinforcement Learning via Policy Gradient Search

Model-free Policy Learning with Reward Gradients

Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL

Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications

Beyond Expected Returns: A Policy Gradient Algorithm for Cumulative Prospect Theoretic Reinforcement Learning

Policy Gradient Method For Robust Reinforcement Learning

Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence

On the Second-Order Convergence of Biased Policy Gradient Algorithms

Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis

Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization

Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and model